Re: [ANNOUNCE] New committer: Matthias J. Sax

[Sriram Subramanian]

Congratulations Matthias ! Well deserved!

On Mon, Jan 15, 2018 at 10:22 PM, Becket Qin  wrote:

> Congrats, Matthias!
>
> On Mon, Jan 15, 2018 at 9:54 PM, Konstantine Karantasis <
> konstant...@confluent.io> wrote:
>
> > Matthias! Congratulations!
> >
> > Konstantine
> >
> > On Mon, Jan 15, 2018 at 4:18 AM, Edoardo Comar 
> wrote:
> >
> > > Congratulations Matthias !
> > > --
> > >
> > > Edoardo Comar
> > >
> > > IBM Message Hub
> > >
> > > IBM UK Ltd, Hursley Park, SO21 2JN
> > >
> > >
> > >
> > > From:   UMESH CHAUDHARY 
> > > To: dev@kafka.apache.org
> > > Date:   15/01/2018 11:47
> > > Subject:Re: [ANNOUNCE] New committer: Matthias J. Sax
> > >
> > >
> > >
> > > Congratulations Matthias :)
> > >
> > > On Mon, 15 Jan 2018 at 17:06 Michael Noll 
> wrote:
> > >
> > > > Herzlichen Glückwunsch, Matthias. ;-)
> > > >
> > > >
> > > >
> > > > On Mon, 15 Jan 2018 at 11:38 Viktor Somogyi  >
> > > > wrote:
> > > >
> > > > > Congrats Matthias!
> > > > >
> > > > > On Mon, Jan 15, 2018 at 9:17 AM, Jorge Esteban Quilcate Otoya <
> > > > > quilcate.jo...@gmail.com> wrote:
> > > > >
> > > > > > Congratulations Matthias!!
> > > > > >
> > > > > > El lun., 15 ene. 2018 a las 9:08, Boyang Chen
> > > ()
> > > > > > escribió:
> > > > > >
> > > > > > > Great news Matthias!
> > > > > > >
> > > > > > >
> > > > > > > 
> > > > > > > From: Kaufman Ng 
> > > > > > > Sent: Monday, January 15, 2018 11:32 AM
> > > > > > > To: us...@kafka.apache.org
> > > > > > > Cc: dev@kafka.apache.org
> > > > > > > Subject: Re: [ANNOUNCE] New committer: Matthias J. Sax
> > > > > > >
> > > > > > > Congrats Matthias!
> > > > > > >
> > > > > > > On Sun, Jan 14, 2018 at 4:52 AM, Rajini Sivaram <
> > > > > rajinisiva...@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Congratulations Matthias!
> > > > > > > >
> > > > > > > > On Sat, Jan 13, 2018 at 11:34 AM, Mickael Maison <
> > > > > > > mickael.mai...@gmail.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Congratulations Matthias !
> > > > > > > > >
> > > > > > > > > On Sat, Jan 13, 2018 at 7:01 AM, Paolo Patierno <
> > > > > ppatie...@live.com>
> > > > > > > > > wrote:
> > > > > > > > > > Congratulations Matthias ! Very well deserved !
> > > > > > > > > > 
> > > > > > > > > > From: Guozhang Wang 
> > > > > > > > > > Sent: Friday, January 12, 2018 11:59:21 PM
> > > > > > > > > > To: dev@kafka.apache.org; us...@kafka.apache.org
> > > > > > > > > > Subject: [ANNOUNCE] New committer: Matthias J. Sax
> > > > > > > > > >
> > > > > > > > > > Hello everyone,
> > > > > > > > > >
> > > > > > > > > > The PMC of Apache Kafka is pleased to announce Matthias
> J.
> > > Sax
> > > > as
> > > > > > our
> > > > > > > > > > newest Kafka committer.
> > > > > > > > > >
> > > > > > > > > > Matthias has made tremendous contributions to Kafka
> Streams
> > > API
> > > > > > since
> > > > > > > > > early
> > > > > > > > > > 2016. His footprint has been all over the places in
> > Streams:
> > > in
> > > > > the
> > > > > > > > past
> > > > > > > > > > two years he has been the main driver on improving the
> join
> > > > > > semantics
> > > > > > > > > > inside Streams DSL, summarizing all their shortcomings
> and
> > > > > bridging
> > > > > > > the
> > > > > > > > > > gaps; he has also been largely working on the
> exactly-once
> > > > > > semantics
> > > > > > > of
> > > > > > > > > > Streams by leveraging on the transaction messaging
> feature
> > > in
> > > > > > 0.11.0.
> > > > > > > > In
> > > > > > > > > > addition, Matthias have been very active in community
> > > activity
> > > > > that
> > > > > > > > goes
> > > > > > > > > > beyond mailing list: he's getting the close to 1000 up
> > votes
> > > > and
> > > > > > 100
> > > > > > > > > > helpful flags on SO for answering almost all questions
> > about
> > > > > Kafka
> > > > > > > > > Streams.
> > > > > > > > > >
> > > > > > > > > > Thank you for your contribution and welcome to Apache
> > Kafka,
> > > > > > > Matthias!
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Guozhang, on behalf of the Apache Kafka PMC
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Kaufman Ng
> > > > > > > +1 646 961 8063 <+1%20646-961-8063> <(646)%20961-8063>
> > > > > > > Solutions Architect | Confluent | www.confluent.io<
> > > https://urldefense.proofpoint.com/v2/url?u=http-3A__www&d=
> > > DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=EzRhmSah4IHsUZVekRUIINhltZK7U0
> > > OaeRo7hgW4_tQ&m=1kMl7t3vDn8RaG7aDYpjxKcgtAA0RVnWDeMBhN7yjzE&s=
> > > qPzEPox7QPw6NE5uYyUmMHo1Z4LPhLoC9hZBexOiBT8&e=
> > > .
> > > > > > confluent.io
> > > > > > > >
> > > > > > > [
> > > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.
> > > confluent.io_wp-2Dcontent_uploads_Untitled-2Ddesign-
> > > 2D12.png&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=
> > > EzRhmS

Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

[Sriram Subramanian]

Congratulations Rajini!

On Wed, Jan 17, 2018 at 11:40 AM, Edoardo Comar  wrote:

> Congratulations Rajini! Bravissima!
> --
>
> Edoardo Comar
>
> IBM Message Hub
>
> IBM UK Ltd, Hursley Park, SO21 2JN
>
>
>
> From:   Guozhang Wang 
> To: dev@kafka.apache.org
> Date:   17/01/2018 19:29
> Subject:Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram
>
>
>
> Congratulations Rajini ！
>
> Guozhang
>
> On Wed, Jan 17, 2018 at 11:08 AM, Bill Bejeck  wrote:
>
> > Congrats Rajini!
> >
> > -Bill
> >
> > On Wed, Jan 17, 2018 at 2:02 PM, Matthias J. Sax 
> > wrote:
> >
> > > Congrats Rajini!!
> > >
> > > -Matthias
> > >
> > > On 1/17/18 10:54 AM, Vahid S Hashemian wrote:
> > > > Great news! Congratulations Rajini!
> > > >
> > > > --Vahid
> > > >
> > > >
> > > >
> > > > From:   Gwen Shapira 
> > > > To: "dev@kafka.apache.org" , Users
> > > > 
> > > > Date:   01/17/2018 10:49 AM
> > > > Subject:[ANNOUNCE] New Kafka PMC Member: Rajini Sivaram
> > > >
> > > >
> > > >
> > > > Dear Kafka Developers, Users and Fans,
> > > >
> > > > Rajini Sivaram became a committer in April 2017.  Since then, she
> > > remained
> > > > active in the community and contributed major patches, reviews and
> KIP
> > > > discussions. I am glad to announce that Rajini is now a member of
> the
> > > > Apache Kafka PMC.
> > > >
> > > > Congratulations, Rajini and looking forward to your future
> > contributions.
> > > >
> > > > Gwen, on behalf of Apache Kafka PMC
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > >
> >
>
>
>
> --
> -- Guozhang
>
>
>
> Unless stated otherwise above:
> IBM United Kingdom Limited - Registered in England and Wales with number
> 741598.
> Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6 3AU
>

Re: [VOTE] Allowing write access to GitHub repositories (aka GitBox)

[Sriram Subramanian]

+1

On Tue, Dec 12, 2017 at 8:22 AM, Manikumar 
wrote:

> +1
>
> On Tue, Dec 12, 2017 at 9:49 PM, Rajini Sivaram 
> wrote:
>
> > +1
> >
> > Thanks, Ismael!
> >
> > On Tue, Dec 12, 2017 at 4:18 PM, Damian Guy 
> wrote:
> >
> > > +1
> > >
> > > On Tue, 12 Dec 2017 at 15:47 Ismael Juma  wrote:
> > >
> > > > Hi all,
> > > >
> > > > The Apache Infra team has started a new project earlier this year
> > called
> > > > GitBox that supports two-way synchronization between GitHub and
> > > > git-wip-us.apache.org and, most importantly, provides GitHub write
> > > access
> > > > to committers. GitBox is not generally available yet, but individual
> > > > projects can ask to be migrated.
> > > >
> > > > I would like to start a vote on migrating kafka and kafka-site to
> > GitBox
> > > > and:
> > > >
> > > > 1. Providing GitHub write access to committers (this requires dual
> > factor
> > > > authentication)
> > > > 2. Allowing merges via the GitHub UI as well as the existing merge
> > script
> > > > 3. Enabling protected branches for trunk and release branches so that
> > > > merges via the GitHub UI can only be done if the tests pass and the
> PR
> > > has
> > > > been approved by a committer
> > > > 4. Only allowing the "squash and merge" strategy for GitHub UI merges
> > > > 5. Updating the merge script so that the GitHub git repo is the
> target
> > of
> > > > the merge
> > > > 6. Disallowing force pushes to trunk and release branches
> > > >
> > > > The discussion thread talks about some of the pros and cons (mostly
> > pros)
> > > > of this change:
> > > >
> > > >
> > > > https://lists.apache.org/thread.html/7031168e7026222169c66fed29f520
> > > 0fc4b561df28c242ccf706f326@%3Cdev.kafka.apache.org%3E
> > > >
> > > > The vote will run for 72 hours.
> > > >
> > > > Ismael
> > > >
> > >
> >
>

Re: [Vote] KIP-150 - Kafka-Streams Cogroup

[Sriram Subramanian]

+1

On Fri, Jun 9, 2017 at 2:24 PM, Jay Kreps  wrote:

> +1
>
> -Jay
>
> On Thu, Jun 8, 2017 at 11:16 AM, Guozhang Wang  wrote:
>
> > I think we can continue on this voting thread.
> >
> > Currently we have one binding vote and 2 non-binging votes. I would like
> to
> > call out for other people especially committers to also take a look at
> this
> > proposal and vote.
> >
> >
> > Guozhang
> >
> >
> > On Wed, Jun 7, 2017 at 6:37 PM, Kyle Winkelman  >
> > wrote:
> >
> > > Just bringing people's attention to the vote thread for my KIP. I
> started
> > > it before another round of discussion happened. Not sure the protocol
> so
> > > someone let me know if I am supposed to restart the vote.
> > > Thanks,
> > > Kyle
> > >
> > > On May 24, 2017 8:49 AM, "Bill Bejeck"  wrote:
> > >
> > > > +1  for the KIP and +1 what Xavier said as well.
> > > >
> > > > On Wed, May 24, 2017 at 3:57 AM, Damian Guy 
> > > wrote:
> > > >
> > > > > Also, +1 for the KIP
> > > > >
> > > > > On Wed, 24 May 2017 at 08:57 Damian Guy 
> > wrote:
> > > > >
> > > > > > +1 to what Xavier said
> > > > > >
> > > > > > On Wed, 24 May 2017 at 06:45 Xavier Léauté 
> > > > wrote:
> > > > > >
> > > > > >> I don't think we should wait for entries from each stream, since
> > > that
> > > > > >> might
> > > > > >> limit the usefulness of the cogroup operator. There are
> instances
> > > > where
> > > > > it
> > > > > >> can be useful to compute something based on data from one or
> more
> > > > > stream,
> > > > > >> without having to wait for all the streams to produce something
> > for
> > > > the
> > > > > >> group. In the example I gave in the discussion, it is possible
> to
> > > > > compute
> > > > > >> impression/auction statistics without having to wait for click
> > data,
> > > > > which
> > > > > >> can typically arrive several minutes late.
> > > > > >>
> > > > > >> We could have a separate discussion around adding inner / outer
> > > > > modifiers
> > > > > >> to each of the streams to decide which fields are optional /
> > > required
> > > > > >> before sending updates if we think that might be useful.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Tue, May 23, 2017 at 6:28 PM Guozhang Wang <
> wangg...@gmail.com
> > >
> > > > > wrote:
> > > > > >>
> > > > > >> > The proposal LGTM, +1
> > > > > >> >
> > > > > >> > One question I have is about when to send the record to the
> > > resulted
> > > > > >> KTable
> > > > > >> > changelog. For example in your code snippet in the wiki page,
> > > before
> > > > > you
> > > > > >> > see the end result of
> > > > > >> >
> > > > > >> > 1L, Customer[
> > > > > >> >
> > > > > >> >   cart:{Item[no:01], Item[no:03],
> > > Item[no:04]},
> > > > > >> >   purchases:{Item[no:07], Item[no:08]},
> > > > > >> >   wishList:{Item[no:11]}
> > > > > >> >   ]
> > > > > >> >
> > > > > >> >
> > > > > >> > You will firs see
> > > > > >> >
> > > > > >> > 1L, Customer[
> > > > > >> >
> > > > > >> >   cart:{Item[no:01]},
> > > > > >> >   purchases:{},
> > > > > >> >   wishList:{}
> > > > > >> >   ]
> > > > > >> >
> > > > > >> > 1L, Customer[
> > > > > >> >
> > > > > >> >   cart:{Item[no:01]},
> > > > > >> >   purchases:{Item[no:07],Item[no:08]},
> > > > > >> >
> > > > > >> >   wishList:{}
> > > > > >> >   ]
> > > > > >> >
> > > > > >> > 1L, Customer[
> > > > > >> >
> > > > > >> >   cart:{Item[no:01]},
> > > > > >> >   purchases:{Item[no:07],Item[no:08]},
> > > > > >> >
> > > > > >> >   wishList:{}
> > > > > >> >   ]
> > > > > >> >
> > > > > >> > ...
> > > > > >> >
> > > > > >> >
> > > > > >> > I'm wondering if it makes more sense to only start sending the
> > > > update
> > > > > if
> > > > > >> > the corresponding agg-key has seen at least one input from
> each
> > of
> > > > the
> > > > > >> > input stream? Maybe it is out of the scope of this KIP and we
> > can
> > > > make
> > > > > >> it a
> > > > > >> > more general discussion in a separate one.
> > > > > >> >
> > > > > >> >
> > > > > >> > Guozhang
> > > > > >> >
> > > > > >> >
> > > > > >> > On Fri, May 19, 2017 at 8:37 AM, Xavier Léauté <
> > > xav...@confluent.io
> > > > >
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > Hi Kyle, I left a few more comments in the discussion
> thread,
> > if
> > > > you
> > > > > >> > > wouldn't mind taking a look
> > > > > >> > >
> > > > > >> > > On Fri, May 19, 2017 at 5:31 AM Kyle Winkelman <
> > > > > >> winkelman.k...@gmail.com
> > > > > >> > >
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > > Hello all,
> > > > > >> > > >
> > > > > >> > > > I would like to start the vote on KIP-150.
> > > > > >> > > >
> > > > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 150+-+
> > > > > >> > > Kafka-Streams+Cogroup
> > > > > >> > > >
> > > > 

Re: [VOTE] KIP-161: streams deserialization exception handlers

[Sriram Subramanian]

+1

On Sat, Jun 24, 2017 at 1:38 AM, Matthias J. Sax 
wrote:

> +1
>
> On 6/23/17 9:45 AM, Guozhang Wang wrote:
> > +1.
> >
> > On Fri, Jun 23, 2017 at 6:42 AM, Bill Bejeck  wrote:
> >
> >> Thanks for the KIP!
> >>
> >> +1
> >>
> >> -Bill
> >>
> >> On Fri, Jun 23, 2017 at 7:15 AM, Damian Guy 
> wrote:
> >>
> >>> Thanks for the KIP Eno.
> >>> +1 (binding)
> >>>
> >>> On Fri, 23 Jun 2017 at 11:00 Eno Thereska 
> >> wrote:
> >>>
>  Starting voting thread for:
> 
>  https://cwiki.apache.org/confluence/display/KAFKA/KIP-161%3A+streams+
> >>> deserialization+exception+handlers
>  <
>  https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>> 161:+streams+deserialization+exception+handlers
> >
> 
>  Thanks
>  Eno
> >>>
> >>
> >
> >
> >
>
>

Re: [VOTE] KIP 130: Expose states of active tasks to KafkaStreams public API

[Sriram Subramanian]

+1

On Fri, Jun 30, 2017 at 12:10 AM, Damian Guy  wrote:

> I know i voted before, but now my vote is binding so...
>
> +1 (binding)
>
> On Thu, 18 May 2017 at 23:46 Matthias J. Sax 
> wrote:
>
> > +1
> >
> > On 5/18/17 8:26 AM, Bill Bejeck wrote:
> > > +1
> > >
> > > On Thu, May 18, 2017 at 6:54 AM, Florian Hussonnois <
> > fhussonn...@gmail.com>
> > > wrote:
> > >
> > >> Hi all,
> > >>
> > >> I've finally found time to update the KIP. The toString() is annotated
> > as
> > >> deprecated. I have also rebase the PR with the current trunk.
> > >> So sorry to have been so long on this KIP.
> > >>
> > >> Thanks.
> > >>
> > >> 2017-04-24 18:56 GMT+02:00 Guozhang Wang :
> > >>
> > >>> Florian, could you also add the part of deprecating
> > >> `KafkaStreams.toString`
> > >>> in your KIP as well?
> > >>>
> > >>>
> > >>> Guozhang
> > >>>
> > >>> On Fri, Apr 21, 2017 at 8:32 AM, Damian Guy 
> > >> wrote:
> > >>>
> >  +1
> > 
> >  On Fri, 21 Apr 2017 at 09:06 Eno Thereska 
> > >>> wrote:
> > 
> > > +1 (non-binding)
> > >
> > > Thanks
> > > Eno
> > >
> > >> On 21 Apr 2017, at 05:58, Guozhang Wang 
> > >> wrote:
> > >>
> > >> +1. Thanks a lot for the KIP!
> > >>
> > >> Guozhang
> > >>
> > >> On Wed, Apr 5, 2017 at 1:57 PM, Florian Hussonnois <
> > > fhussonn...@gmail.com>
> > >> wrote:
> > >>
> > >>> Hi All,
> > >>>
> > >>> I would like to start the vote for the KIP-130 :
> > >>>
> > >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP+
> > >>> 130%3A+Expose+states+of+active+tasks+to+KafkaStreams+public+API
> > >>>
> > >>> Thanks,
> > >>>
> > >>> --
> > >>> Florian HUSSONNOIS
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> -- Guozhang
> > >
> > >
> > 
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> -- Guozhang
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Florian HUSSONNOIS
> > >>
> > >
> >
> >
>

Re: [ANNOUNCE] New Kafka PMC member Ismael Juma

[Sriram Subramanian]

Congratulations Ismael!

On Wed, Jul 5, 2017 at 2:06 PM, Rajini Sivaram 
wrote:

> Congratulations, Ismael!
>
> On Wed, Jul 5, 2017 at 9:55 PM, Jun Rao  wrote:
>
> > Hi, Everyone,
> >
> > Ismael Juma has been active in the Kafka community since he became
> > a Kafka committer about a year ago. I am glad to announce that Ismael is
> > now a member of Kafka PMC.
> >
> > Congratulations, Ismael!
> >
> > Jun
> >
>

Re: [ANNOUNCE] New Kafka PMC member Jason Gustafson

[Sriram Subramanian]

Congratulations Jason!

On Wed, Jul 12, 2017 at 8:02 AM, Rajini Sivaram 
wrote:

> Congratulations, Jason!
>
> On Wed, Jul 12, 2017 at 3:53 PM, Ismael Juma  wrote:
>
> > Congratulations Jason!
> >
> > Ismael
> >
> > On Tue, Jul 11, 2017 at 10:32 PM, Guozhang Wang 
> > wrote:
> >
> > > Hi Everyone,
> > >
> > > Jason Gustafson has been very active in contributing to the Kafka
> > community
> > > since he became a Kafka committer last September and has done lots of
> > > significant work including the most recent exactly-once project. In
> > > addition, Jason has initiated or participated in the design discussion
> of
> > > more than 30 KIPs in which he has consistently brought in great
> judgement
> > > and insights throughout his communication. I am glad to announce that
> > Jason
> > > has now become a PMC member of the project.
> > >
> > > Congratulations, Jason!
> > >
> > > -- Guozhang
> > >
> >
>

Re: [VOTE] KIP-167: Add interface for the state store restoration process

[Sriram Subramanian]

+1

On Sat, Jul 15, 2017 at 9:45 AM, Damian Guy  wrote:

> +1
>
> On Thu, 13 Jul 2017 at 07:13 Eno Thereska  wrote:
>
> > +1 (non-binding).
> >
> > Thanks Bill.
> >
> > Eno
> > > On 12 Jul 2017, at 09:12, Bill Bejeck  wrote:
> > >
> > > All,
> > >
> > > Now that we've concluded a second round of discussion on KIP-167, I'd
> > like
> > > to start a vote.
> > >
> > >
> > > Thanks,
> > > Bill
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 167%3A+Add+interface+for+the+state+store+restoration+process
> >
> >
>

Re: [DISCUSS] 2017 October release planning and release version

[Sriram Subramanian]

+1!

On Fri, Jul 21, 2017 at 12:35 PM, Jay Kreps  wrote:

> 1.0! Let's do it!
>
> -Jay
>
> On Tue, Jul 18, 2017 at 3:36 PM, Guozhang Wang  wrote:
>
> > Hi all,
> >
> > With 0.11.0.0 out of the way, I would like to volunteer to be the
> > release manager
> > for our next time-based feature release. See https://cwiki.apache.org/
> > confluence/display/KAFKA/Time+Based+Release+Plan if you missed
> > previous communication
> > on time-based releases or need a reminder.
> >
> > I put together a draft release plan with October 2017 as the release
> month
> > (as previously agreed) and a list of KIPs that have already been voted:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2017.Oct
> > As of today we already have 10 KIPs voted, including 2 merged and 3 with
> > PRs under review. As we start the process more KIPs are expected to be
> > added until the KIP freeze date.
> >
> > In addition to the current release plan, I would also like to propose to
> > set the release version to 1.0.0. More specifically, we will bump up the
> > major version from 0.11 to 1.0, and change the semantics of release
> digits
> > as:
> >
> > major.minor.bugfix[.release-candidate]
> >
> > To be better aligned with software versioning (https://en.wikipedia.org/
> > wiki/Software_versioning). Moving forward we can use three digits instead
> > of four in most places that do not require to indicate the rc number.
> Here
> > is my motivation:
> >
> > 1) Kafka has significantly evolved from its first Apache release of 0.7.0
> > (incubating) as a pub-sub messaging system into a distributed streaming
> > platform that can enable publish / store / process real-time data
> streams,
> > with the addition of replication (0.8.0), quota / security support for
> > multi-tenancy (0.8.2, 0.9.0), Connect and Streams API (0.9.0, 0.10.0),
> and
> > most recently the exactly-once support to have the desired
> > semantics (0.11.0); I think now is a good time to mark the release as a
> > major milestone in the evolution of Apache Kafka.
> >
> > 2) Some people believe 0.x means that the software is immature or not
> > stable, or the public APIs may subject to change incompatibly regardless
> > the fact that Kafka has been widely adopted in productions and the
> > community has made a great effort on maintaining backward compatibility.
> > Making Kafka 1.x will help with promoting the project for that
> perception.
> >
> > 3) Having three digits as of "major.minor.bugfix" is more natural from a
> > software version understanding pov and aligned with other open source
> > projects as well.
> >
> > How do people feel about 1.0.0.x as the next Kafka version? Please share
> > your thoughts.
> >
> > -- Guozhang
> >
>

Re: [VOTE] KIP-167 (Addendum): Add interface for the state store restoration process

[Sriram Subramanian]

+1

On Fri, Jul 21, 2017 at 12:08 PM, Guozhang Wang  wrote:

> +1
>
> On Thu, Jul 20, 2017 at 11:00 PM, Matthias J. Sax 
> wrote:
>
> > +1
> >
> > On 7/20/17 4:22 AM, Bill Bejeck wrote:
> > > Hi,
> > >
> > > After working on the PR for this KIP I discovered that we need to add
> and
> > > additional parameter (TopicPartition) to the StateRestoreListener
> > interface
> > > methods.
> > >
> > > The addition of the TopicPartition is required as the
> > StateRestoreListener
> > > is for the entire application, thus all tasks with recovering state
> > stores
> > > call the same listener instance.  The TopicPartition is needed to
> > > disambiguate the progress of the state store recovery.
> > >
> > > For those that have voted before, please review the updated KIP
> > >  > 167:+Add+interface+for+the+state+store+restoration+process>
> > > and
> > > re-vote.
> > >
> > > Thanks,
> > > Bill
> > >
> >
> >
>
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-182 - Reduce Streams DSL overloads and allow easier use of custom storage engines

[Sriram Subramanian]

+1

On Thu, Aug 24, 2017 at 10:20 AM, Guozhang Wang  wrote:

> +1. Thanks Damian!
>
> On Thu, Aug 24, 2017 at 9:47 AM, Bill Bejeck  wrote:
>
> > Thanks for the KIP!
> >
> > +1
> >
> > Thanks,
> > Bill
> >
> > On Thu, Aug 24, 2017 at 12:25 PM, Damian Guy 
> wrote:
> >
> > > Hi,
> > >
> > > I'd like to kick off the voting thread for KIP-182:
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 182%3A+Reduce+Streams+DSL+overloads+and+allow+easier+
> > > use+of+custom+storage+engines
> > >
> > > Thanks,
> > > Damian
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-182 - Reduce Streams DSL overloads and allow easier use of custom storage engines

[Sriram Subramanian]

+1

On Tue, Sep 5, 2017 at 1:33 PM, Guozhang Wang  wrote:

> +1
>
> On Fri, Sep 1, 2017 at 3:45 PM, Matthias J. Sax 
> wrote:
>
> > +1
> >
> > On 9/1/17 2:53 PM, Bill Bejeck wrote:
> > > +1
> > >
> > > On Thu, Aug 31, 2017 at 10:20 AM, Damian Guy 
> > wrote:
> > >
> > >> Thanks everyone for voting! Unfortunately i've had to make a bit of an
> > >> update based on some issues found during implementation.
> > >> The main changes are:
> > >> BytesStoreSupplier -> StoreSupplier
> > >> Addition of:
> > >> WindowBytesStoreSupplier, KeyValueBytesStoreSupplier,
> > >> SessionBytesStoreSupplier that will restrict store types to  > byte[]>
> > >> 3 new overloads added to Materialized to enable developers to create a
> > >> Materialized of the appropriate type, i..e, WindowStore etc
> > >> Update DSL where Materialized is used such that the stores have
> generic
> > >> types of 
> > >> Some minor changes to the arguments to Store#persistentWindowStore and
> > >> Store#persistentSessionStore
> > >>
> > >> Please take a look and recast the votes.
> > >>
> > >> Thanks for your time,
> > >> Damian
> > >>
> > >> On Fri, 25 Aug 2017 at 17:05 Matthias J. Sax 
> > >> wrote:
> > >>
> > >>> Thanks Damian. Great KIP!
> > >>>
> > >>> +1
> > >>>
> > >>>
> > >>> -Matthias
> > >>>
> > >>> On 8/25/17 6:45 AM, Damian Guy wrote:
> > >>>> Hi,
> > >>>>
> > >>>> I've just realised we need to add two methods to StateStoreBuilder
> or
> > >> it
> > >>>> isn't going to work:
> > >>>>
> > >>>> Map logConfig();
> > >>>> boolean loggingEnabled();
> > >>>>
> > >>>> These are needed when we are building the topology and determining
> > >>>> changelog topic names and configs.
> > >>>>
> > >>>>
> > >>>> I've also update the KIP to add
> > >>>>
> > >>>> StreamBuilder#stream(String topic)
> > >>>>
> > >>>> StreamBuilder#stream(String topic, Consumed options)
> > >>>>
> > >>>>
> > >>>> Thanks
> > >>>>
> > >>>>
> > >>>> On Thu, 24 Aug 2017 at 22:11 Sriram Subramanian 
> > >>> wrote:
> > >>>>
> > >>>>> +1
> > >>>>>
> > >>>>> On Thu, Aug 24, 2017 at 10:20 AM, Guozhang Wang <
> wangg...@gmail.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> +1. Thanks Damian!
> > >>>>>>
> > >>>>>> On Thu, Aug 24, 2017 at 9:47 AM, Bill Bejeck 
> > >>> wrote:
> > >>>>>>
> > >>>>>>> Thanks for the KIP!
> > >>>>>>>
> > >>>>>>> +1
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>> Bill
> > >>>>>>>
> > >>>>>>> On Thu, Aug 24, 2017 at 12:25 PM, Damian Guy <
> damian@gmail.com
> > >
> > >>>>>> wrote:
> > >>>>>>>
> > >>>>>>>> Hi,
> > >>>>>>>>
> > >>>>>>>> I'd like to kick off the voting thread for KIP-182:
> > >>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >>>>>>>> 182%3A+Reduce+Streams+DSL+overloads+and+allow+easier+
> > >>>>>>>> use+of+custom+storage+engines
> > >>>>>>>>
> > >>>>>>>> Thanks,
> > >>>>>>>> Damian
> > >>>>>>>>
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> -- Guozhang
> > >>>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>>
> > >>
> > >
> >
> >
>
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-197: Include Connector type in Connector REST API

[Sriram Subramanian]

+1

On Fri, Sep 8, 2017 at 4:33 PM, Randall Hauch  wrote:

> +1 (non-binding)
>
> Randall
>
> On Fri, Sep 8, 2017 at 6:32 PM, Ted Yu  wrote:
>
> > Hi,
> > Please take a look at the following and cast your vote:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 197+Connect+REST+API+should+include+the+connector+type+
> > when+describing+a+connector
> >
> > Thanks
> >
>

Re: [VOTE] KIP-198: Remove ZK dependency from Streams Reset Tool

[Sriram Subramanian]

+1

On Fri, Sep 8, 2017 at 3:04 PM, Guozhang Wang  wrote:

> +1, thanks.
>
> On Fri, Sep 8, 2017 at 1:54 PM, Bill Bejeck  wrote:
>
> > +1
> >
> > Thanks,
> > Bill
> >
> > On Fri, Sep 8, 2017 at 4:51 PM, Matthias J. Sax 
> > wrote:
> >
> > > We want to deprecate it for 1.0.0 release. Unclear how long to keep it.
> > >
> > > The point is, that the parameter will just be ignored after it got
> > > deprecated and thus has no effect anymore when running the tool. Thus,
> > > we can keep it as long as we think some people might have scripts that
> > > used the parameter.
> > >
> > > Fixed the typo. Thx.
> > >
> > >
> > > -Matthias
> > >
> > > On 9/8/17 1:20 PM, Ted Yu wrote:
> > > > bq. parameter `--zookeper` that will be deprecated.
> > > >
> > > > Can you clarify in which release the parameter will be deprecated and
> > in
> > > > which release it will be removed ?
> > > >
> > > > On Fri, Sep 8, 2017 at 1:15 PM, Matthias J. Sax <
> matth...@confluent.io
> > >
> > > > wrote:
> > > >
> > > >> Hi,
> > > >>
> > > >> I want to propose KIP-198 for 1.0.0 release. It's about removing ZK
> > > >> dependency from Streams application reset tool. It's a fairly simply
> > > >> KIP, such I want to start the vote immediately.
> > > >>
> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >> 198%3A+Remove+ZK+dependency+from+Streams+Reset+Tool
> > > >>
> > > >>
> > > >> Thanks a lot!
> > > >>
> > > >> -Matthias
> > > >>
> > > >>
> > > >>
> > > >
> > >
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: [VOTE] KIP-199: Add Kafka Connect Offset Tool

[Sriram Subramanian]

+1

On Mon, Sep 11, 2017 at 2:56 PM, Gwen Shapira  wrote:

> +1
>
> On Mon, Sep 11, 2017 at 1:33 PM Ted Yu  wrote:
>
> > +1
> >
> > On Mon, Sep 11, 2017 at 7:43 AM, Randall Hauch  wrote:
> >
> > > I'd like to start the vote on KIP-199 to add a command line tool that
> > will
> > > allow Connect operators to read, modify, and update source connector
> > > offsets. Details are here:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 199%3A+Add+Kafka+Connect+offset+tool
> > >
> > > Thanks, and best regards.
> > >
> > > Randall
> > >
> >
>

Re: [VOTE] KIP-196: Add metrics to Kafka Connect framework

[Sriram Subramanian]

+1

On Mon, Sep 11, 2017 at 2:56 PM, Gwen Shapira  wrote:

> +1
>
> Thanks for this. Can't wait for more complete monitoring for Connect.
>
> On Mon, Sep 11, 2017 at 7:40 AM Randall Hauch  wrote:
>
> > I'd like to start the vote on KIP-196 to add metrics to the Kafka Connect
> > framework so the worker processes can be measured. Details are here:
> >
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 196%3A+Add+metrics+to+Kafka+Connect+framework
> >
> > Thanks, and best regards.
> >
> > Randall
> >
>

Re: [VOTE] KIP-196: Add metrics to Kafka Connect framework

[Sriram Subramanian]

+1

On Tue, Sep 12, 2017 at 12:41 PM, Gwen Shapira  wrote:

> My +1 remains :)
>
> On Tue, Sep 12, 2017 at 12:31 PM Randall Hauch  wrote:
>
> > The KIP was modified (most changes due to reorganization of metrics).
> Feel
> > free to re-vote if you dislike the changes.
> >
> > On Mon, Sep 11, 2017 at 8:40 PM, Sriram Subramanian 
> > wrote:
> >
> > > +1
> > >
> > > On Mon, Sep 11, 2017 at 2:56 PM, Gwen Shapira 
> wrote:
> > >
> > > > +1
> > > >
> > > > Thanks for this. Can't wait for more complete monitoring for Connect.
> > > >
> > > > On Mon, Sep 11, 2017 at 7:40 AM Randall Hauch 
> > wrote:
> > > >
> > > > > I'd like to start the vote on KIP-196 to add metrics to the Kafka
> > > Connect
> > > > > framework so the worker processes can be measured. Details are
> here:
> > > > >
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 196%3A+Add+metrics+to+Kafka+Connect+framework
> > > > >
> > > > > Thanks, and best regards.
> > > > >
> > > > > Randall
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

[Sriram Subramanian]

FWIW, I agree that time metrics have been very useful in the past. The
reasoning around perf overhead seems reasonable as well. Can we agree on a
subset of time metrics that we feel would be super useful for debugging?

On Tue, Sep 12, 2017 at 6:08 PM, Roger Hoover 
wrote:

> Thanks, Ewen.
>
> I agree with you on the overhead of measuring time for SMTs and
> converters.  I'd still argue for keeping other metrics like flush time b/c
> even small batches should still be small overhead compared to writing to a
> sink.
>
> On Tue, Sep 12, 2017 at 3:06 PM, Ewen Cheslack-Postava 
> wrote:
>
> > Requests are generally substantial batches of data, you are not
> guaranteed
> > that for the processing batches both because source connectors can hand
> you
> > batches of whatever size they want and consumer's max.poll.records can be
> > overridden.
> >
> > Both SMTs and converters are a concern because they can both be
> relatively
> > cheap such that just checking the time in between them could possibly
> dwarf
> > the cost of applying them.
> >
> > Also, another thought re: rebalance metrics: we are already getting some
> > info via AbstractCoordinator and those actually provide a bit more detail
> > in some ways (e.g. join & sync vs the entire rebalance). Not sure if we
> > want to effectively duplicate some info so it can all be located under
> > Connect names or rely on the existing metrics for some of these.
> >
> > -Ewen
> >
> > On Tue, Sep 12, 2017 at 2:05 PM, Roger Hoover 
> > wrote:
> >
> > > Ewen,
> > >
> > > I don't know the details of the perf concern.  How is it that the Kafka
> > > broker can keep latency stats per request without suffering too much
> > > performance?  Maybe SMTs are the only concern b/c they are per-message.
> > If
> > > so, let's remove those and keep timing info for everything else like
> > > flushes, which are batch-based.
> > >
> > >
> > > On Tue, Sep 12, 2017 at 1:32 PM, Ewen Cheslack-Postava <
> > e...@confluent.io>
> > > wrote:
> > >
> > > > On Tue, Sep 12, 2017 at 10:55 AM, Gwen Shapira 
> > > wrote:
> > > >
> > > > > Ewen, you gave a nice talk at Kafka Summit where you warned about
> the
> > > > > danger of SMTs that slow down the data pipe. If we don't provide
> the
> > > time
> > > > > metrics, how will users know when their SMTs are causing
> performance
> > > > > issues?
> > > > >
> > > >
> > > > Metrics aren't the only way to gain insight about performance and
> > always
> > > > measuring this even when it's not necessarily being used may not make
> > > > sense. SMT authors are much better off starting out with a JMH or
> > similar
> > > > benchmark. What I was referring to in the talk is more about
> > > understanding
> > > > that the processing for SMTs is entirely synchronous and that means
> > > certain
> > > > classes of operations will just generally be a bad idea, e.g.
> anything
> > > that
> > > > goes out over the network to another service. You don't even really
> > need
> > > > performance info to determine that that type of transformation will
> > cause
> > > > problems.
> > > >
> > > > But my point wasn't that timing info isn't useful. It's that we know
> > that
> > > > getting timestamps is pretty expensive and we'll already be doing so
> > > > elsewhere (e.g. if a source record doesn't include a timestamp). For
> > some
> > > > use cases such as ByteArrayConverter + no SMTs + lightweight
> processing
> > > > (e.g. just gets handed to a background thread that deals with sending
> > the
> > > > data), it wouldn't be out of the question that adding 4 or so more
> > calls
> > > to
> > > > get timestamps could become a bottleneck. Since I don't know if it
> > would
> > > > but we have definitely seen the issue come up before, I would be
> > > > conservative in adding the metrics unless we had some numbers showing
> > it
> > > > doesn't matter or doesn't matter much.
> > > >
> > > > In general, I don't think metrics that require always-on measurement
> > are
> > > a
> > > > good way to get fine grained performance information. Instrumenting
> > > > different phases that imply different types of performance problems
> can
> > > be
> > > > helpful (e.g. "processing time" that should be CPU/memory throughput
> > > bound
> > > > vs. "send time" that, at least for many connectors, is more likely to
> > be
> > > IO
> > > > bound), but if you want finer-grained details, you probably either
> want
> > > > something that can be toggled on/off temporarily or just use a tool
> > > that's
> > > > really designed for the job, i.e. a profiler like perf.
> > > >
> > > > -Ewen
> > > >
> > > >
> > > > >
> > > > > Gwen
> > > > >
> > > > > On Mon, Sep 11, 2017 at 7:50 PM Ewen Cheslack-Postava <
> > > e...@confluent.io
> > > > >
> > > > > wrote:
> > > > >
> > > > > > re: questions about additional metrics, I think we'll undoubtedly
> > > find
> > > > > more
> > > > > > that people want in practice, but as I mentioned earlier I think
> > it's
> > > > > > better to add the ones we kno

Re: [ANNOUCE] Apache Kafka 0.11.0.1 Released

[Sriram Subramanian]

Thanks for driving the release Damian.

> On Sep 13, 2017, at 1:18 PM, Guozhang Wang  wrote:
> 
> Thanks for driving this Damian!
> 
> 
> Guozhang
> 
>> On Wed, Sep 13, 2017 at 4:36 AM, Damian Guy  wrote:
>> 
>> The Apache Kafka community is pleased to announce the release for Apache
>> Kafka 0.11.0.1. This is a bug fix release that fixes 51 issues in 0.11.0.0.
>> 
>> All of the changes in this release can be found in the release notes:
>> *https://archive.apache.org/dist/kafka/0.11.0.1/RELEASE_NOTES.html
>> 
>> > .>
>> 
>> Apache Kafka is a distributed streaming platform with four four core APIs:
>> 
>> ** The Producer API allows an application to publish a stream records to
>> one or more Kafka topics.
>> 
>> ** The Consumer API allows an application to subscribe to one or more
>> topics and process the stream of records produced to them.
>> 
>> ** The Streams API allows an application to act as a stream processor,
>> consuming an input stream from one or more topics and producing an output
>> stream to one or more output topics, effectively transforming the input
>> streams to output streams.
>> 
>> ** The Connector API allows building and running reusable producers or
>> consumers that connect Kafka topics to existing applications or data
>> systems. For example, a connector to a relational database might capture
>> every change to a table.three key capabilities:
>> 
>> 
>> With these APIs, Kafka can be used for two broad classes of application:
>> 
>> ** Building real-time streaming data pipelines that reliably get data
>> between systems or applications.
>> 
>> ** Building real-time streaming applications that transform or react to the
>> streams of data.
>> 
>> 
>> You can download the source release from
>> https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.1/k
>> afka-0.11.0.1-src.tgz
>> > 1/kafka-0.10.2.1-src.tgz>
>> 
>> and binary releases from
>> *https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.1/
>> kafka_2.11-0.11.0.1.tgz
>> > 0/kafka_2.11-0.11.0.0.tgz>
>> > kafka_2.11-0.11.0.1.tgz
>> > 0/kafka_2.11-0.11.0.0.tgz>
>>> *
>> 
>> *https://www.apache.org/dyn/closer.cgi?path=/kafka/0.11.0.1/
>> kafka_2.12-0.11.0.1.tgz
>> > 0/kafka_2.12-0.11.0.0.tgz>
>> > kafka_2.12-0.11.0.1.tgz
>> > 0/kafka_2.12-0.11.0.0.tgz>
>>> *
>> 
>> 
>> A big thank you for the following 33 contributors to this release!
>> 
>> Apurva Mehta, Bill Bejeck, Colin P. Mccabe, Damian Guy, Derrick Or, Dong
>> Lin, dongeforever, Eno Thereska, Ewen Cheslack-Postava, Gregor Uhlenheuer,
>> Guozhang Wang, Hooman Broujerdi, huxihx, Ismael Juma, Jan Burkhardt, Jason
>> Gustafson, Jeff Klukas, Jiangjie Qin, Joel Dice, Konstantine Karantasis,
>> Manikumar Reddy, Matthias J. Sax, Max Zheng, Paolo Patierno, ppatierno,
>> radzish, Rajini Sivaram, Randall Hauch, Robin Moffatt, Stephane Roset,
>> umesh chaudhary, Vahid Hashemian, Xavier Léauté
>> 
>> We welcome your help and feedback. For more information on how to
>> report problems, and to get involved, visit the project website at
>> http://kafka.apache.org/
>> 
>> 
>> Thanks,
>> Damian
> 
> 
> 
> -- 
> -- Guozhang

subscription request

[Sriram Subramanian]

subscription request

[Sriram Subramanian]

Re: subscription request

[Sriram Subramanian]

Sorry, wrong thread.

On Wed, May 25, 2016 at 5:12 PM, Sriram Subramanian 
wrote:

>
>

Re: [HEADS-UP] Modification to KIP Template

[Sriram Subramanian]

++1

Sent from my iPhone

> On May 26, 2016, at 7:24 PM, Gwen Shapira  wrote:
> 
> Hi Kafka Developers,
> 
> Just a quick heads-up that I added a new section to the KIP template: "Test
> Plans".
> I think its a good habit to think about how a feature will be tested while
> planning it. I'm talking about high-level notes on system tests, not gritty
> details.
> 
> This will apply to new KIPs, not ones in discussion/implementation phases
> (although if your KIP is under discussion and you want to add test plans,
> it will be very nice of you).
> 
> I figured we all agree that thinking a bit about tests is a good idea, so I
> added it first and started a discussion later. If you strongly object,
> please respond with strong objections. Wikis are easy to edit :)
> 
> Gwen

Re: [VOTE] KIP-65 Expose timestamps to Connect

[Sriram Subramanian]

+1

On Fri, Jun 24, 2016 at 3:03 PM, Jason Gustafson  wrote:

> +1 (non-binding)
>
> On Fri, Jun 24, 2016 at 11:26 AM, Jay Kreps  wrote:
>
> > +1
> >
> > On Fri, Jun 24, 2016 at 10:59 AM, Shikhar Bhushan 
> > wrote:
> >
> > > Since there isn't much to discuss with this KIP, bringing it to a vote
> > >
> > > KIP:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-65%3A+Expose+timestamps+to+Connect
> > >
> > > Pull request: https://github.com/apache/kafka/pull/1537
> > >
> > > Thanks,
> > >
> > > Shikhar
> > >
> >
>

Re: [VOTE] KIP-67: Queryable state for Kafka Streams

[Sriram Subramanian]

+1

On Thu, Jul 7, 2016 at 9:53 AM, Henry Cai 
wrote:

> +1
>
> On Thu, Jul 7, 2016 at 6:48 AM, Michael Noll  wrote:
>
> > +1 (non-binding)
> >
> > On Thu, Jul 7, 2016 at 10:24 AM, Damian Guy 
> wrote:
> >
> > > Thanks Henry - we've updated the KIP with an example and the new config
> > > parameter required. FWIW the user doesn't register a listener, they
> > provide
> > > a host:port in config. It is expected they will start a service running
> > on
> > > that host:port that they can use to connect to the running KafkaStreams
> > > Instance.
> > >
> > > Thanks,
> > > Damian
> > >
> > > On Thu, 7 Jul 2016 at 06:06 Henry Cai 
> > wrote:
> > >
> > > > It wasn't quite clear to me how the user program interacts with the
> > > > discovery API, especially on the user supplied listener part, how
> does
> > > the
> > > > user program supply that listener to KafkaStreams and how does
> > > KafkaStreams
> > > > know which port the user listener is running, maybe a more complete
> > > > end-to-end example including the steps on registering the user
> listener
> > > and
> > > > whether the user listener needs to be involved with task
> reassignment.
> > > >
> > > >
> > > > On Wed, Jul 6, 2016 at 9:13 PM, Guozhang Wang 
> > > wrote:
> > > >
> > > > > ＋1
> > > > >
> > > > > On Wed, Jul 6, 2016 at 12:44 PM, Damian Guy 
> > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'd like to initiate the voting process for KIP-67
> > > > > > <
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-67%3A+Queryable+state+for+Kafka+Streams
> > > > > > >
> > > > > >
> > > > > > KAFKA-3909  is
> > the
> > > > top
> > > > > > level JIRA for this effort.
> > > > > >
> > > > > > Initial PRs for Step 1 of the process are:
> > > > > > Expose State Store Names <
> > https://github.com/apache/kafka/pull/1526>
> > > > and
> > > > > > Query Local State Stores <
> > https://github.com/apache/kafka/pull/1565>
> > > > > >
> > > > > > Thanks,
> > > > > > Damian
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Best regards,
> > Michael Noll
> >
> >
> >
> > *Michael G. Noll | Product Manager | Confluent | +1 650.453.5860Download
> > Apache Kafka and Confluent Platform: www.confluent.io/download
> > *
> >
>

Re: [VOTE] KIP-67: Queryable state for Kafka Streams

[Sriram Subramanian]

;> >
> > >> > > Addendum in case my previous email wasn't clear:
> > >> > >
> > >> > > > So for any given instance of a streams application there will
> > never
> > >> be
> > >> > > both a v1 and v2 alive at the same time
> > >> > >
> > >> > > That's right.  But the current live instance will be able to tell
> > >> other
> > >> > > instances, via its endpoint setting, whether it wants to be
> > contacted
> > >> at
> > >> > v1
> > >> > > or at v2.  The other instances can't guess that.  Think: if an
> older
> > >> > > instance would manually compose the "rest" of an endpoint URI,
> > having
> > >> > only
> > >> > > the host and port from the endpoint setting, it might not know
> that
> > >> the
> > >> > new
> > >> > > instances have a different endpoint suffix, for example).
> > >> > >
> > >> > >
> > >> > > On Fri, Jul 8, 2016 at 6:37 PM, Michael Noll <
> mich...@confluent.io>
> > >> > wrote:
> > >> > >
> > >> > > > Damian,
> > >> > > >
> > >> > > > about the rolling upgrade comment:  An instance A will contact
> > >> another
> > >> > > > instance B by the latter's endpoint, right?  So if A has no
> > further
> > >> > > > information available than B's host and port, then how should
> > >> instance
> > >> > A
> > >> > > > know whether it should call B at /v1/ or at /v2/?  I agree that
> my
> > >> > > > suggestion isn't foolproof, but it is afaict better than the
> > >> host:port
> > >> > > > approach.
> > >> > > >
> > >> > > >
> > >> > > >
> > >> > > > On Fri, Jul 8, 2016 at 5:15 PM, Damian Guy <
> damian@gmail.com>
> > >> > wrote:
> > >> > > >
> > >> > > >> Michael - i'm ok with changing it to a string. Any one else
> have
> > a
> > >> > > strong
> > >> > > >> opinion on this?
> > >> > > >>
> > >> > > >> FWIW - i don't think it will work fine as is during the rolling
> > >> > upgrade
> > >> > > >> scenario as the service that is listening on the port needs to
> be
> > >> > > embedded
> > >> > > >> within each instance. So for any given instance of a streams
> > >> > application
> > >> > > >> there will never be both a v1 and v2 alive at the same time
> > >> (unless of
> > >> > > >> course the process didn't shutdown properly, but then you have
> > >> another
> > >> > > >> problem...).
> > >> > > >>
> > >> > > >> On Fri, 8 Jul 2016 at 15:26 Michael Noll  >
> > >> > wrote:
> > >> > > >>
> > >> > > >> > I have one further comment about
> > >> > `StreamsConfig.USER_ENDPOINT_CONFIG`.
> > >> > > >> >
> > >> > > >> > I think we should consider to not restricting the value of
> this
> > >> > > setting
> > >> > > >> to
> > >> > > >> > only `host:port` pairs.  By design, this setting is capturing
> > >> > > >> user-driven
> > >> > > >> > metadata to define an endpoint, so why restrict the
> creativity
> > or
> > >> > > >> > flexibility of our users?  I can imagine, for example, that
> > users
> > >> > > would
> > >> > > >> > like to set values such as `https://host:port/api/rest/v1/`
> in
> > >> this
> > >> > > >> field
> > >> > > >> > (e.g. being able to distinguish between `.../v1/` and
> `.../v2/`
> > >> may
> > >> > > >> help in
> > >> > > >> > scenarios such as rolling upgrades, where, during the
> upgrade,
> > >> older
> > >> > > >> > instances may need to coexist with newer instances).
> > >&

Re: [VOTE] KIP-67: Queryable state for Kafka Streams

[Sriram Subramanian]

So, it looks like QueryableStoreTypes would be part of the streams library,
right? If a developer needs to support a new store, would they need to
change code in streams?

On Fri, Jul 15, 2016 at 3:15 PM, Jay Kreps  wrote:

> Cool, I'm +1 after the updates.
>
> -Jay
>
> On Fri, Jul 15, 2016 at 1:50 PM, Damian Guy  wrote:
>
> > Hi Guozhang, KIP updated.
> >
> > Thanks,
> > Damian
> >
> > On Fri, 15 Jul 2016 at 13:15 Guozhang Wang  wrote:
> >
> > > Hi Damian,
> > >
> > > Since the StateStoreProvider is moved into internal packages, how about
> > > just keeping the ReadOnlyXXStores interface for the queryAPI, and
> > > "QueryableStoreType" in the discoverAPI, and move the
> StateStoreProvider
> > > / QueryableStoreTypeMatcher and different implementations of the
> matcher
> > > like KeyValueStoreType / etc in a new section called "developer guide
> for
> > > customized stores"? This way we have a separate guidance for Streams
> > users,
> > > from Streams developers.
> > >
> > > Other than that, all LGTM.
> > >
> > > Guozhang
> > >
> > > On Fri, Jul 15, 2016 at 9:57 AM, Damian Guy 
> > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I've updated the KIP with changes as discussed in this Thread.
> > > >
> > > > Thanks,
> > > > Damian
> > > >
> > > > On Wed, 13 Jul 2016 at 16:51 Ismael Juma  wrote:
> > > >
> > > > > I think it's important to distinguish the use cases of defining new
> > > > stores
> > > > > (somewhat rare) versus using the `store` method (very common). The
> > > > strategy
> > > > > employed here is a common way to use generics to ensure type safety
> > for
> > > > the
> > > > > latter case. In the former case, there are all sorts of weird
> things
> > > one
> > > > > could do to defeat the type system, but spending a bit more effort
> to
> > > get
> > > > > it right so that the common case is safer and more pleasant is
> worth
> > > it,
> > > > in
> > > > > my opinion.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Thu, Jul 14, 2016 at 12:23 AM, Damian Guy  >
> > > > wrote:
> > > > >
> > > > > > Yes, you get compile time errors
> > > > > >
> > > > > > On Wed, 13 Jul 2016 at 16:22 Damian Guy 
> > > wrote:
> > > > > >
> > > > > > > You wont get a runtime error as you wouldn't find a store of
> that
> > > > type.
> > > > > > > The API would return null
> > > > > > >
> > > > > > > On Wed, 13 Jul 2016 at 16:22 Jay Kreps 
> wrote:
> > > > > > >
> > > > > > >> But if "my-store" is not of type MyStoreType don't you still
> > get a
> > > > run
> > > > > > >> time
> > > > > > >> error that in effect is the same as the class cast would be?
> > > > Basically
> > > > > > the
> > > > > > >> question I'm asking is whether this added complexity is
> actually
> > > > > moving
> > > > > > >> runtime errors to compile time errors.
> > > > > > >>
> > > > > > >> -Jay
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>

Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume performance under windows and some old Linux file system

[Sriram Subramanian]

Could you describe how recovery works in this mode? Say, we had a 250 MB
preallocated segment and we wrote till 50MB and crashed. Till what point
do we recover? Also, on startup, how is the append end pointer set even on
a clean shutdown? How does the FileChannel end position get set to 50 MB
instead of 250 MB? The existing code might just work for it but explaining
that would be useful.

On 4/21/15 9:40 AM, "Neha Narkhede"  wrote:

>+1. I've tried this on Linux and it helps reduce the spikes in append (and
>hence producer) latency for high throughput writes. I am not entirely sure
>why but my suspicion is that in the absence of preallocation, you see
>spikes writes need to happen faster than the time it takes Linux to
>allocate the next block to the file.
>
>It will be great to see some performance test results too.
>
>On Tue, Apr 21, 2015 at 9:23 AM, Jay Kreps  wrote:
>
>> I'm also +1 on this. The change is quite small and may actually help
>>perf
>> on Linux as well (we've never tried this).
>>
>> I have a lot of concerns on testing the various failure conditions but I
>> think since it will be off by default the risk is not too high.
>>
>> -Jay
>>
>> On Mon, Apr 20, 2015 at 6:58 PM, Honghai Chen
>>
>> wrote:
>>
>> > I wrote a KIP for this after some discussion on KAFKA-1646.
>> > https://issues.apache.org/jira/browse/KAFKA-1646
>> >
>> >
>> 
>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-20+-+Enable+log+pre
>>allocate+to+improve+consume+performance+under+windows+and+some+old+Linux+
>>file+system
>> > The RB is here: https://reviews.apache.org/r/33204/diff/
>> >
>> > Thanks, Honghai
>> >
>> >
>>
>
>
>
>-- 
>Thanks,
>Neha

Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume performance under windows and some old Linux file system

[Sriram Subramanian]

+1 

Some information on how this will be tested would be useful.

On 4/23/15 9:33 AM, "Jay Kreps"  wrote:

>Yeah if we understand the optimal policy for a config we always want to
>set
>it automatically. In this case I don't think we do yet, but down the road
>that could be nice. I think for now we should consider this option
>experimental to give people a chance to try it out.
>
>-Jay
>
>On Wed, Apr 22, 2015 at 7:32 PM, Honghai Chen 
>wrote:
>
>> Hi Roshan,
>> Use the 'auto' value maybe will break the rule and mess up the
>> configuration. @Jay, any thoughts?
>>
>> Thanks, Honghai Chen
>>
>> -Original Message-
>> From: Sriharsha Chintalapani [mailto:harsh...@fastmail.fm]
>> Sent: Thursday, April 23, 2015 6:27 AM
>> To: dev@kafka.apache.org; Roshan Naik
>> Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume
>> performance under windows and some old Linux file system
>>
>> +1 (non-binding).
>>
>> --
>> Harsha
>>
>>
>> On April 22, 2015 at 2:52:12 PM, Roshan Naik (ros...@hortonworks.com)
>> wrote:
>>
>> I see that it is safe to keep it this off by default due to some
>>concerns.
>> Eventually, for settings such as this whose 'preferred' value is
>>platform
>> specific (or based on other criteria), it might be worth considering
>> having a default value that is not a constant but an 'auto' value ..
>>When
>> kafka boots up it can automatically use the preferred value. Ofcourse it
>> would have to documented as to what auto means for a given platform.
>>
>> -roshan
>>
>>
>> On 4/22/15 1:21 PM, "Jakob Homan"  wrote:
>>
>> >+1. This is an important performance fix for Windows-based clusters.
>> >
>> >-Jakob
>> >
>> >On 22 April 2015 at 03:25, Honghai Chen 
>> >wrote:
>> >> Fix the issue Sriram mentioned. Code review and jira/KIP updated.
>> >>
>> >> Below are detail description for the scenarios:
>> >> 1.If do clear shutdown, the last log file will be truncated to its
>> >>real size since the close() function of FileMessageSet will call
>>trim(),
>> >> 2.If crash, then when restart, will go through the process of
>> >>recover() and the last log file will be truncate to its real size,
>>(and
>> >>the position will be moved to end of the file)
>> >> 3.When service start and open existing file
>> >> a.Will run the LogSegment constructor which has NO parameter
>> >>"preallocate",
>> >> b.Then in FileMessageSet, the "end" in FileMessageSet will be
>> >>Int.MaxValue, and then
>> >>"channel.position(math.min(channel.size().toInt, end))" will make the
>> >>position be end of the file,
>> >> c.If recover needed, the recover function will truncate file to end
>>of
>> >>valid data, and also move the position to it,
>> >>
>> >> 4.When service running and need create new log segment and new
>> >>FileMessageSet
>> >>
>> >> a.If preallocate = truei.the "end" in FileMessageSet will be 0, the
>> >>file size will be "initFileSize", and then
>> >>"channel.position(math.min(channel.size().toInt, end))" will make the
>> >>position be 0,
>> >>
>> >> b.Else if preallocate = falsei.backward compatible, the "end" in
>> >>FileMessageSet will be Int.MaxValue, the file size will be "0", and
>> >>then "channel.position(math.min(channel.size().toInt, end))" will make
>> >>the position be 0,
>> >>
>> >>
>> >>
>> 
>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-20+-+Enable+log+pre
>> 
>>>>allocate+to+improve+consume+performance+under+windows+and+some+old+Linu
>>>>x+
>> >>file+system
>> >> https://issues.apache.org/jira/browse/KAFKA-1646
>> >> https://reviews.apache.org/r/33204/diff/2/
>> >>
>> >> Thanks, Honghai Chen
>> >> http://aka.ms/kafka
>> >> http://aka.ms/manifold
>> >>
>> >> -Original Message-
>> >> From: Honghai Chen
>> >> Sent: Wednesday, April 22, 2015 11:12 AM
>> >> To: dev@kafka.apache.org
>> >> Subject: RE: [DISCUSS] KIP 20 Enable log preallocate to improve
>>consume
>> >>performance under windows and some old L

Re: [VOTE] KIP-89: Allow sink connectors to decouple flush and offset commit

[Sriram Subramanian]

+1 (binding)

On Fri, Nov 11, 2016 at 9:44 PM, Ewen Cheslack-Postava 
wrote:

> +1 (binding)
>
> -Ewen
>
> On Wed, Nov 9, 2016 at 2:16 PM, Shikhar Bhushan 
> wrote:
>
> > Hi,
> >
> > I would like to initiate a vote on KIP-89
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 89%3A+Allow+sink+connectors+to+decouple+flush+and+offset+commit
> >
> > Best,
> >
> > Shikhar
> >
>
>
>
> --
> Thanks,
> Ewen
>

Re: [VOTE] KIP-93: Improve invalid timestamp handling in Kafka Streams

[Sriram Subramanian]

+1 (binding)

I second Ewen's point about compatibility.

On Wed, Nov 30, 2016 at 2:53 AM, Damian Guy  wrote:

> +1
>
> On Wed, 30 Nov 2016 at 05:58 Ewen Cheslack-Postava 
> wrote:
>
> > +1 (binding).
> >
> > Also, see my notes in discussion thread around future compatibility
> > discussions for breaking plugin interface changes like this.
> >
> > -Ewen
> >
> > On Tue, Nov 29, 2016 at 3:54 PM, Guozhang Wang 
> wrote:
> >
> > > +1.
> > >
> > > On Tue, Nov 29, 2016 at 11:05 AM, Bill Bejeck 
> wrote:
> > >
> > > > +1 (non-binding)
> > > >
> > > > Thanks,
> > > >
> > > > Bill
> > > >
> > > > On Tue, Nov 29, 2016 at 1:34 PM, Matthias J. Sax <
> > matth...@confluent.io>
> > > > wrote:
> > > >
> > > > > I’d like to start the voting process for KIP-93:
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 93%3A+Improve+invalid+timestamp+handling+in+Kafka+Streams
> > > > >
> > > > > -Matthias
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
> >
> >
> > --
> > Thanks,
> > Ewen
> >
>

Re: [DISCUSS] KIP-95: Incremental Batch Processing for Kafka Streams

[Sriram Subramanian]

I agree that the metadata topic is required to build a batching semantic
that is intuitive.

One question on the config -

autostop.at

I see one value for it - eol. What other values can be used? Instead, why
would we not have batch.mode=true/false?

On Wed, Nov 30, 2016 at 1:51 PM, Matthias J. Sax 
wrote:

> Both types of intermediate topics are handled the exact same way and
> both types do connect different subtopologies (even if the user might
> not be aware that there are multiple subtopologies in case of internal
> data repartitioning). So there is no distinction between user
> intermediate topics (via through()) and internal intermediate
> repartitioning topics.
>
> I do also not understand your argument about "coupling instances"? The
> only "synchronization" is at startup time until the marker is written.
> Afterwards all instances just run as always. Furthermore, the metadata
> topic will be written within the leader while computing the overall
> partition assignment. Thus, the metadata topic will be fully populated
> (including the marker) before the individual instance will receive their
> assignment via group management protocol. So there is not more
> "synchronization" than before, as group management does synchronize
> instances anyway at startup.
>
> About startup failure. Yes, there is the case that the leader could
> potentially fail before the marker gets written. For this case, we have
> to consider a few things:
>
> 1. the net effect is, that no data will be processed by any instance
>(so application can start up, because no partition assignment will be
> distributed via group management, as the leader did fail while computing
> the assignment)
>
> 2. the failure would occur on partition assignment what would be a
> severe failure anyway and the application has bigger problems than a
> missing marker in the meta data topic (nobody will get partitioned
> assigned as the leader did not finish the assignment computation)
>
> 3. if the leader fails, a different application will become the leader.
>a) thus, if it is a permanent problem, eventually all instances are
> going down
>b) if the problem is transient, the probability is very high that the
> new leader will not fail
>
>
>
> -Matthias
>
> On 11/30/16 1:21 PM, Eno Thereska wrote:
> > In the KIP, two types of intermediate topics are described, 1) ones that
> connect two sub-topologies, and 2) others that are internal repartitioning
> topics (e.g., for joins).
> > I wasn't envisioning stopping the consumption of (2) at the HWM. The HWM
> can be used for the source topics only (so I agree with your "joins"
> scenario, but for a different reason).
> >
> > The case I am worried about is (1) when there are implicit connections
> between application instances where a 2nd instance's source topics would be
> the 1st instances output topics. In that case I was suggesting not to
> couple those instances.
> >
> > In the (corner) case when the application fails repeatedly, it can still
> fail right before we write to the metadata topic, so that corner case can
> still happen. However, it should be extremely rare, and I'd argue if the
> app is failing repeatedly N times there are bigger problems with the app.
> >
> > Eno
> >
> >> On 30 Nov 2016, at 11:52, Damian Guy  wrote:
> >>
> >> I think the KIP looks good. I also think we need the metadata topic
> >> in-order to provide sane guarantees on what data will be processed.
> >>
> >> As Matthias has outlined in the KIP we need to know when to stop
> consuming
> >> from intermediate topics, i.e, topics that are part of the same
> application
> >> but are used for re-partitioning or through etc. Without the metadata
> topic
> >> the consumption from the intermediate topics would always be one run
> >> behind. In the case of a join requiring partitioning this would result
> in
> >> no output for the first run and then in subsequent runs you'd get the
> >> output from the previous run - i'd find this a bit odd.
> >>
> >> Also I think having a fixed HWM IMO is a good idea. If you are running
> your
> >> streams app in some shared environment, then you don't want to get into
> a
> >> situation where the app fails (for random reasons), restarts with a new
> >> HMW, fails, restarts... and then continues to consume resources for
> ever as
> >> the HMW is constantly moving forward. So i think the approach in the KIP
> >> helps batch-mode streams apps to be good-citizens when running in shared
> >> environments.
> >>
> >> Thanks,
> >> Damian
> >>
> >> On Wed, 30 Nov 2016 at 10:40 Eno Thereska 
> wrote:
> >>
> >>> Hi Matthias,
> >>>
> >>> I like the first part of the KIP. However, the second part with the
> >>> failure modes and metadata topic is quite complex and I'm worried it
> >>> doesn't solve the problems you mention under failure. For example, the
> >>> application can fail before writing to the metadata topic. In that
> case, it
> >>> is not clear what the second app instance should do (for 

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

[Sriram Subramanian]

@Jay

1. I totally agree on the naming. The appid for transactions is really an
instance id. Any recommendation for a name is appreciated. We had thought
of instance id, session id or app id and went with app id.
2. We also discussed about init() method but that could add its own set of
confusion to existing users (should I update my existing usage to call
init()? Why should I have this extra step instead of the constructor doing
it?). Transactions is going to be used by a subset of users (probably
small) and it made sense to add the burden of calling
initTransactions/recoverTransactions to only that subset. We are actually
open to suggestions here in terms of naming as well.

@Jonathan
I am not sure it adds more complexity unless you use them. We have
explicitly named them for transactions and the current usage of the
producer remains unchanged.

@Michael
If you look at our idempotent producer implementation in the kip/design,
this is exactly what we do except that the deduplication happens on the
server. We started with separate KIPs but it made it very confusing to read
since there were interdependencies between the concepts.



On Tue, Dec 6, 2016 at 9:04 AM, Michael Pearce 
wrote:

> For dealing with exactly once delivery.
>
> As an alternative option has it been considered to have a message uuid in
> the record, that then is deduped on consumption?
>
> Similar to
> https://activemq.apache.org/artemis/docs/1.0.0/duplicate-detection.html
>
> Agreed this does not deal with transaction support.
>
> But should the two concerns be separated anyhow into two kips?
>
> Cheers
> Mike
>
>
> Sent using OWA for iPhone
> 
> From: Jay Kreps 
> Sent: Tuesday, December 6, 2016 4:47:55 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional
> Messaging
>
> Hey Guozhang,
>
>
>1. My point is that it is a bit confusing to have two things called
>application id that have different meanings, right? Won't the streams
> user
>end up specifying two different application ids?
>2. Makes sense. My two complaints are
>   1. At this point we've jumped through quite a lot of hoops to make
>   the producer lazily initialize, seems sad to get rid of that now.
>   2. The call initTransactions doesn't really make sense to the user
>   unless they understand the details of the protocol (which they
> won't). When
>   do i call this? How many times? etc. Maybe two additional
> options would be
>   to just add a general init() call that could cover metadata
> initialization
>   as well as this and potentially future things or continue to do it
> lazily.
>3. Yeah I get that you need an expiry scheme to limit it to 4 bytes. Is
>there a mechanism to expire them, and hence shrink it?
>
> -Jay
>
> On Fri, Dec 2, 2016 at 12:04 PM, Guozhang Wang  wrote:
>
> > @Jay
> >
> > 1. Stream's applicationId is shared among all instances for the app, and
> is
> > used as part of the consumer group id, while "app.id" is per producer
> > instance. So a Streams app that has a single "applicationID" config will
> > likely contain multiple producers each with a different appID based on
> > their corresponding taskIDs.
> >
> > 2. Another motivation besides the one pointed out by Jason for making
> sure
> > transaction-involved offsets have been committed before resuming, is that
> > we also want to separate the "app.id" config with the transactional
> > mechanism. More concretely, if a user does specify the "app.id" config
> and
> > without using transaction functions (i.e. initTransactions, beginTxn,
> etc),
> > they can still get idempotency guarantee across multiple sessions of the
> > producer identified by the app.id.
> >
> > 4. We thought about the PID length, note that since we do not expire
> PIDs,
> > we are expecting it to cover all possible producers that we have ever
> seen,
> > and producers without an "app.id" can come and go with different PIDs.
> > That
> > is why we feel 4 billion may not be sufficient.
> >
> >
> >
> > Guozhang
> >
> >
> > On Thu, Dec 1, 2016 at 11:10 PM, Jason Gustafson 
> > wrote:
> >
> > > Hey Jay,
> > >
> > > Thanks for the questions! Let me take a couple of them.
> > >
> > > 2. The initTransactions() call is a little annoying. Can we get rid of
> > > >that and call it automatically if you set a transaction.app.id
> when
> > > we
> > > >do the first message send as we do with metadata? Arguably we
> should
> > > > have
> > > >included a general connect() or init() call in the producer, but
> > given
> > > > that
> > > >we didn't do this it seems weird that the cluster metadata
> > initializes
> > > >automatically on demand and the transaction metadata doesn't.
> > >
> > >
> > > The purpose of this call is to fence off any producer with the same
> AppID
> > > and await the completion of any pending transactions. When it returns,
> > you
> > > know that your producer is s

Re: [VOTE] KIP-94: Session Windows

[Sriram Subramanian]

+1 (binding)

On Tue, Dec 6, 2016 at 3:43 PM, Ewen Cheslack-Postava 
wrote:

> +1 binding
>
> -Ewen
>
> On Tue, Dec 6, 2016 at 3:21 PM, Bill Bejeck  wrote:
>
> > +1
> >
> > On Tue, Dec 6, 2016 at 4:55 PM, Guozhang Wang 
> wrote:
> >
> > > +1 (binding)
> > >
> > > On Tue, Dec 6, 2016 at 9:07 AM, Matthias J. Sax  >
> > > wrote:
> > >
> > > > +1
> > > >
> > > > On 12/6/16 7:40 AM, Eno Thereska wrote:
> > > > > +1 (non-binding)
> > > > >
> > > > > Thanks
> > > > > Eno
> > > > >> On 6 Dec 2016, at 12:09, Damian Guy  wrote:
> > > > >>
> > > > >> Hi all,
> > > > >>
> > > > >> I'd like to start the vote for KIP-94:
> > > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 94+Session+Windows
> > > > >>
> > > > >> There is a PR for it here: https://github.com/apache/
> > kafka/pull/2166
> > > > >>
> > > > >> Thanks,
> > > > >> Damian
> > > > >
> > > >
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> Thanks,
> Ewen
>

Re: [VOTE]: KIP-97: The client compatibility KIP

[Sriram Subramanian]

+1 (binding)

On Thu, Dec 8, 2016 at 5:42 AM, Ismael Juma  wrote:

> Thanks for the KIP Colin, +1 (binding) from me.
>
> I agree with Rajini's suggestions. Also, since the command is a public
> interface, it would be good to include details in the KIP.
>
> Ismael
>
> On Wed, Dec 7, 2016 at 5:17 PM, Colin McCabe  wrote:
>
> > Hi all,
> >
> > I heard that the VOTE and DISCUSS threads for the KIP-97 discussion
> > appeared to be in the same email thread for some people using gmail.  So
> > I'm reposting in hopes of getting a separate email thread this time for
> > those viewers. :)
> >
> > I'd like to start voting on KIP-97
> > (https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy
> > ).
> >
> > The discussion thread can be found here:
> > https://www.mail-archive.com/dev@kafka.apache.org/msg60955.html
> >
> > Thanks for your feedback.
> >
> > best,
> > Colin McCabe
> >
>

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

[Sriram Subramanian]

I am not sure if it is a good idea to support both init() and lazy
initialization. The ideal state would have been to implement init as a non
blocking api and have the rest of the methods throw uninitialized exception
if init was not called. This would ensure that init can still be used by
other non blocking frameworks but at the same time enforces that other apis
are not called till init is complete. The problem with supporting both is
that it is going to confuse users. Why should I call init over lazy
initialization? The lazy initialization of transactions will not work if
you want to recover transactions, read offsets and then start a new
transaction. In such cases, you would have to resort to the init api.
Client developers need to provide support for both the lazy initialization
and init method. This problem might be solved if we just renamed the
initTransaction api (eg: recoverTransactions).

On Mon, Dec 12, 2016 at 11:36 PM, Ismael Juma  wrote:

> Hi Jay,
>
> I like the idea of having a single `init`, but I am not sure about the
> specifics of the metadata initialisation (as Jason alluded to). More
> inline.
>
> On Tue, Dec 13, 2016 at 6:01 AM, Jay Kreps  wrote:
>
> >1. Add a generic init() call which initializes both transactions and
> >metadata
> >
>
> Would this initialise metadata for all topics? One advantage of doing the
> metadata call during `send` is that we only retrieve metadata for the
> subset of topics that you are producing to. For large clusters, retrieving
> the metadata for all the topics is relatively expensive and I think users
> would prefer to avoid that unless there are some concrete benefits. We
> could pass the topics to `init`, but that seems a bit clunky.
>
>
> >2. If you don't call init(), metadata is initialized on the first send
> >(as now)
>
>
> We need to maintain the logic to refresh the metadata on `send` anyway if
> you try to send to a topic that is missing from the metadata (e.g. if it's
> added after the `init` method is called, assuming that we don't expect
> people to call `init` more than once) so that seems fine.
>
>
> > and transactions are lazily initialized at the first beginTransaction()
> > call.
>
>
> I'll leave it to Jason to say if this is feasible. However, if it is, it
> seems like we can just do this and avoid the `init` method altogether?
>
> Ismael
>
> On Tue, Dec 13, 2016 at 6:01 AM, Jay Kreps  wrote:
>
> > Hey Jason/Neha,
> >
> > Yeah, clearly having a mandatory, generic init() method that initializes
> > both transactions and topic metadata would be the ideal solution. This
> > would solve the occasional complaint about blocking behavior during
> > initialization of metadata (or at least shift it to a new complaint about
> > an inability to initialize when the cluster isn't up in test
> environments).
> > The challenge is that we can't do this because it isn't backwards
> > compatible with existing apps that don't call init.
> >
> > The alternative of having an optional generic init() call is a bit odd
> > because to figure out if you need to call it you need to discover what it
> > does, which is not generic, it initializes transactions. We can't really
> > add more logic to init because it only gets invoked by transaction users
> so
> > it doesn't really function as a generic init.
> >
> > What do you think of this solution:
> >
> >1. Add a generic init() call which initializes both transactions and
> >metadata
> >2. If you don't call init(), metadata is initialized on the first send
> >(as now) and transactions are lazily initialized at the first
> >beginTransaction() call.
> >
> > -Jay
> >
> >
> >
> >
> >
> >
> >
> > On Mon, Dec 12, 2016 at 9:17 PM, Jason Gustafson 
> > wrote:
> >
> > > @Neha
> > >
> > >
> > > 1. I think we should consider renaming initTransactions to just init()
> > and
> > > > moving the metadata initialization there. Let's make sure we don't
> add
> > > APIs
> > > > that are relevant to this proposal only. Instead, try to think what
> > we'd
> > > > propose if we were writing the producer from scratch today. I suspect
> > we
> > > > would end up with an init() API that would do the metadata
> > initialization
> > > > as well as the transaction stuff lazily. If so, let's make that
> change
> > > now.
> > >
> > >
> > > I think the only awkwardness with `init()` is that it would probably
> have
> > > to be an optional API for non-transactional usage to support existing
> > code.
> > > I'm also not sure what metadata we can actually initialize at that
> point
> > > since we don't know which topics will be produced to. That said, I'm
> also
> > > not fond of the `initTransactions` name, and we may find other uses
> for a
> > > generic `init()` in the future, so I'm in favor this renaming.
> > >
> > >
> > > > 2. Along the same lines, let's think about the role of each id that
> the
> > > > producer will have and see if everything still makes sense. For
> > instance,
> > > > we have quite a

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

[Sriram Subramanian]

Radai,

I think it is important to understand the key requirements that we don’t
want to compromise. We can then understand the tradeoffs of the different
approaches. We did in fact start with the double journal approach couple of
years back. I will highlight the must have requirements first and then
explain the trade offs based on my understanding.

1. End to end latency for stream processing - This is probably one of the
biggest reasons to support transactions in Kafka. We would like to support
very low latency for end to end processing across steam topologies. This
means you would want your downstream processors to see the output of your
processing immediately. The low latency is a requirement even if we only
expose committed messages.

2. Speculative execution - We would like to go one step further for stream
processing. 99% of the transactions will always succeed. We would like to
take advantage of this and process the messages optimistically even if the
transactions are still unfinished. If the transactions abort, we would do a
cascading abort across the topology. This helps us to complete all the
processing and keep the output ready and expose them once the transactions
are committed. This will help us to significantly bring down the latency
for end to end stream processing and provide the ability to keep exactly
once as the default setting.

3. IO and memory constraints - We would want a solution that takes 2x the
number of writes. This will bring down broker utilization by half. I don’t
really understand the in memory solution (would be useful if you can
explain it more if you think it solves these goals) but the same resource
constraints apply. What has made Kafka successful is the ability to run
very high throughput clusters with very few machines. We would like to keep
this true when a cluster is largely dominated by stream processing
workloads.

4. Provide both read committed and read uncommitted isolation levels - This
is actually a desired feature. This is similar to database isolation levels
(except that we provide only two of them for now). Downstream systems that
need strong guarantees with some performance impact can choose read
committed isolation level. Systems that want to optimize for performance
and can live with approximations would choose read uncommitted options.
This helps to nicely decouple downstream users that would like to share
topics but have different end goals.

There are other obvious goals like correctness of the protocol and
simplicity of the design that needs to be true by default.

Given these goals, the double journal approach was a non starter to enable
low end to end latency and did not provide the ability to do speculative
execution in the future. We also found the resource constraints
(specifically IO/Network) to be unacceptable.

We did understand the complexity of the consumers but it was the best
tradeoff considering the other must have goals. We also thought of another
approach to push the consumer buffering to the broker side. This would
enable multiple consumer groups to share the same buffer pool for a
specific topic partition. However, in the worst case, you would need to
bring the entire log into memory to remove the aborted transaction (for a
consumer that is catching up from time 0). This would also make us loose
zero copy semantics.

I would be excited to hear an option that can solve our must have goals and
still keep the consumer really thin. The abstraction seems fine since we
allow the end users to pick the guarantees they need.

On Mon, Dec 19, 2016 at 10:56 AM, Jay Kreps  wrote:

> Hey Radai,
>
> I'm not sure if I fully understand what you are proposing, but I
> interpreted it to be similar to a proposal we worked through back at
> LinkedIn. The proposal was to commit to a central txlog topic, and then
> recopy to the destination topic upon transaction commit. The observation on
> that approach at the time were the following:
>
>1. It is cleaner since the output topics have only committed data!
>2. You need full replication on the txlog topic to ensure atomicity. We
>weren't able to come up with a solution where you buffer in memory or
> use
>renaming tricks the way you are describing. The reason is that once you
>begin committing you must ensure that the commit eventually succeeds to
>guarantee atomicity. If you use a transient store you might commit some
>data and then have a server failure that causes you to lose the rest of
> the
>transaction.
>3. Having a single log allows the reader to choose a "read uncommitted"
>mode that hands out messages immediately. This is important for cases
> where
>latency is important, especially for stream processing topologies where
>these latencies stack up across multiple stages.
>
> For the stream processing use case, item (2) is a bit of a deal killer.
> This takes the cost of a transient message write (say the intermediate
> result of a stream processing t

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

[Sriram Subramanian]

small correction in my third point -

3. IO and memory constraints - We would want a solution that *does not take*
2x the number of writes.

On Mon, Dec 19, 2016 at 12:37 PM, Sriram Subramanian 
wrote:

> Radai,
>
> I think it is important to understand the key requirements that we don’t
> want to compromise. We can then understand the tradeoffs of the different
> approaches. We did in fact start with the double journal approach couple of
> years back. I will highlight the must have requirements first and then
> explain the trade offs based on my understanding.
>
> 1. End to end latency for stream processing - This is probably one of the
> biggest reasons to support transactions in Kafka. We would like to support
> very low latency for end to end processing across steam topologies. This
> means you would want your downstream processors to see the output of your
> processing immediately. The low latency is a requirement even if we only
> expose committed messages.
>
> 2. Speculative execution - We would like to go one step further for stream
> processing. 99% of the transactions will always succeed. We would like to
> take advantage of this and process the messages optimistically even if the
> transactions are still unfinished. If the transactions abort, we would do a
> cascading abort across the topology. This helps us to complete all the
> processing and keep the output ready and expose them once the transactions
> are committed. This will help us to significantly bring down the latency
> for end to end stream processing and provide the ability to keep exactly
> once as the default setting.
>
> 3. IO and memory constraints - We would want a solution that takes 2x the
> number of writes. This will bring down broker utilization by half. I don’t
> really understand the in memory solution (would be useful if you can
> explain it more if you think it solves these goals) but the same resource
> constraints apply. What has made Kafka successful is the ability to run
> very high throughput clusters with very few machines. We would like to keep
> this true when a cluster is largely dominated by stream processing
> workloads.
>
> 4. Provide both read committed and read uncommitted isolation levels -
> This is actually a desired feature. This is similar to database isolation
> levels (except that we provide only two of them for now). Downstream
> systems that need strong guarantees with some performance impact can choose
> read committed isolation level. Systems that want to optimize for
> performance and can live with approximations would choose read uncommitted
> options. This helps to nicely decouple downstream users that would like to
> share topics but have different end goals.
>
> There are other obvious goals like correctness of the protocol and
> simplicity of the design that needs to be true by default.
>
> Given these goals, the double journal approach was a non starter to enable
> low end to end latency and did not provide the ability to do speculative
> execution in the future. We also found the resource constraints
> (specifically IO/Network) to be unacceptable.
>
> We did understand the complexity of the consumers but it was the best
> tradeoff considering the other must have goals. We also thought of another
> approach to push the consumer buffering to the broker side. This would
> enable multiple consumer groups to share the same buffer pool for a
> specific topic partition. However, in the worst case, you would need to
> bring the entire log into memory to remove the aborted transaction (for a
> consumer that is catching up from time 0). This would also make us loose
> zero copy semantics.
>
> I would be excited to hear an option that can solve our must have goals
> and still keep the consumer really thin. The abstraction seems fine since
> we allow the end users to pick the guarantees they need.
>
> On Mon, Dec 19, 2016 at 10:56 AM, Jay Kreps  wrote:
>
>> Hey Radai,
>>
>> I'm not sure if I fully understand what you are proposing, but I
>> interpreted it to be similar to a proposal we worked through back at
>> LinkedIn. The proposal was to commit to a central txlog topic, and then
>> recopy to the destination topic upon transaction commit. The observation
>> on
>> that approach at the time were the following:
>>
>>1. It is cleaner since the output topics have only committed data!
>>2. You need full replication on the txlog topic to ensure atomicity. We
>>weren't able to come up with a solution where you buffer in memory or
>> use
>>renaming tricks the way you are describing. The reason is that once you
>>begin committing you must ensure that the commit eventually succeeds to
>>

Re: [VOTE] KIP-90 Remove zkClient dependency from Streams

[Sriram Subramanian]

+1

On Tue, Dec 20, 2016 at 1:13 PM, Guozhang Wang  wrote:

> +1. Thanks!
>
> On Tue, Dec 20, 2016 at 1:01 PM, Hojjat Jafarpour 
> wrote:
>
> > Hi all,
> >
> > Seems that there is no opposition to this KIP. This email is to start the
> > voting for this KIP.
> > Once again the KIP is for removing zkClient dependency from Streams.
> Please
> > check out the KIP page:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 90+-+Remove+zkClient+
> > dependency+from+Streams
> >
> > Thanks,
> > --Hojjat
> >
>
>
>
> --
> -- Guozhang
>

Re: [DISCUSS] KIP-106 - Change Default unclean.leader.election.enabled from True to False

[Sriram Subramanian]

+1

On Tue, Jan 3, 2017 at 11:56 AM, Ian Wrigley  wrote:

> +1 from me too. When I talk to people in training classes, who are
> typically much newer to Kafka, they tend to be surprised (and
> scared/horrified) that the default is true. Much safer to set it to false
> and let people change it when they really understand the tradeoffs.
>
> Ian.
>
> ---
> Ian Wrigley
> Director, Education Services
> Confluent, Inc
>
> > On Jan 3, 2017, at 1:22 PM, Tom Crayford  wrote:
> >
> > +1. We've been running it in production for a long time and it's the
> right
> > default.
> >
> > On Tue, Jan 3, 2017 at 7:17 PM, Ismael Juma  wrote:
> >
> >> Thanks for the KIP, +1 from me.
> >>
> >> Ismael
> >>
> >> On 3 Jan 2017 6:54 pm, "Ben Stopford"  wrote:
> >>
> >>> Hi All
> >>>
> >>> Please find the below KIP which proposes changing the setting
> >>> unclean.leader.election.enabled from true to false. The motivation for
> >>> this change is that it catches out new Kafka users who don’t realise
> the
> >>> default favours availability over data loss.
> >>>
> >>> This would mean clusters wishing to continue with unclean leader
> election
> >>> enabled would need to add the appropriate configuration on upgrade.
> >>>
> >>> Please let me know if you foresee any issue with this change, agree or
> >>> don’t agree.
> >>>
> >>> https://cwiki.apache.org/confluence/display/KAFKA/%
> >>> 5BWIP%5D+KIP-106+-+Change+Default+unclean.leader.
> >>> election.enabled+from+True+to+False  >>> confluence/display/KAFKA/[WIP]+KIP-106+-+Change+Default+
> >>> unclean.leader.election.enabled+from+True+to+False>
> >>>
> >>> Thanks
> >>>
> >>> B
> >>>
> >>> Ben Stopford
> >>> Confluent, http://www.confluent.io 
> >>>
> >>>
> >>>
> >>>
> >>
>
>

Re: [VOTE] KIP-99: Add Global Tables to Kafka Streams

[Sriram Subramanian]

+1

On Tue, Jan 3, 2017 at 11:43 AM, Neha Narkhede  wrote:

> +1 (binding)
>
> On Tue, Jan 3, 2017 at 11:40 AM Ewen Cheslack-Postava 
> wrote:
>
> > +1
> >
> > -Ewen
> >
> > On Tue, Jan 3, 2017 at 11:16 AM, Gwen Shapira  wrote:
> >
> > > +1
> > >
> > > I'm glad this super-common use-case will become more performant now.
> > >
> > > On Mon, Dec 12, 2016 at 3:45 AM, Damian Guy 
> > wrote:
> > > > Hi all,
> > > >
> > > > I'd like to start the vote for KIP-99:
> > > > https://cwiki.apache.org/confluence/pages/viewpage.
> > > action?pageId=67633649
> > > >
> > > > There is a PR for it here: https://github.com/apache/kafka/pull/2244
> > > >
> > > > Thanks,
> > > > Damian
> > >
> > >
> > >
> > > --
> > > Gwen Shapira
> > > Product Manager | Confluent
> > > 650.450.2760 <(650)%20450-2760> | @gwenshap
> > > Follow us: Twitter | blog
> > >
> >
> --
> Thanks,
> Neha
>

Re: [VOTE] KIP-98: Exactly Once Delivery and Transactional Messaging

[Sriram Subramanian]

+1. Great work in driving this to a consensus. Lots of good constructive
conversations.

On Fri, Feb 24, 2017 at 11:48 AM, Jason Gustafson 
wrote:

> +1 from me (duh). Thanks to all the reviewers. The KIP has been much
> improved because of it!
>
> -Jason
>
> On Wed, Feb 22, 2017 at 8:48 AM, Ismael Juma  wrote:
>
> > Great work on the proposal and iterating on it based on community
> feedback.
> > As Jun (and others) said, it's likely that minor changes will happen as
> the
> > PR is reviewed and additional testing takes place since this is a
> > significant change.
> >
> > I am +1 (binding) on the proposal without optional keys and values to
> keep
> > things consistent. If we show during performance testing that this is
> > worthwhile, we can update the proposal.
> >
> > Ismael
> >
> > On Tue, Feb 21, 2017 at 6:23 PM, Jun Rao  wrote:
> >
> > > It seems that it's simpler and more consistent to avoid optional keys
> and
> > > values. Not sure if it's worth squeezing every byte at the expense of
> > > additional complexity. Other than that, +1 from me.
> > >
> > > Also, since this is a large KIP, minor changes may arise as we start
> the
> > > implementation. It would be good if we can keep the community posted of
> > > those changes, if any.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Tue, Feb 21, 2017 at 4:33 PM, Michael Pearce  >
> > > wrote:
> > >
> > > > If the argument and objective within this KIP is to keep the overhead
> > of
> > > > the protocol as small as possible and remove redundancy, and every
> byte
> > > is
> > > > being counted and the introduction of varInts, then it would make
> sense
> > > to
> > > > use attributes to me.
> > > >
> > > >
> > > > On 22/02/2017, 00:14, "Jason Gustafson"  wrote:
> > > >
> > > > Done. I've left the key and value as optional since we may not
> have
> > > > reached
> > > > consensus on whether to use attributes or not. Perhaps we should
> > just
> > > > keep
> > > > it simple and not do it? The benefit seems small.
> > > >
> > > > -Jason
> > > >
> > > > On Tue, Feb 21, 2017 at 4:05 PM, Michael Pearce <
> > > michael.pea...@ig.com
> > > > >
> > > > wrote:
> > > >
> > > > > Ok, no worries, can you add it back ValueLen on this KIP, and
> > > update
> > > > the
> > > > > doc, then we can work from that ☺
> > > > >
> > > > > Cheers
> > > > > Mike
> > > > >
> > > > > On 22/02/2017, 00:02, "Jason Gustafson" 
> > > wrote:
> > > > >
> > > > > I feel it was a little odd to leave out the value length
> > > anyway,
> > > > so I
> > > > > would
> > > > > rather add it back and put headers at the end. This is more
> > > > consistent
> > > > > with
> > > > > the rest of the Kafka protocol.
> > > > >
> > > > > -Jason
> > > > >
> > > > > On Tue, Feb 21, 2017 at 3:58 PM, Michael Pearce <
> > > > michael.pea...@ig.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Or we keep as is (valuelen removed), and headers are
> added
> > > with
> > > > > headers
> > > > > > length..
> > > > > >
> > > > > > On 21/02/2017, 23:38, "Apurva Mehta" <
> apu...@confluent.io>
> > > > wrote:
> > > > > >
> > > > > > Right now, we don't need the value length: since it
> is
> > > the
> > > > last
> > > > > item
> > > > > > in the
> > > > > > message, and we have the message length, we can
> deduce
> > > the
> > > > value
> > > > > > length.
> > > > > > However, if we are adding record headers to the end,
> we
> > > > would
> > > > > need to
> > > > > > introduce the value length along with that change.
> > > > > >
> > > > > > On Tue, Feb 21, 2017 at 3:34 PM, Michael Pearce <
> > > > > michael.pea...@ig.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > It seems I cannot add comment on the doc.
> > > > > > >
> > > > > > > In the section around the message protocol.
> > > > > > >
> > > > > > > It has stated:
> > > > > > >
> > > > > > > Message =>
> > > > > > > Length => uintVar
> > > > > > > Attributes => int8
> > > > > > > TimestampDelta => intVar
> > > > > > > OffsetDelta => uintVar
> > > > > > > KeyLen => uintVar [OPTIONAL]
> > > > > > > Key => data [OPTIONAL]
> > > > > > > Value => data [OPTIONAL]
> > > > > > >
> > > > > > > Should it not be: (added missing value len)
> > > > > > >
> > > > > > > Message =>
> > > > > > > Length => uintVar
> > > > > > > Attributes => int8
> > > > > > > TimestampDelta => intVar
> > > > > > > OffsetDelta => uintVar
> > > > > > > KeyLen => uintVar [OPTIONAL]
> > > > > > > Ke

Re: [DISCUSS] 0.10.3.0/0.11.0.0 release planning

[Sriram Subramanian]

+1

Thanks Ismael for volunteering to run the next time based release.

On Wed, Mar 1, 2017 at 4:51 PM, Ismael Juma  wrote:

> Thanks everyone for the feedback. Since everyone was in favour and we had
> covered this particular case in the time-based release plan[1], I went
> ahead and updated the wiki page to specify the next version as 0.11.0.0.
> I'll update JIRA versions and the KIP page soon.
>
> Ismael
>
> [1] "We change the message format, in which case we bump to 0.11.x" in
> https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan
>
> On Tue, Feb 28, 2017 at 3:47 AM, Ismael Juma  wrote:
>
> > Hi all,
> >
> > With 0.10.2.0 out of the way, I would like to volunteer to be the release
> > manager for our next time-based release. See https://cwiki.apache.org/c
> > onfluence/display/KAFKA/Time+Based+Release+Plan if you missed previous
> > communication on time-based releases or need a reminder.
> >
> > I put together a draft release plan with June 2017 as the release month
> > (as previously agreed) and a list of KIPs that have already been voted:
> >
> > *https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=68716876
> >  action?pageId=68716876>*
> >
> > I haven't set exact dates for the various stages (feature freeze, code
> > freeze, etc.) for now as Ewen is going to send out an email with some
> > suggested tweaks based on his experience as release manager for 0.10.2.0.
> > We can set the exact dates after that discussion.
> >
> > As we are starting the process early this time, we should expect the
> > number of KIPs in the plan to grow (so don't worry if your KIP is not
> there
> > yet), but it's good to see that we already have 10 (including 2 merged
> and
> > 2 with PR reviews in progress).
> >
> > Out of the KIPs listed, KIP-98 (Exactly-once and Transactions) and
> KIP-101
> > (Leader Generation in Replication) require message format changes, which
> > typically imply a major version bump (i.e. 0.11.0.0). If we do that, then
> > it makes sense to also include KIP-106 (Unclean leader election should be
> > false by default) and KIP-118 (Drop support for Java 7). We would also
> take
> > the chance to remove deprecated code, in that case.
> >
> > Given the above, how do people feel about 0.11.0.0 as the next Kafka
> > version? Please share your thoughts.
> >
> > Thanks,
> > Ismael
> >
>

Re: [VOTE] KIP-128: Add ByteArrayConverter for Kafka Connect

[Sriram Subramanian]

+1. Looks good

On Tue, Mar 7, 2017 at 10:33 PM, Ewen Cheslack-Postava 
wrote:

> Hah, I forgot to do it in the original email, but I suppose I should make
> it explicit: +1 (binding)
>
> -Ewen
>
> On Mon, Mar 6, 2017 at 9:26 PM, Gwen Shapira  wrote:
>
> > +1 (binding)
> >
> > On Mon, Mar 6, 2017 at 7:53 PM, Ewen Cheslack-Postava  >
> > wrote:
> >
> > > Hi all,
> > >
> > > I'd like to kick off voting for KIP-128: Add ByteArrayConverter for
> Kafka
> > > Connect:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 128%3A+Add+ByteArrayConverter+for+Kafka+Connect
> > >
> > > There was a small amount of discussion, see the original thread here:
> > > https://lists.apache.org/thread.html/62fc2245285ac5d15ebb9b2ebed672
> > > b51e391c8dfe9a51be85f685f3@%3Cdev.kafka.apache.org%3E
> > >
> > > The vote will stay open for at least 72 hours.
> > >
> > > -Ewen
> > >
> >
> >
> >
> > --
> > *Gwen Shapira*
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter  | blog
> > 
> >
>

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

[Sriram Subramanian]

StreamsBuilder would be my vote.

> On Mar 13, 2017, at 9:42 PM, Jay Kreps  wrote:
> 
> Hey Matthias,
> 
> Make sense, I'm more advocating for removing the word topology than any
> particular new replacement.
> 
> -Jay
> 
> On Mon, Mar 13, 2017 at 12:30 PM, Matthias J. Sax 
> wrote:
> 
>> Jay,
>> 
>> thanks for your feedback
>> 
>>> What if instead we called it KStreamsBuilder?
>> 
>> That's the current name and I personally think it's not the best one.
>> The main reason why I don't like KStreamsBuilder is, that we have the
>> concepts of KStreams and KTables, and the builder creates both. However,
>> the name puts he focus on KStream and devalues KTable.
>> 
>> I understand your argument, and I am personally open the remove the
>> "Topology" part, and name it "StreamsBuilder". Not sure what others
>> think about this.
>> 
>> 
>> About Processor API: I like the idea in general, but I thinks it's out
>> of scope for this KIP. KIP-120 has the focus on removing leaking
>> internal APIs and do some cleanup how our API reflects some concepts.
>> 
>> However, I added your idea to API discussion Wiki page and we take if
>> from there:
>> https://cwiki.apache.org/confluence/display/KAFKA/
>> Kafka+Streams+Discussions
>> 
>> 
>> 
>> -Matthias
>> 
>> 
>>> On 3/13/17 11:52 AM, Jay Kreps wrote:
>>> Two things:
>>> 
>>>   1. This is a minor thing but the proposed new name for KStreamBuilder
>>>   is StreamsTopologyBuilder. I actually think we should not put
>> topology in
>>>   the name as topology is not a concept you need to understand at the
>>>   kstreams layer right now. I'd think of three categories of concepts:
>> (1)
>>>   concepts you need to understand to get going even for a simple
>> example, (2)
>>>   concepts you need to understand to operate and debug a real
>> production app,
>>>   (3) concepts we truly abstract and you don't need to ever understand.
>> I
>>>   think in the kstream layer topologies are currently category (2), and
>> this
>>>   is where they belong. By introducing the name in even the simplest
>> example
>>>   it means the user has to go read about toplogies to really understand
>> even
>>>   this simple snippet. What if instead we called it KStreamsBuilder?
>>>   2. For the processor api, I think this api is mostly not for end
>> users.
>>>   However this are a couple cases where it might make sense to expose
>> it. I
>>>   think users coming from Samza, or JMS's MessageListener (
>>>   https://docs.oracle.com/javaee/7/api/javax/jms/MessageListener.html)
>>>   understand a simple callback interface for message processing. In
>> fact,
>>>   people often ask why Kafka's consumer doesn't provide such an
>> interface.
>>>   I'd argue we do, it's KafkaStreams. The only issue is that the
>> processor
>>>   API documentation is a bit scary for a person implementing this type
>> of
>>>   api. My observation is that people using this style of API don't do a
>> lot
>>>   of cross-message operations, then just do single message operations
>> and use
>>>   a database for anything that spans messages. They also don't factor
>> their
>>>   code into many MessageListeners and compose them, they just have one
>>>   listener that has the complete handling logic. Say I am a user who
>> wants to
>>>   implement a single Processor in this style. Do we have an easy way to
>> do
>>>   that today (either with the .transform/.process methods in kstreams
>> or with
>>>   the topology apis)? Is there anything we can do in the way of trivial
>>>   helper code to make this better? Also, how can we explain that
>> pattern to
>>>   people? I think currently we have pretty in-depth docs on our apis
>> but I
>>>   suspect a person trying to figure out how to implement a simple
>> callback
>>>   might get a bit lost trying to figure out how to wire it up. A simple
>> five
>>>   line example in the docs would probably help a lot. Not sure if this
>> is
>>>   best addressed in this KIP or is a side comment.
>>> 
>>> Cheers,
>>> 
>>> -Jay
>>> 
>>> On Fri, Feb 3, 2017 at 3:33 PM, Matthias J. Sax 
>>> wrote:
>>> 
 Hi All,
 
 I did prepare a KIP to do some cleanup some of Kafka's Streaming API.
 
 Please have a look here:
 https://cwiki.apache.org/confluence/display/KAFKA/KIP-
 120%3A+Cleanup+Kafka+Streams+builder+API
 
 Looking forward to your feedback!
 
 
 -Matthias
>> 
>> 

Re: [VOTE] KIP-117: Add a public AdminClient API for Kafka admin operations

[Sriram Subramanian]

+1 (binding)

Nice work in driving this.

On Mon, Mar 13, 2017 at 10:31 PM, Gwen Shapira  wrote:

> +1 (binding)
>
> I expressed few concerns in the discussion thread, but in general this is
> super important to get done.
>
> On Fri, Mar 10, 2017 at 10:38 AM, Colin McCabe  wrote:
>
> > Hi all,
> >
> > I'd like to start voting on KIP-117
> > (https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 117%3A+Add+a+public+AdminClient+API+for+Kafka+admin+operations
> > ).
> >
> > The discussion thread can be found here:
> > https://www.mail-archive.com/dev@kafka.apache.org/msg65697.html
> >
> > best,
> > Colin
> >
>
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter  | blog
> 
>

Re: [DISCUSS] KIP-129: Kafka Streams Exactly-Once Semantics

[Sriram Subramanian]

To add to this discussion, I do think we should think about this in
increments. We do a full recovery today and the EOS proposal will not make
this any worse. Using store snapshot is a good option to avoid store
recovery in the future but as Eno points out, all pluggable stores would
need to have this ability. W.r.t transaction failures, this should not be
an issue. We should be simply retrying. There is one optimization we can do
for clean shutdowns. We could store a clean shutdown file that contains the
input offsets. This file gets written when you close the streams instance.
On start, you could can check the offsets from the shutdown file and
compare it with the offsets we get from the consumer and ensure they match.
If they do, you could use the same store instead of recovering. However, if
we go with the snapshot approach, this will not be required. My vote would
be to implement V1 and solve the bootstrap problem which exist today in the
future versions.

On Tue, Mar 21, 2017 at 4:43 PM, Matthias J. Sax 
wrote:

> Thanks for your feedback Eno.
>
> For now, I still think that the KIP itself does not need to talk about
> this in more detail, because we apply the same strategy for EoS as for
> non-EoS (as of 0.10.2).
>
> Thus, in case of a clean shutdown, we write the checkpoint file for a
> store and thus know we can reuse the store. In case of failure, we need
> to recreate the store from the changelog.
>
> > Will a V1 design that relies on plain store recovery from Kafka for
> > each transaction abort be good enough, or usable?
>
> Why should it not be usable? It's the same strategy as used in 0.10.2
> and it runs in production in many companies already.
>
> > however it seems to me we might have a regression of sorts
> > Now we might pay it for a transaction failure.
>
> I would assume transaction failures to be quite rare. Maybe the core EoS
> folks can comment here, too.
>
>
>
> -Matthias
>
>
>
> On 3/20/17 3:16 PM, Eno Thereska wrote:
> > Hi Matthias,
> >
> > I'd like to see some more info on how you propose to handle transactions
> that involve state stores in the KIP itself. The design doc has info about
> various optimisations like RocksDb snapshots and transactions and such, but
> will there be a user-visible interface that indicates whether a store has
> snapshot and/or transactional capabilities? If a user plugs in another
> store, what guarantees are they expected to get?
> >
> > Will a V1 design that relies on plain store recovery from Kafka for each
> transaction abort be good enough, or usable? If your dataset is large
> (e.g., 200GB) the recovery time might be so large as to effectively render
> that Kafka Streams instance unavailable for tens of minutes. You mention
> that is not a regression to what we currently have, however it seems to me
> we might have a regression of sorts: currently we pay the recovery price
> for a Kafka Streams instance failure. Now we might pay it for a transaction
> failure. Will transaction failures be more or less common than the previous
> types of failures? I'd like to see this addressed.
> >
> > Thanks
> > Eno
> >
> >
> >
> >> On 15 Mar 2017, at 22:09, Matthias J. Sax 
> wrote:
> >>
> >> Just a quick follow up:
> >>
> >> Our overall proposal is, to implement KIP-129 as is as a “Stream EoS
> >> 1.0” version. The raised concerns are all valid, but hard to quantify at
> >> the moment. Implementing KIP-129, that provides a clean design, allows
> >> us to gain more insight in the performance implications. This enables
> >> us, to make an educated decision, if the “producer per task” model
> >> perform wells or not, and if a switch to a “producer per thread” model
> >> is mandatory.
> >>
> >> We also want to point out, that we can move incrementally from "producer
> >> per task" to "producer per thread" design or apply some incremental
> >> improvements to "producer per task" (as discussed in the doc). Thus,
> >> there is not issue with regard to upgrading.
> >>
> >>
> >> -Matthias
> >>
> >>
> >>
> >> On 3/15/17 2:36 PM, Matthias J. Sax wrote:
> >>> Hi,
> >>>
> >>> I want to pick up this thread again. As there are some concerns about
> >>> the "producer per task" design, we did write up an alternative
> "producer
> >>> per thread" design and discuss pros/cons of both approaches:
> >>>
> >>> https://docs.google.com/document/d/1CfOJaa6mdg5o7pLf_
> zXISV4oE0ZeMZwT_sG1QWgL4EE
> >>>
> >>>
> >>> Looking forward to your feedback.
> >>>
> >>>
> >>> -Matthias
> >>>
> >>>
> >>> On 3/10/17 3:24 AM, Damian Guy wrote:
>  Hi Matthias,
> 
>  Thanks for the response. I agree with you regarding the use of
>  PartitionGrouper to reduce the number of tasks. It would be good to
> have an
>  idea of any additional load on the brokers as we increase the number
> of
>  tasks and therefore producers.
> 
>  Thanks,
>  Damian
> 
>  On Wed, 8 Mar 2017 at 01:45 Matthias J. Sax 
> wrote:
> 
> > Damian, Jun,
> >
> > Thanks for y

Re: [DISCUSS] KIP-129: Kafka Streams Exactly-Once Semantics

[Sriram Subramanian]

This is a completely new feature which is controlled by a config. It would
be a regression if you upgraded streams and saw a different behavior. That
would not happen in this case. This is similar to how we are introducing
idempotent producer in core kafka with a config to turn it on. There is no
guarantee that the performance of the producer with the config turned on
will be the same although eventually we will like to get to it.

On Wed, Mar 22, 2017 at 7:12 AM, Michael Noll  wrote:

> I second Eno's concern regarding the impact of Streams EOS on state stores.
>
> >  We do a full recovery today and the EOS proposal will not make this any
> worse.
>
> Yes, today we do a full state store recovery under certain failures.
> However, I think the point (or perhaps: open question) is that, with the
> EOS design, there's now an *increased likelihood* of such failures that
> trigger full state store recovery.  If this increase is significant, then I
> would consider this to be a regression that we should address.
>
> As Eno said:
>
> > currently we pay the recovery price for a Kafka Streams instance failure.
> > Now we might pay it for a transaction failure. Will transaction failures
> be
> > more or less common than the previous types of failures?
>
> Damian voiced similar concerns at the very beginning of this discussion,
> not sure what his current opinion is here.
>
> -Michael
>
>
>
>
>
> On Wed, Mar 22, 2017 at 1:04 AM, Sriram Subramanian 
> wrote:
>
> > To add to this discussion, I do think we should think about this in
> > increments. We do a full recovery today and the EOS proposal will not
> make
> > this any worse. Using store snapshot is a good option to avoid store
> > recovery in the future but as Eno points out, all pluggable stores would
> > need to have this ability. W.r.t transaction failures, this should not be
> > an issue. We should be simply retrying. There is one optimization we can
> do
> > for clean shutdowns. We could store a clean shutdown file that contains
> the
> > input offsets. This file gets written when you close the streams
> instance.
> > On start, you could can check the offsets from the shutdown file and
> > compare it with the offsets we get from the consumer and ensure they
> match.
> > If they do, you could use the same store instead of recovering. However,
> if
> > we go with the snapshot approach, this will not be required. My vote
> would
> > be to implement V1 and solve the bootstrap problem which exist today in
> the
> > future versions.
> >
> > On Tue, Mar 21, 2017 at 4:43 PM, Matthias J. Sax 
> > wrote:
> >
> > > Thanks for your feedback Eno.
> > >
> > > For now, I still think that the KIP itself does not need to talk about
> > > this in more detail, because we apply the same strategy for EoS as for
> > > non-EoS (as of 0.10.2).
> > >
> > > Thus, in case of a clean shutdown, we write the checkpoint file for a
> > > store and thus know we can reuse the store. In case of failure, we need
> > > to recreate the store from the changelog.
> > >
> > > > Will a V1 design that relies on plain store recovery from Kafka for
> > > > each transaction abort be good enough, or usable?
> > >
> > > Why should it not be usable? It's the same strategy as used in 0.10.2
> > > and it runs in production in many companies already.
> > >
> > > > however it seems to me we might have a regression of sorts
> > > > Now we might pay it for a transaction failure.
> > >
> > > I would assume transaction failures to be quite rare. Maybe the core
> EoS
> > > folks can comment here, too.
> > >
> > >
> > >
> > > -Matthias
> > >
> > >
> > >
> > > On 3/20/17 3:16 PM, Eno Thereska wrote:
> > > > Hi Matthias,
> > > >
> > > > I'd like to see some more info on how you propose to handle
> > transactions
> > > that involve state stores in the KIP itself. The design doc has info
> > about
> > > various optimisations like RocksDb snapshots and transactions and such,
> > but
> > > will there be a user-visible interface that indicates whether a store
> has
> > > snapshot and/or transactional capabilities? If a user plugs in another
> > > store, what guarantees are they expected to get?
> > > >
> > > > Will a V1 design that relies on plain store recovery from Kafka for
> > each
> > > transaction abort be good enough, or usable? If your dataset is large

Re: [VOTE] KIP-129: Kafka Streams Exactly-Once Semantics

[Sriram Subramanian]

+1

On Wed, Mar 29, 2017 at 9:36 AM, Bill Bejeck  wrote:

> +1 (non-binding)
>
> Thanks Matthias,
> Bill
>
> On Wed, Mar 29, 2017 at 12:18 PM, Apurva Mehta 
> wrote:
>
> > +1 (non-binding)
> >
> > On Wed, Mar 29, 2017 at 9:17 AM, Jay Kreps  wrote:
> >
> > > +1
> > >
> > > -Jay
> > >
> > > On Mon, Mar 20, 2017 at 11:27 AM, Matthias J. Sax <
> matth...@confluent.io
> > >
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I would like to start the vote for KIP-129. Of course, feel free to
> > > > provide some more feedback on the DISCUSS thread.
> > > >
> > > > Thanks a lot!
> > > >
> > > >
> > > > -Matthias
> > > >
> > > >
> > >
> >
>

Re: [VOTE] KIP-120: Cleanup Kafka Streams builder API

[Sriram Subramanian]

+1

On Tue, Apr 4, 2017 at 9:01 PM, Ewen Cheslack-Postava 
wrote:

> +1 (binding)
>
> -Ewen
>
> On Thu, Mar 30, 2017 at 4:03 PM, Guozhang Wang  wrote:
>
> > +1.
> >
> >
> > On Thu, Mar 30, 2017 at 1:18 AM, Damian Guy 
> wrote:
> >
> > > Thanks Matthias.
> > >
> > > +1
> > >
> > > On Thu, 23 Mar 2017 at 22:40 Matthias J. Sax 
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > I would like to start the VOTE on KIP-120:
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-120%
> > > 3A+Cleanup+Kafka+Streams+builder+API
> > > >
> > > > If you have further comments, please reply to the DISCUSS thread.
> > > >
> > > > Thanks a lot!
> > > >
> > > >
> > > > -Matthias
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>

Re: [VOTE] KIP-114: KTable state stores and improved semantics

[Sriram Subramanian]

+1

On Fri, Apr 21, 2017 at 11:06 AM, Guozhang Wang  wrote:

> +1
>
> On Fri, Apr 21, 2017 at 10:52 AM, Bill Bejeck  wrote:
>
> > +1
> > On Fri, Apr 21, 2017 at 1:48 PM Matthias J. Sax 
> > wrote:
> >
> > > +1
> > >
> > > On 4/21/17 10:39 AM, Eno Thereska wrote:
> > > > Hi there,
> > > >
> > > > Unless there are more issues on the discuss thread, I'd like to start
> > > the vote on KIP-114.
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 114%3A+KTable+state+stores+and+improved+semantics
> > > <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 114:+KTable+state+stores+and+improved+semantics
> > > >.
> > > >
> > > > Thanks
> > > > Eno
> > > >
> > >
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: What to do when file.rename fails?

[Sriram Subramanian]

The failure really means that the filesystem is configured incorrectly
(from the link). In such circumstances it is best to fail and let the
operations/admins know instead of working around it.

On 1/24/15 9:42 AM, "Jay Kreps"  wrote:

>Hey guys,
>
>Jaikiran posted a patch on KAFKA-1853 to improve the handling of failures
>during delete.
>https://issues.apache.org/jira/browse/KAFKA-1853
>
>The core problem here is that we are doing File.rename() as part of the
>delete sequence which returns false if the rename failed. Or file delete
>sequence is something like the following:
>1. Remove the file from the index so no new reads can begin on it
>2. Rename the file to xyz.deleted so that if we crash it will get cleaned
>up
>3. Schedule a task to delete the file in 30 seconds or so when any
>in-progress reads have likely completed. The goal here is to avoid errors
>on in progress reads but also avoid locking on all reads.
>
>The question is what to do when rename fails? Previously if this happened
>we actually didn't pay attention and would fail to delete the file
>entirely. This patch changes it so that if the rename fails we log an
>error
>and force an immediate delete.
>
>I think this is the right thing to do, but I guess the real question is
>why
>would rename fail? Some possibilities:
>http://stackoverflow.com/questions/2372374/why-would-a-file-rename-fail-in
>-java
>
>An alternative would be to treat this as a filesystem error and shutdown
>as
>we do elsewhere.
>
>Thoughts?
>
>-Jay

Re: [VOTE] KIP-8: Add a flush() method to the new producer

[Sriram Subramanian]

+1

On 2/19/15 12:08 PM, "Neha Narkhede"  wrote:

>+1 (binding)
>
>On Thu, Feb 19, 2015 at 6:29 AM, Joel Koshy  wrote:
>
>> +1 (binding)
>>
>> On Wed, Feb 18, 2015 at 07:03:26PM -0500, Joe Stein wrote:
>> > +1 binding
>> >
>> > ~ Joestein
>> >  On Feb 18, 2015 6:50 PM, "Jay Kreps"  wrote:
>> >
>> > >
>> > >
>> 
>>https://cwiki.apache.org/confluence/display/KAFKA/KIP-8+-+Add+a+flush+met
>>hod+to+the+producer+API
>> > >
>> > > +1 binding
>> > >
>> > > -Jay
>> > >
>>
>>
>
>
>-- 
>Thanks,
>Neha

Re: Announcing the Confluent Platform built on Apache Kafka

[Sriram Subramanian]

Congratulations! 

On 2/25/15 11:15 AM, "Joe Stein"  wrote:

>Awesome!
>
>The future of schema management, has arrived
>
>~ Joestein
>
>On Wed, Feb 25, 2015 at 1:34 PM, Gwen Shapira 
>wrote:
>
>> Congrats, Confluent team.
>>
>> This is super exciting :)
>>
>>
>> On Wed, Feb 25, 2015 at 10:31 AM, Neha Narkhede 
>>wrote:
>>
>> > Folks,
>> >
>> > We, at Confluent , are excited to announce the
>> > release
>> > of Confluent Platform 1.0 built around Apache Kafka -
>> >
>> 
>>http://blog.confluent.io/2015/02/25/announcing-the-confluent-platform-1-0
>>/
>> >
>> > We also published a detailed two-part guide on how you can put Kafka
>>to
>> use
>> > in your organization -
>> > http://blog.confluent.io/2015/02/25/stream-data-platform-1/
>> >
>> > And, there is a public mailing list where we would love to hear your
>> > feedback: confluent-platf...@googlegroups.com
>> >
>> > Thanks,
>> > Neha
>> >
>>

Re: [VOTE] KIP-66: Single Message Transforms for Kafka Connect

[Sriram Subramanian]

+1

On Wed, Jan 4, 2017 at 1:29 PM, Gwen Shapira  wrote:

> I would have preferred not to bundle transformations, but since SMT
> capability is a much needed feature, I'll take it in its current form.
>
> +1
>
> On Wed, Jan 4, 2017 at 10:47 AM, Shikhar Bhushan 
> wrote:
> > Hi all,
> >
> > I'd like to start voting on KIP-66:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 66%3A+Single+Message+Transforms+for+Kafka+Connect
> >
> > Best,
> >
> > Shikhar
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>

Re: [VOTE] KIP-102 - Add close with timeout for consumers

[Sriram Subramanian]

+1

On Thu, Jan 5, 2017 at 6:30 PM, Ewen Cheslack-Postava 
wrote:

> +1
>
> -Ewen
>
> On Thu, Jan 5, 2017 at 5:48 PM, Neha Narkhede  wrote:
>
> > +1 (binding)
> >
> > On Thu, Jan 5, 2017 at 2:07 PM Rajini Sivaram 
> > wrote:
> >
> > > Hi all,
> > >
> > >
> > > I would like to start the voting process for *KIP-102 - Add close with
> > > timeout for consumers*:
> > >
> > >
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 102+-+Add+close+with+timeout+for+consumers
> > >
> > >
> > >
> > > This KIP adds a new close method with a timeout for consumers similar
> to
> > > the close method in the producer. As described in the discussion thread
> > > <
> > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201612.
> mbox/%3cCAG_+
> > n9us5ohthwmyai9pz4s2j62fmils2ufj8oie9jpmyaf...@mail.gmail.com%3e
> > > >,
> > > the changes are only in the close code path and hence the impact is not
> > too
> > > big. The existing close() method without a timeout will use a default
> > > timeout of 30 seconds.
> > >
> > >
> > > Thank you
> > >
> > >
> > > Regards,
> > >
> > >
> > > Rajini
> > >
> > --
> > Thanks,
> > Neha
> >
>

Re: [VOTE] KIP-105: Addition of Record Level for Sensors

[Sriram Subramanian]

+1

On Fri, Jan 6, 2017 at 8:40 AM, Bill Bejeck  wrote:

> +1
>
> On Fri, Jan 6, 2017 at 11:06 AM, Guozhang Wang  wrote:
>
> > +1
> >
> > On Fri, Jan 6, 2017 at 2:55 AM, Damian Guy  wrote:
> >
> > > +1
> > >
> > > On Fri, 6 Jan 2017 at 10:48 Ismael Juma  wrote:
> > >
> > > > Thanks for the KIP, +1 (binding).
> > > >
> > > > Ismael
> > > >
> > > > On Fri, Jan 6, 2017 at 10:37 AM, Eno Thereska <
> eno.there...@gmail.com>
> > > > wrote:
> > > >
> > > > > The discussion points for KIP-105 are addressed. At this point we'd
> > > like
> > > > > to start the vote for it:
> > > > >
> > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > > 105%3A+Addition+of+Record+Level+for+Sensors <
> > https://cwiki.apache.org/
> > > > > confluence/display/KAFKA/KIP-105:+Addition+of+Record+Level+
> > > for+Sensors>
> > > > >
> > > > > Thanks
> > > > > Eno and Aarti
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>

Re: [VOTE] KIP-103: Separation of Internal and External traffic

[Sriram Subramanian]

+1

On Fri, Jan 6, 2017 at 3:21 AM, Rajini Sivaram 
wrote:

> Ismael,
>
> Thank you for the KIP. It is a very useful feature.
>
> +1 (non-binding)
>
> Regards,
>
> Rajini
>
> On Fri, Jan 6, 2017 at 10:51 AM, Ismael Juma  wrote:
>
> > Hi all,
> >
> > As the discussion seems to have settled down, I would like to initiate
> the
> > voting process for KIP-103: Separation of Internal and External traffic:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 103%3A+Separation+of+Internal+and+External+traffic
> >
> > The vote will run for a minimum of 72 hours.
> >
> > Thanks,
> > Ismael
> >
>

Re: [VOTE] KIP-104: Granular Sensors for Streams

[Sriram Subramanian]

+1

On Fri, Jan 6, 2017 at 9:12 AM, Matthias J. Sax 
wrote:

> +1
>
> On 1/6/17 8:01 AM, Guozhang Wang wrote:
> > +1
> >
> > On Fri, Jan 6, 2017 at 5:05 AM, Bill Bejeck  wrote:
> >
> >> +1
> >>
> >> On Fri, Jan 6, 2017 at 5:57 AM, Damian Guy 
> wrote:
> >>
> >>> +1
> >>>
> >>> On Fri, 6 Jan 2017 at 09:37 Eno Thereska 
> wrote:
> >>>
>  The discussion points for KIP-104 are addressed. At this point we'd
> >> like
>  to start the vote for it:
> 
> 
>  https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>> 104%3A+Granular+Sensors+for+Streams
>  <
>  https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>> 104:+Granular+Sensors+for+Streams
> >
> 
>  Thanks,
>  Eno and Aarti
> >>>
> >>
> >
> >
> >
>
>

Re: [VOTE] KIP-104: Granular Sensors for Streams

[Sriram Subramanian]

+1

On Fri, Jan 6, 2017 at 4:41 PM, Eno Thereska  wrote:

> I light of the recent discussion on KIP-104 I've made changes to the KIP
> to explicitly list all APIs of the Metrics class for completion. I think
> it's fair to re-start the vote on the KIP in light of the new discussion
> for transparency:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 104%3A+Granular+Sensors+for+Streams <https://cwiki.apache.org/
> confluence/display/KAFKA/KIP-104:+Granular+Sensors+for+Streams>
>
> So could you please re-vote even if you've previously voted?
>
> Thank you
> no
>
> > On 6 Jan 2017, at 17:31, Sriram Subramanian  wrote:
> >
> > +1
> >
> > On Fri, Jan 6, 2017 at 9:12 AM, Matthias J. Sax 
> > wrote:
> >
> >> +1
> >>
> >> On 1/6/17 8:01 AM, Guozhang Wang wrote:
> >>> +1
> >>>
> >>> On Fri, Jan 6, 2017 at 5:05 AM, Bill Bejeck  wrote:
> >>>
> >>>> +1
> >>>>
> >>>> On Fri, Jan 6, 2017 at 5:57 AM, Damian Guy 
> >> wrote:
> >>>>
> >>>>> +1
> >>>>>
> >>>>> On Fri, 6 Jan 2017 at 09:37 Eno Thereska 
> >> wrote:
> >>>>>
> >>>>>> The discussion points for KIP-104 are addressed. At this point we'd
> >>>> like
> >>>>>> to start the vote for it:
> >>>>>>
> >>>>>>
> >>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>>> 104%3A+Granular+Sensors+for+Streams
> >>>>>> <
> >>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >>>>> 104:+Granular+Sensors+for+Streams
> >>>>>>>
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Eno and Aarti
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>
> >>
>
>

Re: [VOTE] KIP-108: Create Topic Policy

[Sriram Subramanian]

+1

On Mon, Jan 9, 2017 at 3:29 PM, Apurva Mehta  wrote:

> (hit send too soon)
>
> +1 (non-binding).. that is a very well written KIP!
>
> On Mon, Jan 9, 2017 at 3:28 PM, Apurva Mehta  wrote:
>
> > +1, that1
> >
> > On Mon, Jan 9, 2017 at 2:47 PM, Neha Narkhede  wrote:
> >
> >> +1 - thanks Ismael!
> >>
> >> On Mon, Jan 9, 2017 at 2:30 PM Gwen Shapira  wrote:
> >>
> >> > +1 - thanks for the proposal, I think it will be super useful for
> >> admins.
> >> >
> >> > On Sun, Jan 8, 2017 at 6:50 AM, Ismael Juma 
> wrote:
> >> > > Hi all,
> >> > >
> >> > > As the discussion seems to have settled down, I would like to
> initiate
> >> > the
> >> > > voting process for KIP-108: Create Topic Policy:
> >> > >
> >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-108
> >> > > %3A+Create+Topic+Policy
> >> > >
> >> > > The vote will run for a minimum of 72 hours.
> >> > >
> >> > > Thanks,
> >> > > Ismael
> >> >
> >> >
> >> >
> >> > --
> >> > Gwen Shapira
> >> > Product Manager | Confluent
> >> > 650.450.2760 <(650)%20450-2760> | @gwenshap
> >> > Follow us: Twitter | blog
> >> >
> >> --
> >> Thanks,
> >> Neha
> >>
> >
> >
>

Re: [ANNOUNCE] New committer: Grant Henke

[Sriram Subramanian]

Congratulations Grant!

On Wed, Jan 11, 2017 at 11:51 AM, Gwen Shapira  wrote:

> The PMC for Apache Kafka has invited Grant Henke to join as a
> committer and we are pleased to announce that he has accepted!
>
> Grant contributed 88 patches, 90 code reviews, countless great
> comments on discussions, a much-needed cleanup to our protocol and the
> on-going and critical work on the Admin protocol. Throughout this, he
> displayed great technical judgment, high-quality work and willingness
> to contribute where needed to make Apache Kafka awesome.
>
> Thank you for your contributions, Grant :)
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>

Re: [DISCUSS] KIP-117: Add a public AdministrativeClient API for Kafka admin operations

[Sriram Subramanian]

+1


> On Feb 7, 2017, at 9:17 AM, radai  wrote:
> 
> +1.
> 
> under ismael's "facet" approach we could always start with
> AdminClient.topics() and then pile on more of them later.
> 
>> On Tue, Feb 7, 2017 at 8:57 AM, Grant Henke  wrote:
>> 
>> +1 I think its important to focus this KIP discussion on the "patterns" we
>> would like to see in the client and a few key methods in order to make
>> progress and then iterate from there.
>> 
>> I think we should let Colin drive the APIs he thinks are important since he
>> is volunteering to do the work. And then others can propose and add APIs
>> from there.
>> 
>>> On Tue, Feb 7, 2017 at 10:37 AM, Ismael Juma  wrote:
>>> 
>>> Hi all,
>>> 
>>> I think it's good that we have discussed a number of API that would make
>>> sense in the AdminClient. Having done that, I think we should now narrow
>>> the focus of this KIP to a small set of methods to get us started. Once
>> we
>>> have an AdminClient in the codebase, we can propose subsequent KIPs to
>>> enrich it. I would suggest the following:
>>> 
>>> 1. Let's focus on topic management operations: describe, create, alter
>> and
>>> delete topics.
>>> 2. Let's add an @Unstable annotation to the class and specify in the
>>> javadoc that the methods are subject to change (if necessary).
>>> 
>>> Thoughts?
>>> 
>>> Ismael
>>> 
 On Fri, Feb 3, 2017 at 6:24 PM, Colin McCabe  wrote:
 
> On Thu, Feb 2, 2017, at 21:45, Becket Qin wrote:
> Hi Colin,
> 
> Thanks for the KIP. An admin client is probably a must after we block
> direct access to ZK. Some comments and thoughts below:
> 
> 1. Do we have a clear scope for the admin client? It might be worth
> thinking about the entire user experience of the admin client.
>> Ideally
>>> we
> may want to have a single client to do all the administrative work
> instead
> of having multiple ways to do different things. For example, do we
>> want
> to
> add topic configurations change API in the admin client? What about
> partition movements and preferred leader election? Those are also
> administrative tasks which seem reasonable to be integrated into the
> admin
> client.
 
 Thanks for the comments, Becket!
 
 I agree that topic configuration change should be in the administrative
 client.  I have not thought about partition movement or preferred
>> leader
 election.  It probably makes sense to put them in the client as well,
 but we should probably have a longer discussion about those features
 later when someone is ready to implement them ;)
 
 best,
 Colin
>> 
>> 
>> 
>> --
>> Grant Henke
>> Software Engineer | Cloudera
>> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>> 

Re: [VOTE] 0.10.2.0 RC2

[Sriram Subramanian]

+1
Verified signatures, tests and docs

On Thu, Feb 16, 2017 at 9:44 AM, Neha Narkhede  wrote:

> +1 (binding)
>
> Verified signatures, quickstart, docs.
>
> Thanks for running the release, Ewen!
>
> On Thu, Feb 16, 2017 at 9:42 AM Gwen Shapira  wrote:
>
> > +1 (binding).
> >
> > Verified signatures, ran unit tests, ran quickstart.
> >
> > Nice release :)
> >
> > On Tue, Feb 14, 2017 at 10:39 AM, Ewen Cheslack-Postava
> >  wrote:
> > > Hello Kafka users, developers and client-developers,
> > >
> > > This is the third candidate for release of Apache Kafka 0.10.2.0.
> > >
> > > This is a minor version release of Apache Kafka. It includes 19 new
> KIPs.
> > > See the release notes and release plan (https://cwiki.apache.org/conf
> > > luence/display/KAFKA/Release+Plan+0.10.2.0) for more details. A few
> > feature
> > > highlights: SASL-SCRAM support, improved client compatibility to allow
> > use
> > > of clients newer than the broker, session windows and global tables in
> > the
> > > Kafka Streams API, single message transforms in the Kafka Connect
> > framework.
> > >
> > > Important note: in addition to the artifacts generated using JDK7 for
> > Scala
> > > 2.10 and 2.11, this release also includes experimental artifacts built
> > > using JDK8 for Scala 2.12.
> > >
> > > Important code changes since RC1 (non-docs, non system tests):
> > >
> > > KAFKA-4756; The auto-generated broker id should be passed to
> > > MetricReporter.configure
> > > KAFKA-4761; Fix producer regression handling small or zero batch size
> > >
> > > Release notes for the 0.10.2.0 release:
> > > http://home.apache.org/~ewencp/kafka-0.10.2.0-rc2/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by February 17th 5pm ***
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > http://kafka.apache.org/KEYS
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > http://home.apache.org/~ewencp/kafka-0.10.2.0-rc2/
> > >
> > > * Maven artifacts to be voted upon:
> > > https://repository.apache.org/content/groups/staging/
> > >
> > > * Javadoc:
> > > http://home.apache.org/~ewencp/kafka-0.10.2.0-rc2/javadoc/
> > >
> > > * Tag to be voted upon (off 0.10.2 branch) is the 0.10.2.0 tag:
> > >
> > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> 5712b489038b71ed8d5a679856d1dfaa925eadc1
> > >
> > >
> > > * Documentation:
> > > http://kafka.apache.org/0102/documentation.html
> > >
> > > * Protocol:
> > > http://kafka.apache.org/0102/protocol.html
> > >
> > > * Successful Jenkins builds for the 0.10.2 branch:
> > > Unit/integration tests:
> > https://builds.apache.org/job/kafka-0.10.2-jdk7/77/
> > > System tests:
> > https://jenkins.confluent.io/job/system-test-kafka-0.10.2/29/
> > >
> > > /**
> > >
> > > Thanks,
> > > Ewen
> >
> >
> >
> > --
> > Gwen Shapira
> > Product Manager | Confluent
> > 650.450.2760 <(650)%20450-2760> | @gwenshap
> > Follow us: Twitter | blog
> >
> --
> Thanks,
> Neha
>

Re: [ANNOUNCE] Apache Kafka 0.10.2.0 Released

[Sriram Subramanian]

Thanks Ewen for driving this.

On Wed, Feb 22, 2017 at 12:40 AM, Guozhang Wang  wrote:

> Thanks Ewen for driving the release!
>
> Guozhang
>
> On Wed, Feb 22, 2017 at 12:33 AM, Ewen Cheslack-Postava  >
> wrote:
>
> > The Apache Kafka community is pleased to announce the release for Apache
> > Kafka 0.10.2.0. This is a feature release which includes the completion
> > of 15 KIPs, over 200 bug fixes and improvements, and more than 500 pull
> > requests merged.
> >
> > All of the changes in this release can be found in the release notes:
> > https://archive.apache.org/dist/kafka/0.10.2.0/RELEASE_NOTES.html
> >
> > Apache Kafka is a distributed streaming platform with four four core
> > APIs:
> >
> > ** The Producer API allows an application to publish a stream records to
> > one or more Kafka topics.
> >
> > ** The Consumer API allows an application to subscribe to one or more
> > topics and process the stream of records produced to them.
> >
> > ** The Streams API allows an application to act as a stream processor,
> > consuming an input stream from one or more topics and producing an
> > output
> > stream to one or more output topics, effectively transforming the input
> > streams to output streams.
> >
> > ** The Connector API allows building and running reusable producers or
> > consumers that connect Kafka topics to existing applications or data
> > systems. For example, a connector to a relational database might capture
> > every change to a table.three key capabilities:
> >
> >
> > With these APIs, Kafka can be used for two broad classes of application:
> >
> > ** Building real-time streaming data pipelines that reliably get data
> > between systems or applications.
> >
> > ** Building real-time streaming applications that transform or react to
> > the
> > streams of data.
> >
> >
> > You can download the source release from
> > https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> > 0/kafka-0.10.2.0-src.tgz
> >
> > and binary releases from
> > https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> > 0/kafka_2.11-0.10.2.0.tgz
> > https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> > 0/kafka_2.10-0.10.2.0.tgz
> > https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.
> > 0/kafka_2.12-0.10.2.0.tgz
> > (experimental 2.12 artifact)
> >
> > Thanks to the 101 contributors on this release!
> >
> > Akash Sethi, Alex Loddengaard, Alexey Ozeritsky, amethystic, Andrea
> > Cosentino, Andrew Olson, Andrew Stevenson, Anton Karamanov, Antony
> > Stubbs, Apurva Mehta, Arun Mahadevan, Ashish Singh, Balint Molnar, Ben
> > Stopford, Bernard Leach, Bill Bejeck, Colin P. Mccabe, Damian Guy, Dan
> > Norwood, Dana Powers, dasl, Derrick Or, Dong Lin, Dustin Cote, Edoardo
> > Comar, Edward Ribeiro, Elias Levy, Emanuele Cesena, Eno Thereska, Ewen
> > Cheslack-Postava, Flavio Junqueira, fpj, Geoff Anderson, Guozhang Wang,
> > Gwen Shapira, Hikiko Murakami, Himani Arora, himani1, Hojjat Jafarpour,
> > huxi, Ishita Mandhan, Ismael Juma, Jakub Dziworski, Jan Lukavsky, Jason
> > Gustafson, Jay Kreps, Jeff Widman, Jeyhun Karimov, Jiangjie Qin, Joel
> > Koshy, Jon Freedman, Joshi, Jozef Koval, Json Tu, Jun He, Jun Rao,
> > Kamal, Kamal C, Kamil Szymanski, Kim Christensen, Kiran Pillarisetty,
> > Konstantine Karantasis, Lihua Xin, LoneRifle, Magnus Edenhill, Magnus
> > Reftel, Manikumar Reddy O, Mark Rose, Mathieu Fenniak, Matthias J. Sax,
> > Mayuresh Gharat, MayureshGharat, Michael Schiff, Mickael Maison,
> > MURAKAMI Masahiko, Nikki Thean, Olivier Girardot, pengwei-li, pilo,
> > Prabhat Kashyap, Qian Zheng, Radai Rosenblatt, radai-rosenblatt, Raghav
> > Kumar Gautam, Rajini Sivaram, Rekha Joshi, rnpridgeon, Ryan Pridgeon,
> > Sandesh K, Scott Ferguson, Shikhar Bhushan, steve, Stig Rohde Døssing,
> > Sumant Tambe, Sumit Arrawatia, Theo, Tim Carey-Smith, Tu Yang, Vahid
> > Hashemian, wangzzu, Will Marshall, Xavier Léauté, Xavier Léauté, Xi Hu,
> > Yang Wei, yaojuncn, Yuto Kawamura
> >
> > We welcome your help and feedback. For more information on how to
> > report problems, and to get involved, visit the project website at
> > http://kafka.apache.org/
> >
> > Thanks,
> > Ewen
> >
>
>
>
> --
> -- Guozhang
>

Re: [ANNOUNCE] New committer: Ismael Juma

[Sriram Subramanian]

Congratulations!

On Tue, Apr 26, 2016 at 6:57 AM, Gwen Shapira  wrote:

> Congratulations, very well deserved.
> On Apr 25, 2016 10:53 PM, "Neha Narkhede"  wrote:
>
> > The PMC for Apache Kafka has invited Ismael Juma to join as a committer
> and
> > we are pleased to announce that he has accepted!
> >
> > Ismael has contributed 121 commits
> >  to a wide range of
> > areas, notably within the security and the network layer. His involvement
> > has been phenomenal across the board from mailing lists, JIRA, code
> reviews
> > and helping us move to GitHub pull requests to contributing features, bug
> > fixes and code and documentation improvements.
> >
> > Thank you for your contribution and welcome to Apache Kafka, Ismael!
> >
> > --
> > Thanks,
> > Neha
> >
>

Re: [ANNOUNCE] New Kafka Committer Ewen Cheslack-Postava

[Sriram Subramanian]

Congrats!

On Tue, Dec 8, 2015 at 2:45 PM, Jay Kreps  wrote:

> Congrats Ewen!
>
> -Jay
>
> On Tue, Dec 8, 2015 at 11:37 AM, Neha Narkhede  wrote:
> > I am pleased to announce that the Apache Kafka PMC has voted to
> > invite Ewen Cheslack-Postava as a committer and Ewen has accepted.
> >
> > Ewen is an active member of the community and has contributed and
> reviewed
> > numerous patches to Kafka. His most significant contribution is Kafka
> > Connect which was released few days ago as part of 0.9.
> >
> > Please join me on welcoming and congratulating Ewen.
> >
> > Ewen, we look forward to your continued contributions to the Kafka
> > community!
> >
> > --
> > Thanks,
> > Neha
>

Re: [VOTE] KIP 20 Enable log preallocate to improve consume performance under windows and some old Linux file system

[Sriram Subramanian]

+1


> On Jun 10, 2015, at 9:38 AM, Jay Kreps  wrote:
> 
> +1
> 
> On Tue, Jun 9, 2015 at 11:24 PM, Honghai Chen 
> wrote:
> 
>> Hi Kafka,
>> 
>>After a long discussion, please help vote again for the
>> KIP. Thanks.
>> 
>> 
>> 
>> I wrote a KIP for this after some discussion on KAFKA-1646.
>> https://issues.apache.org/jira/browse/KAFKA-1646
>> 
>> 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-20+-+Enable+log+preallocate+to+improve+consume+performance+under+windows+and+some+old+Linux+file+system
>> 
>> The RB is here: https://reviews.apache.org/r/33204/diff/4/
>> 
>> 
>> 
>> 
>> 
>> Thanks,
>> 
>> Honghai Chen
>> 
>> 
>> 

Re: [DISCUSS] KIP-26 - Add Copycat, a connector framework for data import/export

[Sriram Subramanian]

I am still not convinced why a stream processing framework closely tied to
Kafka will not help with this (since we are also referring to some basic
transformations). The devil is in the details of the design and I would be
able to better comment on it after that. I would love to see a detailed
design doc on the internals!

On 6/23/15 2:59 PM, "Ewen Cheslack-Postava"  wrote:

>There was some discussion on the KIP call today. I'll give my summary of
>what I heard here to make sure this thread has the complete context for
>ongoing discussion.
>
>* Where the project should live, and if in Kafka, where should connectors
>live? If some are in Kafka and some not, how many and which ones? - There
>was little disagreement on the tradeoffs (coding and packaging together
>can
>make things easier for end users especially for a few very popular
>connectors, maintaining internally can lead to messier code base with more
>dependencies that's harder to work with, etc). Seems to be more focus on
>location of connectors than framework right now; we'll probably only make
>progress on this issue with some concrete proposals.
>* Organizational issues within Kafka - subproject? - Jay mentioned desire
>for consistency, which can be a problem even across subprojects.
>* Will streaming data be supported? - Yes, "Streaming and batch" section
>of
>design goals should cover this; this is a very important use case.
>* Additional transformations in copycat - Updated wiki to leave this a bit
>more open. Original motivation for leaving it out was to keep the scope of
>this KIP and the Copycat framework very clear since there is a danger in
>overgeneralizing and ending up with a stream processing framework;
>however,
>it's clear there are some very useful, very common examples like scrubbing
>data during import.
>* Schemas and how the data model works - this requires a more in depth
>answer when we get to a complete proposal, but the prototype we've been
>playing with internally uses something that can work with data roughly
>like
>Avro or JSON, and supports schemas. The goal is for this data model to
>only
>be used at runtime and for the serialization that is used for storing data
>in Kafka to be pluggable. Each type of serialization plugin might handle
>things like schemas in different ways. The reason we are proposing the
>inclusion of schemas is that it lets you cleanly carry important info
>across multiples stages, e.g. the schema for data pulled from a database
>is
>defined by the table the data is read from, intermediate processing steps
>might maintain schemas as well, and then an export to, e.g., a parquet
>file
>in HDFS would also use the schema. There will definitely need to be
>discussion about the details of this data model, what needs to be included
>to make it work across multiple serialization formats, etc.
>* Could mirror maker be implemented in Copycat? Same for Camus? - Yes,
>both
>would make sense in Copycat. One of the motivations is to have fewer tools
>required for a lot of these common tasks. Mirror maker is a case where we
>could easily maintain the connector as part of Kafka, and we could
>probably
>bootstrap one very quickly using lessons learned from mirror maker. The
>experience with mirror maker is also an argument for making sure Kafka
>devs
>are closely involved in Copycat development -- it's actually tricky to get
>it right even when you know Kafka and Copycat has to get everything right
>for more general cases.
>
>I made minor updates to the wiki to reflect some of these notes. Anyone
>else have any specific updates they think should be made to any of the
>sections, especially considerations I may have omitted from the "rejected
>alternatives" (or any "rejected" alternatives that they think still need
>to
>be under consideration)?
>
>Let me know what you think needs to be addressed to get this to a vote --
>I
>don't want to rush people, but I also don't want to just leave this
>lingering unless there are specific issues that can be addressed.
>
>-Ewen
>
>
>On Mon, Jun 22, 2015 at 8:32 PM, Roshan Naik 
>wrote:
>
>> Thanks Jay and Ewen for the response.
>>
>>
>> >@Jay
>> >
>> > 3. This has a built in notion of parallelism throughout.
>>
>>
>>
>> It was not obvious how it will look like or differ from existing
>>systemsŠ
>> since all of existing ones do parallelize data movement.
>>
>>
>> @Ewen,
>>
>> >Import: Flume is just one of many similar systems designed around log
>> >collection. See notes below, but one major point is that they generally
>> >don't provide any sort of guaranteed delivery semantics.
>>
>>
>> I think most of them do provide guarantees of some sort (Ex. Flume &
>> FluentD).
>>
>>
>> >YARN: My point isn't that YARN is bad, it's that tying to any
>>particular
>> >cluster manager severely limits the applicability of the tool. The
>>goal is
>> >to make Copycat agnostic to the cluster manager so it can run under
>>Mesos,
>> >YARN, etc.
>>
>> ok. Got it. Sounds like there is plan to do some wo

Re: [ANNOUNCE] New Committer

[Sriram Subramanian]

Congrats Gwen!


> On Jul 6, 2015, at 6:08 PM, Joe Stein  wrote:
> 
> I am pleased to announce that the Apache Kafka PMC has voted to invite Gwen
> Shapira as a committer and Gwen has accepted.
> 
> Please join me on welcoming and congratulating Gwen.
> 
> Thanks for the contribution both in the project (code, email, etc, etc,
> etc) and in throughout the community too(other projects, conferences, etc,
> etc, etc). I look forward to your continued contributions and much more to
> come!
> 
> ~ Joe Stein
> - - - - - - - - - - - - - - - - - - -
> [image: Logo-Black.jpg]
>  http://www.elodina.net
>http://www.stealth.ly
> - - - - - - - - - - - - - - - - - - -

Re: [VOTE] KIP-26 Add Copycat connector framework for data import/export

[Sriram Subramanian]

+1. This was long due!

On 7/14/15, 4:42 PM, "Guozhang Wang"  wrote:

>+1. Thanks Ewen!!
>
>On Tue, Jul 14, 2015 at 3:01 PM, Neha Narkhede  wrote:
>
>> +1 (binding)
>>
>> Thanks Ewen for taking on something that the Kafka project has long
>>waited
>> for!
>>
>> On Tue, Jul 14, 2015 at 2:58 PM, Jay Kreps  wrote:
>>
>> > +1
>> >
>> > Super excited!
>> >
>> > -Jay
>> >
>> > On Tue, Jul 14, 2015 at 2:09 PM, Ewen Cheslack-Postava <
>> e...@confluent.io>
>> > wrote:
>> >
>> > > Hi all,
>> > >
>> > > Let's start a vote on KIP-26: Add Copycat connector framework for
>>data
>> > > import/export
>> > >
>> > > For reference, here's the wiki:
>> > >
>> >
>> 
>>https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=58851767
>> > > And the mailing list thread (split across two months):
>> > >
>> > >
>> >
>> 
>>http://mail-archives.apache.org/mod_mbox/kafka-dev/201506.mbox/%3CCAE1jLM
>>OEJjnorFK5CtR3g-n%3Dm_AkrFsYeccsB4QimTRfGBrAGQ%40mail.gmail.com%3E
>> > >
>> > >
>> >
>> 
>>http://mail-archives.apache.org/mod_mbox/kafka-dev/201507.mbox/%3CCAHwHRr
>>UeNh%2BnCHwCTUCrcipHM3Po0ECUysO%2B%3DX3nwUeOGrcgdw%40mail.gmail.com%3E
>> > >
>> > > Just to clarify since this is a bit different from the KIPs voted
>>on so
>> > > far, the KIP just covers including Copycat in Kafka (rather than
>>having
>> > it
>> > > as a separate project). While the KIP aimed to be clear about the
>>exact
>> > > scope, the details require further discussion. The aim is to include
>> some
>> > > connectors as well, at a minimum for demonstration purposes, but the
>> > > expectation is that connector development will, by necessity, be
>> > federated.
>> > >
>> > > I'll kick it off with a +1 (non-binding).
>> > >
>> > > --
>> > > Thanks,
>> > > Ewen
>> > >
>> >
>>
>>
>>
>> --
>> Thanks,
>> Neha
>>
>
>
>
>-- 
>-- Guozhang

Re: Kafka Indentation

[Sriram Subramanian]

+1 for consistency.

On Fri, Jul 24, 2015 at 8:51 AM, Neha Narkhede  wrote:

> Personally, I prefer the consistency and 2 spaces for indentation. Although
> I'm in agreement with Jay about the ambition to eventually move the server
> code to Java, I'd hate to make that a blocker for discussing a simple
> change like standardizing on java/scala indentation :-)
>
> On Fri, Jul 24, 2015 at 8:07 AM, Ismael Juma  wrote:
>
> > On Fri, Jul 24, 2015 at 2:00 AM, Jay Kreps  wrote:
> >
> > > I do agree that working with a mixture of scala and java is a pain in
> the
> > > butt. What about considering the more extreme idea of just moving the
> > > remaining server-side scala into java? I like Scala, but the tooling
> and
> > > compatibility story for java is better, and Java 8 addressed some of
> the
> > > gaps. For a system like Kafka I do kind of think that what Scala offers
> > is
> > > less useful, and the kind of boring Java tooling like IDE support,
> > > findbugs, checkstyle, simple exception stack traces, and a good
> > > compatability story is more important.
> >
> >
> > I can certainly see the case for avoiding the complexity of two different
> > languages (assuming that the benefits are not worth it). However, I am
> not
> > sure about the "findbugs, checkstyle" point. Static checking is an area
> > that Scala does quite well (better than Java in many ways): scalastyle,
> > abide, scalariform, wartremover, scapegoat, etc. And Scala 2.11 also has
> a
> > number of Xlint warnings.
> >
> > Best,
> > Ismael
> >
>
>
>
> --
> Thanks,
> Neha
>

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

[Sriram Subramanian]

I think Kafka and streaming are synonymous. Kafka streams or Kafka
streaming really does not indicate "stream processing".

On Wed, Jul 29, 2015 at 6:20 PM, Neha Narkhede  wrote:

> Prefer something that evokes "stream processing on top of Kafka". And since
> I've heard many people conflate "streaming" with "streaming video" (I know,
> duh!), I'd vote for Kafka Streams or a maybe KStream.
>
> Thanks,
> Neha
>
> On Wed, Jul 29, 2015 at 6:08 PM, Jay Kreps  wrote:
>
> > Also, the most important part of any prototype, we should have a name for
> > this producing-consumer-thingamgigy:
> >
> > Various ideas:
> > - Kafka Streams
> > - KStream
> > - Kafka Streaming
> > - The Processor API
> > - Metamorphosis
> > - Transformer API
> > - Verwandlung
> >
> > For my part I think what people are trying to do is stream processing
> with
> > Kafka so I think something that evokes Kafka and stream processing is
> > preferable. I like Kafka Streams or Kafka Streaming followed by KStream.
> >
> > Transformer kind of makes me think of the shape-shifting cars.
> >
> > Metamorphosis is cool and hilarious but since we are kind of envisioning
> > this as more limited scope thing rather than a massive framework in its
> own
> > right I actually think it should have a descriptive name rather than a
> > personality of it's own.
> >
> > Anyhow let the bikeshedding commence.
> >
> > -Jay
> >
> >
> > On Thu, Jul 23, 2015 at 5:59 PM, Guozhang Wang 
> wrote:
> >
> > > Hi all,
> > >
> > > I just posted KIP-28: Add a transform client for data processing
> > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-28+-+Add+a+transform+client+for+data+processing
> > > >
> > > .
> > >
> > > The wiki page does not yet have the full design / implementation
> details,
> > > and this email is to kick-off the conversation on whether we should add
> > > this new client with the described motivations, and if yes what
> features
> > /
> > > functionalities should be included.
> > >
> > > Looking forward to your feedback!
> > >
> > > -- Guozhang
> > >
> >
>
>
>
> --
> Thanks,
> Neha
>

Re: [DISCUSS] KIP-28 - Add a transform client for data processing

[Sriram Subramanian]

I had the same thought. Kafka processor, KProcessor or even Kafka
stream processor is more relevant.



> On Jul 30, 2015, at 2:09 PM, Martin Kleppmann  wrote:
>
> I'm with Sriram -- Kafka is all about streams already (or topics, to be 
> precise, but we're calling it "stream processing" not "topic processing"), so 
> I find "Kafka Streams", "KStream" and "Kafka Streaming" all confusing, since 
> they seem to imply that other bits of Kafka are not about streams.
>
> I would prefer "The Processor API" or "Kafka Processors" or "Kafka Processing 
> Client" or "KProcessor", or something along those lines.
>
>> On 30 Jul 2015, at 15:07, Guozhang Wang  wrote:
>>
>> I would vote for KStream as it sounds sexier (is it only me??), second to
>> that would be Kafka Streaming.
>>
>>> On Wed, Jul 29, 2015 at 6:08 PM, Jay Kreps  wrote:
>>>
>>> Also, the most important part of any prototype, we should have a name for
>>> this producing-consumer-thingamgigy:
>>>
>>> Various ideas:
>>> - Kafka Streams
>>> - KStream
>>> - Kafka Streaming
>>> - The Processor API
>>> - Metamorphosis
>>> - Transformer API
>>> - Verwandlung
>>>
>>> For my part I think what people are trying to do is stream processing with
>>> Kafka so I think something that evokes Kafka and stream processing is
>>> preferable. I like Kafka Streams or Kafka Streaming followed by KStream.
>>>
>>> Transformer kind of makes me think of the shape-shifting cars.
>>>
>>> Metamorphosis is cool and hilarious but since we are kind of envisioning
>>> this as more limited scope thing rather than a massive framework in its own
>>> right I actually think it should have a descriptive name rather than a
>>> personality of it's own.
>>>
>>> Anyhow let the bikeshedding commence.
>>>
>>> -Jay
>>>
>>>
 On Thu, Jul 23, 2015 at 5:59 PM, Guozhang Wang  wrote:

 Hi all,

 I just posted KIP-28: Add a transform client for data processing
 <
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-28+-+Add+a+transform+client+for+data+processing
 .

 The wiki page does not yet have the full design / implementation details,
 and this email is to kick-off the conversation on whether we should add
 this new client with the described motivations, and if yes what features
>>> /
 functionalities should be included.

 Looking forward to your feedback!

 -- Guozhang
>>
>>
>>
>> --
>> -- Guozhang
>

Re: [DISCUSS] KIP-75 - Add per-connector Converters

[Sriram Subramanian]

+1

On Fri, Aug 12, 2016 at 7:36 PM, Gwen Shapira  wrote:

> Looks good to me too - simple, usable and the use-case is clear.
>
> Gwen
>
> On Fri, Aug 12, 2016 at 12:56 PM, Ewen Cheslack-Postava
>  wrote:
> > Hi all,
> >
> > I've added KIP-75 - Add per-connector Converters
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 75+-+Add+per-connector+Converters
> >
> > This is a very small completely backwards compatible change, a patch is
> > already available https://github.com/apache/kafka/pull/1721, and I think
> > probably not very controversial. Any feedback would be appreciated.
> >
> > --
> > Thanks,
> > Ewen
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
>

Re: [VOTE] KIP:71 Enable log compaction and deletion to co-exist

[Sriram Subramanian]

+1 (binding)

On Mon, Aug 15, 2016 at 10:27 AM, Ismael Juma  wrote:

> Thanks for the KIP, +1 (binding)
>
> Ismael
>
> On Mon, Aug 15, 2016 at 2:20 PM, Damian Guy  wrote:
>
> > I would like to initiate the voting process for KIP-71 (
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 71%3A+Enable+log+compaction+and+deletion+to+co-exist
> > ).
> >
> > This change will add a new cleanup.policy, compact_and_delete, that when
> > enabled will run both compaction and deletion.
> >
> > Thanks,
> > Damian
> >
>

Re: [VOTE] KIP-75 - Add per-connector Converters

[Sriram Subramanian]

+1 (binding)

On Mon, Aug 15, 2016 at 11:21 AM, Ewen Cheslack-Postava 
wrote:

> I would like to initiate the voting process for KIP-75:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 75+-+Add+per-connector+Converters
>
> I'll kick things off with a +1 (binding).
>
> --
> Thanks,
> Ewen
>

Re: [ANNOUNCE] New Kafka PMC member Gwen Shapira

[Sriram Subramanian]

Congratulations Gwen!

On Wed, Aug 17, 2016 at 9:24 PM, Neha Narkhede  wrote:

> Congratulations and welcome, Gwen!
> On Wed, Aug 17, 2016 at 6:44 PM Jun Rao  wrote:
>
> > Hi, Everyone,
> >
> > Gwen Shapira has been active in the Kafka community since she became a
> > Kafka committer
> > about a year ago. I am glad to announce that Gwen is now a member of
> Kafka
> > PMC.
> >
> > Congratulations, Gwen!
> >
> > Jun
> >
> --
> Thanks,
> Neha
>

Re: [VOTE] KIP-73 - Replication Quotas

[Sriram Subramanian]

I think the manual way of setting the throttling value is a good first step
and definitely required when things go bad. We should continue discussing
how we can build more intelligence over this incrementally.

+1 (binding)

On Fri, Aug 19, 2016 at 1:21 AM, Ben Stopford  wrote:

> I’d like to initiate the voting process for KIP-73:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 73+Replication+Quotas  confluence/display/KAFKA/KIP-73+Replication+Quotas>
>
> Ben

Re: [ANNOUNCE] New committer: Jason Gustafson

[Sriram Subramanian]

Congratulations Jason!

On Tue, Sep 6, 2016 at 3:40 PM, Vahid S Hashemian  wrote:

> Congratulations Jason on this very well deserved recognition.
>
> --Vahid
>
>
>
> From:   Neha Narkhede 
> To: "dev@kafka.apache.org" ,
> "us...@kafka.apache.org" 
> Cc: "priv...@kafka.apache.org" 
> Date:   09/06/2016 03:26 PM
> Subject:[ANNOUNCE] New committer: Jason Gustafson
>
>
>
> The PMC for Apache Kafka has invited Jason Gustafson to join as a
> committer and
> we are pleased to announce that he has accepted!
>
> Jason has contributed numerous patches to a wide range of areas, notably
> within the new consumer and the Kafka Connect layers. He has displayed
> great taste and judgement which has been apparent through his involvement
> across the board from mailing lists, JIRA, code reviews to contributing
> features, bug fixes and code and documentation improvements.
>
> Thank you for your contribution and welcome to Apache Kafka, Jason!
> --
> Thanks,
> Neha
>
>
>
>
>

Re: [VOTE] KIP-78: Cluster Id

[Sriram Subramanian]

+1 (binding)

On Tue, Sep 6, 2016 at 4:11 PM, Ismael Juma  wrote:

> Hi all,
>
> I would like to initiate the voting process for KIP-78: Cluster Id:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-78%3A+Cluster+Id
>
> As explained in the KIP and discussion thread, we see this as a good first
> step that can serve as a foundation for future improvements.
>
> Thanks,
> Ismael
>

Re: [VOTE] KIP-78 Cluster Id (second attempt)

[Sriram Subramanian]

+1 binding

> On Sep 6, 2016, at 7:46 PM, Ismael Juma  wrote:
> 
> Hi all,
> 
> I would like to (re)initiate[1] the voting process for KIP-78 Cluster Id:
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-78%3A+Cluster+Id
> 
> As explained in the KIP and discussion thread, we see this as a good first
> step that can serve as a foundation for future improvements.
> 
> Thanks,
> Ismael
> 
> [1] Even though I created a new vote thread, Gmail placed the messages in
> the discuss thread, making it not as visible as required. It's important to
> mention that two +1s were cast by Gwen and Sriram:
> 
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201609.mbox/%3CCAD5tkZbLv7fvH4q%2BKe%2B%3DJMgGq%2BZT2t34e0WRUsCT1ErhtKOg1w%40mail.gmail.com%3E

Re: [DISCUSS] Time-based releases for Apache Kafka

[Sriram Subramanian]

Sounds good to me. 

> On Sep 6, 2016, at 8:22 PM, Jason Gustafson  wrote:
> 
> Hey All,
> 
> It sounds like the general consensus is in favor of time-based releases. We
> can continue the discussion about LTS, but I wanted to go ahead and get
> things moving forward by volunteering to manage the next release, which is
> currently slated for October. If that sounds OK, I'll draft a release plan
> and send it out to the community for feedback and a vote.
> 
> Thanks,
> Jason
> 
>> On Thu, Aug 25, 2016 at 2:03 PM, Ofir Manor  wrote:
>> 
>> I happily agree that Kafka is a solid and the community is great :)
>> But I think there is a gap in perception here.
>> For me, LTS means that someone is actively taking care of a release -
>> actively backporting critical fixes (security, stability, data loss,
>> corruption, hangs etc) from trunk to that LTS version periodically for an
>> extended period of time, for example 18-36 months... So people can really
>> rely on the same Kafka version for a long time.
>> Is someone doing it today for 0.9.0? When is 0.9.0.2 expected? When is
>> 0.8.2.3 expected? Will they cover all known critical issues for whoever
>> relies on them in production?
>> In other words, what is the scope of support that the community want to
>> commit for older versions? (upgrade compatibility? investigating bug
>> reports? proactively backporting fixes?)
>> BTW, another legit option is that the Apache Kafka project won't commit to
>> LTS releases. It could let commercial vendors compete on supporting very
>> old versions. I find that actually quite reasonable as well.
>> 
>> Ofir Manor
>> 
>> Co-Founder & CTO | Equalum
>> 
>> Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io
>> 
>> On Thu, Aug 25, 2016 at 8:19 PM, Andrew Schofield <
>> andrew_schofield_j...@outlook.com> wrote:
>> 
>>> I agree that the Kafka community has managed to maintain a very high
>>> quality level, so I'm not concerned
>>> about the quality of non-LTS releases. If the principle is that every
>>> release is supported for 2 years, that
>>> would be good. I suppose that if the burden of having that many
>> in-support
>>> releases proves too heavy,
>>> as you say we could reconsider.
>>> 
>>> Andrew Schofield
>>> 
>>> 
 From: g...@confluent.io
 Date: Thu, 25 Aug 2016 09:57:30 -0700
 Subject: Re: [DISCUSS] Time-based releases for Apache Kafka
 To: dev@kafka.apache.org
 
 I prefer Ismael's suggestion for supporting 2-years (6 releases)
 rather than have designated LTS releases.
 
 The LTS model seems to work well when some releases are high quality
 (LTS) and the rest are a bit more questionable. It is great for
 companies like Redhat, where they have to invest less to support few
 releases and let the community deal with everything else.
 
 Until now the Kafka community has managed to maintain very high
 quality level. Not just for releases, our trunk is often of better
 quality than other project's releases - we don't think of stability as
 something you tuck into a release (and just some releases) but rather
 as an on-going concern. There are costs to doing things that way, but
 in general, I think it has served us well - allowing even conservative
 companies to run on the latest released version.
 
 I hope we can agree to at least try maintaining last 6 releases as LTS
 (i.e. every single release is supported for 2 years) rather than
 designate some releases as better than others. Of course, if this
 totally fails, we can reconsider.
 
 Gwen
 
 On Thu, Aug 25, 2016 at 9:51 AM, Andrew Schofield
  wrote:
> The proposal sounds pretty good, but the main thing currently missing
>>> is a proper long-term support release.
> 
> Having 3 releases a year sounds OK, but if they're all equivalent and
>>> bugfix releases are produced for the most
> recent 2 or 3 releases, anyone wanting to run on an "in support"
>>> release of Kafka has to upgrade every 8-12 months.
> If you don't actually want anything specific from the newer releases,
>>> it's just unnecessary churn.
> 
> Wouldn't it be better to designate one release every 12-18 months as a
>>> long-term support release with bugfix releases
> produced for those for a longer period of say 24 months. That halves
>>> the upgrade work for people just wanting to keep
> "in support". Now that adoption is increasing, there are plenty of
>>> users that just want a dependable messaging system
> without having to be deeply knowledgeable about its innards.
> 
> LTS works nicely for plenty of open-source projects. I think it would
>>> work well for Kafka too.
> 
> Andrew Schofield
> 
> 
>> From: ofir.ma...@equalum.io
>> Date: Thu, 25 Aug 2016 16:07:07 +0300
>> Subject: Re: [DISCUSS] Time-based releases for Apache Kafka

Re: [DISCUSS] Kafka 0.10.1.0 Release Plan

[Sriram Subramanian]

+1

Plan looks good to me. Thanks a lot for putting this together!


On Fri, Sep 9, 2016 at 3:45 PM, Jason Gustafson  wrote:

> Hi All,
>
> I've volunteered to be release manager for the upcoming 0.10.1 release and
> would like to propose the following timeline:
>
> Feature Freeze (Sep. 19): The 0.10.1 release branch will be created.
> Code Freeze (Oct. 3): The first RC will go out.
> Final Release (~Oct. 17): Assuming no blocking issues remain, the final
> release will be cut.
>
> The purpose of the time between the feature freeze and code freeze is to
> stabilize the set of release features. We will continue to accept bug fixes
> during this time and new system tests, but no new features will be merged
> into the release branch (they will continue to be accepted in trunk,
> however). After the code freeze, only blocking bug fixes will be accepted.
> Features which cannot be completed in time will have to await the next
> release cycle.
>
> This is the first iteration of the time-based release plan:
> https://cwiki.apache.org/confluence/display/KAFKA/Time+Based+Release+Plan.
> Note
> that the final release is scheduled for October 17, so we have a little
> more than a month to prepare.
>
> Features which have already been merged to trunk and will be included in
> this release include the following:
>
> KIP-4 (partial): Add request APIs to create and delete topics
> KIP-33: Add time-based index
> KIP-60: Make Java client classloading more flexible
> KIP-62: Allow consumer to send heartbeats from a background thread
> KIP-65: Expose timestamps to Connect
> KIP-67: Queryable state for Kafka Streams
> KIP-71: Enable log compaction and deletion to co-exist
> KIP-75 - Add per-connector Converters
>
> Since this is the first time-based release, we propose to also include the
> following KIPs which already have a patch available and have undergone some
> review:
>
> KIP-58: Make log compaction point configurable
> KIP-63: Unify store and downstream caching in streams
> KIP-70: Revise consumer partition assignment semantics
> KIP-73: Replication quotas
> KIP-74: Add fetch response size limit in bytes
> KIP-78: Add clusterId
>
> One of the goals of time-based releases is to avoid the rush to get
> unstable features in before the release deadline. If a feature is not ready
> now, the next release window is never far away. This helps to ensure the
> overall quality of the release. We've drawn the line for this release based
> on feature progress and code review. For features which can't get in this
> time, don't worry since January will be here soon!
>
> Please let me know if you have any feedback on this plan.
>
> Thanks!
> Jason
>

Re: [VOTE] 0.10.1 Release Plan

[Sriram Subramanian]

+1 (binding)

> On Sep 13, 2016, at 3:03 PM, Guozhang Wang  wrote:
> 
> +1 (binding)
> 
>> On Tue, Sep 13, 2016 at 2:53 PM, Ismael Juma  wrote:
>> 
>> +1 (binding)
>> 
>>> On Tue, Sep 13, 2016 at 10:52 PM, Gwen Shapira  wrote:
>>> 
>>> +1 (binding) - yay for a new release!
>>> 
>>> On Tue, Sep 13, 2016 at 2:40 PM, Jason Gustafson 
>>> wrote:
>>> 
 Hi All,
 
 I'd like to start a vote on the release plan documented here:
 https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.1.
>> I
 went ahead and included KIP-55 since Jun said it may have a chance, but
 note that in-progress KIPs will only be included if they are ready by
>> the
 feature freeze date (Sep. 17). We'll evaluate KIP-79 once we see
>> Becket's
 patch.
 
 Thanks,
 Jason
>>> 
>>> 
>>> 
>>> --
>>> *Gwen Shapira*
>>> Product Manager | Confluent
>>> 650.450.2760 | @gwenshap
>>> Follow us: Twitter  | blog
>>> 
> 
> 
> 
> -- 
> -- Guozhang

Re: [UPDATE] 0.10.1 Release Progress

[Sriram Subramanian]

Thanks Jason!

On Tue, Sep 27, 2016 at 5:38 PM, Ismael Juma  wrote:

> Thanks for the update Jason. :)
>
> Ismael
>
> On Wed, Sep 28, 2016 at 1:28 AM, Jason Gustafson 
> wrote:
>
> > Hi All,
> >
> > Pardon all the JIRA noise, but I've been busy reducing the scope to match
> > the available time since we're now 6 days from the code freeze. I've
> pruned
> > the list to about 30 tickets, some of which probably won't get in:
> > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.1.
> > Other
> > than a few important bug fixes which are nearing completion, the main
> > remaining items are documentation improvements, additional system tests,
> > and transient test failures. If you are looking to help out, addressing
> the
> > transient test failures are the biggest need since I'm sure you've
> noticed
> > all the build failures. Other than that, I think we're on track for the
> > code freeze. Keep up the great work!
> >
> > Thanks,
> > Jason
> >
> > On Tue, Sep 20, 2016 at 3:18 PM, Mayuresh Gharat <
> > gharatmayures...@gmail.com
> > > wrote:
> >
> > > Great !
> > >
> > > Thanks,
> > >
> > > Mayuresh
> > >
> > > On Tue, Sep 20, 2016 at 2:43 PM, Becket Qin 
> > wrote:
> > >
> > > > Awesome!
> > > >
> > > > On Mon, Sep 19, 2016 at 11:42 PM, Neha Narkhede 
> > > wrote:
> > > >
> > > > > Nice!
> > > > > On Mon, Sep 19, 2016 at 11:33 PM Ismael Juma 
> > > wrote:
> > > > >
> > > > > > Well done everyone. :)
> > > > > >
> > > > > > On 20 Sep 2016 5:23 am, "Jason Gustafson" 
> > > wrote:
> > > > > >
> > > > > > > Thanks everyone for the hard work! The 0.10.1 release branch
> has
> > > been
> > > > > > > created. We're now entering the stabilization phase of this
> > release
> > > > > which
> > > > > > > means we'll focus on bug fixes and testing.
> > > > > > >
> > > > > > > -Jason
> > > > > > >
> > > > > > > On Fri, Sep 16, 2016 at 5:00 PM, Jason Gustafson <
> > > ja...@confluent.io
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > Thanks everyone for the hard work! Here's an update on the
> > > > remaining
> > > > > > KIPs
> > > > > > > > that we are hoping to include:
> > > > > > > >
> > > > > > > > KIP-78 (clusterId): Review is basically complete. Assuming no
> > > > > problems
> > > > > > > > emerge, Ismael is planning to merge today.
> > > > > > > > KIP-74 (max fetch size): Review is nearing completion, just a
> > few
> > > > > minor
> > > > > > > > issues remain. This will probably be merged tomorrow or
> Sunday.
> > > > > > > > KIP-55 (secure quotas): The patch has been rebased and
> probably
> > > > needs
> > > > > > one
> > > > > > > > more review pass before merging. Jun is confident it can get
> in
> > > > > before
> > > > > > > > Monday.
> > > > > > > >
> > > > > > > > As for KIP-79, I've made one review pass, but to make it in,
> > > we'll
> > > > > need
> > > > > > > 1)
> > > > > > > > some more votes on the vote thread, and 2) a few review
> > > iterations.
> > > > > > It's
> > > > > > > > looking a bit doubtful, but let's see how it goes!
> > > > > > > >
> > > > > > > > Since we are nearing the feature freeze, it would be helpful
> if
> > > > > people
> > > > > > > > begin setting some priorities on the bugs that need to get in
> > > > before
> > > > > > the
> > > > > > > > code freeze. I am going to make an effort to prune the list
> > early
> > > > > next
> > > > > > > > week, so if there are any critical issues you know about,
> make
> > > sure
> > > > > > they
> > > > > > > > are marked as such.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Jason
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > --
> > > > > Thanks,
> > > > > Neha
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > -Regards,
> > > Mayuresh R. Gharat
> > > (862) 250-7125
> > >
> >
>

Re: [ANNOUNCE] Apache Kafka 0.10.1.0 Released

[Sriram Subramanian]

Fantastic release and on time! Congratulations everyone for our first time
based release.

On Thu, Oct 20, 2016 at 11:34 AM, Ismael Juma  wrote:

> Thanks everyone for your contributions to this release. I think it's
> impressive how much has been achieved in just under 5 months.
>
> And special thanks to Jason who has seemingly just been appointed RM for
> life? :)
>
> Ismael
>
> On Thu, Oct 20, 2016 at 7:22 PM, Guozhang Wang  wrote:
>
> > Thanks for driving the release Jason!
> >
> > You should just be the release person for all the future releases :P
> >
> >
> > Guozhang
> >
> >
> > On Thu, Oct 20, 2016 at 11:12 AM, Jason Gustafson 
> wrote:
> >
> > > Had the wrong address for dev and users (haven't sent from this account
> > > before).
> > >
> > > On Thu, Oct 20, 2016 at 11:05 AM, Jason Gustafson 
> > wrote:
> > >
> > > > The Apache Kafka community is pleased to announce the release for
> > Apache
> > > > Kafka 0.10.1.0. This is a feature release which includes the
> completion
> > > of
> > > > 15 KIPs, over 200 bug fixes and improvements, and more than 500 pull
> > > > requests merged.
> > > >
> > > > All of the changes in this release can be found in the release notes:
> > > > https://archive.apache.org/dist/kafka/0.10.1.0/RELEASE_NOTES.html
> > > >
> > > > Apache Kafka is high-throughput, publish-subscribe messaging system
> > > > rethought of as a distributed commit log.
> > > >
> > > > ** Fast => A single Kafka broker can handle hundreds of megabytes of
> > > reads
> > > > and writes per second from thousands of clients.
> > > >
> > > > ** Scalable => Kafka is designed to allow a single cluster to serve
> as
> > > the
> > > > central data backbone for a large organization. It can be elastically
> > and
> > > > transparently expanded without downtime. Data streams are partitioned
> > > > and spread over a cluster of machines to allow data streams larger
> than
> > > > the capability of any single machine and to allow clusters of
> > > co-ordinated
> > > > consumers.
> > > >
> > > > ** Durable => Messages are persisted on disk and replicated within
> the
> > > > cluster to prevent data loss. Each broker can handle terabytes of
> > > messages
> > > > without performance impact.
> > > >
> > > > ** Distributed by Design => Kafka has a modern cluster-centric design
> > > that
> > > > offers strong durability and fault-tolerance guarantees.
> > > >
> > > > You can download the source release from
> > > > https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.1.0/k
> > > > afka-0.10.1.0-src.tgz
> > > >
> > > > and binary releases from
> > > > https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.1.0/k
> > > > afka_2.11-0.10.1.0.tgz
> > > > https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.1.0/k
> > > > afka_2.10-0.10.1.0.tgz
> > > >
> > > > Thanks to the 115 contributors on this release!
> > > >
> > > > Alex Glikson, Alex Loddengaard, Alexey Ozeritsky, Alexey Romanchuk,
> > > Andrea
> > > > Cosentino, Andrew Otto, Andrey Neporada, Apurva Mehta, Arun
> Mahadevan,
> > > > Ashish Singh, Avi Flax, Ben Stopford, Bharat Viswanadham, Bill
> Bejeck,
> > > > Bryan Baugher, Chen Zhu, Christian Posta, Damian Guy, Dan Norwood,
> Dana
> > > > Powers, David Chen, Derrick Or, Dong Lin, Dustin Cote, Edoardo Comar,
> > > Elias
> > > > Levy, Eno Thereska, Eric Wasserman, Ewen Cheslack-Postava, Filipe
> > > Azevedo,
> > > > Flavio Junqueira, Florian Hussonnois, Geoff Anderson, Grant Henke,
> Greg
> > > > Fodor, Guozhang Wang, Gwen Shapira, Hans Deragon, Henry Cai, Ishita
> > > > Mandhan, Ismael Juma, Jaikiran Pai, Jakub Dziworski, Jakub Pilimon,
> > James
> > > > Cheng, Jan Filipiak, Jason Gustafson, Jay Kreps, Jeff Klukas, Jendrik
> > > > Poloczek, Jeyhun Karimov, Jiangjie Qin, Johnny Lim, Jonathan Bond,
> Jun
> > > Rao,
> > > > Kaufman Ng, Kenji Yoshida, Konstantine Karantasis, Kota Uchida,
> Laurier
> > > > Mantel, Liquan Pei, Luke Zaparaniuk, Magnus Reftel, Manikumar Reddy
> O,
> > > Manu
> > > > Zhang, Mark Grover, Mathieu Fenniak, Matthias J. Sax, Maysam
> Yabandeh,
> > > > Mayuresh Gharat, Michael G. Noll, Mickael Maison, Moritz Siuts, Nafer
> > > > Sanabria, Nihed Bbarek, Onur Karaman, P. Thorpe, Peter Davis,
> Philippe
> > > > Derome, Pierre Coquentin, Rajini Sivaram, Randall Hauch, Rekha Joshi,
> > > Roger
> > > > Hoover, Rollulus, Ryan Pridgeon, Sahil Kharb, Samuel Taylor, Sasaki
> > Toru,
> > > > Satendra Kumar, Sebastien Launay, Shikhar Bhushan, Shuai Zhang, Som
> > Sahu,
> > > > Sriharsha Chintalapani, Sumit Arrawatia, Tao Xiao, Thanasis Katsadas,
> > Tim
> > > > Brooks, Todd Palino, Tom Crayford, Tom Rybak, Vahid Hashemian, Wan
> > Wenli,
> > > > William Thurston, William Yu, Xavier Léauté, Yang Wei, Yeva Byzek,
> > Yukun
> > > > Guo, Yuto Kawamura, Zack Dever, 1ambda, leisore, sven0726
> > > >
> > > > We welcome your help and feedback. For more information on how to
> > > > report problems, and to get involved, visit the project website at
> > > > http://kafka.apache.org/
> > > >
> > > > Thanks,
> > > 

Re: [DISCUSS] KIP-80: Kafka REST Server

[Sriram Subramanian]

FWIW, Apache Kafka has evolved a lot from where it started. It did start as
a messaging system. Over time we realized that that the vision for Kafka is
to build a streaming platform and not just a messaging system. You can take
a look at the site for more description about what comprises the streaming
platform http://kafka.apache.org/ and http://kafka.apache.org/intro.

Can the streaming platform exist without Connect? - No. Data integration is
fundamental to building an end to end platform

Can the streaming platform exist without stream processing? - No.
Processing stream data again is a core part of streaming platform.

Can the streaming platform exist without clients? - We at least need one
client library to complete the platform. Our Java clients help us to
complete the platform story. The rest of the clients are built and
maintained outside the project.

Can the platform exist without the rest proxy? - Yes. The proxy does not
complete the platform vision in anyway. It is just a good to have tool that
might be required by quite a few users and there is an active project that
works on this - https://github.com/confluentinc/kafka-rest




On Fri, Oct 21, 2016 at 11:49 AM, Nacho Solis 
wrote:

> Are you saying Kafka REST is subjective but Kafka Streams and Kafka Connect
> are not subjective?
>
> > "there are likely places that can live without a rest proxy"
>
> There are also places that can live without Kafka Streams and Kafka
> Connect.
>
> Nacho
>
> On Fri, Oct 21, 2016 at 11:17 AM, Jun Rao  wrote:
>
> > At the high level, I think ideally it makes sense to add a component to
> > Apache Kafka if (1) it's widely needed and (2) it needs tight integration
> > with Kafka core. For Kafka Stream, we do expect stream processing will be
> > used widely in the future. Implementation wise, Kafka Stream only
> supports
> > getting data from Kafka and leverages quite a few of the core
> > functionalities in Kafka core. For example, it uses customized rebalance
> > callback in the consumer and uses the compacted topic heavily. So, having
> > Kafka Stream in the same repo makes it easier for testing when those core
> > functionalities evolve over time. Kafka Connect is in the same situation.
> >
> > For rest proxy, whether it's widely used or not is going to be a bit
> > subjective. However, there are likely places that can live without a rest
> > proxy. The rest proxy is just a proxy for the regular clients and doesn't
> > need to be tightly integrated with Kafka core. So, the case for including
> > rest proxy in Apache Kafka is probably not as strong as Kafka Stream and
> > Kafka Connect.
> >
> > Thanks,
> >
> > Jun
> >
> > On Thu, Oct 20, 2016 at 11:28 PM, Michael Pearce 
> > wrote:
> >
> > > So from my reading essentially the first question needs to answered/and
> > > voted on is:
> > >
> > > Is Apache Kafka Community only about the Core or does the apache
> > community
> > > also support some subprojects (and just we need some better way to
> manage
> > > this)
> > >
> > > If vote for Core only wins, then the following should be removed:
> > > Kafka Connect
> > > Kafka Stream
> > >
> > > If vote for Core only loses (aka we will support subprojects) then:
> > > We should look to add Kafka Rest
> > >
> > > And we should look to see how we can manage better govern and manage
> > > submodules.
> > >
> > > A good example which id propose here is how some other communities in
> > > Apache do this.
> > >
> > > Each Module has a Module Management Committee(MMC), this is like almost
> > > the PMC but at a per module basis.
> > >
> > > This MMC should essentially hold the binding votes for that module.
> > > The MMC should be made up of a single representative from each
> > > organisation  (so no single organisation can fully veto the community
> it
> > > has to a genuine consenus)
> > > The MMC requires at least 3 members (so there cant be a tied vote on 2)
> > > For a new Module to be added a MMC committee should be sought
> > > A new Module is only capable of being added if the above requirements
> can
> > > be met (e.g. 3 people wishing to step up, from 3 organisations) so that
> > > only actively support modules would be added
> > >
> > > The PMC reviews each module every 6months or Year. If MMC is inactive,
> a
> > > vote/call to find replacements if raised, if none are forthcoming
> > dropping
> > > the MMC to less than 3 then the module moves to "the attic" (very much
> > like
> > > apache attic but a little more aggressively)
> > >
> > > This way the PMC does not need to micro manage every module
> > > We only add modules where some amount of active support and maintenance
> > > and use is provided by the community
> > > We have an automatic way to retire old or inactive projects.
> > >
> > > Thoughts?
> > > Mike
> > >
> > >
> > > 
> > > From: Harsha Ch 
> > > Sent: Thursday, October 20, 2016 10:26 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP-80: Kafk

Re: [VOTE] Add REST Server to Apache Kafka

[Sriram Subramanian]

-1 for all the reasons that have been described before. This does not need
to be part of the core project.

On Tue, Oct 25, 2016 at 3:25 PM, Suresh Srinivas 
wrote:

> +1.
>
> This is an http access to core Kafka. This is very much needed as part of
> Apache Kafka under ASF governance model.  This would be great for the
> community instead of duplicated and splintered efforts that may spring up.
>
> Get Outlook for iOS
>
> _
> From: Harsha Chintalapani mailto:ka...@harsha.io>>
> Sent: Tuesday, October 25, 2016 2:20 PM
> Subject: [VOTE] Add REST Server to Apache Kafka
> To: mailto:dev@kafka.apache.org>>, <
> us...@kafka.apache.org>
>
>
> Hi All,
>We are proposing to have a REST Server as part of  Apache Kafka
> to provide producer/consumer/admin APIs. We Strongly believe having
> REST server functionality with Apache Kafka will help a lot of users.
> Here is the KIP that Mani Kumar wrote
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 80:+Kafka+Rest+Server.
> There is a discussion thread in dev list that had differing opinions on
> whether to include REST server in Apache Kafka or not. You can read more
> about that in this thread
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201610.mbox/%3CCAMVt_
> aymqeudm39znsxgktpdde46sowmqhsxop-+jmbcuv7...@mail.gmail.com%3E
>
>   This is a VOTE thread to check interest in the community for
> adding REST Server implementation in Apache Kafka.
>
> Thanks,
> Harsha
>
>
>

Re: [ANNOUNCE] New committer: Jiangjie (Becket) Qin

[Sriram Subramanian]

Congratulations!

On Mon, Oct 31, 2016 at 12:23 PM, Ismael Juma  wrote:

> Congratulations Becket. :)
>
> Ismael
>
> On 31 Oct 2016 1:44 pm, "Joel Koshy"  wrote:
>
> > The PMC for Apache Kafka has invited Jiangjie (Becket) Qin to join as a
> > committer and we are pleased to announce that he has accepted!
> >
> > Becket has made significant contributions to Kafka over the last two
> years.
> > He has been deeply involved in a broad range of KIP discussions and has
> > contributed several major features to the project. He recently completed
> > the implementation of a series of improvements (KIP-31, KIP-32, KIP-33)
> to
> > Kafka’s message format that address a number of long-standing issues such
> > as avoiding server-side re-compression, better accuracy for time-based
> log
> > retention, log roll and time-based indexing of messages.
> >
> > Congratulations Becket! Thank you for your many contributions. We are
> > excited to have you on board as a committer and look forward to your
> > continued participation!
> >
> > Joel
> >
>

Re: [VOTE] KIP-143: Controller Health Metrics

[Sriram Subramanian]

+1

On Fri, May 5, 2017 at 6:46 AM, Tom Crayford  wrote:

> +1 (non-binding)
>
> On Fri, 5 May 2017 at 00:28, Michael André Pearce <
> michael.andre.pea...@me.com> wrote:
>
> > +1 , really good to see better operational visibility being added to the
> > broker.
> >
> > Sent from my iPhone
> >
> > > On 5 May 2017, at 03:34, Ismael Juma  wrote:
> > >
> > > Hi everyone,
> > >
> > > It seems like the discussion has converged, so I would like to initiate
> > the
> > > voting process for KIP-143: Controller Health Metrics:
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-143
> > > %3A+Controller+Health+Metrics
> > >
> > > The vote will run for a minimum of 72 hours.
> > >
> > > Thanks,
> > > Ismael
> >
>

Re: [VOTE] KIP-144: Exponential backoff for broker reconnect attempts

[Sriram Subramanian]

+1

On Fri, May 5, 2017 at 6:04 PM, Gwen Shapira  wrote:

> +1
>
> On Fri, May 5, 2017 at 3:32 PM, Ismael Juma  wrote:
>
> > Hi all,
> >
> > Given the simple and non controversial nature of the KIP, I would like to
> > start the voting process for KIP-144: Exponential backoff for broker
> > reconnect attempts:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 144%3A+Exponential+
> > backoff+for+broker+reconnect+attempts
> >
> > The vote will run for a minimum of 72 hours.
> >
> > Thanks,
> > Ismael
> >
>
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter  | blog
> 
>

Re: [VOTE] KIP-153 (separating replication traffic from BytesOutPerSec metric)

[Sriram Subramanian]

+1

> On May 7, 2017, at 7:40 PM, Jun Rao  wrote:
> 
> Hi, Everyone,
> 
> Since this is a relatively simple change, I would like to start the voting
> process for KIP-153 : Include only client traffic in BytesOutPerSec metric.
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-153+%
> 3A+Include+only+client+traffic+in+BytesOutPerSec+metric
> 
> The vote will run for a minimum of 72 hours.
> 
> Thanks,
> 
> Jun

Re: [VOTE] KIP-133: List and Alter Configs Admin APIs (second attempt)

[Sriram Subramanian]

+1

> On May 7, 2017, at 9:01 PM, Ismael Juma  wrote:
> 
> [Seems like the original message ended up in the discuss thread in GMail,
> so trying again]
> 
> Hi everyone,
> 
> I believe I addressed the comments in the discussion thread and given the
> impending KIP freeze, I would like to start the voting process for KIP-133:
> List and Alter Configs Admin APIs:
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-133%
> 3A+List+and+Alter+Configs+Admin+APIs
> 
> As mentioned previously, this KIP and KIP-140 (Add administrative RPCs for
> adding, deleting, and listing ACLs) complete the AdminClient work that was
> originally proposed as part KIP-4.
> 
> If you have additional feedback, please share it in the discuss thread.
> 
> The vote will run for a minimum of 72 hours.
> 
> Thanks,
> Ismael

Re: [VOTE] KIP-154 Add Kafka Connect configuration properties for creating internal topics

[Sriram Subramanian]

+1

On Mon, May 8, 2017 at 2:14 PM, Konstantine Karantasis <
konstant...@confluent.io> wrote:

> +1 (non binding)
>
> On Mon, May 8, 2017 at 1:33 PM, Stephane Maarek <
> steph...@simplemachines.com.au> wrote:
>
> > +1 (non binding)
> >
> >
> >
> > On 9/5/17, 5:51 am, "Randall Hauch"  wrote:
> >
> > Hi, everyone.
> >
> > Given the simple and non-controversial nature of the KIP, I would
> like
> > to
> > start the voting process for KIP-154: Add Kafka Connect configuration
> > properties for creating internal topics:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 154+Add+Kafka+Connect+configuration+properties+for+
> > creating+internal+topics
> >
> > The vote will run for a minimum of 72 hours.
> >
> > Thanks,
> >
> > Randall
> >
> >
> >
> >
>

Re: [VOTE] KIP-146: Isolation of dependencies and classes in Kafka Connect (restarted voting thread)

[Sriram Subramanian]

+1

On Tue, May 9, 2017 at 9:43 PM, dan  wrote:

> +1 (non binding)
>
> thanks Konstantine :thumbsup:
>
> On Mon, May 8, 2017 at 3:24 PM, Guozhang Wang  wrote:
>
> > +1
> >
> > On Mon, May 8, 2017 at 1:36 PM, Stephane Maarek <
> > steph...@simplemachines.com.au> wrote:
> >
> > > +1
> > > Thanks heaps I can’t wait!
> > >
> > >
> > > On 9/5/17, 4:48 am, "Konstantine Karantasis"  >
> > > wrote:
> > >
> > > ** Restarting the voting thread here, with a different title to
> avoid
> > > collapsing this thread's messages with the discussion thread's
> > > messages in
> > > mail clients. Apologies for the inconvenience. **
> > >
> > >
> > > Hi all,
> > >
> > > Given that the comments during the discussion seem to have been
> > > addressed,
> > > I'm pleased to bring
> > >
> > > KIP-146: Classloading Isolation in Connect
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > 146+-+Classloading+Isolation+in+Connect
> > >
> > > up for voting. Again, this KIP aims to bring the highly desired
> > > feature of
> > > dependency isolation in Kafka Connect.
> > >
> > > In the meantime, for any additional feedback, please continue to
> send
> > > your
> > > comments in the discussion thread here:
> > >
> > > https://www.mail-archive.com/dev@kafka.apache.org/msg71453.html
> > >
> > > This voting thread will stay active for a minimum of 72 hours.
> > >
> > > Sincerely,
> > > Konstantine
> > >
> > >
> > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>

Re: [VOTE] KIP-156 Add option "dry run" to Streams application reset tool

[Sriram Subramanian]

+1

On Wed, May 10, 2017 at 9:45 AM, Neha Narkhede  wrote:

> +1
>
> On Wed, May 10, 2017 at 12:32 PM Gwen Shapira  wrote:
>
> > +1. Also not sure that adding a parameter to a CLI requires a KIP. It
> seems
> > excessive.
> >
> >
> > On Tue, May 9, 2017 at 7:57 PM Jay Kreps  wrote:
> >
> > > +1
> > > On Tue, May 9, 2017 at 3:41 PM BigData dev 
> > > wrote:
> > >
> > > > Hi, Everyone,
> > > >
> > > > Since this is a relatively simple change, I would like to start the
> > > voting
> > > > process for KIP-156: Add option "dry run" to Streams application
> reset
> > > tool
> > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/pages/viewpage.
> action?pageId=69410150
> > > >
> > > >
> > > > The vote will run for a minimum of 72 hours.
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Bharat
> > > >
> > >
> >
> --
> Thanks,
> Neha
>

Re: [VOTE] KIP-151: Expose Connector type in REST API (first attempt :)

[Sriram Subramanian]

+1

On Wed, May 10, 2017 at 10:20 AM, Ewen Cheslack-Postava 
wrote:

> +1 binding
>
> Thanks,
> Ewen
>
> On Mon, May 8, 2017 at 4:54 PM, BigData dev 
> wrote:
>
> > +1 (non-binding)
> >
> > Thanks,
> > Bharat
> >
> >
> > On Mon, May 8, 2017 at 4:39 PM, Konstantine Karantasis <
> > konstant...@confluent.io> wrote:
> >
> > > +1 (non-binding)
> > >
> > > On Mon, May 8, 2017 at 3:39 PM, dan  wrote:
> > >
> > > > i'd like to begin voting on
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > > 151+Expose+Connector+type+in+REST+API
> > > >
> > > > discussion should remain on
> > > > http://mail-archives.apache.org/mod_mbox/kafka-dev/201705.
> > > > mbox/%3CCAFJy-U-pF7YxSRadx_zAQYCX2+SswmVPSBcA4tDMPP5834s6Kg@mail.
> > > > gmail.com%3E
> > > >
> > > > This voting thread will stay active for a minimum of 72 hours.
> > > >
> > > > thanks
> > > > dan
> > > >
> > >
> >
>

Re: [VOTE] KIP-155 Add range scan for windowed state stores

[Sriram Subramanian]

+1

On Wed, May 10, 2017 at 11:42 AM, Bill Bejeck  wrote:

> +1
>
> Thanks,
> Bill
>
> On Wed, May 10, 2017 at 2:38 PM, Guozhang Wang  wrote:
>
> > +1. Thank you!
> >
> > On Wed, May 10, 2017 at 11:30 AM, Xavier Léauté 
> > wrote:
> >
> > > Hi everyone,
> > >
> > > Since there aren't any objections to this addition, I would like to
> start
> > > the voting on KIP-155 so we can hopefully get this into 0.11.
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP+
> > > 155+-+Add+range+scan+for+windowed+state+stores
> > >
> > > Voting will stay active for at least 72 hours.
> > >
> > > Thank you,
> > > Xavier
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>