Hey Harsha, I'm trying to understand your specific concern. Is it the fact that the client cannot tell the difference between non-existing topic and hence needlessly retries (instead of perhaps raising an error). Or is it specifically that in some cases, the consumer will block because of this? In other words, would you be satisfied with retrying internally for non-existing topics as long as it doesn't prevent the consumer from making progress on other assigned partitions?
Thanks, Jason On Thu, Sep 1, 2016 at 11:28 AM, Harsha Chintalapani <ka...@harsha.io> wrote: > I would like to see we address > https://issues.apache.org/jira/browse/KAFKA-1894 . This is problematic in > secure cluster when the users try to access the topics that they don't have > ACLs turned on. > > -Harsha > > On Tue, Aug 30, 2016 at 9:40 PM Jaikiran Pai <jai.forums2...@gmail.com> > wrote: > > > We have been using the (new) Java consumer API in 0.9.0.1 for a while > > now. We have some well known issues with it - like heart beats being > > part of the same thread causing the consumer to sometimes be considered > > dead. I understand that this has been fixed in 0.10.0.1 but we haven't > > yet had a chance to migrate to it. We plan to do that in the next month > > or so. > > > > Personally, I would be OK if the beta label is removed from it if the > > dev team is sure the API isn't going to change. I don't know if that's > > true or not post 0.10.0.1. For me the major thing that I think needs to > > be addressed is these JIRAs which actually expose some API > > implementation level issues. Not sure if solving those issues will > > involve changes to API itself: > > > > https://issues.apache.org/jira/browse/KAFKA-1894 > > https://issues.apache.org/jira/browse/KAFKA-3540 > > https://issues.apache.org/jira/browse/KAFKA-3539 > > > > If solving issues like these will not involve changes to the API, I > > think it's safe to move it out of beta label. > > > > -Jaikiran > > > > On Tuesday 30 August 2016 05:09 PM, Ismael Juma wrote: > > > Thanks for the feedback everyone. Since Harsha said that he is OK > either > > > way and everyone else is in favour, I think we should go ahead with > this. > > > Since we committed to API stability for the new Java consumer in > 0.10.0.0 > > > via KIP-45, this is simply a documentation change and I don't think we > > need > > > an official vote thread (we didn't have one for the equivalent producer > > > change). > > > > > > Ismael > > > > > > On Mon, Aug 29, 2016 at 7:37 PM, Jay Kreps <j...@confluent.io> wrote: > > > > > >> +1 I talk to a lot of kafka users, and I would say > 75% of people > doing > > >> new things are on the new consumer despite our warnings :-) > > >> > > >> -Jay > > >> > > >> On Thu, Aug 25, 2016 at 2:05 PM, Jason Gustafson <ja...@confluent.io> > > >> wrote: > > >> > > >>> I'm +1 also. I feel a lot more confident about this with all of the > > >> system > > >>> testing we now have in place (including the tests covering Streams > and > > >>> Connect). > > >>> > > >>> -Jason > > >>> > > >>> On Thu, Aug 25, 2016 at 9:57 AM, Gwen Shapira <g...@confluent.io> > > wrote: > > >>> > > >>>> Makes sense :) > > >>>> > > >>>> On Thu, Aug 25, 2016 at 9:40 AM, Neha Narkhede <n...@confluent.io> > > >>> wrote: > > >>>>> Yeah, I'm supportive of this. > > >>>>> > > >>>>> On Thu, Aug 25, 2016 at 9:26 AM Ismael Juma <ism...@juma.me.uk> > > >> wrote: > > >>>>>> Hi Gwen, > > >>>>>> > > >>>>>> We have a few recent stories of people using Connect and Streams > in > > >>>>>> production. That means the new Java Consumer too. :) > > >>>>>> > > >>>>>> Ismael > > >>>>>> > > >>>>>> On Thu, Aug 25, 2016 at 5:09 PM, Gwen Shapira <g...@confluent.io> > > >>>> wrote: > > >>>>>>> Originally, we suggested keeping the beta label until we know > > >>> someone > > >>>>>>> successfully uses the new consumer in production. > > >>>>>>> > > >>>>>>> We can consider the recent KIPs enough, but IMO it will be better > > >> if > > >>>>>>> someone with production deployment hanging out on our mailing > list > > >>>>>>> will confirm good experience with the new consumer. > > >>>>>>> > > >>>>>>> Gwen > > >>>>>>> > > >>>>>>> On Wed, Aug 24, 2016 at 8:45 PM, Ismael Juma <ism...@juma.me.uk> > > >>>> wrote: > > >>>>>>>> Hi all, > > >>>>>>>> > > >>>>>>>> We currently say the following in our documentation: > > >>>>>>>> > > >>>>>>>> "As of the 0.9.0 release we have added a new Java consumer to > > >>>> replace > > >>>>>> our > > >>>>>>>> existing high-level ZooKeeper-based consumer and low-level > > >>> consumer > > >>>>>> APIs. > > >>>>>>>> This client is considered beta quality."[1] > > >>>>>>>> > > >>>>>>>> Since then, Jason and the community have done a lot of work to > > >>>> improve > > >>>>>> it > > >>>>>>>> (including KIP-41 and KIP-62), we declared it API stable in > > >>> 0.10.0.0 > > >>>>>> and > > >>>>>>>> it's the only option for those that need security support. Yes, > > >> it > > >>>>>> still > > >>>>>>>> has bugs, but so does the old consumer and all development is > > >>>> currently > > >>>>>>>> focused on the new consumer. > > >>>>>>>> > > >>>>>>>> As such, I propose we remove the beta label for the next release > > >>> and > > >>>>>>> switch > > >>>>>>>> our tools to use the new consumer by default unless the > > >> zookeeper > > >>>>>>>> command-line option is present (for compatibility). This is > > >>> similar > > >>>> to > > >>>>>>> what > > >>>>>>>> we did it for the new producer in 0.9.0.0, but backwards > > >>> compatible. > > >>>>>>>> Thoughts? > > >>>>>>>> > > >>>>>>>> Ismael > > >>>>>>>> > > >>>>>>>> [1] http://kafka.apache.org/documentation.html#consumerapi > > >>>>>>> > > >>>>>>> > > >>>>>>> -- > > >>>>>>> Gwen Shapira > > >>>>>>> Product Manager | Confluent > > >>>>>>> 650.450.2760 | @gwenshap > > >>>>>>> Follow us: Twitter | blog > > >>>>>>> > > >>>>> -- > > >>>>> Thanks, > > >>>>> Neha > > >>>> > > >>>> > > >>>> -- > > >>>> Gwen Shapira > > >>>> Product Manager | Confluent > > >>>> 650.450.2760 | @gwenshap > > >>>> Follow us: Twitter | blog > > >>>> > > > > >
