Hyun, Drausin Wulsin, Duncan Sands, Dustin Cote,
> Eamon Zhang, edoardo, Edward Ribeiro, Eno Thereska, Ewen
> Cheslack-Postava, Flavio Junqueira, Francois Visconte, Frank Scholten,
> Gabriel Zhang, gaob13, Geoff Anderson, glikson, Grant Henke, Greg
> Fodor, Guozhang Wang, Gwen Shapira, Igo
have a "max" lag?
> >> And
> >>> is
> >>>>> this "lag" a bad thing?
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Jiangjie (Becket) Qin
> >>>>>
> >>&
wiki.apache.org/confluence/display/KAFKA/KIP-62%3A+Allow+consumer+to+send+heartbeats+from+a+background+thread
> .
> Have a look and let me know what you think.
>
> Thanks,
> Jason
>
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
Response (Version: 0) => [topic_error_codes]
> topic_error_codes => topic error_code
> topic => STRING
> error_code => INT16
>
> CreateTopicResponse contains a map between topic and topic creation
> result error code (see New Protocol Errors
> <https://cwi
ch would
> maintain its own connection pool etc, would be used internally by the
> producer (and potentially consumer) or if they would just reuse the request
> objects.
>
> Just trying to write this down to sanity check that it will work.
>
> -Jay
>
> On Fri, Jun 10, 2016
"ack" from replicas, we will be able to add the new behavior without
> changing the protocol (just the semantics of waiting).
>
> Gwen
>
> On Fri, Jun 10, 2016 at 7:21 PM, Grant Henke wrote:
> > Now that Kafka 0.10 has been released I would like to start work o
the first implementation, it really means
> that the topic metadata is propagated to the controller's metadata cache.
>
> Jun
>
> On Fri, Jun 10, 2016 at 9:21 AM, Grant Henke wrote:
>
> > Now that Kafka 0.10 has been released I would like to start work on the
>
s, and different error codes are returned.
>
> Guozhang
>
>
>
> On Mon, Jun 13, 2016 at 6:54 PM, Grant Henke wrote:
>
> > Thanks for the review Jun.
> >
> > You probably want to make it clearer if timeout > 0, what waiting for
> topic
> > > meta
; Should it be `CreateTopicsRequest` then?
>
Sure, I will update that in the patch and wiki.
P.S. I fixed a couple of typos I spotted on the wiki page, I hope that's OK.
>
Absolutely. Feel free to improve the wiki anytime.
Thanks,
Grant
On Tue, Jun 14, 2016 at 3:09 PM, Ismael
e in the protocol? In the code, negative timeouts are typically
> disallowed or they mean an infinite timeout (we have moved from the latter
> to the former in some of the Java networking code in recent releases).
> Ismael
>
> On Tue, Jun 14, 2016 at 11:51 PM, Grant Henke wrote:
>
&
; > On Wed, Jun 15, 2016 at 6:52 PM, Grant Henke
> wrote:
> >>
> >> The one thing I want to avoid is to many super specific error codes. I
> am
> >> not sure how much of a problem it really is but in the case of wire
> >> protocol errors like multiple inst
nistrativeoperations-request>
>below
>
> Create Topics Response
>
>
>
> CreateTopics Response (Version: 0) => [topic_error_codes]
> topic_error_codes => topic error_code
> topic => STRING
> error_code => INT16
>
> CreateTopicsRes
s may still be using Java 7 by the time Kafka 0.10.1.0 is
> > released.
> > > > > More specifically, we care about the subset who would be able to
> > > upgrade
> > > > to
> > > > > Kafka 0.10.1.0, but would not be able to upgrade the Java version.
> It
> > > > would
> > > > > be great if we could quantify this in some way.
> > > > >
> > > > > What do you think?
> > > > >
> > > > > Ismael
> > > > >
> > > > > [1] https://java.com/en/download/faq/java_7.xml
> > > > > [2]
> > https://blogs.oracle.com/thejavatutorials/entry/jdk_8_is_released
> > > > > [3] http://openjdk.java.net/projects/jdk9/
> > > > > [4] https://github.com/apache/cassandra/blob/trunk/README.asc
> > > > > [5]
> > > https://lucene.apache.org/#highlights-of-this-lucene-release-include
> > > > > [6] http://akka.io/news/2015/09/30/akka-2.4.0-released.html
> > > > > [7] https://issues.apache.org/jira/browse/HADOOP-11858
> > > > > [8] https://webtide.com/jetty-9-3-features/
> > > > > [9] http://markmail.org/message/l7s276y3xkga2eqf
> > > > > [10]
> > > > >
> > > > >
> > > >
> > >
> >
> https://intellij-support.jetbrains.com/hc/en-us/articles/206544879-Selecting-the-JDK-version-the-IDE-will-run-under
> > > > > [11] http://markmail.org/message/l7s276y3xkga2eqf
> > > > >
> > > >
> > >
> >
>
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
t(KAFKA-2945)
<https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-CreateTopicsRequest(KAFKA-2945)>*
A pull request is available implementing the proposed changes here:
https://github.com/apache/kafka/pull/1489
Here is a link to the past discussion on the mailing list:
*http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema
<http://search-hadoop.com/m/uyzND1rfG6v1oixmZ&subj=+DISCUSS+KIP+4+Create+Topic+Schema>*
Thank you,
Grant
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
d+thread
> > > > > > .
> > > > > >
> > > > > > After some discussion on this list, I think we were in agreement
> > that
> > > > > this
> > > > > > change addresses a major part of the problem and we've left the
> > door
> > > > open
> > > > > > for further improvements, such as adding a heartbeat() API or a
> > > > > separately
> > > > > > configured rebalance timeout. Thanks in advance to everyone who
> > helped
> > > > > > review the proposal.
> > > > > >
> > > > > > -Jason
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> >
>
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
te_topic_request, not the entire connection? The
> CreateTopicsResponse returns a map of topics to error codes. You could
> just return the topic that caused the error and an
> InvalidRequestException error code.
>
> -Dana
>
> On Thu, Jun 16, 2016 at 8:37 AM, Grant Henke wrote
r the
> > topics that violated those constraints in the request and handle them
> > accordingly with the right error code w/o disconnecting.
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Jun 17, 2016 at 8:40 AM, Dana Powers
> > wrote:
> &
2016, at 04:15 PM, Guozhang Wang wrote:
> > > +1.
> > >
> > > On Thu, Jun 16, 2016 at 3:47 PM, Ismael Juma
> wrote:
> > >
> > > > +1 (binding)
> > > >
> > > > On Thu, Jun 16, 2016 at 11:50 PM, Grant Henke
> > wrote:
&
operations-request>
>below
>
> Delete Topics Response
>
>
>
> DeleteTopics Response (Version: 0) => [topic_error_codes]
> topic_error_codes => topic error_code
> topic => STRING
> error_code => INT16
>
> DeleteTopicsResponse is si
he topics that did not "complete" in time. This allows the client
to know which topics actions completed and which timed out. I updated the
wiki to explicitly call this out in the response section.
Thanks,
Grant
On Tue, Jun 21, 2016 at 5:44 AM, Ismael Juma wrote:
> Thanks Grant
ume the
> > message was
> > > valid and the topic deletion was triggered.
> >
> > In the timeout <= 0 case, if the client should always ignore and treat
> the timeout
> > error code as "OK", should we just return no error code in this case?
>
gt;
> > > >> On Mon, Jun 20, 2016, at 11:33 AM, Ismael Juma wrote:
> > > >> > +1 (binding)
> > > >> >
> > > >> > On Mon, Jun 20, 2016 at 8:27 PM, Dana Powers <
> dana.pow...@gmail.com
> > >
> > >
ay/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-DeleteTopicsRequest(KAFKA-2946)>*
A sample patch is on github:
https://github.com/granthenke/kafka/tree/delete-wire-new
Note: This branch and patch is based on the CreateTopic request/response
PR. I will open a PR once that patch is complete.
Here is a link to the past discussion on the mailing list:
*http://search-hadoop.com/m/uyzND1fokOBn6uNd2&subj=+DISCUSS+KIP+4+Delete+Topic+Schema
<http://search-hadoop.com/m/uyzND1fokOBn6uNd2&subj=+DISCUSS+KIP+4+Delete+Topic+Schema>*
Thank you,
Grant
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
nningAsBroker".
> >> 2. It's not a useful state.
> >> a. If clients want to use this metric to know whether a broker is
> ready
> >> to receive requests or not, they do not care whether or not the broker
> is
> >> the controller
> >>
://github.com/apache/kafka/pull/1489
On Fri, Jun 24, 2016 at 5:43 PM, Gwen Shapira wrote:
> +1
>
> On Thu, Jun 23, 2016 at 8:32 PM, Grant Henke wrote:
> > I would like to initiate the voting process for the "KIP-4 Delete Topics
> > Schema changes". This is not a vote for all o
Grant
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
; of a transition for people relying on the current behavior.
>
> I kind of feel once you start adding AdminClient methods to the producer
> and consumer it's not really clear where to stop--e.g. if I can create I
> should be able to delete, list, etc.
>
> -Jay
>
n every application.
> >
> > If people like this approach, perhaps we could define a topic spec (if
> all
> > fields besides topic name are empty it use "cluster defaults"). Then the
> > AdminClient would have an idempotent create method that takes a spec
ient` if that's what they need.
> > > > 2. Like Grant, I'm unsure that will generally be used in a positive
> > way.
> > > > However, if this is what we need to be able to deprecate server
> > > auto-topic
> > > > creation,
rrency issues, especially because the Authorizer
> interface
> exposes no details about propagating the ACLs to other nodes.
> - See Request Forwarding
>
> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+op
gt;> - I suggest this be addressed in KIP-50 as well, though it
> has
> > >> some compatibility concerns.
> >
> > Isn't KIP-50 itself one gigantic compatibility concern? I don't see
> > how your suggestions make it any worse...
> >
Hi Parth,
Are you still working on this? If you need any help please don't hesitate
to ask.
Thanks,
Grant
On Thu, Jun 30, 2016 at 4:35 PM, Jun Rao wrote:
> Parth,
>
> Thanks for the reply.
>
> It makes sense to only allow the renewal by users that authenticated using
> *non* delegation token m
>> > >> some compatibility concerns.
> >> >
> >> > Isn't KIP-50 itself one gigantic compatibility concern? I don't see
> >> > how your suggestions make it any worse...
> >> >
> >>
> >> Yes, I also
n the number of messages.
> >
> > This comes after several incidents at Linkedin where a sudden "spike" of
> > large message batches caused an out of memory exception.
> >
> > Thank you,
> >
> >Radai
> >
>
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
(off 0.10.0 branch) is the 0.10.0.1-rc2 tag:
> >> > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> >> f8f56751744ba8e55f90f5c4f3aed8c3459447b2
> >> >
> >> > * Documentation:
> >> > http://kafka.apache.org/0100/documentation.
?
> >
> > The problem w/ being more descriptive is that its possible that
> > it restricts potential use cases if people think that somehow
> > their use case wouldn't fit.
> >
> >> 4. There is no CreateAcls or DeleteAcls (unlike CreateTopics an
in Apache Kafka? Any thoughts on the approach?
Thanks,
Grant
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
the
> consumer
> > group to be stopped before copying the offsets? Some applications may not
> > want to do that.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Tue, Aug 9, 2016 at 10:01 AM, Grant Henke
> wrote:
> >
> >> I had to
creating a new consumer
group instead of joining an existing one.
Note: I just made Jira KAFKA-2216
<https://issues.apache.org/jira/browse/KAFKA-2216>, which is somewhat
related to this discussion.
Thank you,
Grant
--
Grant Henke
Solutions Consultant | Cloudera
ghe...@cloudera.com | twi
the Kafka server when parsing all messages? If so I can open
a Jira.
Thank you,
Grant
--
Grant Henke
Solutions Consultant | Cloudera
ghe...@cloudera.com | twitter.com/ghenke <http://twitter.com/gchenke> |
linkedin.com/in/granthenke
Bumping this message again to get some input before opening a Jira.
On Thu, May 21, 2015 at 11:31 AM, Grant Henke wrote:
> When working on my own implementation of the wire protocol, via a copy and
> paste error, I accidentally sent an OffsetCommit message to the
> ConsumerMetadata
gt; >
> > Thanks,
> > Ismael
> >
>
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> <http://www.confluent.io/blog>
>
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
7;t find anything in JIRA... am I imagining things?
>
> Now that there's API support for creating topics, the version bump to
> 0.11.0 seems like a good time to re-evaluate whether this default should be
> flipped to false.
>
> I'm happy to create a KIP if needed, just
Feel free to create a KIP based on that discussion and drive the jira. I
don't think a KIP exists yet.
On Thu, Mar 2, 2017 at 1:28 PM, Jeff Widman wrote:
> Thanks, that's the ticket I was thinking of.
>
> On Thu, Mar 2, 2017 at 11:19 AM, Grant Henke wrote:
>
> > I
+Add+Reset+Consumer+Group+Offsets+tooling
> >
> >
> > Thanks!
> >
> > Jorge.
> >
>
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> <http://www.confluent.io/blog>
>
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
Colin
> > >
> >
> >
> >
> > --
> > *Gwen Shapira*
> > Product Manager | Confluent
> > 650.450.2760 | @gwenshap
> > Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> > <http://www.confluent.io/blog>
> >
>
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
; codebase
> > with a view to eventually making substantial contributions. Could you
> > please add me as a contributor to the Kafka JIRA so that I can assign
> > myself a newbie ticket?
> >
> > Thanks!
> > Grayson
> > --
> > Grayson Chao
> > Data
;
> ~ Joe Stein
> - - - - - - - - - - - - - - - - -
>
> http://www.stealth.ly
> - - - - - - - - - - - - - - - - -
>
> On Tue, Mar 10, 2015 at 12:54 AM, Grant Henke wrote:
>
> > I am also starting to work with the Kafka codebase with plans to
> contribute
> > more signific
byte[] serialize(String topic, T data);
Thank you,
Grant
--
Grant Henke
Solutions Consultant | Cloudera
ghe...@cloudera.com | 920-980-8979
twitter.com/ghenke <http://twitter.com/gchenke> | linkedin.com/in/granthenke
f passing it in for each append, we probably should
> just pass it in once during the creation RecordAccumulator. Could you file
> a jira to track this?
>
> Thanks,
>
> Jun
>
> On Mon, Mar 23, 2015 at 7:16 PM, Grant Henke wrote:
>
> > I am reading over the new
/SenderTest.java
24274a64885fadd0e9318de2beb232218ddd52cd
Diff: https://reviews.apache.org/r/32440/diff/
Testing
---
Thanks,
Grant Henke
e.org/082/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html
>
> Thanks,
>
> Jun
>
> On Mon, Mar 23, 2015 at 9:23 PM, Grant Henke wrote:
>
> > Thanks for validating that. I was thinking of solving it in the same
> > fashion. Though I was unsure if
l on everything. (Note: I would prefer to put final on everything)
- Grant
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32440/#review77593
-----
--
Grant Henke
Solutions Consultant | Cloudera
ghe...@cloudera.com | 920-980-8979
twitter.com/ghenke <http://twitter.com/gchenke> | linkedin.com/in/granthenke
/apache/kafka/clients/producer/internals/SenderTest.java
24274a64885fadd0e9318de2beb232218ddd52cd
Diff: https://reviews.apache.org/r/32440/diff/
Testing
---
Thanks,
Grant Henke
Log Compaction Point Configurable (blocked - by KIP-33)
> >> - KIP-61: Add a log retention parameter for maximum disk space usage
> >>percentage (dormant)
> >> - KIP-68 Add a consumed log retention before log retention (dormant)
> >
> >
> >
1, 2017 at 2:51 PM, Gwen Shapira wrote:
> >
> > > The PMC for Apache Kafka has invited Grant Henke to join as a
> > > committer and we are pleased to announce that he has accepted!
> > >
> > > Grant contributed 88 patches, 90 code reviews, countless great
c no topics, but supports
> > > topic
> > > >>>> auto-creation as soon as a message is published to the topic
> > > >>>> 2) consumer subscribes using topic string or a regex pattern.
> > > Currently
> > > >> no
> > > >&
a process can't abort them, because the hang is in the kernel.
> It
> > >> is
> > >>>> not fun when threads are stuck in "D state"
> > >>>> http://stackoverflow.com/questions/20423521/process-perminan
> > >>>> tly-stuck-on-d-state
> > >>>> . Even kill -9 cannot abort the thread then. Fortunately, this is
> > >>>> rare.
> > >>>>
> > >>>
> > >>> I agree it is a harder problem and it is rare. We probably don't have
> > to
> > >>> worry about it in this KIP since this issue is orthogonal to whether
> or
> > >> not
> > >>> we use JBOD.
> > >>>
> > >>>
> > >>>>
> > >>>> Another approach we should consider is for Kafka to implement its
> own
> > >>>> storage layer that would stripe across multiple disks. This
> wouldn't
> > >>>> have to be done at the block level, but could be done at the file
> > >> level.
> > >>>> We could use consistent hashing to determine which disks a file
> should
> > >>>> end up on, for example.
> > >>>>
> > >>>
> > >>> Are you suggesting that we should distribute log, or log segment,
> > across
> > >>> disks of brokers? I am not sure if I fully understand this approach.
> My
> > >> gut
> > >>> feel is that this would be a drastic solution that would require
> > >>> non-trivial design. While this may be useful to Kafka, I would prefer
> > not
> > >>> to discuss this in detail in this thread unless you believe it is
> > >> strictly
> > >>> superior to the design in this KIP in terms of solving our use-case.
> > >>>
> > >>>
> > >>>> best,
> > >>>> Colin
> > >>>>
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Dong
> > >>>>>
> > >>>>> On Wed, Jan 25, 2017 at 11:34 AM, Colin McCabe >
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Hi Dong,
> > >>>>>>
> > >>>>>> Thanks for the writeup! It's very interesting.
> > >>>>>>
> > >>>>>> I apologize in advance if this has been discussed somewhere else.
> > >>> But
> > >>>> I
> > >>>>>> am curious if you have considered the solution of running multiple
> > >>>>>> brokers per node. Clearly there is a memory overhead with this
> > >>>> solution
> > >>>>>> because of the fixed cost of starting multiple JVMs. However,
> > >>> running
> > >>>>>> multiple JVMs would help avoid scalability bottlenecks. You could
> > >>>>>> probably push more RPCs per second, for example. A garbage
> > >>> collection
> > >>>>>> in one broker would not affect the others. It would be
> interesting
> > >>> to
> > >>>>>> see this considered in the "alternate designs" design, even if you
> > >>> end
> > >>>>>> up deciding it's not the way to go.
> > >>>>>>
> > >>>>>> best,
> > >>>>>> Colin
> > >>>>>>
> > >>>>>>
> > >>>>>> On Thu, Jan 12, 2017, at 10:46, Dong Lin wrote:
> > >>>>>>> Hi all,
> > >>>>>>>
> > >>>>>>> We created KIP-112: Handle disk failure for JBOD. Please find the
> > >>> KIP
> > >>>>>>> wiki
> > >>>>>>> in the link https://cwiki.apache.org/confl
> > >> uence/display/KAFKA/KIP-
> > >>>>>>> 112%3A+Handle+disk+failure+for+JBOD.
> > >>>>>>>
> > >>>>>>> This KIP is related to KIP-113
> > >>>>>>> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > >>>>>> 113%3A+Support+replicas+movement+between+log+directories>:
> > >>>>>>> Support replicas movement between log directories. They are
> > >> needed
> > >>> in
> > >>>>>>> order
> > >>>>>>> to support JBOD in Kafka. Please help review the KIP. You
> > >> feedback
> > >>> is
> > >>>>>>> appreciated!
> > >>>>>>>
> > >>>>>>> Thanks,
> > >>>>>>> Dong
> > >>>>>>
> > >>>>
> > >>>
> > >>
> >
> >
>
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
, but we know that it won't happen before
> > June 2017.
> >
> > Please take a look at the proposal and share your feedback.
> >
> > Thanks,
> > Ismael
> >
> > [1] http://search-hadoop.com/m/Kafka/uyzND1oIhV61GS5Sf2
> >
>
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
/confluence/display/KAFKA/KIP-
> 119%3A+Drop+Support+for+Scala+2.10+in+Kafka+0.11
>
> Please take a look. Your feedback is appreciated.
>
> Thanks,
> Ismael
>
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
han based
> on
> > > who
> > > >> produced the data, and so it makes sense to me that the durability
> > > should
> > > >> be on the entire topic, not by the producer.
> > > >>
> > > >> What is a use case where you have multiple producers writing to the
> > same
> > > >> topic but would want different durability?
> > > >>
> > > >> -James
> > > >>
> > > >>> The ability to make this tradeoff in different places can seem more
> > > >> complex
> > > >>> (and really by definition *is* more complex), but it also offers
> more
> > > >>> flexibility.
> > > >>>
> > > >>> -Ewen
> > > >>>
> > > >>>
> > > >>>> But I understand your point, min.insync.replicas setting should be
> > > >>>> understood as "if a producer wants to get an error when topics are
> > > under
> > > >>>> replicated, then how many replicas are enough for not raising an
> > > error?"
> > > >>>>
> > > >>>>
> > > >>>> On Thu, Jan 26, 2017 at 4:16 PM, Ewen Cheslack-Postava <
> > > >> e...@confluent.io>
> > > >>>> wrote:
> > > >>>>
> > > >>>>> The acks setting for the producer doesn't affect the final
> > durability
> > > >>>>> guarantees. These are still enforced by the replication and min
> ISR
> > > >>>>> settings. Instead, the ack setting just lets the producer control
> > how
> > > >>>>> durable the write is before *that producer* can consider the
> write
> > > >>>>> "complete", i.e. before it gets an ack.
> > > >>>>>
> > > >>>>> -Ewen
> > > >>>>>
> > > >>>>> On Tue, Jan 24, 2017 at 12:46 PM, Luciano Afranllie <
> > > >>>>> listas.luaf...@gmail.com> wrote:
> > > >>>>>
> > > >>>>>> Hi everybody
> > > >>>>>>
> > > >>>>>> I am trying to understand why Kafka let each individual
> producer,
> > > on a
> > > >>>>>> connection per connection basis, choose the tradeoff between
> > > >>>> availability
> > > >>>>>> and durability, honoring min.insync.replicas value only if
> > producer
> > > >>>> uses
> > > >>>>>> ack=all.
> > > >>>>>>
> > > >>>>>> I mean, for a single topic, cluster administrators can't enforce
> > > >>>> messages
> > > >>>>>> to be stores in a minimum number of replicas without
> coordinating
> > > with
> > > >>>>> all
> > > >>>>>> producers to that topic so all of them use ack=all.
> > > >>>>>>
> > > >>>>>> Is there something that I am missing? Is there any other
> strategy
> > to
> > > >>>>>> overcome this situation?
> > > >>>>>>
> > > >>>>>> Regards
> > > >>>>>> Luciano
> > > >>>>>>
> > > >>>>>
> > > >>>>
> > > >>
> > > >>
> > >
> > >
> >
>
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
ent.
> >
> > Thanks for the comments, Becket!
> >
> > I agree that topic configuration change should be in the administrative
> > client. I have not thought about partition movement or preferred leader
> > election. It probably makes sense to put them in the cli
t;> good reason to go multi-process or consider storing more things off
> >> the
> >>>>>> Java heap.
> >>>>>>
> >>>>>
> >>>>> I see. Now I agree one-broker-per-disk should be more efficient in
> >> te
gt; change
> > it
> > >> to
> > >> > SCRAM?
> > >> >
> > >> >
> > >> Updated the diagram.
> > >>
> > >>
> > >>
> > >> Thanks,
> > >> Manikumar
> > >>
> > >>
> > >&
.11
> >
> >
> > The vote will run for a minimum of 72 hours.
> >
> > Thanks,
> > Ismael
> >
> >
> >
> > Unless stated otherwise above:
> > IBM United Kingdom Limited - Registered in England and Wales with number
> > 741598.
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
> 3AU
>
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
t a better way of managing docs (although
> this may need adjustment once we choose a different format for docs).
> 3. Gets the code checked in so any requests for refinements around protocol
> docs can often be resolved with a patch to the code for autogenerating
> them.
>
> -Ewen
FYI. I have updated the pull request here:
https://github.com/apache/kafka/pull/970
On Wed, Mar 9, 2016 at 10:18 PM, Guozhang Wang wrote:
> Hey Ewen,
>
> For "its own page" which location are you specifically thinking about?
>
> Guozhang
>
> On Wed, Mar 9, 2016
them in this thread.
For reference the last KIP discussion can be viewed here:
https://youtu.be/rFW0-zJqg5I?t=12m30s
Thank you,
Grant
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
ussions on
> synchronous vs asynchronous server support. The wiki contains my current
> perspective on the challenges and approach.
>
> If you have any thoughts or feedback on the "Server-side Admin Request
> handlers" section here
> <https://cwiki.apache.org/conflue
; >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version#KIP-35-Retrievingprotocolversion-SummaryofthechangesproposedaspartofthisKIP
> > >
> > > is a brief summary of the KIP.
> > >
> > > The vote will run for 72 hours.
> &
t; options (such as method overloading) could be brought up again in the
> > future if we find it's needed.
> >
> > Thanks,
> > Jason
> >
>
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
IP+4+Wiki+Update
Thank you,
Grant
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
gt; >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-51+-+List+Connectors+REST+API
> > > >
> > > > I'll obviously kick things off with a +1.
> > > >
> > > > -Ewen
> > > >
> > >
> >
> >
> >
> > --
> > Liquan Pei
> > Department of Physics
> > University of Massachusetts Amherst
> >
>
>
>
> --
> Thanks,
> Neha
>
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
community is following
> and will help anyone who builds against branches.
>
> As an awkward first step, I'm moving the 0.10.0.0 branch back to
> 0.10.0.0-SNAPSHOT.
>
> Sorry for the awkwardness, but this will improve things going forward.
>
> Gwen
>
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
s
> candidates), but since SNAPSHOT has well defined semantics, I think it
> makes sense to keep it.
>
> Gwen
>
>
>
> On Wed, Mar 23, 2016 at 1:01 PM, Grant Henke wrote:
>
> > Hi Gwen,
> >
> > Is the new release process already decided? If so you can ign
Anyone have any feedback on these protocol schemas or patches? I may call a
vote in the next day or two on these to help keep moving KIP-4 forward.
Thanks,
Grant
On Mon, Mar 21, 2016 at 10:29 PM, Grant Henke wrote:
> I have 2 patches available for KIP-4 that implement some of the new
> m
n Tue, Mar 15, 2016 at 9:49 AM, Grant Henke wrote:
> Moving the relevant wiki text here for discussion/tracking:
>>
>> Server-side Admin Request handlers
>>
>> At the highest level, admin requests will be handled on the brokers the
>> same way that all message type
I didn't get anyone in attendance for this meeting. If you would like to
discuss it please let me know.
Thank you,
Grant
On Mon, Mar 28, 2016 at 9:18 AM, Grant Henke wrote:
> I am hoping to get more discussion and feedback around the blocking vs
> async discussion so I can start t
t; > > >
> > > > > > > *** Please download, test and vote by Monday, April 4, 4pm PT
> > > > > > >
> > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
> > > > > > > release:http://kafka.apache.org/KEYS
> > > > > > >
> > > > > > > * Release artifacts to be voted upon (source and
> > > > > > > binary):http://home.apache.org/~gwenshap/0.10.0.0-rc1/
> > > > > > >
> > > > > > > * Maven artifacts to be voted
> > > > > > > upon:https://repository.apache.org/content/groups/staging/
> > > > > > >
> > > > > > > * scala-dochttp://
> > home.apache.org/~gwenshap/0.10.0.0-rc1/scaladoc
> > > > > > >
> > > > > > > * java-dochttp://
> home.apache.org/~gwenshap/0.10.0.0-rc1/javadoc/
> > > > > > >
> > > > > > > * tag to be voted upon (off 0.10.0 branch) is the 0.10.0.0
> > > > > > > tag:
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=759940658d805b1262101dce0ea9a9d562c5f30d
> > > > > > >
> > > > > > > * Documentation:
> http://kafka.apache.org/0100/documentation.html
> > > > > > >
> > > > > > > * Protocol:http://kafka.apache.org/0100/protocol.html
> > > > > > >
> > > > > > > /**
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Gwen
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashish
> >
>
>
>
> --
> Regards,
>
> Rajini
>
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
ly safe way to represent the absence of a value.
> > >
> > > In Java (Scala would be similar) we would convert this to an
> > > Optional> to make it clear that the value could be absent
> > (and
> > > avoid NPEs). The fact that absence of a value means "all topics" makes
> > > sense if one thinks about that field as a filter (absence of a value
> > means
> > > no filter).
> > >
> > > I can see pros and cons for each approach, personally. :)
> > >
> > > Ismael
> > >
> >
>
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
Additionally feel free to review the PR (#1095
<https://github.com/apache/kafka/pull/1095>) to see what the current
proposal/implementation looks like. Right now its using null to signify no
topics. That can be changed based on the discussion here.
On Wed, Mar 30, 2016 at 2:34 PM, Grant
for when it
> matters most (when the number of topics is large).
> Ismael
>
> On Mon, Mar 14, 2016 at 10:07 PM, Grant Henke wrote:
>
> > I have been updating the KIP-4 wiki page based on the last KIP call and
> > wanted to get some review and discussio
towards option A. Keep in mind at the user level, the apis in the various
clients can map this however they like.
Does anyone feel strongly about the choice?
On Thu, Mar 31, 2016 at 9:21 AM, Grant Henke wrote:
> I had a second look at the proposed changes to Metadata Request and
&g
an fix
> > mistakes instead of living with them forever. We should take advantage of
> > that.
> >
> > Ismael
> >
> > On Thu, Mar 31, 2016 at 9:15 PM, Grant Henke
> wrote:
> >
> > > Looking for some resolution on the "No Topics"
available implementing the proposed changes here:
https://github.com/apache/kafka/pull/1095
Here are some links to past discussions on the mailing list:
http://search-hadoop.com/m/uyzND1pd4T52H1m0u1&subj=Re+KIP+4+Wiki+Update
http://search-hadoop.com/m/uyzND1J2IXeSNXAT&subj=Metadata+and+ACLs+wire+protocol+review+KIP+4+
Thank you,
Grant
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
> > > > > > > > > > > > > CallbackHandler class? This seems too limiting to
> > me,
> > > > and
> > > > > > > while
> > > > > > > > > > > Rajini
> > > > > > > > > > > > > > is trying hard to convince me otherwise, I remain
> > > > > doubtful.
> > > > > > > > > Perhaps
> > > > > > > > > > > > > > (since we have similar experience with Hadoop),
> you
> > > can
> > > > > > help
> > > > > > > me
> > > > > > > > > see
> > > > > > > > > > > > > > what I am missing.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Gwen
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Wed, Mar 9, 2016 at 12:02 PM, Harsha <
> > > > ka...@harsha.io
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > > > > > +1 (binding)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Mar 8, 2016, at 02:37 AM, tao xiao
> wrote:
> > > > > > > > > > > > > > >> +1 (non-binding)
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> On Tue, 8 Mar 2016 at 05:33 Andrew Schofield <
> > > > > > > > > > > > > > >> andrew_schofield_j...@outlook.com> wrote:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> > +1 (non-binding)
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > > >> > > From: ism...@juma.me.uk
> > > > > > > > > > > > > > >> > > Date: Mon, 7 Mar 2016 19:52:11 +
> > > > > > > > > > > > > > >> > > Subject: Re: [VOTE] KIP-43: Kafka SASL
> > > > > enhancements
> > > > > > > > > > > > > > >> > > To: dev@kafka.apache.org
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > > +1 (non-binding)
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > > On Thu, Mar 3, 2016 at 10:37 AM, Rajini
> > > Sivaram
> > > > <
> > > > > > > > > > > > > > >> > > rajinisiva...@googlemail.com> wrote:
> > > > > > > > > > > > > > >> > >
> > > > > > > > > > > > > > >> > >> I would like to start the voting process
> > for
> > > > > > *KIP-43:
> > > > > > > > > Kafka
> > > > > > > > > > > > SASL
> > > > > > > > > > > > > > >> > >> enhancements*. This KIP extends the SASL
> > > > > > > implementation
> > > > > > > > > in
> > > > > > > > > > > > Kafka to
> > > > > > > > > > > > > > >> > support
> > > > > > > > > > > > > > >> > >> new SASL mechanisms to enable Kafka to be
> > > > > > integrated
> > > > > > > > with
> > > > > > > > > > > > different
> > > > > > > > > > > > > > >> > >> authentication servers.
> > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > >> > >> The KIP is available here for reference:
> > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-43:+Kafka+SASL+enhancements
> > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > >> > >> And here's is a link to the discussion on
> > the
> > > > > > mailing
> > > > > > > > > list:
> > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201601.mbox/%3CCAOJcB39b9Vy7%3DZEM3tLw2zarCS4A_s-%2BU%2BC%3DuEcWs0712UaYrQ%40mail.gmail.com%3E
> > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > >> > >> Thank you...
> > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > >> > >> Regards,
> > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > >> > >> Rajini
> > > > > > > > > > > > > > >> > >>
> > > > > > > > > > > > > > >> >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > > Regards,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Rajini
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Regards,
> > > > > > > > > >
> > > > > > > > > > Rajini
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Regards,
> > > > > > > >
> > > > > > > > Rajini
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Regards,
> > > > > >
> > > > > > Rajini
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > >
> > > Rajini
> > >
> >
>
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
ternal topic). It seems a bit wasteful and particularly so when using
> regex subscriptions (since we have to retrieve all topics in that case).
>
> 2. A similar comment on topics_marked_for_deletion. Would it be better to
> > only return them when asked for and just return a new T
am not sure I completely follow the issue and requested change. Could you
point me to the discussion?
Thank you,
Grant
On Sun, Apr 3, 2016 at 8:09 PM, Grant Henke wrote:
> Hi Jun and Ismael,
>
> Initially I had 2 booleans used to indicate if a topic was internal and if
> a
rs"
*Kafka Commits:*
Matches: list:"commits.kafka.apache.org"
Do this: Skip Inbox, Apply label "Kafka/Kafka Commits"
If you have any other filters or strategies to keep up, please don't
hesitate to share.
Thanks,
Grant
P.S. I also use Gitify to see/track
g the error handling even though there is nothing wrong with the
> leader. A better behavior is to simply return the list of replica ids with
> no error code in this case.
>
> Thanks,
>
> Jun
>
>
>
> On Sun, Apr 3, 2016 at 6:29 PM, Grant Henke wrote:
>
> > Respond
;
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Sun, Apr 3, 2016 at 12:36 AM, Ismael Juma
> wrote:
> > >
> > > > Hi Grant,
> > > >
> > > > One question that occurred to me is whether we want to take the
> cha
are+of+supported+Principal+Types
> > > >
> > > > Discuss thread: here
> > > > <
> > > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-dev/201603.mbox/%3CCAGQG9cUCLDO0owdziDcL9iStXNF1wURyVNcEZedQJg%3DUuC7j%3DQ%40mail.gmail.com%3E
> > > > >
> > > >
> > > > JIRA: https://issues.apache.org/jira/browse/KAFKA-3186
> > > >
> > > > PR: https://github.com/apache/kafka/pull/861
> > > >
> > > > --
> > > >
> > > > Regards,
> > > > Ashish
> > > >
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashish
> >
>
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
eplicas
> 4,5,6, leader is on replica 4 and replica 5 is down. Currently, the broker
> will send a REPLICA_NOT_AVAILABLE error code and only replicas 4,6 in the
> assigned replicas. It's more intuitive to send no error code and 4,5,6 in
> the assigned replicas in this case.
>
> Th
can request no topics now), and is more simple/flexible to
evolve forward if we ever decide we need to represent more "states".
I will update the patch in the next day or so and post a vote if no issue
is raised with that.
Thanks,
Grant
On Tue, Apr 5, 2016 at 4:05 PM, Grant Henke wr
you please explain the motivation? It did come up in previous
> > discussions that some things like Operation and ResourceType should be in
> > the clients library, but not Authorizer itself. Are we saying that any
> > pluggable interface should be in Java so that users can implement it
> > without including the Scala library?
> >
> > Grant, you originally suggested that some things would have to be in the
> > Java side as well. Can you please elaborate on this?
> >
> > Ismael
> >
>
--
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
r 6, 2016 at 5:12 PM, Grant Henke wrote:
>
> > My work with KIP-4 found that many of the Scala classes used in the
> > Authorizer interface are needed in the Clients package when adding the
> > various ACL requests and responses. I also found that we don't have
> > stan
in the clients for requests/responses should be in the
clients/common package though. Some of this has been done in my preliminary
ACLs request/response pull request:
https://github.com/apache/kafka/pull/1005
On Wed, Apr 6, 2016 at 11:20 AM, Grant Henke wrote:
> KIP-50 as defined is very s
I understand. My thinking is that since this is a major release, if we are
going to break in anyway we should do it now. Otherwise we will need to
wait until 0.11. The work in adding standard exceptions and
an alternative Java interface should be able to be done in a compatible
manner (deprecating
pr 5, 2016 at 8:02 PM, Jun Rao wrote:
> 5. You will return no error and 4,5,6 as replicas. The response also
> includes a list of live brokers. So the client can figure out 5 is not live
> directly w/o relying on the error code.
>
> Thanks,
>
> Jun
>
> On Tue, Apr 5
1 - 100 of 794 matches
Mail list logo
