Re: [ANNOUNCE] New Committer: Manikumar Reddy

2018-10-12 Thread Colin McCabe
Congratulations, Manikumar! Well done. best, Colin On Fri, Oct 12, 2018, at 01:25, Edoardo Comar wrote: > Well done Manikumar ! > -- > > Edoardo Comar > > IBM Event Streams > IBM UK Ltd, Hursley Park, SO21 2JN > > > > > From: "Matthias J.

Re: [VOTE] KIP-376: Implement AutoClosable on appropriate classes that want to be used in a try-with-resource statement

2018-10-12 Thread Colin McCabe
Hi Yishun, Thanks for looking at this. Under "proposed changes," it's not necessary to add a section where you demonstrate adding "implements AutoCloseable" to the code. We know what adding that would look like. Can you create a full, single, list of all the classes that would be affected?

Re: [DISCUSS] KIP-377: TopicCommand to use AdminClient

2018-10-12 Thread Colin McCabe
Hi Viktor, Thanks for bumping this thread. I think we should just focus on transitioning the TopicCommand to use AdminClient, and talk about protocol changes in a separate KIP. Protocol changes often involve a lot of discussion. This does mean that we couldn't implement the "list topics unde

Re: [VOTE] KIP-349 Priorities for Source Topics

2018-10-12 Thread Colin McCabe
ould be like, etc. best, Colin > > On Fri, 2018-10-05 at 11:34 -0700, Colin McCabe wrote: > > On Fri, Oct 5, 2018, at 10:58, Thomas Becker wrote: > > Colin, > > Would you mind sharing your vision for how this looks with multiple > > consumers? I'm still

Re: [VOTE] KIP-376: Implement AutoClosable on appropriate classes that want to be used in a try-with-resource statement

2018-10-12 Thread Colin McCabe
. Thanks for working on this. +1 (binding). best, Colin > > Thanks, > Yishun > > On Fri, Oct 12, 2018 at 11:42 AM Colin McCabe wrote: > > > Hi Yishun, > > > > Thanks for looking at this. > > > > Under "proposed changes," it's n

Re: KAFKA-6654 custom SSLContext

2018-10-15 Thread Colin McCabe
In general Kafka makes an effort to be langauge-neutral so that Kafka clients can be implemented on platforms other than Java. For example, we have things like librdkafka which allow people to access Kafka from C or Python. Unless I'm misunderstanding something, giving direct access to the SSL

Re: KAFKA-6654 custom SSLContext

2018-10-15 Thread Colin McCabe
ch uses all the Kafka ssl properties. > > Is that acceptable to resolve KAFKA-6654? > Can you think of a better solution? > > > -Original Message- > From: Colin McCabe [mailto:cmcc...@apache.org] > Sent: Monday, October 15, 2018 7:58 PM > To: dev@kafka.apache.

Re: [DISCUSS] KIP-377: TopicCommand to use AdminClient

2018-10-20 Thread Colin McCabe
a similar note: is there a KIP for describe topics protocol or have you > been thinking about it? I guess there it's the same problem, we often don't > want to forward the entire metadata. > > Viktor > > On Fri, Oct 12, 2018 at 12:03 PM Colin McCabe wrote: > > &

Re: [VOTE] KIP-349 Priorities for Source Topics

2018-10-21 Thread Colin McCabe
On Thu, Oct 18, 2018, at 09:23, n...@afshartous.com wrote: > > > > On Oct 12, 2018, at 5:06 PM, Colin McCabe wrote: > > > > Maybe there's some really cool use-case that I haven't thought of. But so > > far I can't really think of any time I wou

Re: Throwing away prefetched records optimisation.

2018-10-21 Thread Colin McCabe
Ryanne Dolan wrote: > It sounds to me like this problem is due to Akka attempting to implement > additional backpressure on top of the Consumer API. I'd suggest they not do > that, and then this problem goes away. Imagine a very simple case where you want to consume from three partitions at abou

Re: Throwing away prefetched records optimisation.

2018-10-23 Thread Colin McCabe
On Tue, Oct 23, 2018, at 12:38, Zahari Dichev wrote: > Hi there Matthias, I looked through the code of Kafka Streams. Quite > impressive work ! If I have to put the logic of buffering within the > context of what we are doing in Akka though, I might end up with the > following situation. > > 1. Po

Re: Throwing away prefetched records optimisation.

2018-10-24 Thread Colin McCabe
hread for it or should I start another one ? > > Zahari > > On Wed, Oct 24, 2018 at 6:43 AM Colin McCabe wrote: > > > On Tue, Oct 23, 2018, at 12:38, Zahari Dichev wrote: > > > Hi there Matthias, I looked through the code of Kafka Streams. Quite > > > imp

Re: [VOTE] KIP-377: TopicCommand to use AdminClient

2018-10-24 Thread Colin McCabe
Thanks, Viktor. +1 (binding). One note: can we add a deprecation warning when --zookeeper is used, to indicate that this option will be phased out in the future? best, Colin On Wed, Oct 24, 2018, at 05:47, Mickael Maison wrote: > +1 (non-binding) > Thanks for the KIP! > On Wed, Oct 24, 2018 at

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

2018-10-25 Thread Colin McCabe
Hi Zahari, One question we didn't figure out earlier was who would actually want this cached data to be thrown away. If there's nobody who actually wants this, then perhaps we can simplify the proposal by just unconditionally retaining the cache until the partition is resumed, or we unsubscrib

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

2018-10-26 Thread Colin McCabe
;t seen a > > lot of memory overhead due to this in our systems. We have had this running > > in production for a considerable amount of time without any issues. > > It would be great if you guys can review the PR once its up and see if that > > satisfies your require

Re: [VOTE] KIP-349 Priorities for Source Topics

2018-10-26 Thread Colin McCabe
eams does. best, Colin > > -- > Nick > > > > > > On Oct 18, 2018, at 12:23 PM, n...@afshartous.com wrote: > > > >> On Oct 12, 2018, at 5:06 PM, Colin McCabe wrote: > >> > >> Maybe there's some really cool use-case tha

Re: Kafka-node: Kafka client keeps sending request to broker which went down

2018-10-26 Thread Colin McCabe
Hi Shashank, kafka-node is not developed by Apache or the Apache Kafka project. I don't think anyone here has looked at that code. I have heard reports from the field that kafka-node has some very serious bugs, such as not retrying failed requests at all in some circumstances. I do not recom

Re: [DISCUSS] How hard is it to separate the logic layer and the storage layer of Kafka broker?

2018-11-02 Thread Colin McCabe
On Fri, Nov 2, 2018, at 03:14, Yuanjin Lin wrote: > Hi all, > > I am a software engineer from Zhihu.com. Kafka is so great and used heavily > in Zhihu. There are probably over 2K Kafka brokers in total. > > However, we are suffering from the problem that the performance degrades > rapidly when th

Re: [VOTE] KIP-374: Add '--help' option to all available Kafka CLI commands

2018-11-08 Thread Colin McCabe
+1 (binding) On Wed, Oct 31, 2018, at 05:42, Srinivas Reddy wrote: > Hi All, > > I would like to call for a vote on KIP-374: > https://cwiki.apache.org/confluence/x/FgSQBQ > > Summary: > Currently, the '--help' option is recognized by some Kafka commands > but not all. To provide a consistent us

Re: Kafka Performance Producer - Extension

2018-11-08 Thread Colin McCabe
On Thu, Nov 8, 2018, at 01:38, Srinivas, Kaushik (Nokia - IN/Bangalore) wrote:> Hi All, > > This is in regard to extend few features support for java kafka > performance producer.> Hi Kaushik, Have you created a pull request against the Kafka repo at https://github.com/apache/kafka/ ?

Re: [DISCUSS] KIP-388 Add observer interface to record request and response

2018-11-08 Thread Colin McCabe
Hi Lincong Li, I agree that server-side instrumentation is helpful. However, I don't think this is the right approach. The problem is that RequestChannel.Request and AbstractResponse are internal classes that should not be exposed. These are implementation details that we may change in the f

Re: Heads up: javac warnings are now treated as errors

2018-11-14 Thread Colin McCabe
Sounds good. Thanks for cleaning up the warnings! Colin On Mon, Nov 12, 2018, at 22:29, Ismael Juma wrote: > Hi all, > > As part of KAFKA-7612, all javac warnings were fixed or suppressed. To> > prevent them from reappearing, javac warnings are now treated as > errors. We> still have some scal

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

2018-11-16 Thread Colin McCabe
As Jason said, there are definitely scenarios where we know how many group members we expect ahead of time. It would be nice if we could distinguish between the error case of "we expected 5 clients in the group, but one failed" and a case like "4 clients started up quickly but the 5th took an e

Re: [VOTE] KIP-354 Time-based log compaction policy

2018-12-04 Thread Colin McCabe
gt; > > > >> > > > > > > > >> > > > > Thanks Eno, Brett, Dong, Guozhang, Colin, and Xiaohe for > > > >> feedback. > > > >> > > > > Can I have more feedback or VOTE on this KIP? > > > >>

Re: [VOTE] KIP-354 Time-based log compaction policy

2018-12-05 Thread Colin McCabe
estamp not the maxTimestamp > of the previous log segment to make it simple. > > I have updated the KIP. > > Xiongqi (wesley) Wu > > > On Tue, Dec 4, 2018 at 5:13 PM Colin McCabe wrote: > > > Hi Xiongqi, > > > > Thinking about this a little bit more, it se

Re: Problem in CI for pull request

2018-12-21 Thread Colin McCabe
Try typing "retest this please" as a comment to the PR. best, Colin On Wed, Nov 28, 2018, at 11:05, lk gen wrote: > Hi, > > I made a pull request and it passed CI on JDK 11 but failed on JDK 8 > > I think the JDK 8 error may not related to my commit but an environment > problem on the CI >

Re: [DISCUSS] KIP-388 Add observer interface to record request and response

2018-12-21 Thread Colin McCabe
the internal classes from > >>> being > >>> > > > exposed. These two interfaces contain getters that allow user to > >>> > extract > >>> > > > information from request and response in their own > >>> implementation(

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

2018-12-21 Thread Colin McCabe
post-KIP252 would be an explicit v1 for > > > example. > > > Authorizers declare which versions they are compatible with (though > > > I'd say i backwards compatibility is what we shoud shoot for) and > > > load ACLs of those versions. > > > Introdu

Re: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message Consumption Across Partitions in KafkaConsumer

2018-12-22 Thread Colin McCabe
Hi ChienHsing Wu, Maybe I'm misunderstanding something, but I'm not sure I see the need for a KIP here. You can just set max.partition.fetch.bytes to a very small value. That will cause Kafka to fetch only one message from each partition. This will give you the round robin behavior you want.

Re: [VOTE] KIP-349 Priorities for Source Topics

2019-01-10 Thread Colin McCabe
Hi all, Just as a quick reminder, this is not really a complete proposal. There are a bunch of unresolved issues with this KIP. One example is how this interacts with incremental fetch sessions. It is not mentioned anywhere in the KIP text. Previously we discussed some approaches, but there

Re: [VOTE] KIP-349 Priorities for Source Topics

2019-01-15 Thread Colin McCabe
On Sun, Jan 13, 2019, at 18:13, n...@afshartous.com wrote: > Thanks Colin and Mathias. > > > On Jan 12, 2019, at 8:27 PM, Matthias J. Sax wrote: > > > > Thus, I would suggest to limit this KIP to the consumer only, otherwise, > > the scope will be too large and this KIP will drag on even longer.

Re: [DISCUSS] KIP-402: Improve fairness in SocketServer processors

2019-01-15 Thread Colin McCabe
Hi Rajini, Thanks for this. The KIP looks really useful. > > A new metric will be added to track the amount of time Acceptor is blocked > from accepting connections due to backpressure. This will be a yammer > Meter, consistent with other SocketServer metrics. > > kafka.network:type=Accepto

[DISCUSS] 2.1.1 bug-fix release

2019-01-15 Thread Colin McCabe
Hi all, I'd like to volunteer to be the release manager for the 2.1.1 bug fix release. 2.1 was released November 20, 2018. There are 34 fixes scheduled for inclusion in 2.1.1 so far. Please find all the resolved tickets here: https://issues.apache.org/jira/browse/KAFKA-7818?jql=project%20%3D%2

Re: [DISCUSS] KIP-402: Improve fairness in SocketServer processors

2019-01-15 Thread Colin McCabe
think the point is that we distribute the time more fairly between > > connection handling and other operations where before we could block on the > > TLS handshake for a long time given a large number of connections. > > > > Ismael > > > > On Tue, Jan 15,

Re: [VOTE] KIP-402: Improve fairness in SocketServer processors

2019-01-16 Thread Colin McCabe
+1 (binding) Thanks, Colin On Tue, Jan 15, 2019, at 18:55, Gwen Shapira wrote: > +1 > Thank you! > > > On Tue, Jan 15, 2019, 3:38 PM Rajini Sivaram > > > Hi all, > > > > I would like to start vote on KIP-402 to improve fairness in channel> > > > processing in SocketServer to protect brokers fr

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

2018-05-07 Thread Colin McCabe
tive I'd leave the the command there but put a > warning that the tool is deprecated and should only be used for > setting up SCRAM credentials. > What do you think? What about writing a small script that just handles setting up SCRAM credentials? It would probably be easier

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

2018-05-09 Thread Colin McCabe
Hi Molnar, The points Ismael brought up earlier (and that were brought up on KIP-30) are still relevant here. As Ismael said, the goal is to get rid of external dependencies here. We're going to post more about this soon (sorry for the delay) thanks, Colin On Wed, May 9, 2018, at 07:29, M

Re: [DISCUSS] KIP-278: Add version option to Kafka's commands

2018-05-09 Thread Colin McCabe
> > >> Hi Colin, > >> > >> Thanks for explanation. It's definitely useful to have --version flag. > >> > >> kafka-broker-api-versions.sh gives the API versions, not Kafka release > >> version. > >> Is not easy to figure out release

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

2018-05-10 Thread Colin McCabe
o create a pattern that matches only "foo*". You have to type "foo\*" It would be required if we forced users to specify a version, as well. best, Colin > > Sent from my iPhone > > > On 7 May 2018, at 05:16, Piyush Vijay wrote: > > > > Makes se

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

2018-05-11 Thread Colin McCabe
of several possible fields should be populated. But it's a > minor thing. > > > > > > > On 10 May 2018 at 09:29, Colin McCabe wrote: > > > Hi Andy, > > > > The issue that I was trying to solve here is the Java API. Right now, > > someon

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

2018-05-11 Thread Colin McCabe
cheers, Colin > > > What about writing a small script that just handles setting up SCRAM > > credentials? It would probably be easier to maintain than the old config > > command. Otherwise we have to explain when each tool should be used, which > > will be confusing

Re: [VOTE] KIP-278: Add version option to Kafka's commands

2018-05-11 Thread Colin McCabe
+1 (non-binding) Colin On Fri, May 11, 2018, at 12:35, Attila Sasvári wrote: > +1 (non-binding) > > Thomas Crayford ezt írta (időpont: 2018. máj. > 11., P 18:20): > > > +1 (non-binding) > > > > On Fri, May 11, 2018 at 5:17 PM, Guozhang Wang wrote: > > > > > Thanks Toru-san, +1 (binding) > > >

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

2018-05-15 Thread Colin McCabe
Options options); > > public abstract CreateAclsResult createAcls(Collection > acls, CreateAclsOptions options); > > public abstract DeleteAclsResult > deleteAcls(Collection filters, DeleteAclsOptions > options); > > > Thanks > > Piyush Vijay > > On Mon, Ma

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-15 Thread Colin McCabe
Hi Robert, Thanks for posting this. In the past we've been kind of reluctant to add more complexity to configuration. I think Connect does have a clear need for this kind of functionality, though. As you mention, Connect integrates with external systems, which are very likely to have passwor

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

2018-05-16 Thread Colin McCabe
ndingFilter-- fields can never be null in AclBinding. For example, you can have an AclBindingFilter that matches every AclBinding. There is more discussion of this on the original KIP that added ACL support to AdminClient. best, Colin > >> > >> > >> > >> Piy

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-16 Thread Colin McCabe
Thanks, Robert. Looks good overall. As a clarification about the indirections, what if I have the connect configuration key foo set up as ${vault:bar}, and in Vault, have the bar key set to ${file:baz}? Does connect get foo as the contents of the baz file? I would argue that it should not (a

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-16 Thread Colin McCabe
connectors. In the case of the > VaultConfigProvider, it knows the lease durations and will be able to> > schedule a restart of the Connector using an API in the Herder. The > delayMs will simply be passed to the Herder.restartConnector(long > delayMs,> String connName, Callback cb

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

2018-05-16 Thread Colin McCabe
, it will increment a version number and won't let> other admins to > push changes in with lower than that. Instead it would> return an error. > > I would be also interested what others think about this? > > Cheers, > Viktor > > > On Sat, May 12, 2018 at

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

2018-05-17 Thread Colin McCabe
ite the same as > > a full-fledged user groups, but since it works with all security protocols, > > it could be an alternative to wildcarded principals. > > > > Let us know if we can help in any way to get this KIP updated and ready for > > voting to include in 2.0.0. >

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-17 Thread Colin McCabe
ration parameter to > the ConfigProvider to determine which model (push or poll) to use. > > Thanks, > Robert > > On Wed, May 16, 2018 at 9:56 PM, Colin McCabe wrote: > > > Thanks, Robert. With regard to delayMs, can’t we just restart the > > Connector when the keys are

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

2018-05-17 Thread Colin McCabe
e KIP. cheers, Colin On Wed, May 16, 2018, at 22:06, Colin McCabe wrote: > Hi Viktor, > > The shell command isn’t that easy to integrate into applications. > AdminClient will get integrated into a lot more stuff, which > increases the potential for conflicts. I would argue that

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

2018-05-17 Thread Colin McCabe
ard-suffix ACLs on Zk. +1. Thanks, Piyush. Colin > > > > On Thu, May 17, 2018 at 9:15 AM Colin McCabe wrote: > > > Thanks, Piyush. +1 for starting the vote soon. > > > > Can you please also add a discussion about escaping? For example, earlier >

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

2018-05-17 Thread Colin McCabe
obably just create a flag for alterConfigs which marks it as incremental, like we discussed earlier, and do this as a compatible change that is needed for the shell command. best, Colin On Thu, May 17, 2018, at 09:32, Colin McCabe wrote: > Hi Viktor, > > Since the KIP freeze is coming up

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-17 Thread Colin McCabe
e ConfigChangeCallback { > > > > void willExpire(String path, long ttl); > > > > void onChange(String path, Map values); > > } > > > > > > > > Or we could return a composite object from get(): > > > > ConfigData get(String path); > &g

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

2018-05-18 Thread Colin McCabe
e should probably just create a flag for alterConfigs which marks > >> it as incremental, like we discussed earlier, and do this as a compatible > >> change that is needed for the shell command. > > > > Alright, I missed that about the sensitive configs too, so in this

Re: [VOTE] KIP-290: Support for wildcard suffixed ACLs

2018-05-21 Thread Colin McCabe
Hmm, do we still need one more binding +1, or did I misread the vote thread? +1 (non-binding) from me. As I posted in the other thread, I think it would make sense to name the new ZK hierarchy /kafka-prefix-acls or similar, to reflect the fact that they are ACLs that match by a name prefix. Th

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

2018-05-21 Thread Colin McCabe
Oops, will post on the vote thread. best, Colin On Mon, May 21, 2018, at 21:15, Colin McCabe wrote: > On Mon, May 21, 2018, at 04:53, Andy Coates wrote: > > Hey Piyush, > > > > Thanks for the updated KIP! Couple of minor points from me: > > > > When storing w

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

2018-05-21 Thread Colin McCabe
gt; > > >>>>>> > > > >>>>>> Piyush Vijay > > > >>>>>> > > > >>>>>> On Fri, May 18, 2018 at 12:18 PM, Piyush Vijay < > > > >> piyushvij...@gmail.com > > > >>>>

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

2018-06-05 Thread Colin McCabe
As Jason noted earlier, request.timeout.ms controls something different: the amount of time we're willing to wait for an RPC to complete. Empirically, RPCs sometimes hang for a long time. If the API timeout is the same as the RPC timeout, we are not robust against this failure condition. The

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

2018-06-05 Thread Colin McCabe
? > > Thanks > > On Tue, Jun 5, 2018 at 4:18 PM, Colin McCabe wrote: > > > As Jason noted earlier, request.timeout.ms controls something different: > > the amount of time we're willing to wait for an RPC to complete. > > > > Empirically, RPCs sometimes

Re: Someone to review KAFKA-6919, one line change for faulty documentation

2018-06-05 Thread Colin McCabe
Thanks, Koen. I was on vacation, so I missed this originally. But I'll review it now. cheers, Colin On Fri, Jun 1, 2018, at 13:02, Koen De Groote wrote: > Greetings, > > Poking for someone to have a quick look at this, It's a one-line change. I > noticed the documentation about trogdor was p

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

2018-06-05 Thread Colin McCabe
ple tries if the RPC hangs due to network issues. It should not be longer. best, Colin > > Cheers > > On Tue, Jun 5, 2018 at 4:27 PM, Colin McCabe wrote: > > > I don't think it can be fixed. The RPC duration is something that you > > might reasonably want to

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

2018-06-08 Thread Colin McCabe
; > > bq. request.timeout.ms controls something different: the amount of time > > we're willing to wait for an RPC to complete. > > > > Basically we're in agreement. It is just that figuring out good default is > > non-trivial. > > > > On Tue, Jun 5,

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

2018-06-11 Thread Colin McCabe
> > > On Tue, May 15, 2018 at 8:01 PM, Robert Yokota < > > > rayok...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Colin, > > > > > > > > > > > > > > > > > > Thanks for the feedback! > >

Re: SASL Unit test failing

2018-06-26 Thread Colin McCabe
On trunk, testMultipleServerMechanisms failed for me, as well as testAuthenticateCallbackHandlerMechanisms and testMechanismPluggability. org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > testMultipleServerMechanisms FAILED java.lang.AssertionError at org.junit.A

Re: [VOTE] KIP-324: Add method to get metrics() in AdminClient

2018-06-26 Thread Colin McCabe
Can you add a little more explanation to the KIP for why you are adding this method? Is it something streams needs, for example? Will it help other applications that use admin client and want to expose metrics? What are the thread-safety guarantees for the map which is returned? best, Colin

Re: [VOTE] KIP-324: Add method to get metrics() in AdminClient

2018-06-27 Thread Colin McCabe
tence to the KIP) I agree. Thanks. > Since this returns an unmodifiableMap(like all the other client's metrics() > return), I assume this will be thread-safe, what do you think? Please document that it is thread-safe. thanks, Colin > > Thanks, > Yishun > > > On T

Re: [VOTE] KIP-324: Add method to get metrics() in AdminClient

2018-06-27 Thread Colin McCabe
P.S. +1 (non-binding) once you add the info about it being thread-safe. best, On Wed, Jun 27, 2018, at 15:23, Colin McCabe wrote: > On Tue, Jun 26, 2018, at 13:24, Yishun Guan wrote: > > Hi Colin, > > > > I agree with what Guozhang's opinion that because all the

Re: Kafka system tests contribution

2018-07-10 Thread Colin McCabe
Hi Andriy, Try looking at the logs to see why the test failed. best, Colin On Wed, May 16, 2018, at 07:08, Andriy Sorokhtey wrote: > Hi, > > Did anyone had a chance to take a look at this issue? > > 2018-05-08 15:01 GMT+03:00 Andriy Sorokhtey : > > > Hello Kafka team > > > > I’d like to contr

[DISCUSS]: KIP-339: Create a new ModifyConfigs API

2018-07-11 Thread Colin McCabe
Hi all, Previously, we discussed some issues with alterConfigs here on the mailing list, and eventually came to the conclusion that the RPC as implemented can't be used for a shell command modifying configurations. I wrote up a small KIP to fix the issues with the RPC. Please take a look at h

Re: [DISCUSS]: KIP-339: Create a new ModifyConfigs API

2018-07-13 Thread Colin McCabe
): > > bq. If a configuration key is specified in both *changes* and > *removals*> > *Cheers* > > On Wed, Jul 11, 2018 at 10:54 AM Colin McCabe > wrote:> > > Hi all, > > > > Previously, we discussed some issues with alterConfigs here on the > > mailin

Re: [DISCUSS]: KIP-339: Create a new ModifyConfigs API

2018-07-13 Thread Colin McCabe
ng. > Anyway, to signify that null value is supported, value type can be > declared as Optional.> > FYI Yeah, now that we're on Java 8, Optional could be a good choice here. best, Colin > On Fri, Jul 13, 2018 at 5:35 PM Colin McCabe > wrote:>> __ >> Hi Te

Re: KIP-327: Add describe all topics API to AdminClient

2018-07-13 Thread Colin McCabe
As Jason wrote, this won't scale as the number of partitions increases. We already have users who have tens of thousands of topics, or more. If you multiply that by 100x over the next few years, you end up with this API returning full information about millions of topics, which clearly doesn't

Re: KIP-327: Add describe all topics API to AdminClient

2018-07-13 Thread Colin McCabe
the regex doesn't filter ("*") ? > > Cheers > > On Fri, Jul 13, 2018 at 6:02 PM Colin McCabe > wrote:> > > As Jason wrote, this won't scale as the number of partitions > > increases.> > We already have users who have tens of thousands of to

Re: KIP-327: Add describe all topics API to AdminClient

2018-07-15 Thread Colin McCabe
red ? > > On Sat., 14 Jul. 2018, 4:15 pm Colin McCabe, wrote: > > > Good point. We should probably have a maximum number of results like > > 1000 or something. That can go in the request RPC as well... > > Cheers, > > Colin > > > > On Fri, Jul 13, 20

Re: KIP-327: Add describe all topics API to AdminClient

2018-07-15 Thread Colin McCabe
, Jul 14, 2018 at 2:42 PM Ted Yu wrote: > > > What if broker crashes before all the pages can be returned ? > > > > Cheers > > > > On Sat, Jul 14, 2018 at 1:07 AM Stephane Maarek < > > steph...@simplemachines.com.au> wrote: > > > > > Why

Re: [DISCUSS]: KIP-339: Create a new ModifyConfigs API

2018-07-16 Thread Colin McCabe
> What do you think? alterConfigs needs to be deprecated, though. Code using alterConfigs is almost certainly buggy because of the possibility of simultaneous read-modify-write cycles, and the fact that some configs can't be read. best, Colin > > Thanks, > Dong > > On

Re: [DISCUSS]: KIP-339: Create a new ModifyConfigs API

2018-07-16 Thread Colin McCabe
uld we include specify in the "Compatibility, > > Deprecation, and Migration Plan" section that we intend to deprecate the > > existing API? And do we plan to deprecate the > > AlterConfigsRequest/AlterConfigsResponse > > as well? The latter may be importa

Re: KIP-327: Add describe all topics API to AdminClient

2018-07-17 Thread Colin McCabe
Thanks, Manikumar. best, Colin On Tue, Jul 17, 2018, at 19:44, Manikumar wrote: > Closing this KIP in favor of adding filtering support to the Metadata API > and KIP-142. Will open a new KIP when ready. > Thanks for your reviews. > > On Mon, Jul 16, 2018 at 8:38 AM Coli

Re: [VOTE] KIP-338 Support to exclude the internal topics in kafka-topics.sh command

2018-07-19 Thread Colin McCabe
+1 (non-binding) cheers, Colin On Mon, Jul 16, 2018, at 01:10, Chia-Ping Tsai wrote: > hi folks, > > The discussion[1] of KIP-338[2] did not get any objection for last 6 > days so it is time to start the voting thread. > > Thanks for your time! > > [1] > https://lists.apache.org/thread.html/

Re: [DISCUSS]: KIP-339: Create a new ModifyConfigs API

2018-07-20 Thread Colin McCabe
27; idea of supporting "append" and "subtract" on configuration keys that contain lists. best, Colin On Mon, Jul 16, 2018, at 14:12, Colin McCabe wrote: > Hi Magnus, > > Thanks for taking a look. > > On Mon, Jul 16, 2018, at 11:43, Magnus Edenhill wrote: > &g

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

2018-07-25 Thread Colin McCabe
On Mon, Jul 23, 2018, at 23:20, James Cheng wrote: > Hi Stanislav! Thanks for this KIP! > > I agree that it would be good if the LogCleaner were more tolerant of > errors. Currently, as you said, once it dies, it stays dead. > > Things are better now than they used to be. We have the metric >

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

2018-07-27 Thread Colin McCabe
Sorry if this is a silly question, but what's the rationale for switching to using null for the default group id, rather than the empty string? Continuing to use the empty string seems like less churn. And after all, we're not using the empty string group name for anything else. The big incom

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

2018-07-30 Thread Colin McCabe
Hi Manikumar, It's great that you are taking a look at this! Much needed. Just one note: I assume that --authorizer-properties is no longer required if the --bootstrap-server option is specified. We should probably spell this out somewhere in the KIP. thanks, Colin On Mon, Jul 30, 2018, at

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

2018-08-02 Thread Colin McCabe
re is sort of clunky. What does everybody think about this? > >>>>>> > >>>>>>> 4) The only improvement I can think of is that if such an > >>>>>>> error occurs, then have the option (configuration setting?) to > &

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

2018-08-02 Thread Colin McCabe
t > > be changed, will continue to work because the old versions of the > > OffsetCommit API still accept the empty group id. The null group id is > > different from the empty group id: it is not allowed in any version of the > > API. It is basically a way to indicate that

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

2018-08-02 Thread Colin McCabe
+1 for starting the vote cheers, Colin On Wed, Aug 1, 2018, at 08:46, Manikumar wrote: > Hi all, > > If there are no concerns, I will start the voting process soon. > > Thanks > > On Tue, Jul 31, 2018 at 9:08 AM Manikumar wrote: > > > Hi Colin, > > > > Yes, "--authorizer-properties" option

Re: [DISCUSS] KIP-348 Eliminate null from SourceTask#poll()

2018-08-03 Thread Colin McCabe
"No changes to public interface" doesn't seem accurate here. SourceTask#poll is a public interface, right? Kafka connectors that are out-of-tree would certainly break if we disallowed returning null from this method. However, reading the KIP more closely, it seems like both null and the empty

Re: [DISCUSS] KIP-350: Allow kafka-topics.sh to take brokerid as parameter to show partitions associated with it

2018-08-03 Thread Colin McCabe
Thanks for the KIP, Ratish. We should probably specify that it is an error when --broker is specified with operations other than --list, right? best, Colin On Wed, Aug 1, 2018, at 21:28, Ratish Ravindran wrote: > Hi, > > I would like to open a discussion thread on KIP-350: > > https://cwiki.

Re: [VOTE] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

2018-08-03 Thread Colin McCabe
Hi Stanislav, Thanks for the KIP. As as user, "FaultyRecordException" seems a little vague. What's faulty about it? Is it too big? Does it not match the schema of the other records? Was it not found? Of course, we know that the specific problem is with [de]serializing the record, not with

Re: [VOTE] KIP-332: Update AclCommand to use AdminClient API

2018-08-03 Thread Colin McCabe
+1 (non-binding) regards, Colin On Fri, Aug 3, 2018, at 02:27, Rajini Sivaram wrote: > Hi Manikumar, > > +1 (binding) > > Thanks for the KIP! > > On Fri, Aug 3, 2018 at 3:46 AM, Ted Yu wrote: > > > +1 > > > > On Thu, Aug 2, 2018 at 7:33 PM Manikumar > > wrote: > > > > > Hi All, > > > > > >

Re: [DISCUSS] KIP-240: AdminClient.listReassignments AdminClient.describeReassignments

2018-01-09 Thread Colin McCabe
What if we had an internal topic which watchers could listen to for information about partition reassignments? The information could be in JSON, so if we want to add new fields later, we always could. This avoids introducing a new AdminClient API. For clients that want to be notified about

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

2018-01-09 Thread Colin McCabe
gt; >> > >> > > > > > >> > >> > I am of course happy to withdraw the KIP if you think it is > > > > >> > unnecessary, > > > > >> > >> > I've also updated the pull request for KAFKA-4930 to reflect > &g

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

2018-01-09 Thread Colin McCabe
> > > > > > > > > > > > > > On Wed, Dec 6, 2017 at 11:50 PM, Jun Rao wrote: > > > > > > > > > > > > > A couple more things. > > > > > > > > > > > > > > 6. For the SSL/SASL con

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

2018-01-09 Thread Colin McCabe
arnings however. > > > On Tue, Jan 9, 2018 at 9:38 PM, Colin McCabe wrote: > > > On Mon, Dec 18, 2017, at 13:40, Jason Gustafson wrote: > > > Hi Rajini, > > > > > > Looking good. Just a few questions. > > > > > > 1. (Related to Jay'

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

2018-01-12 Thread Colin McCabe
gt; > needed). > > > > If you have a set of characters you think we should exclude, I think it > > would be good to add them here or in a subsequent KIP! > > > > -Ewen > > > > On Tue, Jan 9, 2018 at 1:30 PM, Colin McCabe wrote: > > > >> On Sat

Re: Vote for KIP-245: Use Properties instead of StreamsConfig in KafkaStreams constructor

2018-01-16 Thread Colin McCabe
Why not just have a StreamsConfig constructor that takes a Properties object? This has a few advantages. Firstly, because it's purely additive, it doesn't create any deprecated functions or compatibility issues that we have to clean up later. Secondly, if we decide to do something interestin

Re: [VOTE] KIP-229: DeleteGroups API

2018-01-16 Thread Colin McCabe
Thanks for this KIP, Vahid. +1 (non-binding). How about creating a GROUP_ID_NOT_FOUND error, rather than re-using INVALID_GROUP_ID here? INVALID_GROUP_ID is used to indicate that the group id itself is bad (contains invalid characters, is an empty string, etc.). A group ID not being found o

Re: [DISCUSS] KIP-228 Negative record timestamp support

2018-01-16 Thread Colin McCabe
I think we should just keep -1 as a special value, and allow negative timestamps. It just means one missing millisecond in 1969, right? There is just a lot of code everywhere checking for -1, and changing it now would be really likely to be buggy. We probably also want to limit the range of ti

Re: [VOTE] KIP-222 - Add "describe consumer group" to KafkaAdminClient

2018-01-18 Thread Colin McCabe
+1 (non-binding) Colin On Thu, Jan 18, 2018, at 07:36, Ted Yu wrote: > +1 > Original message From: Bill Bejeck > Date: 1/18/18 6:59 AM (GMT-08:00) To: dev@kafka.apache.org Subject: > Re: [VOTE] KIP-222 - Add "describe consumer group" to KafkaAdminClient > Thanks for the KI

  1   2   3   4   5   6   7   8   9   10   >