Re: Kafka Command Line & Shell

2014-10-17 Thread Colin
+100 -- Colin Clark +1-320-221-9531 > On Oct 17, 2014, at 7:10 PM, Steve Morin wrote: > > Joe I think this is great! > >> On Fri, Oct 17, 2014 at 5:03 PM, Joe Stein wrote: >> >> Hi, I have been thinking about the ease of use for operations with Kafka. >&g

Re: Compile failure going from kafka 0.8.1.1 to 0.8.2

2014-10-30 Thread Colin
+1 on two methods -- Colin Clark +1-320-221-9531 > On Oct 30, 2014, at 11:24 AM, Jonathan Weeks wrote: > > +1 on the two methods suggestion > > -JW > >> On Oct 30, 2014, at 9:20 AM, Jay Kreps wrote: >> >> But Jun, >> >> That change breaks

Re: [ANNOUNCE] New Committer: Manikumar Reddy

2018-10-12 Thread Colin McCabe
Congratulations, Manikumar! Well done. best, Colin On Fri, Oct 12, 2018, at 01:25, Edoardo Comar wrote: > Well done Manikumar ! > -- > > Edoardo Comar > > IBM Event Streams > IBM UK Ltd, Hursley Park, SO21 2JN > >

Re: [VOTE] KIP-376: Implement AutoClosable on appropriate classes that want to be used in a try-with-resource statement

2018-10-12 Thread Colin McCabe
n particular, it would be good to highlight that we are not renaming or changing the existing "close" methods. Under "rejected alternatives" we could explain why we chose to implement AutoCloseable rather than Closeable. cheers, Colin On Thu, Oct 11, 2018, at 13:48

Re: [DISCUSS] KIP-377: TopicCommand to use AdminClient

2018-10-12 Thread Colin McCabe
ist topics under deletion" feature when using AdminClient at the moment. We could add a note to the tool output indicating this. We should move the protocol discussion to a separate thread. Probably also look at KIP-142 as well. best, Colin On Tue, Oct 9, 2018, at 07:45, Viktor Somogyi-V

Re: [VOTE] KIP-349 Priorities for Source Topics

2018-10-12 Thread Colin McCabe
can't really think of any time I would need topic priorities if I was muting topics and offloading blocking operations in a reasonable way. It would be good to identify use-cases because it would motivate choices like how many priorities do we want (2? 256? 4 billion?) and what the API w

Re: [VOTE] KIP-376: Implement AutoClosable on appropriate classes that want to be used in a try-with-resource statement

2018-10-12 Thread Colin McCabe
On Fri, Oct 12, 2018, at 15:45, Yishun Guan wrote: > Hi Colin, > > Thanks for your suggestions. I have modified the current KIP with your > comments. However, I still think I should keep the entire list, because it > is a good way to keep track of which class need to be change, a

Re: KAFKA-6654 custom SSLContext

2018-10-15 Thread Colin McCabe
o the SSLContext and SSLSocketFactory seems like it would make that kind of compatibility harder, if it were even still possible. I'm curious if there's a way to do this by adding configuration entries for what you need? best, Colin On Mon, Oct 15, 2018, at 13:20, Pellerin, Clement wr

Re: KAFKA-6654 custom SSLContext

2018-10-15 Thread Colin McCabe
Hi Clement, Thanks for the clarification. Perhaps a pluggable interface makes sense here. Maybe someone more familiar with the SSL code can comment. best, Colin On Mon, Oct 15, 2018, at 19:53, Pellerin, Clement wrote: > OK, I can see why passing an instance is not language neutral. &g

Re: [DISCUSS] KIP-377: TopicCommand to use AdminClient

2018-10-20 Thread Colin McCabe
't need). best, Colin On Tue, Oct 16, 2018, at 10:11, Viktor Somogyi-Vass wrote: > Hi Colin, > > Thanks, it makes sense and simplifies this KIP tremendously. I'll move this > section to the rejected alternatives with a note that KIP-142 will have > this feature. > On

Re: [VOTE] KIP-349 Priorities for Source Topics

2018-10-21 Thread Colin McCabe
On Thu, Oct 18, 2018, at 09:23, n...@afshartous.com wrote: > > > > On Oct 12, 2018, at 5:06 PM, Colin McCabe wrote: > > > > Maybe there's some really cool use-case that I haven't thought of. But so > > far I can't really think of any time I wou

Re: Throwing away prefetched records optimisation.

2018-10-21 Thread Colin McCabe
is a KafkaConsumer#wakeup method which might help here. > * Wakeup the consumer. This method is thread-safe and is useful in > particular to abort a long poll. best, Colin On Sun, Oct 21, 2018, at 14:59, Matthias J. Sax wrote: > It's spread out multiple classes... > >

Re: Throwing away prefetched records optimisation.

2018-10-23 Thread Colin McCabe
. I have a hard time thinking of a case where we actually want to throw away data for paused partitions. If you're still subscribed to it, presumably you'll eventually unpause it and use the cache, right? It makes sense for unsubscribe to clear those records, but not pause, as far as I can

Re: Throwing away prefetched records optimisation.

2018-10-24 Thread Colin McCabe
context for everything. best, Colin On Tue, Oct 23, 2018, at 23:21, Zahari Dichev wrote: > Colin, I agree > > I will take a closer looks at the Fetcher itself and see whether that is > feasible and update my KIP accordingly. I guess we can label this one, the > official discussion t

Re: [VOTE] KIP-377: TopicCommand to use AdminClient

2018-10-24 Thread Colin McCabe
Thanks, Viktor. +1 (binding). One note: can we add a deprecation warning when --zookeeper is used, to indicate that this option will be phased out in the future? best, Colin On Wed, Oct 24, 2018, at 05:47, Mickael Maison wrote: > +1 (non-binding) > Thanks for the KIP! > On Wed, Oct

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

2018-10-25 Thread Colin McCabe
med, or we unsubscribe from the partition. This would avoid adding a new configuration. best, Colin On Sun, Oct 21, 2018, at 11:54, Zahari Dichev wrote: > Hi there, although it has been discussed briefly already in this thread > <https://lists.apa

Re: [DISCUSS] KIP-385: Provide configuration allowing consumer to no throw away prefetched data

2018-10-26 Thread Colin McCabe
nk you did an excellent job raising this issue and discussing it. It's a very good contribution to the project even if you don't end up writing the patch yourself. I'm going to take a look at the patch today. If you want to take a look, that would also be good. best, Colin On

Re: [VOTE] KIP-349 Priorities for Source Topics

2018-10-26 Thread Colin McCabe
cause starvation by design, because the high priority partition always takes priority. In general the best solution would probably be to have a work queue between the consumer and the event handler, and manage the backpressure as appropriate. This could be done with pause and resume, as Str

Re: Kafka-node: Kafka client keeps sending request to broker which went down

2018-10-26 Thread Colin McCabe
o not recommend using this client. https://github.com/Blizzard/node-rdkafka is a better choice. Keep in mind that node-rdkafka is still not part of the Apache Kafka project, so you might want to discuss any issues with that particular project. best, Colin On Tue, Oct 23, 2018, at 22:04, Shashan

Re: [DISCUSS] How hard is it to separate the logic layer and the storage layer of Kafka broker?

2018-11-02 Thread Colin McCabe
ions. This is great for writing, but not so good if you want to read historical data from a single topic. regards, Colin

Re: [VOTE] KIP-374: Add '--help' option to all available Kafka CLI commands

2018-11-08 Thread Colin McCabe
+1 (binding) On Wed, Oct 31, 2018, at 05:42, Srinivas Reddy wrote: > Hi All, > > I would like to call for a vote on KIP-374: > https://cwiki.apache.org/confluence/x/FgSQBQ > > Summary: > Currently, the '--help' option is recognized by some Kafka commands > but not all. To provide a consistent us

Re: Kafka Performance Producer - Extension

2018-11-08 Thread Colin McCabe
ment, and help with feedback on how > well this could be integrated. I'm sorry, but I do not open MS Word attachments because of security issues. best, Colin > > > These features would help any team load testing kafka brokers/schema > registry components efficiently

Re: [DISCUSS] KIP-388 Add observer interface to record request and response

2018-11-08 Thread Colin McCabe
should also use java here to avoid the compatibility issues we have had with Scala APIs in the past. best, Colin On Thu, Nov 8, 2018, at 11:34, radai wrote: > another downside to client instrumentation (beyond the number of > client codebases one would need to cover) is that in a large &g

Re: Heads up: javac warnings are now treated as errors

2018-11-14 Thread Colin McCabe
Sounds good. Thanks for cleaning up the warnings! Colin On Mon, Nov 12, 2018, at 22:29, Ismael Juma wrote: > Hi all, > > As part of KAFKA-7612, all javac warnings were fixed or suppressed. To> > prevent them from reappearing, javac warnings are now treated as > errors. W

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

2018-11-16 Thread Colin McCabe
t be spoofed or accidentally reused by a misconfigured client. best, Colin On Fri, Nov 16, 2018, at 00:04, Jason Gustafson wrote: > > > > If we initialize a set of member names (I assume ids = names here) on > > broker through Admin API, the client needs to pick up this informat

Re: [VOTE] KIP-354 Time-based log compaction policy

2018-12-04 Thread Colin McCabe
fference.max.ms. best, Colin On Mon, Nov 26, 2018, at 13:10, xiongqi wu wrote: > Thanks for binding and non-binding votes. > Can I get one more binding vote? > > Thanks in advance! > > Xiongqi (Wesley) Wu > > > On Wed, Nov 14, 2018 at 7:29 PM Matt Farmer wrote:

Re: [VOTE] KIP-354 Time-based log compaction policy

2018-12-05 Thread Colin McCabe
Thanks, Xiongqi Wu. +1 (binding) regards, Colin On Tue, Dec 4, 2018, at 20:58, xiongqi (wesley) wu wrote: > Colin, > > Thanks for comments. > Out of ordered message timestamp is a very good point. > We can combine max.compaction.lag.ms with > log.message.timestamp.di

Re: Problem in CI for pull request

2018-12-21 Thread Colin McCabe
Try typing "retest this please" as a comment to the PR. best, Colin On Wed, Nov 28, 2018, at 11:05, lk gen wrote: > Hi, > > I made a pull request and it passed CI on JDK 11 but failed on JDK 8 > > I think the JDK 8 error may not related to my commit but an enviro

Re: [DISCUSS] KIP-388 Add observer interface to record request and response

2018-12-21 Thread Colin McCabe
that we don't really need to perform. This issue has shown up a lot when we profile Kafka performance. Since the amount of extra garbage scales with the number of partitions, it's especially bad in large enterprise clusters. best, Colin > > > On Sun, Nov 18, 2018 at 9:35 PM Don

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

2018-12-21 Thread Colin McCabe
hope we don't keep changing the ZK path each time there is a version bump. best, Colin On Thu, Nov 29, 2018, at 14:25, Sönke Liebau wrote: > This has been dormant for a while now, can I interest anybody in chiming in > here? > > I think we need to come up with an idea of ho

Re: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message Consumption Across Partitions in KafkaConsumer

2018-12-22 Thread Colin McCabe
A B C D... etc. order always repeats. best, Colin On Wed, Dec 19, 2018, at 09:00, ChienHsing Wu wrote: > Looking back the email thread I think one of the comments from Mayuresh > was the question about needing KIP for this change or not as the > KafkaConsumer does not guarantee the end u

Re: [VOTE] KIP-349 Priorities for Source Topics

2019-01-10 Thread Colin McCabe
some examples. Or perhaps we should consider some kind of API to do pluggable buffering on the client side. In any case, this needs more discussion. We need to be clear and definite about what use cases we want to solve, and the tradeoffs we're making to solve them. For now, I have to

Re: [VOTE] KIP-349 Priorities for Source Topics

2019-01-15 Thread Colin McCabe
On Sun, Jan 13, 2019, at 18:13, n...@afshartous.com wrote: > Thanks Colin and Mathias. > > > On Jan 12, 2019, at 8:27 PM, Matthias J. Sax wrote: > > > > Thus, I would suggest to limit this KIP to the consumer only, otherwise, > > the scope will be too large and thi

Re: [DISCUSS] KIP-402: Improve fairness in SocketServer processors

2019-01-15 Thread Colin McCabe
uess it is unfair that a lot of incoming connections can swamp the network threads right now. But limiting the rate of new connections is unfair to people connecting. Overall the goal seems to be usability, not fairness. best, Colin On Tue, Jan 15, 2019, at 04:27, Rajini Sivaram wrote: > H

[DISCUSS] 2.1.1 bug-fix release

2019-01-15 Thread Colin McCabe
20%3D%20KAFKA%20AND%20status%20in%20(Resolved%2C%20Closed)%20AND%20fixVersion%20%3D%202.1.1%20%20 Please find the Release plan: https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+2.1.1 regards, Colin

Re: [DISCUSS] KIP-402: Improve fairness in SocketServer processors

2019-01-15 Thread Colin McCabe
On Tue, Jan 15, 2019, at 12:59, Rajini Sivaram wrote: > Hi Colin, > > `AcceptorIdlePercent` indicates the total amount of time the acceptor is > inactive and not accepting any connections because it is blocked on > Processors. But I agree the name could be improved. There is bac

Re: [VOTE] KIP-402: Improve fairness in SocketServer processors

2019-01-16 Thread Colin McCabe
+1 (binding) Thanks, Colin On Tue, Jan 15, 2019, at 18:55, Gwen Shapira wrote: > +1 > Thank you! > > > On Tue, Jan 15, 2019, 3:38 PM Rajini Sivaram > > > Hi all, > > > > I would like to start vote on KIP-402 to improve fairness in channel> > > &g

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

2018-05-07 Thread Colin McCabe
On Fri, May 4, 2018, at 05:49, Viktor Somogyi wrote: > Hi Colin, > > > Rather than breaking compatibility, we should simply add a new > > "incremental" boolean to AlterConfigsOptions. Callers can set this boolean > > to true when they want the update to be

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

2018-05-09 Thread Colin McCabe
Hi Molnar, The points Ismael brought up earlier (and that were brought up on KIP-30) are still relevant here. As Ismael said, the goal is to get rid of external dependencies here. We're going to post more about this soon (sorry for the delay) thanks, Colin On Wed, May 9, 2018, at

Re: [DISCUSS] KIP-278: Add version option to Kafka's commands

2018-05-09 Thread Colin McCabe
+1. Thanks, Sasaki. Colin On Wed, May 9, 2018, at 09:15, Jason Gustafson wrote: > Hi Sasaki, > > Thanks for the update. The KIP looks good to me. I'd suggest moving to a > vote. > > Thanks, > Jason > > On Mon, May 7, 2018 at 7:08 AM, Sasaki Toru >

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

2018-05-10 Thread Colin McCabe
o create a pattern that matches only "foo*". You have to type "foo\*" It would be required if we forced users to specify a version, as well. best, Colin > > Sent from my iPhone > > > On 7 May 2018, at 05:16, Piyush Vijay wrote: > > > > Makes se

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

2018-05-11 Thread Colin McCabe
Hi Andy, I see what you mean. I guess my thought here is that if the fields are private, we can change it later if we need to. I definitely agree that we should use the scheme you describe for sending ACLs over the wire (just the string + version number) cheers, Colin On Fri, May 11, 2018

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

2018-05-11 Thread Colin McCabe
On Wed, May 9, 2018, at 05:41, Viktor Somogyi wrote: > Hi Colin, > > > We are going to need to create a new version of AlterConfigsRequest to add > > the "incremental" boolean. So while we're doing that, maybe we can change > > the type to NULLAB

Re: [VOTE] KIP-278: Add version option to Kafka's commands

2018-05-11 Thread Colin McCabe
+1 (non-binding) Colin On Fri, May 11, 2018, at 12:35, Attila Sasvári wrote: > +1 (non-binding) > > Thomas Crayford ezt írta (időpont: 2018. máj. > 11., P 18:20): > > > +1 (non-binding) > > > > On Fri, May 11, 2018 at 5:17 PM, Guozhang Wang wrote: >

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

2018-05-15 Thread Colin McCabe
ce name, it should be an error, unless they are escaped with a backslash. That way, we can use them in the future. We should create a Resource.escapeName function which adds the correct escape characters to resource names (so it would translate foo* into foo\*, foo+bar into foo\+bar, etc. et

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-15 Thread Colin McCabe
ing key, String value); > } With regard to ConfigTransformer: do we need to include all this code in the KIP? Seems like an implementation detail. > Other connectors such as the S3 connector are tightly coupled with a > particular secret manager, and may > wish to handle rotation o

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

2018-05-16 Thread Colin McCabe
&, *. > > On Tue, May 15, 2018 at 7:15 PM, Piyush Vijay > > wrote: > > > >> Hi Colin, > >> > >> Escaping at this level is making sense to me but let me think more and get > >> back to you. Thanks, Piyush. What questions do you think a

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-16 Thread Colin McCabe
callback with delayMs set, you don't know what the new values will be, only that an update is coming, but not yet here. best, Colin On Wed, May 16, 2018, at 17:05, Robert Yokota wrote: > Hello everyone, > > After a good round of discussions with excellent feedback and no major

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-16 Thread Colin McCabe
, Colin On Wed, May 16, 2018, at 19:13, Robert Yokota wrote: > Hi Colin, > > Good questions. > > > > As a clarification about the indirections, what if I have the > > connect> configuration key foo set up as ${vault:bar}, and in Vault, > have the bar> key set to ${fi

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

2018-05-16 Thread Colin McCabe
the ConfigCommand tool for now, and leave AC unchanged. Best, Colin On Wed, May 16, 2018, at 04:57, Viktor Somogyi wrote: > Hi Colin, > > > Doing get-merge-set is buggy, though. If someone else does get-merge- > > set at the same time as you, you might overwrite that person'

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

2018-05-17 Thread Colin McCabe
names. So you can feed this method your "foo\*" consumer group name, and it will give you "foo\\\*", which is what you would need to use to create an ACL for this consumer group in AdminClient. I think that's the only change we need to admin client regards, Colin On T

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-17 Thread Colin McCabe
Or is this like a period after which a password is automatically rotated? On Wed, May 16, 2018, at 22:25, Robert Yokota wrote: > Hi Colin, > > > With regard to delayMs, can’t we just restart the > > Connector when the keys are actually changed? > > Currently the VaultConfigProv

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

2018-05-17 Thread Colin McCabe
e KIP. cheers, Colin On Wed, May 16, 2018, at 22:06, Colin McCabe wrote: > Hi Viktor, > > The shell command isn’t that easy to integrate into applications. > AdminClient will get integrated into a lot more stuff, which > increases the potential for conflicts. I would argue that

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

2018-05-17 Thread Colin McCabe
ard-suffix ACLs on Zk. +1. Thanks, Piyush. Colin > > > > On Thu, May 17, 2018 at 9:15 AM Colin McCabe wrote: > > > Thanks, Piyush. +1 for starting the vote soon. > > > > Can you please also add a discussion about escaping? For example, earlier >

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

2018-05-17 Thread Colin McCabe
obably just create a flag for alterConfigs which marks it as incremental, like we discussed earlier, and do this as a compatible change that is needed for the shell command. best, Colin On Thu, May 17, 2018, at 09:32, Colin McCabe wrote: > Hi Viktor, > > Since the KIP freeze is coming up

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

2018-05-17 Thread Colin McCabe
Thanks, Robert! +1 (non-binding) Colin On Thu, May 17, 2018, at 14:15, Robert Yokota wrote: > Hi Colin, > > I've changed the KIP to have a composite object returned from get(). It's > probably the most straightforward option. Please let me know if you have > any ot

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

2018-05-18 Thread Colin McCabe
Hi Viktor, Thanks, this looks good. The boolean should default to false if not set, to ensure that existing clients continue to work as-is, right? Might be good to add a note specifying that. +1 (non-binding) best, Colin On Fri, May 18, 2018, at 08:16, Viktor Somogyi wrote: > Updated

Re: [VOTE] KIP-290: Support for wildcard suffixed ACLs

2018-05-21 Thread Colin McCabe
rly explain how each type works. best, Colin On Mon, May 21, 2018, at 13:40, Piyush Vijay wrote: > Thanks for the +1s. I'll send out a PR shortly. > > Thanks > > > Piyush Vijay > > On Mon, May 21, 2018 at 8:59 AM, Guozhang Wang wrote: > > > Thanks fo

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

2018-05-21 Thread Colin McCabe
Oops, will post on the vote thread. best, Colin On Mon, May 21, 2018, at 21:15, Colin McCabe wrote: > On Mon, May 21, 2018, at 04:53, Andy Coates wrote: > > Hey Piyush, > > > > Thanks for the updated KIP! Couple of minor points from me: > > > > When storing w

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

2018-05-21 Thread Colin McCabe
inevitably comes up with a new NameType. Does this still have a chance to get in, or has the KIP window closed? I would argue with one or two minor changes it's ready to go. Pretty much all of the compatibility problems are solved with the separate ZK hierarchy. best, Colin >

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

2018-06-05 Thread Colin McCabe
their API calls. best, Colin On Tue, Jun 5, 2018, at 15:29, Ted Yu wrote: > bq. we were already doing with request.timeout.ms > > I would vote for using existing config. > > Any new config parameter needs to go thru long process of digestion: > documentation, etc in order

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

2018-06-05 Thread Colin McCabe
be set to something like max(1000, 0.5 * request.timeout.ms) best, Colin On Tue, Jun 5, 2018, at 16:21, Ted Yu wrote: > bq. we must make the API timeout longer than the RPC timeout > > I agree with the above. > > How about adding a fixed duration on top of request.timeout.ms

Re: Someone to review KAFKA-6919, one line change for faulty documentation

2018-06-05 Thread Colin McCabe
Thanks, Koen. I was on vacation, so I missed this originally. But I'll review it now. cheers, Colin On Fri, Jun 1, 2018, at 13:02, Koen De Groote wrote: > Greetings, > > Poking for someone to have a quick look at this, It's a one-line change. I > noticed the documenta

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

2018-06-05 Thread Colin McCabe
ple tries if the RPC hangs due to network issues. It should not be longer. best, Colin > > Cheers > > On Tue, Jun 5, 2018 at 4:27 PM, Colin McCabe wrote: > > > I don't think it can be fixed. The RPC duration is something that you > > might reasonably want to

Re: [VOTE] KIP-266: Add TimeoutException for KafkaConsumer#position

2018-06-08 Thread Colin McCabe
alue. otherwise... B. If request.timeout.ms is longer than 2 minutes, we set default.api.timeout.ms to request.timeout.ms + 1500. otherwise... we set default.api.timeout.ms to request.timeout.ms best, Colin > > > Guozhang > > > On Tue, Jun 5, 2018 at 5:18 PM, Ted Yu wrot

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

2018-06-11 Thread Colin McCabe
Sounds good. Thanks, Konstantin. Colin On Mon, Jun 11, 2018, at 13:41, Rajini Sivaram wrote: > Hi Konstantine, > > Sounds reasonable to me too. > > Regards, > > Rajini > > On Mon, Jun 11, 2018 at 7:55 PM, Robert Yokota wrote: > > > Hi Ko

Re: SASL Unit test failing

2018-06-26 Thread Colin McCabe
ggability(SaslAuthenticatorTest.java:257) best, Colin On Mon, Jun 25, 2018, at 18:02, Ted Yu wrote: > I ran the test on Linux as well. > > cat /etc/redhat-release > CentOS Linux release 7.2.1511 (Core) > > Java version: 1.8.0_161, vendor: Oracle Corporation > Java home: /jdk1.8.0_161/jre >

Re: [VOTE] KIP-324: Add method to get metrics() in AdminClient

2018-06-26 Thread Colin McCabe
Can you add a little more explanation to the KIP for why you are adding this method? Is it something streams needs, for example? Will it help other applications that use admin client and want to expose metrics? What are the thread-safety guarantees for the map which is returned? best, Colin

Re: [VOTE] KIP-324: Add method to get metrics() in AdminClient

2018-06-27 Thread Colin McCabe
On Tue, Jun 26, 2018, at 13:24, Yishun Guan wrote: > Hi Colin, > > I agree with what Guozhang's opinion that because all the other clients > have it (producer, consumer..) and this will gain more visibility for those > application that use admin client. (Now I added this sen

Re: [VOTE] KIP-324: Add method to get metrics() in AdminClient

2018-06-27 Thread Colin McCabe
P.S. +1 (non-binding) once you add the info about it being thread-safe. best, On Wed, Jun 27, 2018, at 15:23, Colin McCabe wrote: > On Tue, Jun 26, 2018, at 13:24, Yishun Guan wrote: > > Hi Colin, > > > > I agree with what Guozhang's opinion that because all the

Re: Kafka system tests contribution

2018-07-10 Thread Colin McCabe
Hi Andriy, Try looking at the logs to see why the test failed. best, Colin On Wed, May 16, 2018, at 07:08, Andriy Sorokhtey wrote: > Hi, > > Did anyone had a chance to take a look at this issue? > > 2018-05-08 15:01 GMT+03:00 Andriy Sorokhtey : > > > Hello Kafka

[DISCUSS]: KIP-339: Create a new ModifyConfigs API

2018-07-11 Thread Colin McCabe
ok at https://cwiki.apache.org/confluence/display/KAFKA/KIP-339%3A+Create+a+new+ModifyConfigs+API best, Colin

Re: [DISCUSS]: KIP-339: Create a new ModifyConfigs API

2018-07-13 Thread Colin McCabe
Hi Ted, That’s a fair question. I think the main reason I didn’t propose that originally is that many people find null values in maps confusing. Also, some newer java maps don’t support null values, such as ConcuurentHashMap. I’m curious what others think about this. Best, Colin On Wed, Jul 11

Re: [DISCUSS]: KIP-339: Create a new ModifyConfigs API

2018-07-13 Thread Colin McCabe
ng. > Anyway, to signify that null value is supported, value type can be > declared as Optional.> > FYI Yeah, now that we're on Java 8, Optional could be a good choice here. best, Colin > On Fri, Jul 13, 2018 at 5:35 PM Colin McCabe > wrote:>> __ >> Hi Te

Re: KIP-327: Add describe all topics API to AdminClient

2018-07-13 Thread Colin McCabe
ferent releases of the JDK (there were some changes in java 8, for example). Also, re2 regular expressions are linear time, never exponential time. See https://github.com/google/re2j regards, Colin On Fri, Jul 13, 2018, at 05:00, Andras Beni wrote: > The KIP looks good to me. >

Re: KIP-327: Add describe all topics API to AdminClient

2018-07-13 Thread Colin McCabe
Good point. We should probably have a maximum number of results like 1000 or something. That can go in the request RPC as well... Cheers, Colin On Fri, Jul 13, 2018, at 18:15, Ted Yu wrote: > bq. describe topics by a regular expression on the server side > > Should caution be taken if

Re: KIP-327: Add describe all topics API to AdminClient

2018-07-15 Thread Colin McCabe
ave a huge load on the network and the broker from sending all that unnecessary data. With server-side regular expressions, you could only send the stuff you need. best, Colin On Sat, Jul 14, 2018, at 01:06, Stephane Maarek wrote: > Why not paginate ? Then one can retrieve as many topics as desi

Re: KIP-327: Add describe all topics API to AdminClient

2018-07-15 Thread Colin McCabe
Thanks, Manikumar. I've been meaning to bring up KIP-142 again. It would definitely be a nice improvement. best, Colin On Sat, Jul 14, 2018, at 08:51, Manikumar wrote: > Hi Jason and Colin, > > Thanks for the feedback. I agree that having filtering support to the > Meta

Re: [DISCUSS]: KIP-339: Create a new ModifyConfigs API

2018-07-16 Thread Colin McCabe
On Fri, Jul 13, 2018, at 23:20, Dong Lin wrote: > Hey Colin, > > Thanks for the KIP! > > It seems that the AlterConfigsResult is pretty much the same as > ModifyConfigsResult. Instead of adding ModifyConfigs API and deprecating > AlterConfigs API, would it be simpler to just

Re: [DISCUSS]: KIP-339: Create a new ModifyConfigs API

2018-07-16 Thread Colin McCabe
Hi Magnus, Thanks for taking a look. On Mon, Jul 16, 2018, at 11:43, Magnus Edenhill wrote: > Thanks for driving this KIP, Colin. > > I agree with Dong that a new similar modifyConfigs API (and protocol API) > is confusing and that > we should try to extend the current alterConfi

Re: KIP-327: Add describe all topics API to AdminClient

2018-07-17 Thread Colin McCabe
Thanks, Manikumar. best, Colin On Tue, Jul 17, 2018, at 19:44, Manikumar wrote: > Closing this KIP in favor of adding filtering support to the Metadata API > and KIP-142. Will open a new KIP when ready. > Thanks for your reviews. > > On Mon, Jul 16, 2018 at 8:38 AM Coli

Re: [VOTE] KIP-338 Support to exclude the internal topics in kafka-topics.sh command

2018-07-19 Thread Colin McCabe
+1 (non-binding) cheers, Colin On Mon, Jul 16, 2018, at 01:10, Chia-Ping Tsai wrote: > hi folks, > > The discussion[1] of KIP-338[2] did not get any objection for last 6 > days so it is time to start the voting thread. > > Thanks for your time! > > [1] > https://l

Re: [DISCUSS]: KIP-339: Create a new ModifyConfigs API

2018-07-20 Thread Colin McCabe
27; idea of supporting "append" and "subtract" on configuration keys that contain lists. best, Colin On Mon, Jul 16, 2018, at 14:12, Colin McCabe wrote: > Hi Magnus, > > Thanks for taking a look. > > On Mon, Jul 16, 2018, at 11:43, Magnus Edenhill wrote: > &g

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

2018-07-25 Thread Colin McCabe
ead or > write) the log directory. > - Broker will be offline if all log directories are bad. > - Broker will stop serving replicas in any bad log directory. New replicas > will only be created > on good log directory. The behavior Stanislav is proposing for the log cleaner is actual

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

2018-07-27 Thread Colin McCabe
nd eventually have to do more protocol downgrading, etc. best, Colin On Thu, Jul 26, 2018, at 11:50, Vahid S Hashemian wrote: > Hi Jason, > > That makes sense. > I have updated the KIP based on the recent feedback. > > Thanks! > --Vahid > > > > > From: Jason

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

2018-07-30 Thread Colin McCabe
Hi Manikumar, It's great that you are taking a look at this! Much needed. Just one note: I assume that --authorizer-properties is no longer required if the --bootstrap-server option is specified. We should probably spell this out somewhere in the KIP. thanks, Colin On Mon, Jul 30,

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

2018-08-02 Thread Colin McCabe
ce." Most of the ones I've looked at only affect one or two partitions. In particular the ones that resulted from over-eagerness to use 32-bit math on 64-bit values. If the log cleaner is so buggy that it's useless (the scenario you're describing), and you want to p

Re: [DISCUSS] KIP-289: Improve the default group id behavior in KafkaConsumer

2018-08-02 Thread Colin McCabe
Thanks, Jason. I don't have a very strong opinion on this. But like you said, if we skip bumping the RPC versions, this would be a smaller change, which might be good. best, Colin On Wed, Aug 1, 2018, at 17:43, Jason Gustafson wrote: > Hey Vahid, > > I talked with Colin of

Re: [DISCUSS] KIP-332: Update AclCommand to use AdminClient API

2018-08-02 Thread Colin McCabe
+1 for starting the vote cheers, Colin On Wed, Aug 1, 2018, at 08:46, Manikumar wrote: > Hi all, > > If there are no concerns, I will start the voting process soon. > > Thanks > > On Tue, Jul 31, 2018 at 9:08 AM Manikumar wrote: > > > Hi Colin, > >

Re: [DISCUSS] KIP-348 Eliminate null from SourceTask#poll()

2018-08-03 Thread Colin McCabe
l and the empty list will be supported. Perhaps you should discuss this in the compatibility section? Also, the KIP should probably be renamed "deprecate null" rather than "eliminate null" since it will still be possible to return null here, right? best, Colin On Tue, Jul

Re: [DISCUSS] KIP-350: Allow kafka-topics.sh to take brokerid as parameter to show partitions associated with it

2018-08-03 Thread Colin McCabe
Thanks for the KIP, Ratish. We should probably specify that it is an error when --broker is specified with operations other than --list, right? best, Colin On Wed, Aug 1, 2018, at 21:28, Ratish Ravindran wrote: > Hi, > > I would like to open a discussion thread on KIP-350: &

Re: [VOTE] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

2018-08-03 Thread Colin McCabe
he record, not with any of those those things. So we should choose a name that reflects that serialization is the problem. How about RecordSerializationException? best, Colin On Thu, Aug 2, 2018, at 15:11, Stanislav Kozlovski wrote: > Hi Jason and Ted, > > @Jason > I did not opp

Re: [VOTE] KIP-332: Update AclCommand to use AdminClient API

2018-08-03 Thread Colin McCabe
+1 (non-binding) regards, Colin On Fri, Aug 3, 2018, at 02:27, Rajini Sivaram wrote: > Hi Manikumar, > > +1 (binding) > > Thanks for the KIP! > > On Fri, Aug 3, 2018 at 3:46 AM, Ted Yu wrote: > > > +1 > > > > On Thu, Aug 2, 2018 at 7:33

Re: [DISCUSS] KIP-240: AdminClient.listReassignments AdminClient.describeReassignments

2018-01-09 Thread Colin McCabe
partition reassignments in a timely fashion, this avoids the "polling an AdminClient API in a tight loop" antipattern. It allows watchers to be notified in a simple and natural way about what is going on. Access can be controlled by the existing topic ACL mechanisms. best, Colin O

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

2018-01-09 Thread Colin McCabe
. Because of the (in my opinion, mistaken) decision to allow control characters in UNIX filenames, even echoing a file name to your terminal is a security vulnerability. best, Colin > > In general my default approach (without context of a specific system) would > be to accept anything t

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

2018-01-09 Thread Colin McCabe
t seems like, with the current proposal, the broker will use the value found in the local configuration (config file) rather than the new ZK version. Should the broker print out a WARN message in that scenario? best, Colin > 2. Should Reconfigurable extend Configurable or is the initial >

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

2018-01-09 Thread Colin McCabe
Thanks, Rajini. That makes sense. regards, Colin On Tue, Jan 9, 2018, at 14:38, Rajini Sivaram wrote: > Hi Colin, > > Thank you for reviewing. > > Yes, validation is done on the broker, not the client. > > All configs from ZooKeeper are processed and any config that c

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

2018-01-12 Thread Colin McCabe
ior, so no issue with it. > TL/DR: I'm happy either way :) > > Colin brought up control characters and that we should disallow these > in connector names. The specific one that is mentioned in his link is > Ascii 27 - ESC - \e so one possibility would be to explicitly > blacklist

Re: Vote for KIP-245: Use Properties instead of StreamsConfig in KafkaStreams constructor

2018-01-16 Thread Colin McCabe
something interesting with StreamsConfig later, we still have it. For example, we could have a builder, or some interesting ways to serialize it or send it to a string. best, Colin On Tue, Jan 16, 2018, at 09:54, Matthias J. Sax wrote: > Thanks for updating the KIP. > > I am recasting

Re: [VOTE] KIP-229: DeleteGroups API

2018-01-16 Thread Colin McCabe
are > we going to add a `--delete` option on the `kafka-consumer-groups.sh` > script? I thought it is "yes" but the Proposed Changes section does not > explicitly mention it. +1 for a --delete option on kafka-consumer-groups.sh regards, Colin > > > Guozhang > >

  1   2   3   4   5   6   7   8   9   10   >