[jira] [Created] (KAFKA-14573) RoundRobinPartitioner doesn't work as expected with topic that has 6 partitions

2023-01-04 Thread Stephane Maarek (Jira)
Stephane Maarek created KAFKA-14573: --- Summary: RoundRobinPartitioner doesn't work as expected with topic that has 6 partitions Key: KAFKA-14573 URL: https://issues.apache.org/jira/browse/KAFKA-

Re: Preliminary blog post for the Apache Kafka 2.3.0 release

2019-06-25 Thread Stephane Maarek
Here it is: https://youtu.be/YutjYKSGd64 Thanks in advance!! On Tue., 25 Jun. 2019, 12:39 am Colin McCabe, wrote: > Hi Stephane, > > Sounds interesting! Do you have a link to the video you made for 2.3? > > best, > Colin > > > On Sun, Jun 23, 2019, at 15:10, Ste

Re: Preliminary blog post for the Apache Kafka 2.3.0 release

2019-06-23 Thread Stephane Maarek
The video is ready :) waiting for the release of Kafka 2.3 to make it public. @colin if you want to link it in the blog at some point that'd be great! On Wed., 19 Jun. 2019, 4:03 pm Ron Dagostino, wrote: > Looks great, Colin. > > I have also enjoyed Stephane Maarek's "What's New in Kafka..." ser

Re: [ANNOUNCE] Apache Kafka 2.2.0

2019-03-26 Thread Stephane Maarek
Congratulations on this amazing release! Lots of cool new features :) I've also released a YouTube video that will hopefully help the community get up to speed: https://www.youtube.com/watch?v=kaWbp1Cnfo4&t=5s Happy watching! On Tue, Mar 26, 2019 at 7:02 PM Matthias J. Sax wrote: > The Apache

Re: [VOTE] KIP-392: Allow consumers to fetch from the closest replica

2019-03-26 Thread Stephane Maarek
It's going to change quite a few things for learners, but this is an awesome idea! +1 (non-binding) On Tue, Mar 26, 2019 at 3:35 PM Viktor Somogyi-Vass wrote: > +1 (non-binding) very good proposal. > > On Mon, Mar 25, 2019 at 7:19 PM David Arthur wrote: > > > +1 > > > > Thanks, Jason! > > > > O

Kafka-1194

2019-03-20 Thread Stephane Maarek
Hi all Kafka-1194 has been a longstanding issue and now there seems to be a viable fix that also passes tests. I know Windows isn't a priority, but as I mentioned before, avoiding a fatal un-recoverable crash for newcomers learning Kafka is very important. I don't have enough core Kafka expertise

Re: [VOTE] 2.2.0 RC0

2019-02-24 Thread Stephane Maarek
Hi Matthias Thanks for this Running through the list of KIPs. I think this is not included in 2.2: - Allow clients to suppress auto-topic-creation Regards Stephane On Sun, Feb 24, 2019 at 1:03 AM Matthias J. Sax wrote: > Hello Kafka users, developers and client-developers, > > This is the fir

Re: [VOTE] KIP-382 MirrorMaker 2.0

2018-12-24 Thread Stephane Maarek
+1 ! Great stuff Stephane On Mon., 24 Dec. 2018, 12:07 pm Edoardo Comar +1 non-binding > > thanks for the KIP > -- > > Edoardo Comar > > IBM Event Streams > > > Harsha wrote on 21/12/2018 20:17:03: > > > From: Harsha > > To: dev@kafka.apache.org

Re: [ANNOUNCE] New committer: Colin McCabe

2018-09-25 Thread Stephane Maarek
Congrats Colin ! On Tue., 25 Sep. 2018, 3:33 pm Bill Bejeck, wrote: > Congrats Colin! > > On Tue, Sep 25, 2018 at 8:11 AM Manikumar > wrote: > > > Congrats Colin! > > > > On Tue, Sep 25, 2018 at 4:39 PM Mickael Maison > > > wrote: > > > > > Congratulations Colin! > > > > > > On Tue, Sep 25, 20

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

2018-09-12 Thread Stephane Maarek
> > > > > > >> >> > > > Best regards, > > > >> >> > > > > > > >> >> > > > Randall > > > >> >> > > > > > > >> >> >

Re: [EXTERNAL] [VOTE] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

2018-09-11 Thread Stephane Maarek
+1 (non binding) On Tue., 11 Sep. 2018, 10:48 am Mickael Maison, wrote: > +1 (non-binding) > Thanks for the KIP! > On Tue, Sep 11, 2018 at 8:40 AM McCaig, Rhys > wrote: > > > > Looks great Randall > > +1 (non-binding) > > > > > On Sep 9, 2018, at 7:17 PM, Gwen Shapira wrote: > > > > > > +1 > >

Re: KAFKA-1194

2018-09-03 Thread Stephane Maarek
el Juma wrote: > > > Hi Stephane, > > > > Does https://github.com/apache/kafka/pull/4947/files fix it for you? > > > > Ismael > > > > On Sun, Sep 2, 2018 at 11:25 AM Stephane Maarek < > > steph...@simplemachines.com.au> wrote: > > > > &

Re: KAFKA-1194

2018-09-03 Thread Stephane Maarek
discussion there? On 3 September 2018 at 08:24, Stephane Maarek < steph...@simplemachines.com.au> wrote: > Hi Ismael, thanks for having a look! > > https://github.com/apache/kafka/pull/4947/files does not fix it for me. > > But this one does: https://github.com/apache/ka

Re: KAFKA-1194

2018-09-02 Thread Stephane Maarek
ately. On 3 September 2018 at 03:26, Ismael Juma wrote: > Hi Stephane, > > Does https://github.com/apache/kafka/pull/4947/files fix it for you? > > Ismael > > On Sun, Sep 2, 2018 at 11:25 AM Stephane Maarek < > steph...@simplemachines.com.au> wrote: > > > Hi,

KAFKA-1194

2018-09-02 Thread Stephane Maarek
Hi, I've seen Dong has done some work on https://issues.apache.org/jira/browse/KAFKA-7278 which is said from the comments that it could have possibly fixed https://issues.apache.org/jira/browse/KAFKA-1194. I tested and it is unfortunately not the case... I have posted in KAFKA-1194 a way to repro

[jira] [Created] (KAFKA-7355) Topic Configuration Changes are not applied until reboot

2018-08-29 Thread Stephane Maarek (JIRA)
Stephane Maarek created KAFKA-7355: -- Summary: Topic Configuration Changes are not applied until reboot Key: KAFKA-7355 URL: https://issues.apache.org/jira/browse/KAFKA-7355 Project: Kafka

Re: [EXTERNAL] [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

2018-08-27 Thread Stephane Maarek
ng" even though technically it is > not) > >>>>> shouldn't be done blindly due to the partitioning impacts, and IIRC > you > >>>>> can't reduce them (which we could verify before applying). Also, I > >>> don't > >&

Re: [EXTERNAL] [VOTE] KIP-310: Add a Kafka Source Connector to Kafka Connect

2018-08-10 Thread Stephane Maarek
Hi Rhys, Overall I'm +1 (non binding), but you're going to need 3 binding votes for this KIP to pass. I don't feel there has been enough discussion on this from the community. Can we get some input from other people? Thanks for starting the vote nonetheless :) Stephane On 8 August 2018 at 20:28,

Re: [ANNOUNCE] Apache Kafka 2.0.0 Released

2018-07-30 Thread Stephane Maarek
at transform or react > > > > > > > > to the streams of data. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Apache Kafka is in use at large and small compani

Re: KIP-327: Add describe all topics API to AdminClient

2018-07-14 Thread Stephane Maarek
Why not paginate ? Then one can retrieve as many topics as desired ? On Sat., 14 Jul. 2018, 4:15 pm Colin McCabe, wrote: > Good point. We should probably have a maximum number of results like > 1000 or something. That can go in the request RPC as well... > Cheers, > Colin > > On Fri, Jul 13, 2

Re: [DISCUSS] KIP-318: Make Kafka Connect Source idempotent

2018-06-20 Thread Stephane Maarek
been addressed to > date: it's already possible for users to enable this feature, but doing it > by default has compatibility concerns. > > Thoughts? > > Best regards, > > Randall > > > On Wed, Jun 20, 2018 at 1:17 AM, Stephane Maarek < > steph...@simplemachin

[DISCUSS] KIP-318: Make Kafka Connect Source idempotent

2018-06-19 Thread Stephane Maarek
KIP link: https://cwiki.apache.org/confluence/display/KAFKA/KIP-318%3A+Make+Kafka+Connect+Source+idempotent By looking at the code, it seems Worker.java is where the magic happens, but do we need to also operate changes to KafkaBasedLog.java (~line 241) ? Love to hear your thoughts!

[jira] [Created] (KAFKA-7077) producerProps.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "5") producerProps.put(ProducerConfig.ENABLE_IDEMPOTENCE, true)

2018-06-19 Thread Stephane Maarek (JIRA)
Stephane Maarek created KAFKA-7077: -- Summary: producerProps.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "5") producerProps.put(ProducerConfig.ENABLE_IDEMPOTENCE, true) Key: KAFKA-7077

Re: [DISCUSS] KIP-317: Transparent Data Encryption

2018-06-18 Thread Stephane Maarek
Hi Sonke Very much needed feature and discussion. FYI the image links seem broken. My 2 cents (if I understood correctly): you say "This process will be implemented after Serializer and Interceptors are done with the message right before it is added to the batch to be sent, in order to ensure tha

Re: Are defaults serde in Kafka streams doing more harm then good ?

2018-06-16 Thread Stephane Maarek
Thank Guozhang! Let's pick this up in https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-7066 and https://github.com/apache/kafka/pull/5239 Cheers Stephane Kind regards, Stephane [image: Simple Machines] Stephane Maarek | Developer +61 416 575 980 steph...@simplemachines.c

[jira] [Created] (KAFKA-7066) Make Streams Runtime Error User Friendly in Case of Serialisation exception

2018-06-16 Thread Stephane Maarek (JIRA)
Stephane Maarek created KAFKA-7066: -- Summary: Make Streams Runtime Error User Friendly in Case of Serialisation exception Key: KAFKA-7066 URL: https://issues.apache.org/jira/browse/KAFKA-7066

Re: Are defaults serde in Kafka streams doing more harm then good ?

2018-06-13 Thread Stephane Maarek
release we have a Scala API for Streams where > default serdes are indeed dropped since with Scala we can safely rely on > implicit typing inference to override the serdes automatically. > > > > Guozhang > > > On Tue, Jun 12, 2018 at 6:32 PM, Stephane Maarek < > steph..

Are defaults serde in Kafka streams doing more harm then good ?

2018-06-12 Thread Stephane Maarek
Hi Coming from a user perspective, I see a lot of beginners not understanding the need for serdes and misusing the default serde settings. I believe default serdes do more harm than good. At best, they save a bit of boilerplate code but hide the complexity of serde happening at each step. At wors

Re: [DISCUSS] KIP-304: Connect runtime mode improvements for container platforms

2018-06-06 Thread Stephane Maarek
offer some insight?). > > > > > > Regarding Rahul's questions as to why these topics are a problem, I can > > > also re-iterate what Stephane said: it's a needless burden to manage. > > > Though I'm pretty sure different connectors can use the same topics, &g

Re: [DISCUSS] KIP-310: Add a Kafka Source Connector to Kafka Connect

2018-06-06 Thread Stephane Maarek
nd regards, Stephane [image: Simple Machines] Stephane Maarek | Developer +61 416 575 980 steph...@simplemachines.com.au simplemachines.com.au Level 2, 145 William Street, Sydney NSW 2010 On 5 June 2018 at 09:04, McCaig, Rhys wrote: > Hi All, > > As I didn’t get any comment on this KI

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

2018-05-21 Thread Stephane Maarek
+1 non binding On Mon., 21 May 2018, 2:44 pm Rajini Sivaram, wrote: > Hi Piyush, Thanks for the KIP! > > +1 (binding) > > Regards, > > Rajini > > On Sun, May 20, 2018 at 2:53 PM, Andy Coates wrote: > > > Awesome last minute effort Piyush. > > > > Really appreciate your time and input, > > > > A

Re: [DISCUSS] KIP-304: Connect runtime mode improvements for container platforms

2018-05-17 Thread Stephane Maarek
Fat fingered too... "Connect source should be able to achieve the same centrality of offsets " On Thu., 17 May 2018, 10:27 pm Stephane Maarek, < steph...@simplemachines.com.au> wrote: > Say you have 50 connectors all with different ACLs and service account. > That&#x

Re: [DISCUSS] KIP-304: Connect runtime mode improvements for container platforms

2018-05-17 Thread Stephane Maarek
r Kafka > Connect.” > > -- > Rahul Singh > rahul.si...@anant.us > > Anant Corporation > > On May 17, 2018, 6:17 AM -0400, Stephane Maarek < > steph...@simplemachines.com.au>, wrote: > > Hi Salius > > > > I think you're on the money, but you&#x

Re: [DISCUSS] KIP-304: Connect runtime mode improvements for container platforms

2018-05-17 Thread Stephane Maarek
ge may be big to implement but probably worthwhile. I'd be happy to provide more "user feedback" on a PR, but probably won't be able to implement a PR myself. More than happy to discuss this Best, Stephane Kind regards, Stephane [image: Simple Machines] Stephane Maarek |

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

2018-05-01 Thread Stephane Maarek
Hi, thanks for this badly needed feature 1) Why introduce two new APIs in authorizer instead of replacing the implementation for simple ACL authorizer with adding the wildcard capability? 2) is there an impact to performance as now we're evaluating more rules ? A while back I had evaluated the co

Re: [VOTE] KIP-235 Add DNS alias support for secured connection

2018-04-26 Thread Stephane Maarek
+1 as a user BUT I am no security expert. I have experienced that issue while setting up a cluster and while I would have liked a feature like that (I opened a JIRA at the time), I always guessed that the reason was because of some security protection. Now from a setup point of view this helps a

Re: [VOTE] KIP-277 - Fine Grained ACL for CreateTopics API

2018-04-25 Thread Stephane Maarek
+1 On Thu., 26 Apr. 2018, 4:19 am Ted Yu, wrote: > +1 > > On Wed, Apr 25, 2018 at 3:24 PM, Guozhang Wang wrote: > > > Thanks Edo, +1 from me. > > > > > > Guozhang > > > > On Wed, Apr 25, 2018 at 2:45 AM, Edoardo Comar > wrote: > > > > > Hi, > > > > > > The discuss thread on KIP-277 ( > > > htt

Re: [VOTE] Kafka 2.0.0 in June 2018

2018-04-19 Thread Stephane Maarek
+1 (non binding). Thanks Ismael! On Thu., 19 Apr. 2018, 2:47 pm Gwen Shapira, wrote: > +1 (binding) > > On Wed, Apr 18, 2018 at 11:35 AM, Ismael Juma wrote: > > > Hi all, > > > > I started a discussion last year about bumping the version of the June > 2018 > > release to 2.0.0[1]. To reiterate

Re: Dead link in Kafka Streams documentation

2018-04-07 Thread Stephane Maarek
Thanks Guozhang, everything works as expected now! Kind regards, Stephane [image: Simple Machines] Stephane Maarek | Developer +61 416 575 980 steph...@simplemachines.com.au simplemachines.com.au Level 2, 145 William Street, Sydney NSW 2010 On 8 April 2018 at 06:38, Guozhang Wang wrote: >

Dead link in Kafka Streams documentation

2018-04-06 Thread Stephane Maarek
https://kafka.apache.org/documentation/streams/developer-guide/ "Testing a Streams application" points to https://kafka.apache.org/documentation/streams/developer-guide/testing.html Which is a dead link. The working link is https://kafka.apache.org/11/documentation/streams/developer-guide/testin

Re: [DISCUSS] KIP-277 - Fine Grained ACL for CreateTopics API

2018-03-29 Thread Stephane Maarek
Not against, but this needs to support regex for support for Kafka streams application that create many topics with complex names On Thu., 29 Mar. 2018, 7:21 pm Edoardo Comar, wrote: > Hi all, > > We have submitted KIP-277 to give users permission to manage the lifecycle > of a defined set of to

Re: Kafka version and JDK version

2018-03-20 Thread Stephane Maarek
n. > The bump to 2.0 also gives us an opportunity to drop the old Scala clients. > This will be a huge win in tech debt reduction so the project will be able > to move faster after that. > > Ismael > > On Tue, 20 Mar 2018, 20:24 Stephane Maarek, < > steph...@simplemachi

Kafka version and JDK version

2018-03-20 Thread Stephane Maarek
Hi If I remember correctly, Kafka 2.0 is targeted this summer as it'll drop support for java 7 and dropping a java version is supposed to imply a major version bump in Kafka. Now that Java has a very quick release cycle for JDK (version 10 today), my question is: how fast will Kafka versioning go

Re: [DISCUSS] KIP-263: Allow broker to skip sanity check of inactive segments on broker startup

2018-02-27 Thread Stephane Maarek
This is great and definitely needed. I'm not exactly sure of what goes in the process of checking log files at startup, but is there something like signature checks of files (especially closed, immutable ones) that can be saved on disk and checked against at startup ? Wouldn't that help speed up bo

Re: [VOTE] KIP-171 - Extend Consumer Group Reset Offset for Stream Application

2018-02-27 Thread Stephane Maarek
Bootstrap servers vs broker list is the biggest hurdle when teaching to beginners. A standardized set of parameters would incredibly help On 27 Feb. 2018 9:44 am, "Matthias J. Sax" wrote: I agree on consistency, too. However, I am not sure if we should introduce an explicit --execute option. An

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

2018-02-27 Thread Stephane Maarek
Sounds awesome ! Are you planning to have auto scaling of partitions in a following KIP ? That would be the holy grail On 28 Feb. 2018 5:13 pm, "Dong Lin" wrote: > Hey Jan, > > I am not sure if it is acceptable for producer to be stopped for a while, > particularly for online application which r

RE: [DISCUSS]KIP-235 DNS alias and secured connections

2017-12-06 Thread Stephane Maarek
Hi Jonathan I think this will be very useful. I reported something similar here : https://issues.apache.org/jira/browse/KAFKA-4781 Please confirm your kip will address it ? Stéphane On 6 Dec. 2017 8:20 pm, "Skrzypek, Jonathan" wrote: > True, amended the KIP, thanks. > > Jonathan Skrzypek > Mi

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

2017-11-22 Thread Stephane Maarek
We have seen clusters with a few thousand topic getting idle CPU of 20 percent or more. These may be due to these fetch request. It seems your kip would address scalability by a lot in terms of (dormant) partitions, so I'm excited for this change On 22 Nov. 2017 10:01 pm, "Mickael Maison" wrote:

Re: [DISCUSS] Kafka 2.0.0 in June 2018

2017-11-09 Thread Stephane Maarek
I'm very happy with the milestones but worried about the versioning number. It seems it will mostly bring stuff out of deprecation vs actually bringing in breaking features. A 2.0 to me should bring something major to the table, possibly breaking, which would justify a big number hop. I'm still new

Re: [VOTE] KIP-201: Rationalising policy interfaces

2017-11-07 Thread Stephane Maarek
el Juma" wrote: The idea is that you only depend on a Java jar. The core jar includes the Scala version in the name and you should not care about that when implementing a Java interface. Ismael On Tue, Nov 7, 2017 at 10:37 AM, Stephane Maarek < step

Re: [VOTE] KIP-201: Rationalising policy interfaces

2017-11-07 Thread Stephane Maarek
do that and it's mostly compatible (users would only have to update their build dependency). Generally, all public APIs going forward will be in Java. Ismael On Tue, Nov 7, 2017 at 9:44 AM, Stephane Maarek < steph...@simplemachines.com.au> wrote: > Hi Tom, > > Regarding th

Re: [VOTE] KIP-201: Rationalising policy interfaces

2017-11-07 Thread Stephane Maarek
, Stephane [image: Simple Machines] Stephane Maarek | Developer +61 416 575 980 steph...@simplemachines.com.au simplemachines.com.au Level 2, 145 William Street, Sydney NSW 2010 On 7 November 2017 at 20:27, Tom Bentley wrote: > Hi Stephane, > > The vote on this KIP is on-going. >

Re: [VOTE] KIP-201: Rationalising policy interfaces

2017-11-07 Thread Stephane Maarek
rt of the kafka-clients package, and would love to see the next classes you're implementing appear in core/src/main/scala/kafka/policy or server/policy. Unless I'm missing something? Thanks for driving this Stephane Kind regards, Stephane [image: Simple Machines] Stephane Maarek | D

Re: [VOTE] KIP-215: Add topic regex support for Connect sinks

2017-11-03 Thread Stephane Maarek
+1! The S3 or hdfs connector will now be super powerful ! On 4 Nov. 2017 11:27 am, "Konstantine Karantasis" wrote: > Nice addition! > > +1 (non-binding) > > Konstantine > > On Fri, Nov 3, 2017 at 4:52 PM, Jeff Klukas wrote: > > > So sorry for skirting the process there. I wasn't aware of the 72

Re: [DISCUSS] KIP-217: Expose a timeout to allow an expired ZK session to be re-created

2017-11-02 Thread Stephane Maarek
h this KIP, we will keep retrying the > > creation of Zookeeper until success. > > > > Thanks, > > > > Jun > > > > On Tue, Oct 31, 2017 at 3:28 PM, Stephane Maarek < > > steph...@simplemachines.com.au> wrote: > > > > > Hi Jun, > > &

Re: [DISCUSS] KIP-217: Expose a timeout to allow an expired ZK session to be re-created

2017-11-01 Thread Stephane Maarek
P, we will keep retrying the > creation of Zookeeper until success. > > Thanks, > > Jun > > On Tue, Oct 31, 2017 at 3:28 PM, Stephane Maarek < > steph...@simplemachines.com.au> wrote: > > > Hi Jun, > > > > Thanks for the reply. > > > > 1) The re

Re: [DISCUSS] KIP-217: Expose a timeout to allow an expired ZK session to be re-created

2017-10-31 Thread Stephane Maarek
x the issue of unresolved host. In some of the cases, human intervention is needed. 3) The default timeout is infinite though. Jun On Sat, Oct 28, 2017 at 11:48 PM, Stephane Maarek < steph...@simplemachines.com.au> wrote: > Hi Jun, >

Re: [DISCUSS] KIP-217: Expose a timeout to allow an expired ZK session to be re-created

2017-10-28 Thread Stephane Maarek
Hi Jun, I think this is very helpful. Restarting Kafka brokers in case of zookeeper host change is not a well known operation. Few questions: 1) would it not be worth fixing the problem at the source ? This has been stuck for a while though, maybe a little push would help : https://issues.apache.

[jira] [Created] (KAFKA-6092) Time passed in punctuate call is currentTime, not punctuate schedule time.

2017-10-19 Thread Stephane Maarek (JIRA)
Stephane Maarek created KAFKA-6092: -- Summary: Time passed in punctuate call is currentTime, not punctuate schedule time. Key: KAFKA-6092 URL: https://issues.apache.org/jira/browse/KAFKA-6092

Kafka Connect Source questions

2017-10-11 Thread Stephane Maarek
Hi, I had a look at the Connect Source Worker code and have two questions: When a Source Task commits offsets, does it perform compaction / optimisation before sending off? E.g.  I read from 1 source partition, and I read 1000 messages. Will the offset flush send 1000 messages to the offset s

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

2017-10-04 Thread Stephane Maarek
xisting topic, I do think it's advantageous to at least notify the connector (or task) prior to the first record sent to a given topic so that the connector can fail or issue a warning if it doesn't meet its requirements. Best regards, Randall On

[jira] [Resolved] (KAFKA-5993) Kafka Admin Client throws a warning for sasl.jaas.config

2017-10-03 Thread Stephane Maarek (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephane Maarek resolved KAFKA-5993. Resolution: Not A Problem > Kafka Admin Client throws a warning for sasl.jaas.con

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

2017-10-03 Thread Stephane Maarek
Hi Randall, Thanks for the KIP. I like it What happens when the target topic is already created but the configs do not match? i.e. wrong RF, num partitions, or missing / additional configs? Will you attempt to apply the necessary changes or throw an error? Thanks! Stephane On 24/5/17, 5:59

[jira] [Created] (KAFKA-6007) Connect can't validate against transforms in plugins.path

2017-10-03 Thread Stephane Maarek (JIRA)
Stephane Maarek created KAFKA-6007: -- Summary: Connect can't validate against transforms in plugins.path Key: KAFKA-6007 URL: https://issues.apache.org/jira/browse/KAFKA-6007 Project:

[jira] [Created] (KAFKA-5993) Kafka AdminClient does not support standard security settings

2017-09-28 Thread Stephane Maarek (JIRA)
Stephane Maarek created KAFKA-5993: -- Summary: Kafka AdminClient does not support standard security settings Key: KAFKA-5993 URL: https://issues.apache.org/jira/browse/KAFKA-5993 Project: Kafka

[jira] [Created] (KAFKA-5992) Better Java Documentation for AdminClient Exceptions

2017-09-28 Thread Stephane Maarek (JIRA)
Stephane Maarek created KAFKA-5992: -- Summary: Better Java Documentation for AdminClient Exceptions Key: KAFKA-5992 URL: https://issues.apache.org/jira/browse/KAFKA-5992 Project: Kafka Issue

Speeding up Kafka Start cold startup time

2017-09-26 Thread Stephane Maarek
Hi, We have our Kafka Brokers in AWS backed by st1 EBS drives. These are optimised for throughput On a kafka restart (warm restart), if some of the appropriate data is in the pagecache, everything goes well and the broker boots in just a few seconds. (1000 partitions) On a cold start (new

[jira] [Created] (KAFKA-5851) CPU overhead to having partitions (even inactive)

2017-09-07 Thread Stephane Maarek (JIRA)
Stephane Maarek created KAFKA-5851: -- Summary: CPU overhead to having partitions (even inactive) Key: KAFKA-5851 URL: https://issues.apache.org/jira/browse/KAFKA-5851 Project: Kafka Issue

Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days

2017-08-09 Thread Stephane Maarek
Any interest on having offsets.retention.minutes= log.retention.(ms|minutes|hours) as a dynamic setting if not set, and having the option the override to a constant value? That would address different types of deployments as well, who modify the default log retention period On 10/8/17, 5:11 a

Re: Minimum Replication Factor

2017-06-23 Thread Stephane Maarek
IBM Message Hub eco...@uk.ibm.com IBM UK Ltd, Hursley Park, SO21 2JN From: Stephane Maarek To: "dev@kafka.apache.org" Date: 23/06/2017 01:48 Subject:Minimum Replication Factor Hi all,

Minimum Replication Factor

2017-06-22 Thread Stephane Maarek
Hi all, Interested in getting people’s opinion on something. The problem I have is that some people launch streams app in our cluster but forget to set a replication factor > 1. Then it’s a pain to increase the topic’s RF, when we do notice some topic partitions go offline because we reboot

[jira] [Created] (KAFKA-5468) Change Source offset commit message to info to match Sink behaviour

2017-06-18 Thread Stephane Maarek (JIRA)
Stephane Maarek created KAFKA-5468: -- Summary: Change Source offset commit message to info to match Sink behaviour Key: KAFKA-5468 URL: https://issues.apache.org/jira/browse/KAFKA-5468 Project: Kafka

[jira] [Commented] (KAFKA-5261) Performance improvement of SimpleAclAuthorizer

2017-05-17 Thread Stephane Maarek (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16014945#comment-16014945 ] Stephane Maarek commented on KAFKA-5261: this and possibly {code} private

[jira] [Updated] (KAFKA-5261) Performance improvement of SimpleAclAuthorizer

2017-05-17 Thread Stephane Maarek (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephane Maarek updated KAFKA-5261: --- Summary: Performance improvement of SimpleAclAuthorizer (was: Performance improvement of

[jira] [Created] (KAFKA-5261) Performance improvement of SimpleAclAuthoriser

2017-05-17 Thread Stephane Maarek (JIRA)
Stephane Maarek created KAFKA-5261: -- Summary: Performance improvement of SimpleAclAuthoriser Key: KAFKA-5261 URL: https://issues.apache.org/jira/browse/KAFKA-5261 Project: Kafka Issue Type

Re: [VOTE] KIP-146: Isolation of dependencies and classes in Kafka Connect (restarted voting thread)

2017-05-08 Thread Stephane Maarek
+1 Thanks heaps I can’t wait! On 9/5/17, 4:48 am, "Konstantine Karantasis" wrote: ** Restarting the voting thread here, with a different title to avoid collapsing this thread's messages with the discussion thread's messages in mail clients. Apologies for the inconvenience. **

Re: [VOTE] KIP-154 Add Kafka Connect configuration properties for creating internal topics

2017-05-08 Thread Stephane Maarek
+1 (non binding) On 9/5/17, 5:51 am, "Randall Hauch" wrote: Hi, everyone. Given the simple and non-controversial nature of the KIP, I would like to start the voting process for KIP-154: Add Kafka Connect configuration properties for creating internal topics: h

Re: [DISCUSS] KIP-146: Classloading Isolation in Connect

2017-05-03 Thread Stephane Maarek
m adding my replies below Ewen's comments. On Tue, May 2, 2017 at 10:15 PM, Ewen Cheslack-Postava wrote: > On Tue, May 2, 2017 at 10:01 PM, Stephane Maarek < > steph...@simplemachines.com.au> wrote: > > Excellent feedback, Stephane!

Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface

2017-05-03 Thread Stephane Maarek
efinitely think that the constructors could have been better > > designed, > > >>> but I think given that they're in heavy use I don't think this > proposal > > >>> will improve things. Deprecating constructors just leaves everyone > with

Re: [DISCUSS] KIP-146: Classloading Isolation in Connect

2017-05-02 Thread Stephane Maarek
Thanks for the work, it’s definitely needed! I’d like to suggest to take it one step further. To me, I’d like to see Kafka Connect the same way we have Docker and Docker repositories. Here’s how I would envision the flow: - Kafka Connect workers are just workers. They come with no jars whatsoeve

Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface

2017-04-30 Thread Stephane Maarek
of warnings and crossed out things. We can't actually delete the >>> methods because lots of code needs to be usable across multiple Kafka >>> versions, right? So we aren't picking between the original approach >> (worse) >>> and the

Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface

2017-04-28 Thread Stephane Maarek
petual > mingling of the original style and the new style with a bunch of deprecated > stuff, which I think is worst of all. > > I'd vote for just documenting the meaning of null in the ProducerRecord > constructor. >

Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface

2017-04-22 Thread Stephane Maarek
Good call. That’s when I heavily miss Scala Case classes and options. You get clarity on optional vs mandatory fields, one constructor, and immutability. If losing Immutability is an issue, then the with pattern is a no-go and then I’ll just add a missing constructor the way Ismael described it.

Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface

2017-04-21 Thread Stephane Maarek
t 12:57 AM, Stephane Maarek < steph...@simplemachines.com.au> wrote: > Matthias: I was definitely on board with you at first, but Ismael made the > comment that for: > > public ProducerRecord(String topic, K key, V value, Integer partition) > public Produc

[jira] [Updated] (KAFKA-5092) KIP 141 - ProducerRecord Interface Improvements

2017-04-21 Thread Stephane Maarek (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephane Maarek updated KAFKA-5092: --- Description: See KIP here: https://cwiki.apache.org/confluence/display/KAFKA/KIP+141

[jira] [Updated] (KAFKA-5092) KIP 141 - ProducerRecord Interface Improvements

2017-04-21 Thread Stephane Maarek (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephane Maarek updated KAFKA-5092: --- Summary: KIP 141 - ProducerRecord Interface Improvements (was: KIP 141

Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface

2017-04-21 Thread Stephane Maarek
cerRecord(topic, key, value).withTimestamp(ts).withForcedPartition(p); WDYT? -Matthias On 4/20/17 4:57 PM, Stephane Maarek wrote: > Matthias: I was definitely on board with you at first, but Ismael made the comment that for: > > public ProducerRecord(Str

Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface

2017-04-20 Thread Stephane Maarek
(String topic, V value) > > You can then remove the withValue and withTopic methods > > I also think withForcedPartition should just be withPartition > > Thanks, > Damian > > On Wed, 19 Apr 2017 at 23:34 Stephane Maarek

[jira] [Updated] (KAFKA-5092) KIP 141 - ProducerRecordBuilder Interface

2017-04-19 Thread Stephane Maarek (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephane Maarek updated KAFKA-5092: --- Summary: KIP 141 - ProducerRecordBuilder Interface (was: KIP 141- ProducerRecordBuilder

[jira] [Updated] (KAFKA-5092) KIP 141- ProducerRecordBuilder Interface improvements

2017-04-19 Thread Stephane Maarek (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephane Maarek updated KAFKA-5092: --- Summary: KIP 141- ProducerRecordBuilder Interface improvements (was: Producer record

[jira] [Created] (KAFKA-5092) Producer record interface

2017-04-19 Thread Stephane Maarek (JIRA)
Stephane Maarek created KAFKA-5092: -- Summary: Producer record interface Key: KAFKA-5092 URL: https://issues.apache.org/jira/browse/KAFKA-5092 Project: Kafka Issue Type: Improvement

[DISCUSS] KIP 141 - ProducerRecordBuilder Interface

2017-04-19 Thread Stephane Maarek
Hi all, My first KIP, let me know your thoughts! https://cwiki.apache.org/confluence/display/KAFKA/KIP+141+-+ProducerRecordBuilder+Interface Cheers, Stephane

Re: Credentials for Confluence

2017-04-12 Thread Stephane Maarek
It’s simplesteph Thanks for adding me! On 13 April 2017 at 9:28:35 am, Gwen Shapira (g...@confluent.io) wrote: What's your Apache wiki username? On Mon, Apr 10, 2017 at 11:48 PM, Stephane Maarek wrote: > Hi, > > I’d like to create a KIP on Confluence but according to Matth

Credentials for Confluence

2017-04-10 Thread Stephane Maarek
Hi, I’d like to create a KIP on Confluence but according to Matthias Sax I need credentials. Can you please provide me some? Looking forward to redacting my first KIP :) Regards, Stephane

[jira] [Commented] (KAFKA-3436) Speed up controlled shutdown.

2017-03-28 Thread Stephane Maarek (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946392#comment-15946392 ] Stephane Maarek commented on KAFKA-3436: [~onurkaraman] I can't find

[jira] [Comment Edited] (KAFKA-3083) a soft failure in controller may leave a topic partition in an inconsistent state

2017-03-28 Thread Stephane Maarek (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946384#comment-15946384 ] Stephane Maarek edited comment on KAFKA-3083 at 3/29/17 1:4

[jira] [Commented] (KAFKA-3083) a soft failure in controller may leave a topic partition in an inconsistent state

2017-03-28 Thread Stephane Maarek (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15946384#comment-15946384 ] Stephane Maarek commented on KAFKA-3083: Just hit this issue in prod, and

[jira] [Commented] (KAFKA-4943) SCRAM secret's should be better protected with Zookeeper ACLs

2017-03-23 Thread Stephane Maarek (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939916#comment-15939916 ] Stephane Maarek commented on KAFKA-4943: [~rsivaram] if Zookeeper has only c

[jira] [Comment Edited] (KAFKA-4942) Kafka Connect: Offset committing times out before expected

2017-03-23 Thread Stephane Maarek (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939453#comment-15939453 ] Stephane Maarek edited comment on KAFKA-4942 at 3/23/17 11:5

[jira] [Commented] (KAFKA-4942) Kafka Connect: Offset committing times out before expected

2017-03-23 Thread Stephane Maarek (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15939453#comment-15939453 ] Stephane Maarek commented on KAFKA-4942: [~ewencp] nothing weird from the

  1   2   >