[jira] [Created] (KAFKA-8153) Streaming application with state stores takes up to 1 hour to restart

Michael Melsen created KAFKA-8153: - Summary: Streaming application with state stores takes up to 1 hour to restart Key: KAFKA-8153 URL: https://issues.apache.org/jira/browse/KAFKA-8153 Project: Kafka

Source connectors stops working after the broker crashes.

Configuration: 3 br 1 zk 1 connect worker(distributed mode) 8 connectors(sink and source) Description of connect topics: [image: Снимок экрана 2019-03-22 в 17.00.56.png] Environment: K8s, gcloud, confluent images After crash of first broker there are several messages that broker is not a

Re: [VOTE] KIP-349 Priorities for Source Topics

Hi Colin, that is definitely a good option and will cover 90% of all use cases (probaby more). However strictly speaking it only addresses one half of the issue unless I am mistaken. The internal behavior of the KafkaConsumer (which partition the fetcher gets data from next and which buffered dat

Re: [VOTE] KIP-396: Add Commit/List Offsets Operations to AdminClient

Bumping this thread once again Ismael, have I answered your questions? While this has received a few non-binding +1s, no committers have voted yet. If you have concerns or questions, please let me know. Thanks On Mon, Feb 11, 2019 at 11:51 AM Mickael Maison wrote: > > Bumping this thread as it'

Re: [DISCUSS] KIP-409: Allow creating under-replicated topics and partitions

Thanks Colin for the feedback. The idea was to allow both users and administrator to decide if they wanted to opt-in and if so under what conditions. Maybe we could do something simpler and just allow the creation if at least min-in-sync replicas are available? That should not require changes to

Re: [DISCUSS] KIP-411: Add option to make Kafka Connect task client ID values unique

Randall, Paul, the proposal looks great, thanks. Ryanne On Mon, Mar 25, 2019, 9:03 AM Randall Hauch wrote: > Paul, > > Thanks for updating the KIP with the proposal. I do think the KIP should at > least mention that the prior behavior is to allow the worker to override > the `producer.client.id

Re: [DISCUSS] KIP-411: Add option to make Kafka Connect task client ID values unique

Paul, Thanks for updating the KIP with the proposal. I do think the KIP should at least mention that the prior behavior is to allow the worker to override the `producer.client.id` or `consumer.client.id`, which is entirely possible (though unlikely since there would be an MBean conflict, as pointe

Re: [DISCUSSION] KIP-418: A method-chaining way to branch KStream

Paul, I see your point when you are talking about stream..branch..branch...default.. Still, I believe that this cannot not be implemented the easy way. Maybe we all should think further. Let me comment on two of your ideas. user could specify a terminal method that assumes nothing will re

[jira] [Resolved] (KAFKA-8014) Extend Connect integration tests to add and remove workers dynamically

[ https://issues.apache.org/jira/browse/KAFKA-8014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randall Hauch resolved KAFKA-8014. -- Resolution: Fixed > Extend Connect integration tests to add and remove workers dynamically > --

Create SSL Kafka AdminClient object using keystore.jks from database

I need to create kafkaAmdinClient SSL object using java.security.keystore object. i.e I have to read the keystore and truststore files from database as a clob and get the keystore in java.security.keystore object and use this to create Admin client object. I am able to create AdminClient obj

[VOTE] KIP-392: Allow consumers to fetch from the closest replica

Hi All, discussion on the KIP seems to have died down, so I'd like to go ahead and start a vote. Here is a link to the KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Allow+consumers+to+fetch+from+closest+replica . +1 from me (duh) -Jason

Re: [VOTE] KIP-392: Allow consumers to fetch from the closest replica

+1 (non-binding) Great stuff, thanks. Ryanne On Mon, Mar 25, 2019, 11:08 AM Jason Gustafson wrote: > Hi All, discussion on the KIP seems to have died down, so I'd like to go > ahead and start a vote. Here is a link to the KIP: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-392%3A+Al

[jira] [Resolved] (KAFKA-8150) Fix bugs in handling null arrays in generated RPC code

[ https://issues.apache.org/jira/browse/KAFKA-8150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin P. McCabe resolved KAFKA-8150. Resolution: Fixed Fix Version/s: 2.2.1 The code path that this fixes isn't used in 2

Re: [VOTE] KIP-392: Allow consumers to fetch from the closest replica

+1 (binding). A very well written proposal and a pleasant read. Thanks Jason! Guozhang On Mon, Mar 25, 2019 at 9:07 AM Jason Gustafson wrote: > Hi All, discussion on the KIP seems to have died down, so I'd like to go > ahead and start a vote. Here is a link to the KIP: > > https://cwiki.apache

Re: [VOTE] KIP-392: Allow consumers to fetch from the closest replica

+1 (non-binding) Thanks for updating the KIP and addressing my previous comments. Eno On Mon, Mar 25, 2019 at 4:35 PM Ryanne Dolan wrote: > +1 (non-binding) > > Great stuff, thanks. > > Ryanne > > On Mon, Mar 25, 2019, 11:08 AM Jason Gustafson wrote: > > > Hi All, discussion on the KIP seems t

Re: [VOTE] KIP-392: Allow consumers to fetch from the closest replica

+1 Thanks, Jason! On Mon, Mar 25, 2019 at 1:23 PM Eno Thereska wrote: > +1 (non-binding) > Thanks for updating the KIP and addressing my previous comments. > > Eno > > On Mon, Mar 25, 2019 at 4:35 PM Ryanne Dolan > wrote: > > > +1 (non-binding) > > > > Great stuff, thanks. > > > > Ryanne > > >

[jira] [Created] (KAFKA-8154) Buffer Overflow exceptions between brokers and with clients

Rajesh Nataraja created KAFKA-8154: -- Summary: Buffer Overflow exceptions between brokers and with clients Key: KAFKA-8154 URL: https://issues.apache.org/jira/browse/KAFKA-8154 Project: Kafka

Jenkins build is back to normal : kafka-1.1-jdk7 #254

See

Re: MirrorMaker 2.0 and Streams interplay (topic naming control)

Hi Paul, Thanks for the email. This does seem like a good setup to support. This might seem a little low-fi, but do you think it would work to handle this use case entirely in "user space"? I may be missing something because this is off the cuff... In the code for your Streams app, I'm wondering

Re: MirrorMaker 2.0 and Streams interplay (topic naming control)

Hey Paul, thanks for the kind words re MM2. I'm not a Streams expert first off, but I think I understand your question: if a Streams app can switch between topics with and without a cluster alias prefix when you migrate between prod and pre-prod, while preserving state. Streams supports regexes an

[jira] [Created] (KAFKA-8155) Update Streams system tests for 2.2.0 and 2.1.1 releases

John Roesler created KAFKA-8155: --- Summary: Update Streams system tests for 2.2.0 and 2.1.1 releases Key: KAFKA-8155 URL: https://issues.apache.org/jira/browse/KAFKA-8155 Project: Kafka Issue Ty

Request to add recent versions to the s3 bucket

Hi Kafka Devs, While updating the Streams system tests, I noticed that 2.1.1 and 2.2.0 are not uploaded to https://s3-us-west-2.amazonaws.com/kafka-packages yet. The instructions in tests/kafkatest/tests/streams/streams_upgrade_test.py say to ping this list! Can someone with access take care of t

Re: Request to add recent versions to the s3 bucket

I'm a bit surprised that these are the instructions. The release process doesn't even mention S3, IIRC. Can you use an official Apache repo for testing? ( https://www.apache.org/dyn/closer.cgi?path=/kafka/) On Mon, Mar 25, 2019 at 1:59 PM John Roesler wrote: > Hi Kafka Devs, > > While updating

Re: [DISCUSS] KIP-360: Improve handling of unknown producer

Hi John What is the status of this KIP? My teammates and I are running into the "UNKNOWN_PRODUCER_ID" error on 2.1.1 for a multitude of our internal topics, and I suspect that a proper fix is needed. Adam On Mon, Jan 7, 2019 at 7:42 PM Guozhang Wang wrote: > Thanks Jason. The proposed solutio

Re: [DISCUSS] KIP-360: Improve handling of unknown producer

Ach - Sorry. I meant Jason. I had just read a John Roesler email. On Mon, Mar 25, 2019 at 5:21 PM Adam Bellemare wrote: > Hi John > > What is the status of this KIP? > > My teammates and I are running into the "UNKNOWN_PRODUCER_ID" error on > 2.1.1 for a multitude of our internal topics, and I s

Re: Request to add recent versions to the s3 bucket

Thanks Gwen, I will try that! If we can remove a link from the chain, it would be a win. Thanks, -John On Mon, Mar 25, 2019 at 4:12 PM Gwen Shapira wrote: > I'm a bit surprised that these are the instructions. The release process > doesn't even mention S3, IIRC. > Can you use an official Apach

Re: [DISCUSS] KIP-236 Interruptible Partition Reassignment

Thanks Ismael.  One question, even switch to submitting reassignments via RPC instead of Zookeeper.  The reassignment data will still persist in ZooKeeper node /admin/reassign_partitions (e.g. when Controller failover it can resume reassignments)?  If yes, how this can keep someone from modifyi

an article on kafka vs pulsar

Hi Team, I like to share this article on kafka vs pulsar. https://www.infoworld.com/article/3379120/pubsub-messaging-apache-kafka-vs-apache-pulsar.html Your comments are valuable. Thank you./Sree

[jira] [Created] (KAFKA-8156) Client id when provided is not suffixed with an index

Nagaraj Gopal created KAFKA-8156: Summary: Client id when provided is not suffixed with an index Key: KAFKA-8156 URL: https://issues.apache.org/jira/browse/KAFKA-8156 Project: Kafka Issue Typ

Jenkins build is back to normal : kafka-1.0-jdk7 #265

See

Re: [DISCUSS] KIP-236 Interruptible Partition Reassignment

Hi George, The goal is not to prevent people from updating ZK directly. The goal is to offer a solution where people don't have to. If people then decide to avoid the recommended path, they can deal with the consequences. However, if we add another structure in ZK and no RPC mechanism, then there

Please add me to the contributor list

My JIRA userid is JBFletcher Thank you, Anna

Re: Please add me to the contributor list

Done. On 3/25/19 3:08 PM, am wrote: > My JIRA userid is JBFletcher > > Thank you, > Anna > signature.asc Description: OpenPGP digital signature

Re: MirrorMaker 2.0 and Streams interplay (topic naming control)

John and Ryanne, Thanks for the responses! I think Ryanne's way of describing the question is actually a much better summary than my long winded description: "a Streams app can switch between topics with and without a cluster alias prefix when you migrate between prod and pre-prod, while preservin