Re: How Kafka handle partition leader change?

2023-11-21 Thread Andrew Grant
st is > very likely during leadership change unless producer uses idempotency? Is > this a generic situation that no intent to provide data integration guarantee > upon metadata change? > > From: Andrew Grant > Sent: 20 November 2023 12:26 > T

Re: How Kafka handle partition leader change?

2023-11-20 Thread Andrew Grant
Hey De Gao, The controller is the one that always elects a new leader. When that happens that metadata is changed on the controller and once committed it’s broadcast to all brokers in the cluster. In KRaft this would be via a PartitonChange record that each broker will fetch from the controller

[jira] [Resolved] (KAFKA-15149) Fix not sending UMR and LISR RPCs in dual-write mode when there are new partitions

2023-07-07 Thread Andrew Grant (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Grant resolved KAFKA-15149. -- Resolution: Fixed > Fix not sending UMR and LISR RPCs in dual-write mode when there are

[jira] [Created] (KAFKA-15149) Fix not sending RPCs in dual-write mode when there are new partitions

2023-07-05 Thread Andrew Grant (Jira)
Andrew Grant created KAFKA-15149: Summary: Fix not sending RPCs in dual-write mode when there are new partitions Key: KAFKA-15149 URL: https://issues.apache.org/jira/browse/KAFKA-15149 Project: Kafka

Re: [DISCUSS] Adding non-committers as Github collaborators

2023-06-06 Thread Andrew Grant
t; >>> > > >>> > > >>> On Wed, May 31, 2023 at 6:46 PM Luke Chen wrote: > > >>> > > > >>> > Hi Greg and Andrew, > > >>> > > > >>> > Sorry, I don't know how to re-sent the invitation. > &g

[jira] [Resolved] (KAFKA-14791) Create a builder class for PartitionRegistration

2023-06-06 Thread Andrew Grant (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Grant resolved KAFKA-14791. -- Resolution: Fixed > Create a builder class for PartitionRegistrat

Re: [DISCUSS] Adding non-committers as Github collaborators

2023-05-31 Thread Andrew Grant
Hi all, Like Greg I also received an invitation to collaborate but was too slow to accept the invite :( I'm also wondering if there's a way to resend the invite? I'm andymg3 on GitHub. Thanks, Andrew On Tue, May 30, 2023 at 12:12 PM Greg Harris wrote: > Hey all, > > I received an invitation to c

[jira] [Created] (KAFKA-14829) Consolidate reassignment logic in PartitionReassignmentReplicas

2023-03-20 Thread Andrew Grant (Jira)
Andrew Grant created KAFKA-14829: Summary: Consolidate reassignment logic in PartitionReassignmentReplicas Key: KAFKA-14829 URL: https://issues.apache.org/jira/browse/KAFKA-14829 Project: Kafka

[jira] [Created] (KAFKA-14791) Create a builder class for PartitionRegistration

2023-03-07 Thread Andrew Grant (Jira)
Andrew Grant created KAFKA-14791: Summary: Create a builder class for PartitionRegistration Key: KAFKA-14791 URL: https://issues.apache.org/jira/browse/KAFKA-14791 Project: Kafka Issue Type

[jira] [Created] (KAFKA-14608) Make sure reassignment does not cause under min ISR

2023-01-08 Thread Andrew Grant (Jira)
Andrew Grant created KAFKA-14608: Summary: Make sure reassignment does not cause under min ISR Key: KAFKA-14608 URL: https://issues.apache.org/jira/browse/KAFKA-14608 Project: Kafka Issue

[jira] [Resolved] (KAFKA-14386) Change ReplicaPlacer place method to return a class instead of list of list of integers

2022-12-10 Thread Andrew Grant (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Grant resolved KAFKA-14386. -- Resolution: Fixed > Change ReplicaPlacer place method to return a class instead of list

[jira] [Created] (KAFKA-14456) Fix AdminUtils startIndex for rack aware partition creations

2022-12-08 Thread Andrew Grant (Jira)
Andrew Grant created KAFKA-14456: Summary: Fix AdminUtils startIndex for rack aware partition creations Key: KAFKA-14456 URL: https://issues.apache.org/jira/browse/KAFKA-14456 Project: Kafka

[jira] [Created] (KAFKA-14437) Enhance StripedReplicaPlacer to account for existing partition assignments

2022-12-02 Thread Andrew Grant (Jira)
Andrew Grant created KAFKA-14437: Summary: Enhance StripedReplicaPlacer to account for existing partition assignments Key: KAFKA-14437 URL: https://issues.apache.org/jira/browse/KAFKA-14437 Project

[jira] [Created] (KAFKA-14386) Change ReplicaPlacer to return a class instead of list of list of integers

2022-11-13 Thread Andrew Grant (Jira)
Andrew Grant created KAFKA-14386: Summary: Change ReplicaPlacer to return a class instead of list of list of integers Key: KAFKA-14386 URL: https://issues.apache.org/jira/browse/KAFKA-14386 Project

Re: [DISCUSS] KIP-866 ZooKeeper to KRaft Migration

2022-10-24 Thread Andrew Grant
Hey David, Thanks for the KIP. I had a few small questions. “The ZK data migration will copy the existing ZK data into the KRaft metadata log and establish the new KRaft active controller as the active controller from a ZK perspective.” How will the code get all the ZooKeeper config? Will it d

[jira] [Resolved] (KAFKA-13892) Dedupe RemoveAccessControlEntryRecord in deleteAcls of AclControlManager

2022-05-10 Thread Andrew Grant (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Grant resolved KAFKA-13892. -- Resolution: Fixed > Dedupe RemoveAccessControlEntryRecord in deleteAcls of AclControlMana

[jira] [Created] (KAFKA-13892) Dedupe response objects in deleteAcls of AclControlManager

2022-05-10 Thread Andrew Grant (Jira)
Andrew Grant created KAFKA-13892: Summary: Dedupe response objects in deleteAcls of AclControlManager Key: KAFKA-13892 URL: https://issues.apache.org/jira/browse/KAFKA-13892 Project: Kafka

[jira] [Created] (KAFKA-13889) Broker can't handle ACCESS_CONTROL_ENTRY_RECORD quickly followed by REMOVE_ACCESS_CONTROL_ENTRY_RECORD for same ACL

2022-05-09 Thread Andrew Grant (Jira)
Andrew Grant created KAFKA-13889: Summary: Broker can't handle ACCESS_CONTROL_ENTRY_RECORD quickly followed by REMOVE_ACCESS_CONTROL_ENTRY_RECORD for same ACL Key: KAFKA-13889 URL: https://issues.apache.org

Re: Why does Kafka have a higher throughput than Redis?

2021-10-14 Thread Andrew Grant
M PROCESSING > PIPELINES by Sebastian Tallberg) > showing Kafka's throughput hitting 3x the amount of msg/s if compared to > Redis for a 1kB payload. I would like to understand what is in Kafka's > architecture that allows it to be a lot faster than other message > brokers/Redis in particular > > Thanks! > -- Andrew Grant 8054482621

Re: Fetching rows from DB and insert it into multiple topics

2021-05-07 Thread Andrew Grant
Hi, I’d look into Kafka streams https://kafka.apache.org/documentation/streams/. You could use your connector to dump all records into a single topic and then write a streams application that can use arbitrary logic to route records to different downstream topics. A nice benefit, in my opinion,