[jira] [Created] (KAFKA-19212) Fix ElectionWasClean method bug when partition reassignment completes

2025-04-28 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-19212: -- Summary: Fix ElectionWasClean method bug when partition reassignment completes Key: KAFKA-19212 URL: https://issues.apache.org/jira/browse/KAFKA-19212 Project: Kafka

Re: [VOTE] KIP-1152: Add transactional ID pattern filter to ListTransactions API

2025-04-28 Thread Calvin Liu
t 10:12 AM Jun Rao wrote: > > > Hi, Calvin, > > > > Thanks for the KIP. +1 > > > > Jun > > > > On Thu, Apr 24, 2025 at 8:30 AM Calvin Liu > > wrote: > > > > > Hi, > > > I would like to start a vote on KIP-1152, thank you! &

[VOTE] KIP-1152: Add transactional ID pattern filter to ListTransactions API

2025-04-24 Thread Calvin Liu
Hi, I would like to start a vote on KIP-1152, thank you! KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1152%3A+Add+transactional+ID+pattern+filter+to+ListTransactions+API Discussion thread: https://lists.apache.org/thread/m5n934wlwmqploz6ox8t1xo9sp50x11h Thanks Calvin

Re: [DISCUSS] KIP-1152: Add transactional ID prefix filter to ListTransactions API

2025-04-24 Thread Calvin Liu
Hi Thanks for the comments so far. Since they mainly involve minor changes, I'd like to start a vote for this KIP. Thanks! https://lists.apache.org/thread/nwongf7bptvo6z1dfbo0ro63y48sxzqb On Wed, Apr 23, 2025 at 8:27 AM Calvin Liu wrote: > Hi > Just coming back from my vacation. So

Re: [DISCUSS] KIP-1152: Add transactional ID prefix filter to ListTransactions API

2025-04-23 Thread Calvin Liu
rn is > > null instead of empty? Or do we want to do it for both null and empty? > > > > DJ06: In the motivation, could you please explain that the pattern will > > rely on re2j like KIP-848? Or you could actually add this to the > "Proposed > > Changes" sec

[jira] [Resolved] (KAFKA-18940) Fix uncleanLeaderElection metric due to ELR election

2025-04-05 Thread Calvin Liu (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-18940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Calvin Liu resolved KAFKA-18940. Resolution: Fixed > Fix uncleanLeaderElection metric due to ELR elect

Re: [DISCUSS] KIP-1152: Add transactional ID prefix filter to ListTransactions API

2025-04-05 Thread Calvin Liu
; > Best, > Chia-Ping > > > > Calvin Liu 於 2025年4月5日 凌晨2:35 寫道: > > > > Do you mean the TransactionalIdPatternFilter in the ListTransaction > request > > can be nullable? >

Re: [DISCUSS] KIP-1152: Add transactional ID prefix filter to ListTransactions API

2025-04-04 Thread Calvin Liu
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-994%3A+Minor+Enhancements+to+ListTransactions+and+DescribeTransactions+APIs > did, when it added a new filter: the client should only fail when the new > filter is used and the broker doesn't support the new filter. > >

Re: [DISCUSS] KIP-1152: Add transactional ID prefix filter to ListTransactions API

2025-04-04 Thread Calvin Liu
Filter` be renamed to > filterTransactionalIdPattern for consistency? > > chia_3: > > the re2j used by consumer is nullable, and the null means nothing. Should > we align with it? > > Best, > Chia-Ping > > On 2025/04/04 16:31:08 Calvin Liu wrote: > > Hi Chia-Ping and D

Re: [DISCUSS] KIP-1152: Add transactional ID prefix filter to ListTransactions API

2025-04-04 Thread Calvin Liu
gt; > > thanks for filing this KIP. Some questions are listed below. PTAL > > > > chia_0: > > > > could you please add discussion link to the KIP? > > > > chia_1: > > > > Have you consider using regex or multi-prefixes (similar to > >

Re: [DISCUSS] KIP-1152: Add transactional ID prefix filter to ListTransactions API

2025-04-03 Thread Calvin Liu
Hi Jun Good catch, the typo is corrected. Thanks! On Thu, Apr 3, 2025 at 4:17 PM Jun Rao wrote: > Hi, Calvin, > > Thanks for the KIP. Just a minor comment. > > ListTransactionRequest no longer supports zkBroker. > > Jun > > On Wed, Apr 2, 2025 at 9:43 AM Calvin L

[DISCUSS] KIP-1152: Add transactional ID prefix filter to ListTransactions API

2025-04-01 Thread Calvin Liu
Hi Kafka Community, I would like to start a discussion on KIP-1152: Add transactional ID prefix filter to ListTransactions API The main motivation behind the KIP is to reduce the transactions included in the ListTransactions response when the transactional IDs are well-formatted and can be fetched

[jira] [Created] (KAFKA-19073) Add Transactional ID prefix filter to ListTransactions API

2025-04-01 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-19073: -- Summary: Add Transactional ID prefix filter to ListTransactions API Key: KAFKA-19073 URL: https://issues.apache.org/jira/browse/KAFKA-19073 Project: Kafka Issue

[jira] [Created] (KAFKA-19072) Add a system test

2025-04-01 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-19072: -- Summary: Add a system test Key: KAFKA-19072 URL: https://issues.apache.org/jira/browse/KAFKA-19072 Project: Kafka Issue Type: Sub-task Reporter

[jira] [Created] (KAFKA-18966) Don't honor controller_num_nodes_override in combined controller test mode

2025-03-15 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-18966: -- Summary: Don't honor controller_num_nodes_override in combined controller test mode Key: KAFKA-18966 URL: https://issues.apache.org/jira/browse/KAFKA-18966 Pr

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2025-03-15 Thread Calvin Liu
al pattern here instead of nullables. > > > > --- > > Summary: > > > > So far, it seems like we are looking at these different options. The main > > difference in terms of API design is if the user will need to implement > > more than one method, or if a lambda can suffic

[jira] [Created] (KAFKA-18954) Add a metric to track the election based on ELR

2025-03-11 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-18954: -- Summary: Add a metric to track the election based on ELR Key: KAFKA-18954 URL: https://issues.apache.org/jira/browse/KAFKA-18954 Project: Kafka Issue Type: Sub

[jira] [Created] (KAFKA-18940) Fix uncleanLeaderElection metric due to ELR election

2025-03-07 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-18940: -- Summary: Fix uncleanLeaderElection metric due to ELR election Key: KAFKA-18940 URL: https://issues.apache.org/jira/browse/KAFKA-18940 Project: Kafka Issue Type

[jira] [Resolved] (KAFKA-18635) Re-enable the unclean shutdown detection when in ELR mode

2025-02-04 Thread Calvin Liu (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-18635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Calvin Liu resolved KAFKA-18635. Resolution: Fixed > Re-enable the unclean shutdown detection when in ELR m

[jira] [Resolved] (KAFKA-16540) Update partitions when the min isr config is updated.

2025-02-04 Thread Calvin Liu (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Calvin Liu resolved KAFKA-16540. Resolution: Fixed > Update partitions when the min isr config is upda

[jira] [Resolved] (KAFKA-18649) Complete ClearElrRecord handling in topic delta

2025-02-04 Thread Calvin Liu (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-18649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Calvin Liu resolved KAFKA-18649. Resolution: Fixed > Complete ClearElrRecord handling in topic de

[jira] [Reopened] (KAFKA-18635) Re-enable the unclean shutdown detection when in ELR mode

2025-02-04 Thread Calvin Liu (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-18635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Calvin Liu reopened KAFKA-18635: > Re-enable the unclean shutdown detection when in ELR m

[jira] [Reopened] (KAFKA-16540) Update partitions when the min isr config is updated.

2025-02-04 Thread Calvin Liu (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Calvin Liu reopened KAFKA-16540: > Update partitions when the min isr config is upda

[jira] [Reopened] (KAFKA-18649) Complete ClearElrRecord handling in topic delta

2025-02-04 Thread Calvin Liu (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-18649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Calvin Liu reopened KAFKA-18649: > Complete ClearElrRecord handling in topic de

[jira] [Resolved] (KAFKA-18635) Re-enable the unclean shutdown detection when in ELR mode

2025-02-04 Thread Calvin Liu (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-18635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Calvin Liu resolved KAFKA-18635. Resolution: Fixed > Re-enable the unclean shutdown detection when in ELR m

[jira] [Resolved] (KAFKA-18649) Complete ClearElrRecord handling in topic delta

2025-01-30 Thread Calvin Liu (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-18649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Calvin Liu resolved KAFKA-18649. Resolution: Fixed > Complete ClearElrRecord handling in topic de

[jira] [Created] (KAFKA-18649) Complete ClearElrRecord handling in topic delta

2025-01-25 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-18649: -- Summary: Complete ClearElrRecord handling in topic delta Key: KAFKA-18649 URL: https://issues.apache.org/jira/browse/KAFKA-18649 Project: Kafka Issue Type: Bug

[jira] [Resolved] (KAFKA-16540) Update partitions when the min isr config is updated.

2025-01-24 Thread Calvin Liu (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Calvin Liu resolved KAFKA-16540. Resolution: Fixed > Update partitions when the min isr config is upda

[jira] [Resolved] (KAFKA-15820) Add a metric to track the number of partitions under min ISR

2025-01-21 Thread Calvin Liu (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Calvin Liu resolved KAFKA-15820. Resolution: Won't Fix We can use the kafka.replication.under_min_isr to track this info &g

[jira] [Resolved] (KAFKA-18062) Use feature version to enable ELR

2025-01-15 Thread Calvin Liu (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-18062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Calvin Liu resolved KAFKA-18062. Resolution: Fixed > Use feature version to enable

Re: [DISCUSS] Apache Kafka 4.0.0 release

2024-12-12 Thread Calvin Liu
Hi David, For KIP-966, we have 2 open PRs to complete the feature. The estimated completion time is 2 weeks, we are still trying to catch the 4.0 release. https://github.com/apache/kafka/pull/17952 https://github.com/apache/kafka/pull/18148 On Wed, Dec 11, 2024 at 12:22 PM Chia-Ping Tsai wrote:

[jira] [Created] (KAFKA-18062) Use feature version to enable ELR

2024-11-21 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-18062: -- Summary: Use feature version to enable ELR Key: KAFKA-18062 URL: https://issues.apache.org/jira/browse/KAFKA-18062 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-18042) Reject the produce request with lower producer epoch early.

2024-11-19 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-18042: -- Summary: Reject the produce request with lower producer epoch early. Key: KAFKA-18042 URL: https://issues.apache.org/jira/browse/KAFKA-18042 Project: Kafka

[jira] [Created] (KAFKA-17877) IllegalStateException: missing producer id from the WriteTxnMarkersResponse

2024-10-25 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-17877: -- Summary: IllegalStateException: missing producer id from the WriteTxnMarkersResponse Key: KAFKA-17877 URL: https://issues.apache.org/jira/browse/KAFKA-17877 Project

Re: [DISCUSS] Single broker failures causing offline partitions

2024-09-10 Thread Calvin Liu
Hi Martin. Yes, the KIP-966 does not resolve your concern for the degraded leader. To fill the gap, 1. As you have mentioned, we need a leader degradation detection mechanism. So that the controller can promote another replica to the leader. 2. The controller needs to know which replica is a valid

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-06-12 Thread Calvin Liu
Hi Josep Due to the limit of the review resources, KIP-966 may have to be delayed to the next release. Thanks. On Wed, Jun 12, 2024 at 2:05 AM Rajini Sivaram wrote: > > > > Yes, we could backport this one to the 3.8 branch. Would you be able to > do > > this once you merge this PR? > > > Thanks

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-05-30 Thread Calvin Liu
Hi Mickael, Part 1 adds the ELR and enables the leader election improvements related to ELR. It does not change unclean leader election behavior which I think is hard-coded to be disabled. Part 2 should replace the current unclean leader election with the unclean recovery. Colin McCabe will help wi

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-05-29 Thread Calvin Liu
Hi Regarding the KIP-966, the implementation is split into 2 parts. Part 1 where we introduce the Eligible Leader Replicas to help increase the data durability is pretty close to finish. We have 1 feature PR pending. Currently, Colin McCabe is helping the review and hopefully we can get it merged w

[jira] [Resolved] (KAFKA-16217) Transactional producer stuck in IllegalStateException during close

2024-04-29 Thread Calvin Liu (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Calvin Liu resolved KAFKA-16217. Fix Version/s: (was: 3.6.3) Resolution: Fixed > Transactional producer stuck

[jira] [Resolved] (KAFKA-15585) DescribeTopic API

2024-04-19 Thread Calvin Liu (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Calvin Liu resolved KAFKA-15585. Resolution: Fixed > DescribeTopic API > - > > Key:

[jira] [Created] (KAFKA-16540) Update partitions when the min isr config is updated.

2024-04-11 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-16540: -- Summary: Update partitions when the min isr config is updated. Key: KAFKA-16540 URL: https://issues.apache.org/jira/browse/KAFKA-16540 Project: Kafka Issue Type

[jira] [Resolved] (KAFKA-15586) Clean shutdown detection, server side

2024-04-11 Thread Calvin Liu (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Calvin Liu resolved KAFKA-15586. Resolution: Fixed > Clean shutdown detection, server s

[jira] [Created] (KAFKA-16479) Add pagination supported describeTopic interface

2024-04-05 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-16479: -- Summary: Add pagination supported describeTopic interface Key: KAFKA-16479 URL: https://issues.apache.org/jira/browse/KAFKA-16479 Project: Kafka Issue Type

[jira] [Resolved] (KAFKA-15583) High watermark can only advance if ISR size is larger than min ISR

2024-04-05 Thread Calvin Liu (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Calvin Liu resolved KAFKA-15583. Resolution: Fixed > High watermark can only advance if ISR size is larger than min

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2024-02-23 Thread Calvin Liu
describeTopics(TopicCollection topics, DescribeTopicsOptions options); David and I would prefer the first Consumer version which works better as a stream purposes. On Wed, Oct 11, 2023 at 4:28 PM Calvin Liu wrote: > Hi David, > Thanks for the comment. > Yes, we can separate the ELR enablement

[jira] [Created] (KAFKA-16250) Consumer group coordinator should perform sanity check on the offset commits.

2024-02-13 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-16250: -- Summary: Consumer group coordinator should perform sanity check on the offset commits. Key: KAFKA-16250 URL: https://issues.apache.org/jira/browse/KAFKA-16250 Project

[jira] [Created] (KAFKA-16217) Transactional producer stuck in IllegalStateException during close

2024-02-01 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-16217: -- Summary: Transactional producer stuck in IllegalStateException during close Key: KAFKA-16217 URL: https://issues.apache.org/jira/browse/KAFKA-16217 Project: Kafka

[jira] [Created] (KAFKA-15873) Improve the performance of the DescribeTopicPartitions API

2023-11-21 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-15873: -- Summary: Improve the performance of the DescribeTopicPartitions API Key: KAFKA-15873 URL: https://issues.apache.org/jira/browse/KAFKA-15873 Project: Kafka Issue

[jira] [Created] (KAFKA-15820) Add a metric to track the number of partitions under min ISR

2023-11-13 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-15820: -- Summary: Add a metric to track the number of partitions under min ISR Key: KAFKA-15820 URL: https://issues.apache.org/jira/browse/KAFKA-15820 Project: Kafka

[jira] [Resolved] (KAFKA-15584) ELR leader election

2023-11-06 Thread Calvin Liu (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Calvin Liu resolved KAFKA-15584. Resolution: Fixed > ELR leader election > --- > >

[jira] [Created] (KAFKA-15762) ClusterConnectionStatesTest.testSingleIP is flaky

2023-10-30 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-15762: -- Summary: ClusterConnectionStatesTest.testSingleIP is flaky Key: KAFKA-15762 URL: https://issues.apache.org/jira/browse/KAFKA-15762 Project: Kafka Issue Type

[jira] [Created] (KAFKA-15761) ConnectorRestartApiIntegrationTest.testMultiWorkerRestartOnlyConnector is flaky

2023-10-30 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-15761: -- Summary: ConnectorRestartApiIntegrationTest.testMultiWorkerRestartOnlyConnector is flaky Key: KAFKA-15761 URL: https://issues.apache.org/jira/browse/KAFKA-15761 Project

[jira] [Created] (KAFKA-15760) org.apache.kafka.trogdor.coordinator.CoordinatorTest.testTaskRequestWithOldStartMsGetsUpdated is flaky

2023-10-30 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-15760: -- Summary: org.apache.kafka.trogdor.coordinator.CoordinatorTest.testTaskRequestWithOldStartMsGetsUpdated is flaky Key: KAFKA-15760 URL: https://issues.apache.org/jira/browse/KAFKA

[jira] [Created] (KAFKA-15759) DescribeClusterRequestTest is flaky

2023-10-30 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-15759: -- Summary: DescribeClusterRequestTest is flaky Key: KAFKA-15759 URL: https://issues.apache.org/jira/browse/KAFKA-15759 Project: Kafka Issue Type: Bug

[jira] [Created] (KAFKA-15700) FetchFromFollowerIntegrationTest is flaky

2023-10-26 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-15700: -- Summary: FetchFromFollowerIntegrationTest is flaky Key: KAFKA-15700 URL: https://issues.apache.org/jira/browse/KAFKA-15700 Project: Kafka Issue Type: Bug

[jira] [Created] (KAFKA-15699) MirrorConnectorsIntegrationBaseTest is flaky

2023-10-26 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-15699: -- Summary: MirrorConnectorsIntegrationBaseTest is flaky Key: KAFKA-15699 URL: https://issues.apache.org/jira/browse/KAFKA-15699 Project: Kafka Issue Type: Bug

[jira] [Created] (KAFKA-15690) Flaky integration tests

2023-10-26 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-15690: -- Summary: Flaky integration tests Key: KAFKA-15690 URL: https://issues.apache.org/jira/browse/KAFKA-15690 Project: Kafka Issue Type: Bug Reporter

[jira] [Created] (KAFKA-15665) Enforce min ISR when complete partition reassignment

2023-10-21 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-15665: -- Summary: Enforce min ISR when complete partition reassignment Key: KAFKA-15665 URL: https://issues.apache.org/jira/browse/KAFKA-15665 Project: Kafka Issue Type

[jira] [Resolved] (KAFKA-15581) Introduce ELR

2023-10-19 Thread Calvin Liu (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Calvin Liu resolved KAFKA-15581. Reviewer: David Arthur Resolution: Fixed > Introduce

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-10-11 Thread Calvin Liu
a cluster has updated to > its MetadataVersion. It's simpler to disable the feature through config > rather than going through a MetadataVersion downgrade (once that's > supported). > > Does that make sense? > > -David > > On Wed, Oct 11, 2023 at 1:40 PM Calvin Liu >

[jira] [Created] (KAFKA-15590) Replica.updateFetchState should also fence updates with stale leader epoch

2023-10-11 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-15590: -- Summary: Replica.updateFetchState should also fence updates with stale leader epoch Key: KAFKA-15590 URL: https://issues.apache.org/jira/browse/KAFKA-15590 Project

[jira] [Created] (KAFKA-15586) Clean shutdown detection, server side

2023-10-11 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-15586: -- Summary: Clean shutdown detection, server side Key: KAFKA-15586 URL: https://issues.apache.org/jira/browse/KAFKA-15586 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-15585) DescribeTopic API

2023-10-11 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-15585: -- Summary: DescribeTopic API Key: KAFKA-15585 URL: https://issues.apache.org/jira/browse/KAFKA-15585 Project: Kafka Issue Type: Sub-task Reporter

[jira] [Created] (KAFKA-15584) ELR leader election

2023-10-11 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-15584: -- Summary: ELR leader election Key: KAFKA-15584 URL: https://issues.apache.org/jira/browse/KAFKA-15584 Project: Kafka Issue Type: Sub-task Reporter

[jira] [Created] (KAFKA-15583) High watermark can only advance if ISR size is larger than min ISR

2023-10-11 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-15583: -- Summary: High watermark can only advance if ISR size is larger than min ISR Key: KAFKA-15583 URL: https://issues.apache.org/jira/browse/KAFKA-15583 Project: Kafka

[jira] [Created] (KAFKA-15582) Clean shutdown detection, broker side

2023-10-11 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-15582: -- Summary: Clean shutdown detection, broker side Key: KAFKA-15582 URL: https://issues.apache.org/jira/browse/KAFKA-15582 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-15581) Introduce ELR

2023-10-11 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-15581: -- Summary: Introduce ELR Key: KAFKA-15581 URL: https://issues.apache.org/jira/browse/KAFKA-15581 Project: Kafka Issue Type: Sub-task Reporter: Calvin

[jira] [Created] (KAFKA-15580) KIP-966: Unclean Recovery

2023-10-11 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-15580: -- Summary: KIP-966: Unclean Recovery Key: KAFKA-15580 URL: https://issues.apache.org/jira/browse/KAFKA-15580 Project: Kafka Issue Type: New Feature

[jira] [Created] (KAFKA-15579) KIP-966: Eligible Leader Replicas

2023-10-11 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-15579: -- Summary: KIP-966: Eligible Leader Replicas Key: KAFKA-15579 URL: https://issues.apache.org/jira/browse/KAFKA-15579 Project: Kafka Issue Type: New Feature

Re: [VOTE]KIP-966: Eligible Leader Replicas

2023-10-11 Thread Calvin Liu
Hi, The KIP has received 3 binding votes from Justine Olshan, Jun Rao and Colin McCabe. Also thanks to Jeff Kim, Jack Vanlightly, Artem Livshits, David Arthur, David Jacot for the comments to make the KIP better. Thanks for the help! On Mon, Oct 9, 2023 at 10:20 AM Calvin Liu wrote: > Hi Co

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-10-11 Thread Calvin Liu
> > > One more comment. > > > > "The first partition to fetch details for. -1 means to fetch all > > partitions." It seems that FirstPartitionId of 0 naturally means fetching > > all partitions? > > > > Thanks, > > > > Jun >

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-10-10 Thread Calvin Liu
the range query has errors for some of the partitions, do we expect > different responses when querying particular partitions? > > Thanks, > > Jun > > On Tue, Oct 10, 2023 at 10:50 AM Calvin Liu > wrote: > > > Hi Jun > > 60. Yes, it is a good question. I was t

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-10-10 Thread Calvin Liu
m DescribeTopicRequest. > > 62. Should ElectLeadersRequest.DesiredLeaders be at the same level as > ElectLeadersRequest.TopicPartitions.Partitions? In the KIP, it looks like > it's at the same level as ElectLeadersRequest.TopicPartitions. > > Thanks, > > Jun > > On Wed, Oct 4, 2023 at

Re: [VOTE]KIP-966: Eligible Leader Replicas

2023-10-09 Thread Calvin Liu
Justine Olshan > > > wrote: > > > >> Thanks Calvin. > >> I think this will be very helpful going forward to minimize data loss. > >> > >> +1 from me (binding) > >> > >> Justine > >> > >> On Wed, Sep 20, 2023 a

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-10-04 Thread Calvin Liu
fy the changes to the admin api to > accommodate the new or updated apis. Do you plan to add them? > > Best, > David > > Le mer. 4 oct. 2023 à 20:39, Calvin Liu a > écrit : > > > Hi Jun, > > After the MV downgrade, the controller will write in the old versi

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-10-04 Thread Calvin Liu
ng MV > is > > > not supported? I think Kafka typically tries to support downgrades, > and I > > > couldn't think of other examples. > > > > > > Thanks, > > > Justine > > > > > > On Wed, Oct 4, 2023 at 9:40 AM

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-10-04 Thread Calvin Liu
; 53. ElectLeadersRequest.DesiredLeaders: Should it be nullable since a > > > user > > > > may not specify DesiredLeaders? > > > > > > > > 54. Downgrade: Is that indeed possible? I thought earlier you said > that > > > > once the new version of th

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-10-03 Thread Calvin Liu
> > > > > > 54. Downgrade: Is that indeed possible? I thought earlier you said that > > > once the new version of the records are in the metadata log, one can't > > > downgrade since the old broker doesn't know how to parse the new > version > &

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-09-25 Thread Calvin Liu
idea of using none, balanced, > > aggressive. We also had an offline discussion about why it is good to > use a > > new config key (basically, so that we can deprecate the old one which had > > only false/true values in 4.0) With these changes, I am +1. >

[VOTE]KIP-966: Eligible Leader Replicas

2023-09-20 Thread Calvin Liu
Hi all, I'd like to call for a vote on KIP-966 which includes a series of enhancements to the current ISR model. - Introduce the new HWM advancement requirement which enables the system to have more potentially data-safe replicas. - Introduce Eligible Leader Replicas(ELR) to represent the

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-09-18 Thread Calvin Liu
, 2023 at 3:51 PM Calvin Liu wrote: > Hi Colin, > For the unclean.recovery.strategy config name, how about we use the > following > None. It basically means no unclean recovery will be performed. > Aggressive. It means availability goes first. Whenever the partition can't >

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-09-18 Thread Calvin Liu
overy. Balanced. It is the balance point of the availability first(Aggressive) and least availability(None). The controller performs unclean recovery when both ISR and ELR are empty. On Fri, Sep 15, 2023 at 11:42 AM Calvin Liu wrote: > Hi Colin, > > > So, the proposal is tha

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-09-15 Thread Calvin Liu
ecovery.strategy, the cluster level config unclean.recovery.Enabled, the controller can calculate the correct methods to use right? On Fri, Sep 15, 2023 at 10:02 AM Colin McCabe wrote: > On Thu, Sep 14, 2023, at 22:23, Calvin Liu wrote: > > Hi Colin > > 1. I think using the new

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-09-14 Thread Calvin Liu
s discuss this. > > 3. You still need to describe the changes to AdminClient that are needed > to use DescribeTopicRequest. > > Keep at it. It's looking better. :) > > best, > Colin > > > On Thu, Sep 14, 2023, at 11:03, Calvin Liu wrote: > > Hi Colin > >

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-09-14 Thread Calvin Liu
. kafka.controller.unclean_recovery_finished_count On Wed, Sep 13, 2023 at 10:46 AM Colin McCabe wrote: > On Tue, Sep 12, 2023, at 17:21, Calvin Liu wrote: > > Hi Colin > > Thanks for the comments! > > > > Hi Calvin, > > Thanks again for the KIP. > > One meta-comment: it's usually better to jus

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-09-12 Thread Calvin Liu
he controller. Just like with the other metadata > > requests like MetadataRequest, this should be served by brokers. > > > > It's a bit confusing why both topic ID and topic name are provided to > > this API. Is the intention that callers should set one but not the >

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-09-12 Thread Calvin Liu
Hi Jun, I have a second thought on the kafka.replication.electable_replicas_count. I don't think it is a useful metric and it seems wasteful to have a per partition metric to track the sum(ISR, ELR). I will remove this metric. On Tue, Sep 12, 2023 at 3:35 PM Calvin Liu wrote: > Hi Jun,

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-09-12 Thread Calvin Liu
> LastKnownLeader is preferred." Do we wait strictly for the lastKnownLeader > until it's unfenced or do we select the first unfenced replica from > LastKnownELR? > > Thanks, > > Jun > > On Fri, Sep 8, 2023 at 2:56 PM Calvin Liu > wrote: > > > Hi Artem

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-09-08 Thread Calvin Liu
tion and replica ids -- what are those? > Would those be just the whole replica assignment or just LastKnownELR? > > -Artem > > On Thu, Aug 10, 2023 at 3:47 PM Calvin Liu > wrote: > > > Hi everyone, > > I'd like to discuss a series of enhancement to t

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-09-08 Thread Calvin Liu
oes it ever get unset? > > 27. "min.insync.replicas now applies to the replication of all kinds of > messages." Not sure that I follow. Could you explain a bit more? > > 28. "But later, the current leader will put the follower into the pending > ISR" : It would be useful t

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-09-06 Thread Calvin Liu
s. The only thing > that is not modeled is the unclean recovery though I may leave that as the > specification is already very large. > > Jack > > On 2023/09/01 22:27:10 Calvin Liu wrote: > > Hi Justine > > 1. With the new proposal, in order to let the consumer consume

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-09-01 Thread Calvin Liu
d from > "delivery plan" to the bullet points. > > Justine > > On Thu, Aug 31, 2023 at 4:04 PM Calvin Liu > wrote: > > > Hi Justine > > Thanks for the questions! > > *a. For my understanding, will we block replication? Or just the high > > wa

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-08-31 Thread Calvin Liu
ing is once metadata version > is updated we will always take the more thoughtful unclean election process > (ie, inspect the logs) > > Overall though, the general KIP is pretty solid. Looking at the rejected > alternatives, it looks like a lot was considered, so it's nice to

Re: [DISCUSS] KIP-966: Eligible Leader Replicas

2023-08-14 Thread Calvin Liu
you referring to broker 2? > > Thanks, > Jeff > > On Thu, Aug 10, 2023 at 6:48 PM Calvin Liu > wrote: > > > Hi everyone, > > I'd like to discuss a series of enhancement to the replication protocol. > > > > A partition replica can experience local da

[DISCUSS] KIP-966: Eligible Leader Replicas

2023-08-10 Thread Calvin Liu
Hi everyone, I'd like to discuss a series of enhancement to the replication protocol. A partition replica can experience local data loss in unclean shutdown scenarios where unflushed data in the OS page cache is lost - such as an availability zone power outage or a server error. The Kafka replicat

[jira] [Created] (KAFKA-15332) Eligible Leader Replicas

2023-08-10 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-15332: -- Summary: Eligible Leader Replicas Key: KAFKA-15332 URL: https://issues.apache.org/jira/browse/KAFKA-15332 Project: Kafka Issue Type: New Feature

[jira] [Created] (KAFKA-15221) Potential race condition between requests from rebooted followers

2023-07-19 Thread Calvin Liu (Jira)
Calvin Liu created KAFKA-15221: -- Summary: Potential race condition between requests from rebooted followers Key: KAFKA-15221 URL: https://issues.apache.org/jira/browse/KAFKA-15221 Project: Kafka

[jira] [Resolved] (KAFKA-14139) Replaced disk can lead to loss of committed data even with non-empty ISR

2023-04-13 Thread Calvin Liu (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Calvin Liu resolved KAFKA-14139. Resolution: Fixed > Replaced disk can lead to loss of committed data even with non-empty

Re: [VOTE] KIP-903: Replicas with stale broker epoch should not be allowed to join the ISR

2023-03-07 Thread Calvin Liu
The KIP is accepted with 3 binding votes (David, Jun, Jason) and 1 non-binding vote (Alexandre). Thanks! On Tue, Feb 28, 2023 at 3:06 PM Jason Gustafson wrote: > Thanks Calvin, +1 from me. > > On Mon, Feb 27, 2023 at 9:41 AM Calvin Liu > wrote: > > > Hi Jason, &

Re: [VOTE] KIP-903: Replicas with stale broker epoch should not be allowed to join the ISR

2023-02-27 Thread Calvin Liu
4+", > "entityType": "brokerId", > "about": "The broker ID of the follower, of -1 if this request is > from a consumer." }, > { "name": "BrokerEpoch", "type": "int64", "versions": &qu

  1   2   >