Re: [DISCUSS] KIP-868 Metadata Transactions (new thread)

2022-09-28 Thread David Arthur
Thanks Luke, I failed to mention in my reply to Ziming, but I did add this sentence to the end of the "visibility" section: The metadata shell must also avoid showing partially committed metadata > transactions. -David On Tue, Sep 27, 2022 at 10:20 PM Luke Chen wrote: > Hi David, > > I think

Re: [DISCUSS] KIP-868 Metadata Transactions (new thread)

2022-09-27 Thread Luke Chen
Hi David, I think there's still an unanswered question from Ziming: > I think we can also describe visibility in the MetadataShell since it is also a public interface. I think this suggestion makes sense. But I'm still +1 for current proposal. Thanks. Luke On Tue, Sep 27, 2022 at 11:11 PM David

Re: [DISCUSS] KIP-868 Metadata Transactions (new thread)

2022-09-27 Thread David Arthur
Thanks for the reviews, everyone. I’ve started a VOTE thread -David On Fri, Sep 23, 2022 at 00:55 deng ziming wrote: > David, > Thanks for the feedback about #2 and #3, I'm OK with them. > I also mentioned the visibility in the MetadataShell in #1, do you have any > thoughts? > > -- > Best, > Z

Re: [DISCUSS] KIP-868 Metadata Transactions (new thread)

2022-09-22 Thread deng ziming
David, Thanks for the feedback about #2 and #3, I'm OK with them. I also mentioned the visibility in the MetadataShell in #1, do you have any thoughts? -- Best, Ziming On Wed, Sep 21, 2022 at 10:56 PM David Arthur wrote: > Ziming, thanks for the feedback! Let me know your thoughts on #2 and #3

Re: [DISCUSS] KIP-868 Metadata Transactions (new thread)

2022-09-21 Thread David Arthur
Ziming, thanks for the feedback! Let me know your thoughts on #2 and #3 1. Good idea. I consolidated all the details of record visibility into that section. 2. I'm not sure we can always know the number of records ahead of time for a transaction. One future use case is likely for the ZK data migr

Re: [DISCUSS] KIP-868 Metadata Transactions (new thread)

2022-09-19 Thread deng ziming
Hello David, Thanks for the KIP, certainly it makes sense, I left some minor questions. 1. In “Record Visibility” section you declare visibility in the controller, in “Broker Support” you mention visibility in the broker, we can put them together, and I think we can also describe visibility in t

Re: [DISCUSS] KIP-868 Metadata Transactions (new thread)

2022-09-19 Thread David Arthur
Thanks, Luke :) Colin -- I updated the KIP with your feedback. Do you think we would expose the "last stable offset" outside of the controller? Or would it just be an internal concept? -David On Sun, Sep 18, 2022 at 9:05 AM Luke Chen wrote: > Hi David, > > Thanks for the KIP! > It's a light-we

Re: [DISCUSS] KIP-868 Metadata Transactions (new thread)

2022-09-18 Thread Luke Chen
Hi David, Thanks for the KIP! It's a light-weight transactional proposal for single producer, cool! +1 for it! Luke On Sat, Sep 10, 2022 at 1:14 AM David Arthur wrote: > Starting a new thread to avoid issues with mail client threading. > > Original thread follows: > > Hey folks, I'd like to s

[DISCUSS] KIP-868 Metadata Transactions (new thread)

2022-09-09 Thread David Arthur
Starting a new thread to avoid issues with mail client threading. Original thread follows: Hey folks, I'd like to start a discussion on the idea of adding transactions in the KRaft controller. This will allow us to overcome the current limitation of atomic batch sizes in Raft which lets us do thi