Re: Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect

2022-10-17 Thread Chris Egerton
Thanks Greg! I think sorting results (at least w/r/t pagination) is only valuable if it's part of a formal contract; otherwise, a REST extension wouldn't really be able to take advantage of that behavior without becoming brittle across non-major version changes. I'd opt on the side of leaving it o

Re: Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect

2022-10-14 Thread Greg Harris
Hey Chris, Thanks for your quick reply! Design 1: I think this is a completely reasonable way to implement a V1, without any more complicated pagination. I had not considered that a REST extension would be able to add pagination, but upon reflection that makes a lot of sense. I like the best-ef

Re: Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect

2022-10-14 Thread Chris Egerton
Hi Greg, Thanks for your thoughts. RE your design questions: 1. The responses in the REST API may grow fairly large for sink connectors that consume from a large number of Kafka topic partitions, and source connectors that store a wide range of source partitions. If there is a large amount of da

RE: Re: [DISCUSS] KIP-875: First-class offsets support in Kafka Connect

2022-10-13 Thread Greg Harris
Hey Chris, Thanks for the KIP! I think this is an important feature for both development and operations use-cases, and it's an obvious gap in the REST feature set. I also appreciate the incremental nature of the KIP and the future extensions that will now be possible. I had a couple of questions