Hey y'all, resurrecting an old KIP for the benefit of KIP-382, which
depends on an additional parameter in SourceTask.commitRecord(). I've
updated KIP-416 according to consensus reached in PR-6295. Let's finish the
vote so we can formally approve this minor KIP, please!
Ryanne
On Mon, Jan 21, 2019, 4:25 PM Ryanne Dolan <ryannedo...@gmail.com> wrote:
> Andrew, I agree it's a better commitRecord, but with the slightly
> different semantics you mentioned. I suppose we could document that well
> enough that reusing the same name would be fine.
>
> I'll resend the discussion email. Maybe it got lost somehow.
>
> Ryanne
>
> On Mon, Jan 21, 2019, 4:37 AM Andrew Schofield <andrew_schofi...@live.com
> wrote:
>
>> Hi,
>> I'm not quite sure about the etiquette here but I wonder whether the KIP
>> could be improved. I think I missed the DISCUSS thread.
>>
>> I think that really your recordLogged(SourceRecord, RecordMetadata)
>> method is actually a better version of commitRecord() and perhaps it ought
>> to be an overload. This is similar to the situation in which the Serializer
>> interface was enhanced when record headers were added.
>>
>> public abstract class SourceTask implements Task {
>> public void commitRecord(SourceRecord sourceRecord, RecordMetadata
>> recordMetadata) {
>> this.commitRecord();
>> }
>>
>> public void commitRecord() {
>> }
>> }
>>
>> Or something like that. I do understand that the KIP mentions that
>> recordLogged() is only called for records that are actually ACKed, but it's
>> very similar in intent to commitRecord() in my view.
>>
>> Just my 2 cents.
>>
>> Andrew Schofield
>> IBM Event Streams
>>
>>
>> On 17/01/2019, 23:54, "Ryanne Dolan" <ryannedo...@gmail.com> wrote:
>>
>> Hey y'all, please vote for KIP-416 by replying +1 to this thread.
>>
>> Right now, there is no way for a SourceConnector/Task to know:
>>
>> - whether a record was successfully sent to Kafka, vs filtered out or
>> skipped.
>> - the downstream offsets and metadata of sent records
>>
>> KIP-416 proposes adding a recordLogged() callback for this purpose.
>>
>>
>> https://nam05.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-416%253A%2BNotify%2BSourceTask%2Bof%2BACK%2527d%2Boffsets%252C%2Bmetadata&data=02%7C01%7C%7C9fa617754cce4bab7ba508d67cd7128f%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C636833660500817715&sdata=udEP27%2FrshuP5sWthvZmUIdt13whM5XqKMoia1wE93c%3D&reserved=0
>>
>> Thanks!
>> Ryanne
>>
>>
>>
