Re: [DISCUSS] KIP-472: Add header to RecordContext

2019-05-31 Thread Matthias J. Sax
Sure thing :) For the original issue to preserve partition-time within a task across restarts, we don't need this KIP. And I agree that downstream time propagation via heartbeats is nothing critical atm but may require a big change. Hence, maybe not worth the effort right now. If you are fine wi

Re: [DISCUSS] KIP-472: Add header to RecordContext

2019-05-31 Thread Richard Yu
Hi Matthias, Thanks for responding. :) I suppose from the scope of the change that is needed to fix the timestamp propagation bug is too complex (in other words, probably not worth it). So should this KIP be closed since it might be a little excessive? There probably is no need for too big of a

Re: [DISCUSS] KIP-472: Add header to RecordContext

2019-05-31 Thread Matthias J. Sax
Thanks for the KIP. However, I have some doubts that this would work. In the example, you mention 4 records > r1(2), r2(3), r3(7), and r4(9) and that if r3 and r4 would be filtered, the downstream task would not advance partition time from 3 to 9. However, if r3 and r4 are filtered, no record

[DISCUSS] KIP-472: Add header to RecordContext

2019-05-20 Thread Richard Yu
Hello, I wish to introduce a minor addition present in RecordContext (a public facing API). This addition works to both provide the user with more information regarding the processing state of the partition, but also help resolve a bug which Kafka is currently experiencing. Here is the KIP Link: h