Re: [DISCUSS][TABLE] How to handle empty delete for UpsertSource

Hi Hequn, That's great! Yes, let's go with option 2 (from the source's point of view) and later extend Join and Aggregation to discard empty deletes. I agree that the filtering at the sink should be optional and configurable via the query configuration. Again, thanks for starting this discussion.

Re: [DISCUSS][TABLE] How to handle empty delete for UpsertSource

Hi Fabian Thanks for your update. The opinions on upsert streams are highly enlightening. I think now I am agree with you that we can choose option 2 to solve the problem: Throw away empty deletes when source generate retractions, otherwise pass empty deletes down. As for the UpsertSink, I think

Re: [DISCUSS][TABLE] How to handle empty delete for UpsertSource

Hi Hequn, hi Piotr, Thanks for pushing this discussion forward and sorry for not responding earlier. After reading the thread, I agree that we do not *need* to (but may) forward empty deletes. As I pointed out before, I see empty deletes not as a semantic problem that needs to be exactly solved b

Re: [DISCUSS][TABLE] How to handle empty delete for UpsertSource

Hi Piotrek, Great to see your replies, and really thanks for all your suggestions. Inline is a good way, i will do it same as you :-) *> I’m assuming that we are talking about event time and that `(delete 1, a, 1)` happened before `(add 1, a, 2)`, right?* We are talking about processing time(FLI

Re: [DISCUSS][TABLE] How to handle empty delete for UpsertSource

Hi, Thanks for very detailed explanation :) Please check my inlined responses. > On 22 Aug 2018, at 14:28, Hequn Cheng wrote: > > Hi all, > Thanks a lot for your replies, @Piotr Nowojski > @Fabian Hueske > @Xingcan Cui

Re: [DISCUSS][TABLE] How to handle empty delete for UpsertSource

Hi all, Thanks a lot for your replies, @Piotr Nowojski @Fabian Hueske @Xingcan Cui Option 3 will send empty deletes to downstream operators, but downstream operators seems don't know how to deal with it. Assume that upsert source forward deletes by default. Let's see what problems we have. *Ta

Re: [DISCUSS][TABLE] How to handle empty delete for UpsertSource

For the record I was also in favour of option 3. regarding empty deletes. About stateful Filters on upsert stream. Hequn, I don’t fully understand this: > However, if UpsertSource can also output empty deletes, these > empty deletes will be more difficult to control. We don't know where these > d

Re: [DISCUSS][TABLE] How to handle empty delete for UpsertSource

Hi Hequn, Thanks for this discussion. Personally, I’m also in favor of option 3. There are two reasons for that: (1) A proctime-based upsert table source does not guarantee the records’ order, which means empty delete messages may not really be "empty". Simply discarding them may cause semanti

Re: [DISCUSS][TABLE] How to handle empty delete for UpsertSource

Hi, Thanks fort starting this discussion Hequn! These are a tricky questions. 1) Handling empty deletes in UpsertSource. I think forwarding empty deletes would only have a semantic difference if the output is persisted in a non-empty external table, e.g., a Cassandra table with entries. If we wou

Re: [DISCUSS][TABLE] How to handle empty delete for UpsertSource

Hi, Thanks for bringing up this issue here. I’m not sure whether sometimes swallowing empty deletes could be a problem or always swallowing/forwarding them is better. I guess for most use cases it doesn't matter. Maybe the best for now would be to always forward them, since if they are a probl

[DISCUSS][TABLE] How to handle empty delete for UpsertSource

Hi all, Currently, I am working on FLINK-8577 Implement proctime DataStream to Table upsert conversion . And a design doc can be found here . It recei