Barriers in Flink SQL

[Ralph Matthias Debusmann]

Hi,

one question - it seems that "barriers" are perfectly supported by Flink,
but not yet supported in Flink SQL.

When I e.g. do a UNION of two views derived from one source table fed by
Kafka, I get thousands of intermediate results which are incorrect (the
example I am using is this one:
https://www.scattered-thoughts.net/writing/internal-consistency-in-streaming-systems/),
and I can only be sure to get the correct result if I stop producing new
messages into the source table.

1) Is my understanding correct that barriers are not implemented for Flink
SQL?
2) Why is it not implemented/is this on the roadmap?

Best, Ralph

Invalid Null Check in DefaultFileFilter

[Chirag Dewan via user]

Hi,
I was looking at this check in DefaultFileFilter:
public boolean test(Path path) {
final String fileName = path.getName();
if (fileName == null || fileName.length() == 0) {
return true;
}Was wondering how can a file name be null?
And if null, shouldnt this be return false?
I created a JIRA for this - [FLINK-33367] Invalid Check in DefaultFileFilter - 
ASF JIRA

| 
| 
|  | 
[FLINK-33367] Invalid Check in DefaultFileFilter - ASF JIRA


 |

 |

 |

Any input is appreciated.
Thanks

Re: Barriers in Flink SQL

[Giannis Polyzos]

Hi Ralph,
can you explain a bit more? When you say "barriers" you should be referring
to the checkpoints, but from your description seems more like watermarks.
What functionality is supported in Flink and not Flink SQL? In terms of
watermarks, there were a few shortcomings between the two APIs which are
addressed in the upcoming 1.18 release
https://cwiki.apache.org/confluence/display/FLINK/FLIP-296%3A+Extend+watermark-related+features+for+SQL
What operations are you running and you see incorrect results?

Best

On Wed, Oct 25, 2023 at 9:51 PM Ralph Matthias Debusmann <
matthias.debusm...@gmail.com> wrote:

> Hi,
>
> one question - it seems that "barriers" are perfectly supported by Flink,
> but not yet supported in Flink SQL.
>
> When I e.g. do a UNION of two views derived from one source table fed by
> Kafka, I get thousands of intermediate results which are incorrect (the
> example I am using is this one:
> https://www.scattered-thoughts.net/writing/internal-consistency-in-streaming-systems/),
> and I can only be sure to get the correct result if I stop producing new
> messages into the source table.
>
> 1) Is my understanding correct that barriers are not implemented for Flink
> SQL?
> 2) Why is it not implemented/is this on the roadmap?
>
> Best, Ralph
>
>