Hi, Daniel

> 1. The file source appears to not always read all of the files that are
in my source directory. There are 3 files that I'm reading in a directory.
It will sometimes skip one, seemingly at random.

What version of Flink are you using? This looks strange, I think you could
check the logs and you might find info.

> 2. When I do a join on the kafka source, the process that does the join
will quickly stop emitting data and the file source process appears to end.
The file source process never emits a final watermark before it ends. It
works correctly when I use the file monitor and configure it to emit
watermarks at an interval.

The Join operator does not depend on the watermark trigger data being
computed and emit, so intuitively I understand that the two should be
unrelated.

Best,
Ron

Daniel Henneberger <henneberger.dan...@gmail.com> 于2023年8月11日周五 07:16写道:

> Dear Apache Flink community,
>
> When I run Flink locally in my test cases on my Mac, I observe different
> behavior compared to running it in my Docker-backed build instance or using
> the official Docker-compose image. The processes complete as expected when
> i run it in-process but not always when i use docker. I'm using the
> datastream file connector with the table api. Specifically, I'm observing
> these two behaviors:
>
> 1. The file source appears to not always read all of the files that are in
> my source directory. There are 3 files that I'm reading in a directory. It
> will sometimes skip one, seemingly at random.
>
> 2. When I do a join on the kafka source, the process that does the join
> will quickly stop emitting data and the file source process appears to end.
> The file source process never emits a final watermark before it ends. It
> works correctly when I use the file monitor and configure it to emit
> watermarks at an interval.
>
> I'm not sure if these two issues are related. Let me know what you think!
>
> Thanks,
> Daniel Henneberger
>

Reply via email to