Hi, Daniel > 1. The file source appears to not always read all of the files that are in my source directory. There are 3 files that I'm reading in a directory. It will sometimes skip one, seemingly at random.
What version of Flink are you using? This looks strange, I think you could check the logs and you might find info. > 2. When I do a join on the kafka source, the process that does the join will quickly stop emitting data and the file source process appears to end. The file source process never emits a final watermark before it ends. It works correctly when I use the file monitor and configure it to emit watermarks at an interval. The Join operator does not depend on the watermark trigger data being computed and emit, so intuitively I understand that the two should be unrelated. Best, Ron Daniel Henneberger <henneberger.dan...@gmail.com> 于2023年8月11日周五 07:16写道： > Dear Apache Flink community, > > When I run Flink locally in my test cases on my Mac, I observe different > behavior compared to running it in my Docker-backed build instance or using > the official Docker-compose image. The processes complete as expected when > i run it in-process but not always when i use docker. I'm using the > datastream file connector with the table api. Specifically, I'm observing > these two behaviors: > > 1. The file source appears to not always read all of the files that are in > my source directory. There are 3 files that I'm reading in a directory. It > will sometimes skip one, seemingly at random. > > 2. When I do a join on the kafka source, the process that does the join > will quickly stop emitting data and the file source process appears to end. > The file source process never emits a final watermark before it ends. It > works correctly when I use the file monitor and configure it to emit > watermarks at an interval. > > I'm not sure if these two issues are related. Let me know what you think! > > Thanks, > Daniel Henneberger >
