Hi Austin, could you share with us the exact job you are running (including the custom window trigger)? This would help us to better understand your problem.
I am also pulling in Klou and Timo who might help with the windowing logic and the Table to DataStream conversion. Cheers, Till On Mon, Sep 28, 2020 at 11:49 PM Austin Cawley-Edwards < austin.caw...@gmail.com> wrote: > Hey all, > > I'm not sure if I've missed something in the docs, but I'm having a bit of > trouble with a streaming SQL job that starts w/ raw SQL queries and then > transitions to a more traditional streaming job. I'm on Flink 1.10 using > the Blink planner, running locally with no checkpointing. > > The job looks roughly like: > > CSV 1 --> > CSV 2 --> SQL Query to Join --> toRetractStream --> keyed time window w/ > process func & custom trigger --> some other ops > CSV 3 --> > > > When I remove the windowing directly after the `toRetractStream`, the > records make it to the "some other ops" stage, but with the windowing, > those operations are sometimes not sent any data. I can also get data sent > to the downstream operators by putting in a no-op map before the window and > placing some breakpoints in there to manually slow down processing. > > > The logs don't seem to indicate anything went wrong and generally look > like: > > 4819 [Source: Custom File source (1/1)] INFO > org.apache.flink.runtime.taskmanager.Task - Source: Custom File source > (1/1) (3578629787c777320d9ab030c004abd4) switched from RUNNING to > FINISHED.\4819 [Source: Custom File source (1/1)] INFO > org.apache.flink.runtime.taskmanager.Task - Freeing task resources for > Source: Custom File source (1/1) (3578629787c777320d9ab030c004abd4). > 4819 [Source: Custom File source (1/1)] INFO > org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem > streams are closed for task Source: Custom File source (1/1) > (3578629787c777320d9ab030c004abd4) [FINISHED] > 4820 [flink-akka.actor.default-dispatcher-5] INFO > org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task > and sending final execution state FINISHED to JobManager for task Source: > Custom File source (1/1) 3578629787c777320d9ab030c004abd4. > ... > 4996 [Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, > ProcessWindowFunction$1) (1/1)] INFO > org.apache.flink.runtime.taskmanager.Task - > Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, > ProcessWindowFunction$1) (1/1) (907acf9bfa2f4a9bbd13997b8b34d91f) switched > from RUNNING to FINISHED. > 4996 [Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, > ProcessWindowFunction$1) (1/1)] INFO > org.apache.flink.runtime.taskmanager.Task - Freeing task resources for > Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, > ProcessWindowFunction$1) (1/1) (907acf9bfa2f4a9bbd13997b8b34d91f). > 4996 [Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, > ProcessWindowFunction$1) (1/1)] INFO > org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem > streams are closed for task Window(TumblingProcessingTimeWindows(60000), > TimedCountTrigger, ProcessWindowFunction$1) (1/1) > (907acf9bfa2f4a9bbd13997b8b34d91f) [FINISHED] > ... > rest of the shutdown > ... > Program execution finished > Job with JobID 889b161e432c0e69a8d760bbed205d5d has finished. > Job Runtime: 783 ms > > > Is there something I'm missing in my setup? Could it be my custom window > trigger? Bug? I'm stumped. > > > Thanks, > Austin > > >
