Perfect, thanks so much Till! On Thu, Oct 1, 2020 at 5:13 AM Till Rohrmann <trohrm...@apache.org> wrote:
> Hi Austin, > > I believe that the problem is the processing time window. Unlike for event > time where we send a MAX_WATERMARK at the end of the stream to trigger all > remaining windows, this does not happen for processing time windows. Hence, > if your stream ends and you still have an open processing time window, then > it will never get triggered. > > The problem should disappear if you use event time or if you process > unbounded streams which never end. > > Cheers, > Till > > On Thu, Oct 1, 2020 at 12:01 AM Austin Cawley-Edwards < > austin.caw...@gmail.com> wrote: > >> Hey all, >> >> Thanks for your patience. I've got a small repo that reproduces the issue >> here: https://github.com/austince/flink-1.10-sql-windowing-error >> >> Not sure what I'm doing wrong but it feels silly. >> >> Thanks so much! >> Austin >> >> On Tue, Sep 29, 2020 at 3:48 PM Austin Cawley-Edwards < >> austin.caw...@gmail.com> wrote: >> >>> Hey Till, >>> >>> Thanks for the reply -- I'll try to see if I can reproduce this in a >>> small repo and share it with you. >>> >>> Best, >>> Austin >>> >>> On Tue, Sep 29, 2020 at 3:58 AM Till Rohrmann <trohrm...@apache.org> >>> wrote: >>> >>>> Hi Austin, >>>> >>>> could you share with us the exact job you are running (including the >>>> custom window trigger)? This would help us to better understand your >>>> problem. >>>> >>>> I am also pulling in Klou and Timo who might help with the windowing >>>> logic and the Table to DataStream conversion. >>>> >>>> Cheers, >>>> Till >>>> >>>> On Mon, Sep 28, 2020 at 11:49 PM Austin Cawley-Edwards < >>>> austin.caw...@gmail.com> wrote: >>>> >>>>> Hey all, >>>>> >>>>> I'm not sure if I've missed something in the docs, but I'm having a >>>>> bit of trouble with a streaming SQL job that starts w/ raw SQL queries and >>>>> then transitions to a more traditional streaming job. I'm on Flink 1.10 >>>>> using the Blink planner, running locally with no checkpointing. >>>>> >>>>> The job looks roughly like: >>>>> >>>>> CSV 1 --> >>>>> CSV 2 --> SQL Query to Join --> toRetractStream --> keyed time window >>>>> w/ process func & custom trigger --> some other ops >>>>> CSV 3 --> >>>>> >>>>> >>>>> When I remove the windowing directly after the `toRetractStream`, the >>>>> records make it to the "some other ops" stage, but with the windowing, >>>>> those operations are sometimes not sent any data. I can also get data sent >>>>> to the downstream operators by putting in a no-op map before the window >>>>> and >>>>> placing some breakpoints in there to manually slow down processing. >>>>> >>>>> >>>>> The logs don't seem to indicate anything went wrong and generally look >>>>> like: >>>>> >>>>> 4819 [Source: Custom File source (1/1)] INFO >>>>> org.apache.flink.runtime.taskmanager.Task - Source: Custom File source >>>>> (1/1) (3578629787c777320d9ab030c004abd4) switched from RUNNING to >>>>> FINISHED.\4819 [Source: Custom File source (1/1)] INFO >>>>> org.apache.flink.runtime.taskmanager.Task - Freeing task resources for >>>>> Source: Custom File source (1/1) (3578629787c777320d9ab030c004abd4). >>>>> 4819 [Source: Custom File source (1/1)] INFO >>>>> org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem >>>>> streams are closed for task Source: Custom File source (1/1) >>>>> (3578629787c777320d9ab030c004abd4) [FINISHED] >>>>> 4820 [flink-akka.actor.default-dispatcher-5] INFO >>>>> org.apache.flink.runtime.taskexecutor.TaskExecutor - Un-registering task >>>>> and sending final execution state FINISHED to JobManager for task Source: >>>>> Custom File source (1/1) 3578629787c777320d9ab030c004abd4. >>>>> ... >>>>> 4996 [Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, >>>>> ProcessWindowFunction$1) (1/1)] INFO >>>>> org.apache.flink.runtime.taskmanager.Task - >>>>> Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, >>>>> ProcessWindowFunction$1) (1/1) (907acf9bfa2f4a9bbd13997b8b34d91f) switched >>>>> from RUNNING to FINISHED. >>>>> 4996 [Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, >>>>> ProcessWindowFunction$1) (1/1)] INFO >>>>> org.apache.flink.runtime.taskmanager.Task - Freeing task resources for >>>>> Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, >>>>> ProcessWindowFunction$1) (1/1) (907acf9bfa2f4a9bbd13997b8b34d91f). >>>>> 4996 [Window(TumblingProcessingTimeWindows(60000), TimedCountTrigger, >>>>> ProcessWindowFunction$1) (1/1)] INFO >>>>> org.apache.flink.runtime.taskmanager.Task - Ensuring all FileSystem >>>>> streams are closed for task Window(TumblingProcessingTimeWindows(60000), >>>>> TimedCountTrigger, ProcessWindowFunction$1) (1/1) >>>>> (907acf9bfa2f4a9bbd13997b8b34d91f) [FINISHED] >>>>> ... >>>>> rest of the shutdown >>>>> ... >>>>> Program execution finished >>>>> Job with JobID 889b161e432c0e69a8d760bbed205d5d has finished. >>>>> Job Runtime: 783 ms >>>>> >>>>> >>>>> Is there something I'm missing in my setup? Could it be my custom >>>>> window trigger? Bug? I'm stumped. >>>>> >>>>> >>>>> Thanks, >>>>> Austin >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > >
