Can you share some more details, such as code? We may identify something that relies upon assumptions from batch execution style.
Also notably the Java DirectRunner does not have separate batch/streaming mode. It always executes in a "streaming" sort of way. It is also simpler in some ways so if you can reproduce it on the DirectRunner that might help. Kenn On Tue, May 11, 2021 at 3:41 PM Raman Gupta <rocketra...@gmail.com> wrote: > I have a Dataflow pipeline that reads data from JDBC and Pub/Sub. My ideal > pipeline backfills its state and output from historical data via the JDBC > input, and then continues processing new elements arriving via pub/sub. > Conceptually, this seems easy to do with a filter on each source > before/after some specific cutoff instant. > > However, when I add pub/sub into the pipeline, it runs in streaming mode, > and the pipeline does not produce the expected results -- all of the > results that would be produced based on looping timers seem to be missing. > > I thought this might be related to the post-inputs Flatten, but I've taken > pub/sub out of the equation, and run the same exact JDBC-based pipeline in > batch vs streaming mode, and the JDBC-only pipeline in streaming mode > produces the same partial results. > > What could be happening? > > Regards, > Raman > >
