I don’t think it exists. Do you really need to have an unbounded pipeline, meaning that the data will continuously arrive, or just re-running a batch pipeline once per some amount of time or externally triggered by some signal shouldn’t be enough?
— Alexey > On 22 Apr 2022, at 13:40, Eric Berryman <eric.berry...@gmail.com> wrote: > > Does an unbounded JdbcIO exist, or would I need to wrap the existing one in a > spilttable DoFn? Or maybe there is an easier way to do it? > > Thank you again, > Eric > > > > On Wed, Apr 20, 2022, 21:59 Ahmet Altay <al...@google.com > <mailto:al...@google.com>> wrote: > /cc @Pablo Estrada <mailto:pabl...@google.com> @John Casey > <mailto:johnjca...@google.com> > > On Wed, Apr 20, 2022 at 6:29 PM Eric Berryman <eric.berry...@gmail.com > <mailto:eric.berry...@gmail.com>> wrote: > Hello, > > I have a rather simple use case where I would like to read a db table, which > acts as a queue (~ hundreds millions events in initial load, but only > thousands of events per day), and write that data out to a sink. This > pipeline would be unbounded. > > I'm looking for reading material, and or code, which displays reading from > the JdbcIO API with checkpoints. I would like to avoid the initial load on > restarts, upgrades, etc. :) > > Thank you for your time! > Eric
