In general there is no generic source/sink fake/mock/emulator that runs for all sources/sinks locally. Configuration and data is on a case by case basis. Most of the Apache Beam integration tests either launch a local implementation that is specific to the source/sink (such as a DB for JdbcIO) or use the real service and populate it with test data (such as BigQuery).
You might want to take a look at the integration tests that are part of Apache Beam to gather more details. On Thu, Sep 12, 2019 at 7:47 PM deepak kumar <[email protected]> wrote: > Hi All, > I am trying to come up with a framework that would help debugging dataflow > job end to end locally. > Dataflow can work with 100s of source and sinks. > Is there a framework already to setup these sources and sinks locally e.g. > if my dataflow reads from BQ and inserts to Bigtable. > Please let me know if someone is already doing this. > If ues then how. > > Thanks > Deepak >
