Greetings, I have an issue with the existing Java Beam pipeline that I need to edit. I haven't used Beam for some time and Java not to mention :) The pipeline reads data from Cloud SQL and inserts data into BigQuery. I need to provide to the template as input options table name and date based on which: 1. query for the database will be constructed, something like this: SELECT * FROM <table name> WHERE created_at < "date". This is used in step when reading from Cloud SQL. 2. BigQuery table name for partition, i.e. "tablename_yyyymmdd", i.e. table name + date from input. this is used when writing data to BigQuery.
I have an issue with putting together the table name and date, doing simple string formatting, and then using it further down the pipeline. Since this is evaluated during runtime, I need to use them as ValueProvider. I'm aware of NestedValueProvider, but as far as I understand only one input variable can be used. I tried to play with some custom DoFn but without success. I also tried to search on the web for similar examples but without success, so I'm not sure if I'm not missing something obvious, again I don't have much experience with Java Beam SDK. I would be grateful for any advice or pointing to some similar example. Best regards, -- Zdenko Hrček _______________________ https://www.the-swamp.info https://www.gcpweekly.com/
