Hi, We have, by now, a large set of different Beam jobs all written in Python that all write to a set of BigQuery tables that more or less behave the same way in a single dataset. These tables aren't partitioned at all, but going forward, we need them to be.
I partitioned a single table to start with, and was very surprised to find that a Beam job that wrote to it couldn't do so: Failed to copy Non partitioned table to Column partitioned table: not supported. We have a bunch of pre-created tables, and I would've thought I could just keep writing to those without changing settings even after setting up partitioning on them. It doesn't seem to matter whether the create_disposition is CREATE_IF_NEEDED or CREATE_NEVER. It does work when I set the additional_bq_parameters to add time_partitioning, but it would be a huge undertaking not only to update all of our currently running jobs across many projects, but also to make sure to synchronize these changes with updating the underlying BigQuery tables. And it doesn't seem like it should be necessary to specify if we're not creating any new tables? Is there any way to just write the data we have to pre-created, partitioned tables without having to set time_partitioning in additional_bq_parameters, or potentially if there's some other recommended way to solve this problem? Thanks, -Lina
