Re: Batch load with BigQueryIO fails because of a few bad records.

Thank you for responding Evan. It looks like these options will only work for STREAMING_INSERTS. Are there any options for BATCH_LOADS, and if not are there any plans for it? On Thu, May 6, 2021 at 6:11 PM Evan Galpin wrote: > Hey Matthew, > > I believe you might also need to use the “ignoreUn

Re: Batch load with BigQueryIO fails because of a few bad records.

ignoreUnknownValues is supported for BATCH_LOADS as well. On Fri, May 7, 2021 at 7:08 AM Matthew Ouyang wrote: > Thank you for responding Evan. It looks like these options will only work > for STREAMING_INSERTS. Are there any options for BATCH_LOADS, and if not > are there any plans for it? >

How Beam SQL Side Input refresh/update

Hi, Based on Join documentation. If I have a Join with Unbounded and Bounded > For this type of JOIN bounded input is treated as a side-input by the > implementation. This means that window/trigger is inherented from upstreams. On my pipeline I dont have any triggering or window. I use a global

Re: How Beam SQL Side Input refresh/update

We call this a Slowly Changing Dimensions join, there was a previous effort to add this to Beam that is partially implemented in Java . Unfortunately we haven't finished

DirectRunner, Fusion, and Triggers

Hi Beam-users, *TL;DR;* I wonder if DirectRunner does any fusion optimization and whether this has any impact on triggers/panes? *Details* (the context for everything below is *DirectRunner* and this is a *batch* job): I hav