[Question] proper type for beam udf for complex types

2022-04-06 Thread Balogh , György
Hi, I'd like to use BeamSqlUdf-s with complex (potential nested) types. It is not clear how to specify the type to achieve this. Eg.: I've a rectangle with x,y,width,height and want to operate on this. I'm getting this error: No match found for function signature iou() -> [Help 1] So I'd like to

Go SDK - Docker image for Dataflow

2022-04-06 Thread Pawel
Hi, I'm wondering if it is possible to create custom docker image with compiled go binary that can be deployed to Dataflow? Currently I'm using it as below myjob   --runner dataflow   --output gs://prog-test-bucket/out   --project ***   --region europe-west1   --temp_location gs://prog-

Re: [Question] infer schema from proto in java?

2022-04-06 Thread Balogh , György
Correct, SqlTransform works well without explicit conversion. The Beam SQL Walkthrough page was a bit misleading. It says: "Before applying a SQL query to a PCollection, the data in the collection must be in Row format" and shows examples how to achieve it. Thank you! Gyorgy On Wed, Apr 6, 2022 a

Re: Breaking change for FileIO WriteDynamic in Beam 2.34?

2022-04-06 Thread Steve Niemitz
Without the full logs it's hard to say, but I've definitely seen that error in the past when the worker disks are full. ApplianceShuffleWriter needs to extract a native library to a temp location, and if the disk is full that'll fail, resulting in the NoClassDefFoundError. On Wed, Apr 6, 2022 at