Thanks @Ahmed Abualsaud <[email protected]> but how do I get around this error for now if I want to use beam schema?
On Mon, Sep 30, 2024 at 4:31 PM Ahmed Abualsaud via user < [email protected]> wrote: > Hey Siyuan, > > We use the descriptor because it is derived from the BQ table's schema In > a previous step [1]. We are essentially checking against the table schema. > You're seeing this error because *nullable* and *repeated* modes are > mutually exclusive. I think we can reduce friction though by defaulting > null values to an empty list, which seems to be in line with GoogleSQL's > behavior [2]. > > Opened a PR for this: https://github.com/apache/beam/pull/32604. > Hopefully we can get this in for the upcoming Beam version 2.60.0 > > For now, you can work around this by converting your null array values to > empty lists. > > [1] > https://github.com/apache/beam/blob/111f4c34ab2efd166de732c32d99ff615abf6064/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiDynamicDestinationsBeamRow.java#L66-L67 > [2] > https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#array_nulls > > On Mon, Sep 30, 2024 at 6:57 PM [email protected] <[email protected]> wrote: > >> I'm trying to write Beam row directly to bigquery because it would go >> through less conversion and more efficient but there is some weird error >> happening >> A nullable array field would throw >> >> Caused by: java.lang.IllegalArgumentException: Received null value for >> non-nullable field >> >> If I set null for that field >> >> Here is code in beam I found related >> >> >> https://github.com/apache/beam/blob/111f4c34ab2efd166de732c32d99ff615abf6064/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BeamRowToStorageApiProto.java#L277 >> >> private static Object messageValueFromRowValue( >> FieldDescriptor fieldDescriptor, Field beamField, int index, Row >> row) { >> @Nullable Object value = row.getValue(index); >> if (value == null) { >> if (fieldDescriptor.isOptional()) { >> return null; >> } else { >> throw new IllegalArgumentException( >> "Received null value for non-nullable field " + >> fieldDescriptor.getName()); >> } >> } >> return toProtoValue(fieldDescriptor, beamField.getType(), value); >> } >> >> line 277 why not use beamField.isNullable() instead of >> fieldDescriptior.isOptional() It it's useing beam schema it should stick to >> nullable setting on beam schema field, correct? >> >> And how do I avoid this? >> >> Regards, >> Siyuan >> >
