I mean how do I create empty list if the element type is unknown at compile time.
On Tue, Oct 1, 2024 at 12:42 AM [email protected] <[email protected]> wrote: > Thanks @Ahmed Abualsaud <[email protected]> but how do I get > around this error for now if I want to use beam schema? > > On Mon, Sep 30, 2024 at 4:31 PM Ahmed Abualsaud via user < > [email protected]> wrote: > >> Hey Siyuan, >> >> We use the descriptor because it is derived from the BQ table's schema In >> a previous step [1]. We are essentially checking against the table schema. >> You're seeing this error because *nullable* and *repeated* modes are >> mutually exclusive. I think we can reduce friction though by defaulting >> null values to an empty list, which seems to be in line with GoogleSQL's >> behavior [2]. >> >> Opened a PR for this: https://github.com/apache/beam/pull/32604. >> Hopefully we can get this in for the upcoming Beam version 2.60.0 >> >> For now, you can work around this by converting your null array values to >> empty lists. >> >> [1] >> https://github.com/apache/beam/blob/111f4c34ab2efd166de732c32d99ff615abf6064/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/StorageApiDynamicDestinationsBeamRow.java#L66-L67 >> [2] >> https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#array_nulls >> >> On Mon, Sep 30, 2024 at 6:57 PM [email protected] <[email protected]> >> wrote: >> >>> I'm trying to write Beam row directly to bigquery because it would go >>> through less conversion and more efficient but there is some weird error >>> happening >>> A nullable array field would throw >>> >>> Caused by: java.lang.IllegalArgumentException: Received null value for >>> non-nullable field >>> >>> If I set null for that field >>> >>> Here is code in beam I found related >>> >>> >>> https://github.com/apache/beam/blob/111f4c34ab2efd166de732c32d99ff615abf6064/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BeamRowToStorageApiProto.java#L277 >>> >>> private static Object messageValueFromRowValue( >>> FieldDescriptor fieldDescriptor, Field beamField, int index, Row >>> row) { >>> @Nullable Object value = row.getValue(index); >>> if (value == null) { >>> if (fieldDescriptor.isOptional()) { >>> return null; >>> } else { >>> throw new IllegalArgumentException( >>> "Received null value for non-nullable field " + >>> fieldDescriptor.getName()); >>> } >>> } >>> return toProtoValue(fieldDescriptor, beamField.getType(), value); >>> } >>> >>> line 277 why not use beamField.isNullable() instead of >>> fieldDescriptior.isOptional() It it's useing beam schema it should stick to >>> nullable setting on beam schema field, correct? >>> >>> And how do I avoid this? >>> >>> Regards, >>> Siyuan >>> >>
