Thanks that looks like it is the issue, I appreciate the help.
From: Thomas Fredriksen(External)
Sent: Monday, June 7, 2021 12:53 AM
To: user@beam.apache.org
Subject: Re: [2.28.0] [Java] [protobuf] ProtoMessageSchema doesn't create
fields as nullable
Hi Andrew,
I believe that as of proto 3.12, optional fields are supported directly -
https://github.com/pseudomuto/protoc-gen-doc/issues/422 . _think_ this
should be supported by Beam (assuming Beam uses a new-enough proto
library), but I'm not sure if it's been tested.
On Mon, Jun 7, 2021 at 8:53 AM Andrew
The problem is that protobuf primitives are represented in Java as
primitives, which are not nullable.
Ideally, they should be objects instead, but alas - no.
The wrapper is a decent (but not perfect) workaround.
On Mon, Jun 7, 2021, 18:01 Reuven Lax wrote:
> I believe that as of proto 3.12, o
That's why separate has_xxx methods are generated to test whether the
specified field is present or not.
On Mon, Jun 7, 2021 at 9:17 AM Thomas Fredriksen(External) <
thomas.fredrik...@cognite.com> wrote:
> The problem is that protobuf primitives are represented in Java as
> primitives, which are
Unfortunately, the problem is that we are taking serialized protobuf messages
from pubsub and writing them to Avro, so I was taking the message, using the
payload to create the object, then converting -> (Beam) Row -> GenericRecord
(Avro) -> Write to storage. I was using the ProtoMessageSchema.s
Regarding the ProtoSchemaTranslator, I can't use that class as it was changed
from public to default in 2.20 and above.
https://github.com/apache/beam/blame/v2.20.0/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoSchemaTranslator.java#L128
__
I understand. However if the field is marked as optional in the proto
definition, Beam should create a nullable field.
On Mon, Jun 7, 2021 at 10:54 AM Andrew Kettmann
wrote:
> Unfortunately, the problem is that we are taking serialized protobuf
> messages from pubsub and writing them to Avro, so
FYI - this should be fixed by https://github.com/apache/beam/pull/14960
On Thu, Jun 3, 2021 at 10:00 AM Reuven Lax wrote:
> Correct.
>
> On Thu, Jun 3, 2021 at 9:51 AM Kenneth Knowles wrote:
>
>> I still don't quite grok the details of how this succeeds or fails in
>> different situations. The
Yes please!
On Thu, Jun 3, 2021, 18:32 Ratnakar Malla wrote:
> +1
>
>
> --
> *From:* Austin Bennett
> *Sent:* Thursday, June 3, 2021 6:20:25 PM
> *To:* user@beam.apache.org
> *Cc:* dev
> *Subject:* Re: Allyship workshops for open source contributors
>
> +1, assumin
