I believe that as of proto 3.12, optional fields are supported directly - https://github.com/pseudomuto/protoc-gen-doc/issues/422 . _think_ this should be supported by Beam (assuming Beam uses a new-enough proto library), but I'm not sure if it's been tested.
On Mon, Jun 7, 2021 at 8:53 AM Andrew Kettmann <akettm...@evolve24.com> wrote: > Thanks that looks like it is the issue, I appreciate the help. > ------------------------------ > *From:* Thomas Fredriksen(External) <thomas.fredrik...@cognite.com> > *Sent:* Monday, June 7, 2021 12:53 AM > *To:* user@beam.apache.org <user@beam.apache.org> > *Subject:* Re: [2.28.0] [Java] [protobuf] ProtoMessageSchema doesn't > create fields as nullable > > Hi Andrew, > > From the documentation ( > https://beam.apache.org/releases/javadoc/2.19.0/org/apache/beam/sdk/extensions/protobuf/ProtoSchemaTranslator.html > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Freleases%2Fjavadoc%2F2.19.0%2Forg%2Fapache%2Fbeam%2Fsdk%2Fextensions%2Fprotobuf%2FProtoSchemaTranslator.html&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013163228%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=iFaa1RyJbbRVIU%2FqnDoxvvJzXuxSrkMwilc8jYFh%2Btc%3D&reserved=0> > ): > > Protobuf wrapper classes are translated to nullable types, as follows. > > - google.protobuf.Int32Value maps to a nullable FieldType.INT32 > - google.protobuf.Int64Value maps to a nullable FieldType.INT64 > - google.protobuf.UInt32Value maps to a nullable > FieldType.logicalType(new UInt32()) > - google.protobuf.UInt64Value maps to a nullable Field.logicalType(new > UInt64()) > - google.protobuf.FloatValue maps to a nullable FieldType.FLOAT > - google.protobuf.DoubleValue maps to a nullable FieldType.DOUBLE > - google.protobuf.BoolValue maps to a nullable FieldType.BOOLEAN > - google.protobuf.StringValue maps to a nullable FieldType.STRING > - google.protobuf.BytesValue maps to a nullable FieldType.BYTES > > This means that you should use the google wrapper-types in order to > achieve nullable fields. > The wrapper is available here: > https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/wrappers.proto > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fprotocolbuffers%2Fprotobuf%2Fblob%2Fmaster%2Fsrc%2Fgoogle%2Fprotobuf%2Fwrappers.proto&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013173227%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ym8iX%2F4BnQsnbDfsj4sSa6F4xJEIinWSsPGnDsIc8ng%3D&reserved=0> > . > > Hope it helps :) > > On Thu, Jun 3, 2021 at 6:48 PM Andrew Kettmann <akettm...@evolve24.com> > wrote: > > Using org.apache.beam.sdk.extensions.protobuf.ProtoMessageSchema to create > a beam schema from generated protobuf3 classes. However, > org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator#beamFieldTypeFromSingularProtoField > doesn't apply nullable to fields in the message. My understanding is that > by default protobuf fields ARE optional, is that incorrect? Converting from > a serialized message without values for some fields crashes when it tries > to cast them to a Row since the Row is not expecting a field as nullable. > > Anyone have any advice regarding this? Modify the schema after it is > generated by ProtoMessageSchema or is there another method/option I am > missing? > > > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.evolve24.com%2F&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013183214%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Rwn8bQVGLtAX5efP0JWFaABkvFRDw6Ql0WizkQ6ouR0%3D&reserved=0> > *Andrew > Kettmann* > DevOps Engineer > P: 1.314.596.2836 <(314)%20596-2836> > [image: LinkedIn] > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinkedin.com%2Fcompany%2Fevolve24&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013193221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=fFRlXtpBwHpYmeVwJL5DZ4cjFEpnXfW%2FZDVUBVLYbLc%3D&reserved=0> > [image: > Twitter] > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fevolve24&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013193221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=sOKlR8T6dH88boG5C6JS3muO3AQxBUWr%2B9Q0w2AG934%3D&reserved=0> > [image: > Instagram] > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.instagram.com%2Fevolve_24&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013203210%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=yBQR%2BPmDOURPaeHR5lVY00vADdu1OYpxda64UaxOmzI%3D&reserved=0> > > evolve24 Confidential & Proprietary Statement: This email and any > attachments are confidential and may contain information that is > privileged, confidential or exempt from disclosure under applicable law. It > is intended for the use of the recipients. If you are not the intended > recipient, or believe that you have received this communication in error, > please do not read, print, copy, retransmit, disseminate, or otherwise use > the information. Please delete this email and attachments, without reading, > printing, copying, forwarding or saving them, and notify the Sender > immediately by reply email. No confidentiality or privilege is waived or > lost by any transmission in error. > >
