That's why separate has_xxx methods are generated to test whether the specified field is present or not.
On Mon, Jun 7, 2021 at 9:17 AM Thomas Fredriksen(External) < thomas.fredrik...@cognite.com> wrote: > The problem is that protobuf primitives are represented in Java as > primitives, which are not nullable. > > Ideally, they should be objects instead, but alas - no. > > The wrapper is a decent (but not perfect) workaround. > > On Mon, Jun 7, 2021, 18:01 Reuven Lax <re...@google.com> wrote: > >> I believe that as of proto 3.12, optional fields are supported directly - >> https://github.com/pseudomuto/protoc-gen-doc/issues/422 . _think_ this >> should be supported by Beam (assuming Beam uses a new-enough proto >> library), but I'm not sure if it's been tested. >> >> On Mon, Jun 7, 2021 at 8:53 AM Andrew Kettmann <akettm...@evolve24.com> >> wrote: >> >>> Thanks that looks like it is the issue, I appreciate the help. >>> ------------------------------ >>> *From:* Thomas Fredriksen(External) <thomas.fredrik...@cognite.com> >>> *Sent:* Monday, June 7, 2021 12:53 AM >>> *To:* user@beam.apache.org <user@beam.apache.org> >>> *Subject:* Re: [2.28.0] [Java] [protobuf] ProtoMessageSchema doesn't >>> create fields as nullable >>> >>> Hi Andrew, >>> >>> From the documentation ( >>> https://beam.apache.org/releases/javadoc/2.19.0/org/apache/beam/sdk/extensions/protobuf/ProtoSchemaTranslator.html >>> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Freleases%2Fjavadoc%2F2.19.0%2Forg%2Fapache%2Fbeam%2Fsdk%2Fextensions%2Fprotobuf%2FProtoSchemaTranslator.html&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013163228%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=iFaa1RyJbbRVIU%2FqnDoxvvJzXuxSrkMwilc8jYFh%2Btc%3D&reserved=0> >>> ): >>> >>> Protobuf wrapper classes are translated to nullable types, as follows. >>> >>> - google.protobuf.Int32Value maps to a nullable FieldType.INT32 >>> - google.protobuf.Int64Value maps to a nullable FieldType.INT64 >>> - google.protobuf.UInt32Value maps to a nullable >>> FieldType.logicalType(new UInt32()) >>> - google.protobuf.UInt64Value maps to a nullable >>> Field.logicalType(new UInt64()) >>> - google.protobuf.FloatValue maps to a nullable FieldType.FLOAT >>> - google.protobuf.DoubleValue maps to a nullable FieldType.DOUBLE >>> - google.protobuf.BoolValue maps to a nullable FieldType.BOOLEAN >>> - google.protobuf.StringValue maps to a nullable FieldType.STRING >>> - google.protobuf.BytesValue maps to a nullable FieldType.BYTES >>> >>> This means that you should use the google wrapper-types in order to >>> achieve nullable fields. >>> The wrapper is available here: >>> https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/wrappers.proto >>> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fprotocolbuffers%2Fprotobuf%2Fblob%2Fmaster%2Fsrc%2Fgoogle%2Fprotobuf%2Fwrappers.proto&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013173227%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ym8iX%2F4BnQsnbDfsj4sSa6F4xJEIinWSsPGnDsIc8ng%3D&reserved=0> >>> . >>> >>> Hope it helps :) >>> >>> On Thu, Jun 3, 2021 at 6:48 PM Andrew Kettmann <akettm...@evolve24.com> >>> wrote: >>> >>> Using org.apache.beam.sdk.extensions.protobuf.ProtoMessageSchema to >>> create a beam schema from generated protobuf3 classes. However, >>> org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator#beamFieldTypeFromSingularProtoField >>> doesn't apply nullable to fields in the message. My understanding is that >>> by default protobuf fields ARE optional, is that incorrect? Converting from >>> a serialized message without values for some fields crashes when it tries >>> to cast them to a Row since the Row is not expecting a field as nullable. >>> >>> Anyone have any advice regarding this? Modify the schema after it is >>> generated by ProtoMessageSchema or is there another method/option I am >>> missing? >>> >>> >>> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.evolve24.com%2F&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013183214%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Rwn8bQVGLtAX5efP0JWFaABkvFRDw6Ql0WizkQ6ouR0%3D&reserved=0> >>> *Andrew >>> Kettmann* >>> DevOps Engineer >>> P: 1.314.596.2836 <(314)%20596-2836> >>> [image: LinkedIn] >>> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinkedin.com%2Fcompany%2Fevolve24&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013193221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=fFRlXtpBwHpYmeVwJL5DZ4cjFEpnXfW%2FZDVUBVLYbLc%3D&reserved=0> >>> [image: >>> Twitter] >>> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fevolve24&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013193221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=sOKlR8T6dH88boG5C6JS3muO3AQxBUWr%2B9Q0w2AG934%3D&reserved=0> >>> [image: >>> Instagram] >>> <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.instagram.com%2Fevolve_24&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013203210%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=yBQR%2BPmDOURPaeHR5lVY00vADdu1OYpxda64UaxOmzI%3D&reserved=0> >>> >>> evolve24 Confidential & Proprietary Statement: This email and any >>> attachments are confidential and may contain information that is >>> privileged, confidential or exempt from disclosure under applicable law. It >>> is intended for the use of the recipients. If you are not the intended >>> recipient, or believe that you have received this communication in error, >>> please do not read, print, copy, retransmit, disseminate, or otherwise use >>> the information. Please delete this email and attachments, without reading, >>> printing, copying, forwarding or saving them, and notify the Sender >>> immediately by reply email. No confidentiality or privilege is waived or >>> lost by any transmission in error. >>> >>>
