Regarding the ProtoSchemaTranslator, I can't use that class as it was changed from public to default in 2.20 and above. https://github.com/apache/beam/blame/v2.20.0/sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoSchemaTranslator.java#L128
________________________________ From: Thomas Fredriksen(External) <thomas.fredrik...@cognite.com> Sent: Monday, June 7, 2021 12:53 AM To: user@beam.apache.org <user@beam.apache.org> Subject: Re: [2.28.0] [Java] [protobuf] ProtoMessageSchema doesn't create fields as nullable Hi Andrew, >From the documentation >(https://beam.apache.org/releases/javadoc/2.19.0/org/apache/beam/sdk/extensions/protobuf/ProtoSchemaTranslator.html<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Freleases%2Fjavadoc%2F2.19.0%2Forg%2Fapache%2Fbeam%2Fsdk%2Fextensions%2Fprotobuf%2FProtoSchemaTranslator.html&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013163228%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=iFaa1RyJbbRVIU%2FqnDoxvvJzXuxSrkMwilc8jYFh%2Btc%3D&reserved=0>): Protobuf wrapper classes are translated to nullable types, as follows. * google.protobuf.Int32Value maps to a nullable FieldType.INT32 * google.protobuf.Int64Value maps to a nullable FieldType.INT64 * google.protobuf.UInt32Value maps to a nullable FieldType.logicalType(new UInt32()) * google.protobuf.UInt64Value maps to a nullable Field.logicalType(new UInt64()) * google.protobuf.FloatValue maps to a nullable FieldType.FLOAT * google.protobuf.DoubleValue maps to a nullable FieldType.DOUBLE * google.protobuf.BoolValue maps to a nullable FieldType.BOOLEAN * google.protobuf.StringValue maps to a nullable FieldType.STRING * google.protobuf.BytesValue maps to a nullable FieldType.BYTES This means that you should use the google wrapper-types in order to achieve nullable fields. The wrapper is available here: https://github.com/protocolbuffers/protobuf/blob/master/src/google/protobuf/wrappers.proto<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fprotocolbuffers%2Fprotobuf%2Fblob%2Fmaster%2Fsrc%2Fgoogle%2Fprotobuf%2Fwrappers.proto&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013173227%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ym8iX%2F4BnQsnbDfsj4sSa6F4xJEIinWSsPGnDsIc8ng%3D&reserved=0>. Hope it helps :) On Thu, Jun 3, 2021 at 6:48 PM Andrew Kettmann <akettm...@evolve24.com<mailto:akettm...@evolve24.com>> wrote: Using org.apache.beam.sdk.extensions.protobuf.ProtoMessageSchema to create a beam schema from generated protobuf3 classes. However, org.apache.beam.sdk.extensions.protobuf.ProtoSchemaTranslator#beamFieldTypeFromSingularProtoField doesn't apply nullable to fields in the message. My understanding is that by default protobuf fields ARE optional, is that incorrect? Converting from a serialized message without values for some fields crashes when it tries to cast them to a Row since the Row is not expecting a field as nullable. Anyone have any advice regarding this? Modify the schema after it is generated by ProtoMessageSchema or is there another method/option I am missing? [https://storage.googleapis.com/e24-email-images/e24logonotag.png]<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.evolve24.com%2F&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013183214%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Rwn8bQVGLtAX5efP0JWFaABkvFRDw6Ql0WizkQ6ouR0%3D&reserved=0> Andrew Kettmann DevOps Engineer P: 1.314.596.2836 [LinkedIn]<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Flinkedin.com%2Fcompany%2Fevolve24&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013193221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=fFRlXtpBwHpYmeVwJL5DZ4cjFEpnXfW%2FZDVUBVLYbLc%3D&reserved=0> [Twitter] <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fevolve24&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013193221%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=sOKlR8T6dH88boG5C6JS3muO3AQxBUWr%2B9Q0w2AG934%3D&reserved=0> [Instagram] <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.instagram.com%2Fevolve_24&data=04%7C01%7Cakettmann%40evolve24.com%7C4bbc2c07b6594eaaf30308d929789be6%7Ce36287f1b44849498093fe543a560976%7C0%7C0%7C637586421013203210%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=yBQR%2BPmDOURPaeHR5lVY00vADdu1OYpxda64UaxOmzI%3D&reserved=0> evolve24 Confidential & Proprietary Statement: This email and any attachments are confidential and may contain information that is privileged, confidential or exempt from disclosure under applicable law. It is intended for the use of the recipients. If you are not the intended recipient, or believe that you have received this communication in error, please do not read, print, copy, retransmit, disseminate, or otherwise use the information. Please delete this email and attachments, without reading, printing, copying, forwarding or saving them, and notify the Sender immediately by reply email. No confidentiality or privilege is waived or lost by any transmission in error.
