Ok thanks, I was trying to avoid this, to compile and use the generated classes. Now we are currently processing the raw protobuf message as a map of UnknownFieldSet, and I was wondering if there was a choice to make this "rewrite" without the need to compile and use the class, just traversing the protobuf.
I will check and try. Thanks, Joan. On Wednesday, September 20, 2023 at 12:30:26 PM UTC+2 Florian Enner wrote: > As the very first step you should compile your schema and work with the > generated classes: > https://protobuf.dev/getting-started/javatutorial/#compiling-protocol-buffers > > Don't mess with the reflection API if you can avoid it. > > > On Wednesday, September 20, 2023 at 12:19:38 PM UTC+2 Joan Balagueró wrote: > >> Hi Florian, >> >> Not sure what I can do with this code. I only have a byte array >> representing the above protobuf, I don't have any object or message. At >> most I have a map with the list of the "UnknownFieldSet.Field" fields after >> parsing the byte array (this is java code). >> >> I'm relatively new to protobuf, I read a lot on the Internet trying to >> find a solution but I did not find anything. >> >> That's why I tried to traverse the map of fields and write them to the >> new byte array. It's the only solution I could think of, but I'm doing >> something wrong. >> >> Not sure if you can help me a bit more to solve this. >> >> Anyways thanks. >> >> Joan. >> >> >> >> >> >> On Wednesday, September 20, 2023 at 11:56:19 AM UTC+2 Florian Enner wrote: >> >>> Messages are serialized with a length delimiter, so changing the content >>> produces a mismatch and invalid message. >>> >>> Your schema has no affected repeated fields, so appending a delta should >>> work. I've never used the C# API, but here is some hopefully understandable >>> pseudo code: >>> >>> var delta = Request.newInstance(); >>> delta.getMutableMeta().getMutableCutOffTime() >>> .setValue(value) >>> .setScale(TimeSpanScale.MINMAX) >>> byte[] output = append(unmodifiedInputBytes, delta.toByteArray()); >>> >>> If the server expects a length delimiter you'd need to update it to the >>> new combined length. >>> >>> - Florian >>> >>> On Wednesday, September 20, 2023 at 11:03:46 AM UTC+2 Joan Balagueró >>> wrote: >>> >>>> Hi Florian, >>>> >>>> Thanks for your quick response. I'm stuck on this. >>>> >>>> 1) It's not working. When I send the protobuf to the backend server >>>> (it's not our api nor server) using the first method, I get a right >>>> response. But using the second method I receive this error: >>>> ProtoBuf.ProtoException: Invalid wire-type; this usually means you have >>>> over-written a file without truncating or setting the length; see >>>> https://stackoverflow.com/q/2152978/23354 >>>> at ProtoBuf.ProtoReader.StartSubItem(ProtoReader reader) in >>>> C:\code\protobuf-net\src\protobuf-net\ProtoReader.cs:line 637 >>>> at ProtoBuf.ProtoReader.ReadTypedObject(Object value, Int32 key, >>>> ProtoReader reader, Type type) in >>>> C:\code\protobuf-net\src\protobuf-net\ProtoReader.cs:line 584 >>>> at proto_40(Object , ProtoReader ) >>>> at ProtoBuf.Meta.TypeModel.DeserializeCore(ProtoReader reader, Type >>>> type, Object value, Boolean noAutoCreate) in >>>> C:\code\protobuf-net\src\protobuf-net\Meta\TypeModel.cs:line 722 >>>> at ProtoBuf.Meta.TypeModel.Deserialize(Stream source, Object value, >>>> Type type, SerializationContext context) in >>>> C:\code\protobuf-net\src\protobuf-net\Meta\TypeModel.cs:line 599 >>>> at >>>> WebBeds.Connect.AspNetCore.Formatters.ProtobufInputFormatter.ReadRequestBodyAsync(InputFormatterContext >>>> >>>> context) >>>> at >>>> Microsoft.AspNetCore.Mvc.Formatters.InputFormatter.ReadAsync(InputFormatterContext >>>> >>>> context) >>>> at >>>> Microsoft.AspNetCore.Mvc.ModelBinding.Binders.BodyModelBinder.BindModelAsync(ModelBindingContext >>>> >>>> bindingContext) >>>> at >>>> Microsoft.AspNetCore.Mvc.ModelBinding.ParameterBinder.BindModelAsync(ActionContext >>>> >>>> actionContext, IModelBinder modelBinder, IValueProvider valueProvider, >>>> ParameterDescriptor parameter, ModelMetadata metadata, Object value, >>>> Object >>>> container) >>>> at >>>> Microsoft.AspNetCore.Mvc.Controllers.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<<CreateBinderDelegate>g__Bind|0>d.MoveNext() >>>> --- End of stack trace from previous location --- >>>> at >>>> Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker >>>> >>>> invoker, Task lastTask, State next, Scope scope, Object state, Boolean >>>> isCompleted) >>>> at >>>> Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextExceptionFilterAsync>g__Awaited|26_0(ResourceInvoker >>>> >>>> invoker, Task lastTask, State next, Scope scope, Object state, Boolean >>>> isCompleted >>>> >>>> >>>> 2) This is the proto: >>>> >>>> The request with the 'Meta' element: >>>> >>>> message Request { >>>> Meta Meta = 1; >>>> repeated int32 Hotels = 2 [packed = false]; >>>> Country Market = 3; >>>> repeated Room Rooms = 4; >>>> .bcl.DateTime CheckIn = 5; >>>> .bcl.DateTime CheckOut = 6; >>>> OptionalCriteria OptionalCriteria = 7; >>>> } >>>> >>>> The 'Meta' element that contains the 'CutOffTime' that we want to >>>> modify: >>>> >>>> message Meta { >>>> int32 Client = 1; >>>> int32 Brand = 2; >>>> bool UseCache = 3; >>>> .bcl.TimeSpan CutOffTime = 4; >>>> bool B2C = 5; >>>> Language Language = 6; >>>> Currency Currency = 7; >>>> bool IncludeProviderAuditData = 8; >>>> SalesChannel SalesChannel = 9; >>>> string AgentId = 10; >>>> } >>>> >>>> The 'CutOffTime': >>>> >>>> message TimeSpan { >>>> sint64 value = 1; // the size of the timespan (in units of the >>>> selected scale) >>>> TimeSpanScale scale = 2; // the scale of the timespan [default = DAYS] >>>> enum TimeSpanScale { >>>> DAYS = 0; >>>> HOURS = 1; >>>> MINUTES = 2; >>>> SECONDS = 3; >>>> MILLISECONDS = 4; >>>> TICKS = 5; >>>> >>>> MINMAX = 15; // dubious >>>> } >>>> } >>>> >>>> Thanks, >>>> >>>> Joan. >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Wednesday, September 20, 2023 at 10:40:08 AM UTC+2 Florian Enner >>>> wrote: >>>> >>>>> 1) A "varint" is a "variable length integer". When you replace a large >>>>> number with a small one, it's entirely possible to lose some bytes and >>>>> still be valid. You need to check the actual output. >>>>> >>>>> 2) Can you provide the proto definition of the field you want to >>>>> modify? Scalar fields get set to the last encountered value, so the >>>>> easiest >>>>> option may be to copy the original bytes and append a delta containing >>>>> the >>>>> differences. >>>>> >>>>> >>>>> >>>>> >>>>> On Wednesday, September 20, 2023 at 10:19:07 AM UTC+2 Joan Balagueró >>>>> wrote: >>>>> >>>>>> Hello, >>>>>> >>>>>> I have a protobuf message like this into a byte array: >>>>>> >>>>>> 1: { // META element >>>>>> 1: 2 >>>>>> 2: 1 >>>>>> 3: 1 >>>>>> 4: { // CutOffTime element within META >>>>>> 1: 10 >>>>>> 2: 3 >>>>>> } >>>>>> 5: 1 >>>>>> 6: 40 >>>>>> } >>>>>> 2: 9836 // HOTEL element >>>>>> 3: 724 // MARKET element >>>>>> >>>>>> >>>>>> We need to traverse this message and write it to a >>>>>> 'CodeOutputStream', but changing the values of the 'cutoff' element to, >>>>>> for >>>>>> example to '4: { 1: 7, 2: 4 }'. I'm not able to do it, I need some help. >>>>>> >>>>>> A basic algorithm that writes the same protobuf to another byte array >>>>>> but without changing anything works. Here I try the special case of >>>>>> 'META' >>>>>> (key = 1) that contains the 'cutoff' element. >>>>>> >>>>>> Map<Integer, UnknownFieldSet.Field> rootFields = >>>>>> UnknownFieldSet.parseFrom(document).asMap(); >>>>>> >>>>>> for (Map.Entry<Integer, UnknownFieldSet.Field> entry : >>>>>> rootFields.entrySet()) { >>>>>> if (entry.getKey() == 1) { >>>>>> ByteString bs = >>>>>> entry.getValue().getLengthDelimitedList().get(0); >>>>>> output.writeByteArray(1, bs.toByteArray()); >>>>>> } >>>>>> else { >>>>>> entry.getValue().writeTo(entry.getKey(), output); >>>>>> } >>>>>> } >>>>>> >>>>>> >>>>>> Now I try to go a bit further, trying to read the cuotff element, >>>>>> change the values and rewrite them to the 'output'. And here is when I'm >>>>>> not able to solve the problem. Below my try that does not work, it >>>>>> generates a byte array >>>>>> of 69 bytes instead of 73 (I'm losing 4 bytes): >>>>>> >>>>>> Map<Integer, UnknownFieldSet.Field> rootFields = >>>>>> UnknownFieldSet.parseFrom(document).asMap(); >>>>>> >>>>>> for (Map.Entry<Integer, UnknownFieldSet.Field> entry : >>>>>> rootFields.entrySet()) { >>>>>> if (entry.getKey() == 1) { >>>>>> ByteString bs = >>>>>> entry.getValue().getLengthDelimitedList().get(0); >>>>>> Map<Integer, UnknownFieldSet.Field> ufs = >>>>>> UnknownFieldSet.parseFrom(bs).asMap(); >>>>>> >>>>>> for (Map.Entry<Integer, UnknownFieldSet.Field> >>>>>> item : ufs.entrySet()) { >>>>>> if (item.getKey() == 4) { >>>>>> Map<Integer, UnknownFieldSet.Field> >>>>>> cutoff = >>>>>> UnknownFieldSet.parseFrom(item.getValue().getLengthDelimitedList().get(0)).asMap(); >>>>>> cutoff.put(1, >>>>>> UnknownFieldSet.Field.newBuilder().addVarint(7).build()).writeTo(1, >>>>>> output); >>>>>> cutoff.put(2, >>>>>> UnknownFieldSet.Field.newBuilder().addVarint(4).build()).writeTo(1, >>>>>> output); >>>>>> } >>>>>> else { >>>>>> item.getValue().writeTo(item.getKey(), >>>>>> output); >>>>>> } >>>>>> } >>>>>> } >>>>>> else { >>>>>> entry.getValue().writeTo(entry.getKey(), output); >>>>>> } >>>>>> } >>>>>> >>>>>> Any help would be really appreciated. >>>>>> >>>>>> Thanks, >>>>>> >>>>>> Joan. >>>>>> >>>>> -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/301116cd-25eb-4360-84b8-e659e3c41c49n%40googlegroups.com.
