Thanks for the answer. I thought I was doing something wrong. I was actually looking into protobuf.js thinking I could use that alongside grpc (it was able to handle Struct/Value/Any types); so it's good to know that grpc indeed uses that. It seems there are a few similar issues on GH already, and the library might be getting a rewrite so maybe it will be solved at some point in the future. I'll look further into it.
This isn't really a real world scenario for me at the moment... I am just doing some experimenting with grpc and wanted to try to handle variable JSON data. It's not really a stretch if for example we are working with schemaless / document based databases and depending on app context where you may want to store some variable metadata for objects. I agree the workround is to do manual encode / decode . Thanks for the help, it's appreciated! On Wednesday, 16 November 2016 17:52:04 UTC-4, Adam Cozzette wrote: > > I think the problem you're running into is that unfortunately we don't > currently support proto3 JSON > <https://developers.google.com/protocol-buffers/docs/proto3#json> for > Javascript. Proto3 JSON is the JSON-based format that understands the > special representations for well-known types such as Struct and Any. > > Actually gRPC's dynamic codegen uses protobuf.js > <https://github.com/dcodeIO/protobuf.js>, which is a totally separate > implementation from the main one we maintain at > https://github.com/google/protobuf. But neither implementation has > support for proto3 JSON, so if you're using Node.js then Struct, Value, and > Any won't be of much help for now. > > Could you describe a bit more about what the metadata field needs to > contain? Does it really need to support completely arbitrary JSON or would > it possibly be something where there are just a handful of different basic > schemas you expect? If the field can be arbitrary JSON then perhaps the > best solution would be to just use a bytes field and store the raw JSON > blob inside it. > > On Wed, Nov 16, 2016 at 10:47 AM, Bojan D <[email protected] <javascript:>> > wrote: > >> Here's a simpler server.js that doesn't require a client.js or deals with >> any of the grpc stuff really. >> >> const path = require('path') >> const grpc = require('grpc') >> >> const PROTO_PATH = path.resolve(__dirname, './user.proto') >> const HOSTPORT = '0.0.0.0:50051' >> >> const UserProto = grpc.load(PROTO_PATH); >> const User = UserProto.User >> >> const data = { >> "[email protected] <javascript:>": { >> "email": "[email protected] <javascript:>", >> "firstName": "Bob", >> "lastName": "Smith", >> "metadata": { >> "foo": "bar", >> "active": true >> } >> }, >> "[email protected] <javascript:>": { >> "email": "[email protected] <javascript:>", >> "firstName": "Jane", >> "lastName": "Smith" >> } >> } >> >> function main() { >> const u = new User(data['[email protected] <javascript:>']) >> console.log(u) >> } >> >> main() >> >> change the email to '[email protected] <javascript:>' and it would crash. >> >> On Wednesday, 16 November 2016 14:38:55 UTC-4, Bojan D wrote: >>> >>> Sure sorry... >>> >>> I am working with node.js and trying to use grpc. Reproducible steps: >>> >>> Install node 6.9.1 >>> >>> mkdir grpctest >>> cd grpctest >>> npm install grpc >>> >>> user.proto: >>> >>> import "google/protobuf/struct.proto"; >>> >>> syntax = "proto3"; >>> >>> package user; >>> >>> message User { >>> string email = 1; >>> string firstName = 2; >>> string lastName = 3; >>> google.protobuf.Value metadata = 4; >>> } >>> >>> message GetUserRequest { >>> string email = 1; >>> } >>> >>> service UserService { >>> rpc GetUser(GetUserRequest) returns (User); >>> } >>> >>> server.js: >>> >>> const path = require('path') >>> const grpc = require('grpc') >>> >>> const PROTO_PATH = path.resolve(__dirname, './user.proto') >>> const HOSTPORT = '0.0.0.0:50051' >>> >>> const UserService = grpc.load(PROTO_PATH).user.UserService >>> >>> const data = { >>> "[email protected] <javascript:>": { >>> "email": "[email protected] <javascript:>", >>> "firstName": "Bob", >>> "lastName": "Smith", >>> "metadata": { >>> "foo": "bar", >>> "active": true >>> } >>> }, >>> "[email protected] <javascript:>": { >>> "email": "[email protected] <javascript:>", >>> "firstName": "Jane", >>> "lastName": "Smith" >>> } >>> } >>> >>> function getUser(call, callback) { >>> const user = data[call.request.email] >>> if (!user) { >>> return callback(new Error('User Not Found')) >>> } >>> return callback(null, user) >>> } >>> >>> function main() { >>> const server = new grpc.Server() >>> server.addProtoService(UserService.service, { getUser }) >>> server.bind(HOSTPORT, grpc.ServerCredentials.createInsecure()) >>> server.start() >>> console.log(`User service running @ ${HOSTPORT}`) >>> } >>> >>> main() >>> >>> client.js >>> >>> const path = require('path') >>> const grpc = require('grpc') >>> >>> const PROTO_PATH = path.resolve(__dirname, './user.proto') >>> const HOSTPORT = '0.0.0.0:50051' >>> >>> const UserService = grpc.load(PROTO_PATH).user.UserService >>> const client = new UserService(HOSTPORT, >>> grpc.credentials.createInsecure()) >>> >>> client.getUser({ email: '[email protected] <javascript:>' }, (err, user) >>> => { >>> console.log(user) >>> process.exit() >>> }) >>> >>> >>> Run the server using command: >>> >>> node server.js >>> User service running @ 0.0.0.0:50051 >>> >>> Run the client: >>> >>> node client.js >>> { email: '[email protected] <javascript:>', >>> firstName: 'Jane', >>> lastName: 'Smith', >>> metadata: null } >>> >>> change the email in client request to '[email protected] <javascript:>' >>> >>> >>> /Users/bojand/dev/nodejs/grpctest/node_modules/protobufjs/dist/protobuf.js:2472 >>> throw Error(this+"#"+keyOrObj+" is not a >>> field: undefined"); >>> ^ >>> >>> Error: .google.protobuf.Value#foo is not a field: undefined >>> at Error (native) >>> ... >>> >>> Within the server part once we load proto we do have a constructor for >>> the User class, and I've tried playing around with trying to create an >>> instance of that in different ways and it crashes when trying to create >>> that instance. I've also tried to encode from stringified JSON and still >>> fails. When there is "metadata" it crashes. I think I am doing something >>> dumb but I am not sure what. >>> >>> Thanks. >>> >>> On Wednesday, 16 November 2016 13:58:44 UTC-4, Tim Kientzle wrote: >>>> >>>> I think we need more details: >>>> >>>> Are you using C++ or Java or Python or Ruby or C# or some other >>>> language? >>>> >>>> What version of protoc? >>>> >>>> Can you show the code you’re using to create the message? >>>> >>>> Tim >>>> >>>> >>>> On Nov 16, 2016, at 5:02 AM, Bojan D <[email protected]> wrote: >>>> >>>> Thanks for the answer. Somehow I missed the Protocol Buffers Well-Known >>>> Types page in the docs / protobuf website. >>>> >>>> I still have issue creating an instance of the message. For example if >>>> I have a plain object: >>>> >>>> { >>>> firstName: 'Bob' >>>> lastName: 'Smith', >>>> email: '[email protected]', >>>> metadata: { >>>> foo: 'bar', >>>> active: true >>>> } >>>> } >>>> >>>> And I try to create an instance of the message (to be sent via grpc) I >>>> get error >>>> >>>> .google.protobuf.Struct#foo is not a field: undefined >>>> >>>> Same thing if I use google.protobuf.Value. I've tried numerous ways of >>>> doing this but can't seem to accomplish it. I must be missing something. >>>> Anyone have any ideas. >>>> >>>> Thanks for the help again. >>>> >>>> Bojan >>>> >>>> On Tuesday, 15 November 2016 15:57:20 UTC-4, Tim Kientzle wrote: >>>>> >>>>> >>>>> On Nov 15, 2016, at 9:15 AM, Bojan D <[email protected]> wrote: >>>>> >>>>> Newbie question... If I have the following sample "User" JSON data: >>>>> >>>>> { >>>>> firstName: 'Bob' >>>>> lastName: 'Smith', >>>>> email: '[email protected]', >>>>> metadata: { >>>>> // a plain JS object that >>>>> // - will always exist and be at least an empty {} object >>>>> // - could potentially contain any number of properties and >>>>> values, depending on specific "user" >>>>> } >>>>> } >>>>> >>>>> How do I represent the metadata property within proto definition? >>>>> >>>>> >>>>> >>>>> You want to use the well-known types “Struct” or “Value”, which are >>>>> specifically designed to support ad hoc JSON parsing. “Struct” supports >>>>> parsing any valid JSON object structure, “Value” can parse any valid JSON: >>>>> >>>>> message User { >>>>> string email = 1; >>>>> string firstName = 2; >>>>> string lastName = 3; >>>>> google.protobuf.Struct metadata = 4; >>>>> } >>>>> >>>>> >>>>> Tim >>>>> >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Protocol Buffers" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at https://groups.google.com/group/protobuf. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> -- >> You received this message because you are subscribed to the Google Groups >> "Protocol Buffers" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To post to this group, send email to [email protected] >> <javascript:>. >> Visit this group at https://groups.google.com/group/protobuf. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "Protocol Buffers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/protobuf. For more options, visit https://groups.google.com/d/optout.
