Protobuf does not do inter-message delimiters. You have to handle that aspect yourself.
In the streambuf, put in the length of your protobuf message (in network format), then the stream the message into it. On the other end they should extract the length, convert to host format and then read the data in and ship to protobuf. That's the 10,000 foot view. http://code.google.com/apis/protocolbuffers/docs/techniques.html --snip If you want to write multiple messages to a single file or stream, it is up to you to keep track of where one message ends and the next begins. The Protocol Buffer wire format is not self-delimiting, so protocol buffer parsers cannot determine where a message ends on their own. The easiest way to solve this problem is to write the size of each message before you write the message itself. When you read the messages back in, you read the size, then read the bytes into a separate buffer, then parse from that buffer. (If you want to avoid copying bytes to a separate buffer, check out the CodedInputStream class (in both C++ and Java) which can be told to limit reads to a certain number of bytes.) --end snip You do have to parse a message in its entirety; there is no incremental parsing. If you have a message which wraps 10,000 messages you'll have to load the whole thing in before you can parse. HTH. -mox On Thu, May 5, 2011 at 3:02 PM, David Leimbach <leim...@gmail.com> wrote: > I haven't completely abandoned mine yet either, but I've been finding that > some of the I wanted to use with streambufs aren't going to pan out due to > the nature of a socket backed stream and the way protocol buffers expects it > to work. Protocol Buffers expects the stream to be terminated when done > receiving a message *THEN* parse it rather than building the parsing in (I > think that's what happened anyway... I really need to check my notes I took > when I ran into this). This does appear to make a simple socket fd backed > streambuf way to plug into any protocol buffers stream a bit more difficult > than I was hoping. > > Then again, I may just not be overriding a particular member function that > I should be either. > > Might get some time on the weekend to poke at this some more. > > On the plus side I definitely had some Riak RPC binary protocol bits > working. I was encoding/decoding Ping/Pong just fine (and they don't need > any Protocol Buffer anything to work). > > Dave > > On Sat, Apr 9, 2011 at 8:47 AM, Sean Cribbs <s...@basho.com> wrote: > >> I didn't permanently abandon it, but it was much more fiddly than doing >> the same thing in pure Ruby. I have plans to deliver separate "native" >> Protocol Buffers libraries for MRI and JRuby (at least) in 1.0 of the Ruby >> client. >> >> Because it's being confused in this conversation, I think it merits >> clarification -- the "protocol" that is used to talk to Riak and Google's >> Protocol Buffers are NOT the same thing. Riak uses a simple length- and >> message-code-prefixed binary protocol, in which the complex messages (ones >> that have bodies and not just the message code) are serialized via Google's >> Protocol Buffers. So, while we don't use the RPC facilities in Google's >> library, the *serialization format* DOES use Protocol Buffers. >> >> Sorry for the confusion, we'll work to make that clearer in the wiki. >> >> Sean Cribbs <s...@basho.com> >> Developer Advocate >> Basho Technologies, Inc. >> http://basho.com/ >> >> On Apr 8, 2011, at 9:17 PM, Scott Gonyea wrote: >> >> They are the same and you can actually see me plugging into the C++ code >> here: >> >> https://github.com/sgonyea/pabst/tree/master/ext >> >> But as part of an Objective-C library (called ObjFW). So, the code is >> actually an Objective-C++ wrapper around the C++ PB code, that exchanges >> messages with Objective-C code (that hooks into Ruby). >> >> I believe Sean Cribbs has some initial C++-wrapper code in his Ripple >> repo... Though he eventually abandoned it after C++ left him permanently >> cross-eyed (I think that's why). >> >> Scott >> >> On Apr 8, 2011, at 5:20 PM, Mike Oxford wrote: >> >> Be careful here.. >> >> I do not thing Riak's "protocol buffers" are the same as Google's protocol >> buffers. >> Google's does bit-level packing and some other tricks that Riak does not >> do, even though they both use the ".proto" file extension and very very >> similar proto semantics. >> >> That said, if they ARE the same, then you can take the .proto files and >> generate C++ classes, and use the secondary library "protobuf-c" to generate >> C structs for the wire format. >> >> -mox >> >> On Fri, Apr 8, 2011 at 4:43 PM, David Leimbach <leim...@gmail.com> wrote: >> >>> Spent a little time poking at this today... Kind of surprised that there >>> was no message defined for PingReq or for listing buckets. >>> >>> I realize these messages really have no usable payload, and just sort of >>> have a tag and length, but for completeness it kind of feels like they >>> should be there. >>> >>> Of course I'm not a Protocol Buffers expert in any sense, so I can't say >>> whether this is a normal kind of choice or not. >>> >>> Dave >>> >>> >>> On Fri, Apr 8, 2011 at 2:49 PM, Scott Gonyea <sc...@aitrus.org> wrote: >>> >>>> If we had this then a C-wrapper would be that much more attainable. So, >>>> the author of such a lib would be a superstar in my book :). >>>> >>>> Sent from my iPhone >>>> >>>> On Apr 8, 2011, at 1:46 PM, David Leimbach <leim...@gmail.com> wrote: >>>> >>>> > I've been writing a bit of code in Haskell to push data to Riak, and >>>> the bindings are pretty easy to use (Thanks Brian!), but getting >>>> penetration >>>> at my company for Haskell is going to take a little time. >>>> > >>>> > As such I'm just wondering if anyone knows of anyone working on a >>>> protocol buffers version of a Riak client in C++, or if this is going to be >>>> something I'll have to take on. >>>> > >>>> > I've found a few generic looking C++ projects that use Boost's >>>> asynchronous IO stuff with protocol buffers to make an RPC system, but I'm >>>> not sure if any of those are implicitly compatible. >>>> > >>>> > Guess I'm just looking for a pointer... >>>> > >>>> > Dave >>>> > _______________________________________________ >>>> > riak-users mailing list >>>> > riak-users@lists.basho.com >>>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>> >>> >>> >>> _______________________________________________ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >> _______________________________________________ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> >> _______________________________________________ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> >> >> _______________________________________________ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> > > _______________________________________________ > riak-users mailing list > riak-users@lists.basho.com > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
