Thanks! On Thu, May 5, 2011 at 3:11 PM, Mike Oxford <moxf...@gmail.com> wrote:
> Protobuf does not do inter-message delimiters. You have to handle that > aspect yourself. > > In the streambuf, put in the length of your protobuf message (in network > format), then the stream the message into it. > On the other end they should extract the length, convert to host format and > then read the data in and ship to protobuf. > Ah this seems like some part I hadn't read up to yet... It's difficult to do this stuff without contiguous blocks of time :-). Some of this makes me want to also investigate using BERT or BERT-RPC straight to erlang's "native" client for C++ bindings and just skip protocol buffers altogether :-). Yes I know protocol buffers are trendy and all, but even Google's Go programming language doesn't support them natively (without external packages). They did their own encoding as well (gob). https://github.com/blog/531-introducing-bert-and-bert-rpc http://bert-rpc.org/ Dave > > That's the 10,000 foot view. > > http://code.google.com/apis/protocolbuffers/docs/techniques.html > > --snip > If you want to write multiple messages to a single file or stream, it is up > to you to keep track of where one message ends and the next begins. The > Protocol Buffer wire format is not self-delimiting, so protocol buffer > parsers cannot determine where a message ends on their own. The easiest way > to solve this problem is to write the size of each message before you write > the message itself. When you read the messages back in, you read the size, > then read the bytes into a separate buffer, then parse from that buffer. (If > you want to avoid copying bytes to a separate buffer, check out the > CodedInputStream class (in both C++ and Java) which can be told to limit > reads to a certain number of bytes.) > > --end snip > > You do have to parse a message in its entirety; there is no incremental > parsing. If you have a message which wraps 10,000 messages you'll have to > load the whole thing in before you can parse. > > HTH. > > -mox > > > > > > On Thu, May 5, 2011 at 3:02 PM, David Leimbach <leim...@gmail.com> wrote: > >> I haven't completely abandoned mine yet either, but I've been finding that >> some of the I wanted to use with streambufs aren't going to pan out due to >> the nature of a socket backed stream and the way protocol buffers expects it >> to work. Protocol Buffers expects the stream to be terminated when done >> receiving a message *THEN* parse it rather than building the parsing in (I >> think that's what happened anyway... I really need to check my notes I took >> when I ran into this). This does appear to make a simple socket fd backed >> streambuf way to plug into any protocol buffers stream a bit more difficult >> than I was hoping. >> >> Then again, I may just not be overriding a particular member function that >> I should be either. >> >> Might get some time on the weekend to poke at this some more. >> >> On the plus side I definitely had some Riak RPC binary protocol bits >> working. I was encoding/decoding Ping/Pong just fine (and they don't need >> any Protocol Buffer anything to work). >> >> Dave >> >> On Sat, Apr 9, 2011 at 8:47 AM, Sean Cribbs <s...@basho.com> wrote: >> >>> I didn't permanently abandon it, but it was much more fiddly than doing >>> the same thing in pure Ruby. I have plans to deliver separate "native" >>> Protocol Buffers libraries for MRI and JRuby (at least) in 1.0 of the Ruby >>> client. >>> >>> Because it's being confused in this conversation, I think it merits >>> clarification -- the "protocol" that is used to talk to Riak and Google's >>> Protocol Buffers are NOT the same thing. Riak uses a simple length- and >>> message-code-prefixed binary protocol, in which the complex messages (ones >>> that have bodies and not just the message code) are serialized via Google's >>> Protocol Buffers. So, while we don't use the RPC facilities in Google's >>> library, the *serialization format* DOES use Protocol Buffers. >>> >>> Sorry for the confusion, we'll work to make that clearer in the wiki. >>> >>> Sean Cribbs <s...@basho.com> >>> Developer Advocate >>> Basho Technologies, Inc. >>> http://basho.com/ >>> >>> On Apr 8, 2011, at 9:17 PM, Scott Gonyea wrote: >>> >>> They are the same and you can actually see me plugging into the C++ code >>> here: >>> >>> https://github.com/sgonyea/pabst/tree/master/ext >>> >>> But as part of an Objective-C library (called ObjFW). So, the code is >>> actually an Objective-C++ wrapper around the C++ PB code, that exchanges >>> messages with Objective-C code (that hooks into Ruby). >>> >>> I believe Sean Cribbs has some initial C++-wrapper code in his Ripple >>> repo... Though he eventually abandoned it after C++ left him permanently >>> cross-eyed (I think that's why). >>> >>> Scott >>> >>> On Apr 8, 2011, at 5:20 PM, Mike Oxford wrote: >>> >>> Be careful here.. >>> >>> I do not thing Riak's "protocol buffers" are the same as Google's >>> protocol buffers. >>> Google's does bit-level packing and some other tricks that Riak does >>> not do, even though they both use the ".proto" file extension and very very >>> similar proto semantics. >>> >>> That said, if they ARE the same, then you can take the .proto files and >>> generate C++ classes, and use the secondary library "protobuf-c" to generate >>> C structs for the wire format. >>> >>> -mox >>> >>> On Fri, Apr 8, 2011 at 4:43 PM, David Leimbach <leim...@gmail.com>wrote: >>> >>>> Spent a little time poking at this today... Kind of surprised that there >>>> was no message defined for PingReq or for listing buckets. >>>> >>>> I realize these messages really have no usable payload, and just sort of >>>> have a tag and length, but for completeness it kind of feels like they >>>> should be there. >>>> >>>> Of course I'm not a Protocol Buffers expert in any sense, so I can't say >>>> whether this is a normal kind of choice or not. >>>> >>>> Dave >>>> >>>> >>>> On Fri, Apr 8, 2011 at 2:49 PM, Scott Gonyea <sc...@aitrus.org> wrote: >>>> >>>>> If we had this then a C-wrapper would be that much more attainable. So, >>>>> the author of such a lib would be a superstar in my book :). >>>>> >>>>> Sent from my iPhone >>>>> >>>>> On Apr 8, 2011, at 1:46 PM, David Leimbach <leim...@gmail.com> wrote: >>>>> >>>>> > I've been writing a bit of code in Haskell to push data to Riak, and >>>>> the bindings are pretty easy to use (Thanks Brian!), but getting >>>>> penetration >>>>> at my company for Haskell is going to take a little time. >>>>> > >>>>> > As such I'm just wondering if anyone knows of anyone working on a >>>>> protocol buffers version of a Riak client in C++, or if this is going to >>>>> be >>>>> something I'll have to take on. >>>>> > >>>>> > I've found a few generic looking C++ projects that use Boost's >>>>> asynchronous IO stuff with protocol buffers to make an RPC system, but I'm >>>>> not sure if any of those are implicitly compatible. >>>>> > >>>>> > Guess I'm just looking for a pointer... >>>>> > >>>>> > Dave >>>>> > _______________________________________________ >>>>> > riak-users mailing list >>>>> > riak-users@lists.basho.com >>>>> > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>>> >>>> >>>> >>>> _______________________________________________ >>>> riak-users mailing list >>>> riak-users@lists.basho.com >>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>>> >>>> >>> _______________________________________________ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >>> _______________________________________________ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >>> >>> _______________________________________________ >>> riak-users mailing list >>> riak-users@lists.basho.com >>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >>> >>> >> >> _______________________________________________ >> riak-users mailing list >> riak-users@lists.basho.com >> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com >> >> >
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
