Ok, cool. Makes sense, thanks for the confirmation. I'm toying with an idea of patching Riak to hard-code the backend for certain buckets. (A custom patch just for our needs, not general-purpose.) Along the lines of, "If the bucket name starts with foo_, use backend foo". Then those buckets don't have properties that accumulate in the ring state, and we avoid the gossip lag problem altogether.
What do you think? -Greg On Friday, June 10, 2011 at 6:50 AM, Sean Cribbs wrote: > Greg, > > Yes, it is most likely gossip lag causing that. For your test cluster you > could decrease the gossip_interval setting in riak_core from its default of > 60000 ms to something smaller. > > There has long been talk of moving the bucket properties out of the ring into > something like a "system bucket" or "catalog", which could be gossiped on > demand or at a different interval than the ring. I don't recall the exact > issue number in our Bugzilla, but it's < 100. > > Sean Cribbs <s...@basho.com (mailto:s...@basho.com)> > Developer Advocate > Basho Technologies, Inc. > http://basho.com/ > > On Jun 10, 2011, at 2:17 AM, Greg Nelson wrote: > > > Hello, > > > > I have been debugging something I've seen popping up intermittently when > > running my application's functional tests against Riak (local 5 node devrel > > cluster). The behavior is basically that sometimes an object which was PUT > > will seemingly disappear. Any future GETs will 404. Even if waiting seconds > > or minutes between PUT and GET. > > > > After staring at this and pulling out some hair I finally figured out what > > was happening (I think). I noticed it was always the first few objects > > written that were lost, and only on a certain bucket. My application uses > > multi-backend with two bitcask backends. That bucket is the only one which > > uses the non-default backend. > > > > What's happening is the application first gets the bucket properties and > > then sets the "backend" prop if it's not set. You can probably guess the > > rest. (PUTs come into nodes which don't have the property in their ring > > state yet and store the objects in the default backend) > > > > I don't think this is necessarily a "bug". It's expected behavior when you > > think about it, as long as you know how bucket properties are propagated. > > But even knowing that, this is pretty subtle. > > > > Is there a good way for a client to know when the property has been > > gossiped to all the nodes? Seems like the only approach is to wait a bit > > after setting a property before doing a PUT... > > > > Also, does this sound right? It's very possible I'm wrong about what's > > causing this behavior. > > > > -Greg > > _______________________________________________ > > riak-users mailing list > > riak-users@lists.basho.com (mailto:riak-users@lists.basho.com) > > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
