Thanks Gary (and Himanshu), correct, using batch operations is just an optimization and not a semantic difference to single row ops.
Should then all RPC triggered by a coprocessor be avoided (and hence the use the env-provided HTableInterface be generally discouraged)? The 2ndary index scenario (with separate tables) was just a "bad" example, which is better handled at the client anyway. I still think a RegionServer or RpcServer level "preRequest" and "postRequest" (or whatever) hooks would be useful for a variety of scenarios. Thanks. -- Lars ________________________________ From: Gary Helmling <[email protected]> To: [email protected] Sent: Thursday, August 11, 2011 11:24 AM Subject: Re: Coprocessors and batch processing On Wed, Aug 10, 2011 at 10:46 PM, lars hofhansl <[email protected]> wrote: > > I guess there could either be a {pre|post}Multi on RegionObserver (although > HRegionServer.multi does a lot of munging). > Or maybe a general {pre|post}Request with no arguments - in which case it > would be at least possible to write code in the coprocessor > to collect the puts/deletes/etc through the normal single > prePut/preDelete/etc hooks and then batch-process them in postRequest(). > > This same question came up recently in an internal discussion as well. Since multi() only exists at the HRegionServer level (there is no HRegion.multi()), I don't think that this belongs in RegionObserver at all. So this would be the main reason why there currently are no preMulti()/postMulti() hooks. There is something of a gap here between the way the client sees things and the way the coprocessor sees things. But I really see this as more of an RPC-layer listener or filter. After all, "multi" is really just an RPC operation for efficiency. It's not really a core HBase operation. As I see it there are a couple motivations behind the current limitation: 1) we generally only want a single set of coprocessor hooks to be involved in a given operation 2) the coprocessor API should be as conceptually simple as possible, while still reflecting what is happening in HBase I think adding in some multi representation in the coprocessor API poses challenges on each of these fronts. Not necessarily unresolvable challenges, but there are trade-offs involved. Re (1): representing multi in cp hooks means that you now have layering in the cp hooks handling a given operation. Say all the actions in the multi request are Puts, you go from having: [prePut() ... postPut()] x N to preMulti() [prePut() ... postPut()] x N postMulti() For me, this implies some confusion about where I should handle the actions in my coprocessor. Should I put all handling in pre/postMulti() or do I also need to implement pre/postPut()? This gets at (2), for me it's easier to think of "multi" as just an aggregation of operations, not as an operation in itself. The actual operation in the above example is a Put. It's just that there are a lot of them. Back to the original situation you raise, I think it's a really bad idea to immediately trigger RPC operations within a coprocessor, _especially_ in the case of multi. Say you are doing a secondary indexing transformation on the Puts you receive. You get a multi batch of 1000 puts. You transform that into a batch of 1000 secondary index puts, potentially going to every region server in your cluster holding a region in the secondary indexing table. You've just multiplied the RPC operations triggered by a single request and exposed yourself to triggering a distributed deadlock, where the RPC handler thread running in one RS is waiting for an RPC handler to become available in another RS, which in turn has all handlers occupied waiting on other servers. I think the better approach to doing these kind of updates would be to have the RegionObserver.pre/postPut() implementation queue them up and have them batched and processed by a separate background thread so that you're not tying up resources directly in the RPC handling path (and also making clients wait on a synchronous response). It may be that a higher level (RegionServer or RpcServer level) observer-type interface would be useful. But I think that adds some complexity to understanding what coprocessors are and how they interact. --gh
