Hi, re: pools and detecting node failure... Pooling is handled by ThriftPool. This class maintains a separate NodeContext object for each known node. This in turn maintains a pool of connections to its node.
Each NodeContext has a single "poolRefiller" object/thread, which runs either when signalled, or every ~2s, whichever is the sooner. Whenever it runs, the first thing it does is check which of its existing pooled connections are open. This is necessary for it to correctly calculate the number of new connections to open (assuming it has to) To check whether a connection is open, it calls TTransport.isOpen, which is TSocket.isOpen, which is Socket.isConnected. If a connection is not open, then it is binned. Therefore, pretty quickly if a node has failed, the NodeContext will not be holding any connections to it. This causes the NodeContext.isAvailable method to return false. When this is the case, that node is not considered by ThriftPool when it is seeking to return a connection to an operand (Mutator, Selector, KeyDeletor etc object) The pool refiller thread keeps on trying to create connections to a node, even after all connections to it have failed. When/if it becomes available again, then as soon as a connection is made NodeContext.isAvailable will return true and it comes "back online" for the purposes of the operands. NOTE: Some of my colleagues were working on Windows machines separated from our local development servers by low-end NAT routers. After some period using this Cassandra, inside Pelops even though TSocket.isOpen was returning true, when an operand tried using connections they were getting a timeout or other network exception. Calling setKeepAlive(true) on the underlying socket does not prevent this (although this option is best set because in general it should force timely detection of connection failure). Hector also experienced similar problems and we adopt a similar response - by default you'll see that Pelops sets Policy.getKillNodeConnsOnException() to true by default. What this means is that if a network exception is thrown when an operand interacts with a node, the NodeContext destroys all pooled connections to that node on the basis that the general failure of connections to that node may not be detectable because of the network setup. Of course, not many people will be running their Cassandra clients from Windows behind NAT in production, but the option is set by default because otherwise a segment of developers trying the library will experience persistent problems due to this network (and/or Thrift) strangeness and in production we are ourselves will switch it off (although note the worse downside is that the occasional network error to a node will cause the refreshing of its pool) Hope this makes sense. Best, Dominic On 14 June 2010 15:32, Kochheiser,Todd W - TO-DITT1 <twkochhei...@bpa.gov>wrote: > Great API that looks easy and intuitive to use. Regarding your > connection pool implementation, how does it handle failed/crashed nodes? > Will the pool auto-detect failed nodes via a “tester” thread or will a > failed node, and hence its pooled connection(s), be removed only when they > are used? Conversely, how will the pool be repopulated once the > failed/crashed node becomes available? > > > > Todd > > > ------------------------------ > > *From:* Dominic Williams [mailto:thedwilli...@googlemail.com] > *Sent:* Friday, June 11, 2010 7:05 AM > *To:* user@cassandra.apache.org > *Subject:* Re: Pelops - a new Java client library paradigm > > > > Hi good question. > > > > The scalability of Pelops is dependent on Cassandra, not the library > itself. The library aims to provide an more effective access layer on top of > the Thrift API. > > > > The library does perform connection pooling, and you can control the size > of the pool and other parameters using a policy object. But connection > pooling itself does not increase scalability, only efficiency. > > > > Hope this helps. > > BEst, Dominic > > > > On 11 June 2010 14:47, Ian Soboroff <isobor...@gmail.com> wrote: > > Sounds nice. Can you say something about the scales at which you've used > this library? Both write and read load? Size of clusters and size of data? > > Ian > > > > On Fri, Jun 11, 2010 at 9:41 AM, Dominic Williams < > thedwilli...@googlemail.com> wrote: > > Pelops is a new high quality Java client library for Cassandra. > > > > It has a design that: > > * reveals the full power of Cassandra through an elegant "Mutator and > Selector" paradigm > > * generates better, cleaner, less bug prone code > > * reduces the learning curve for new users > > * drives rapid application development > > * encapsulates advanced pooling algorithms > > > > An article introducing Pelops can be found at > > > http://ria101.wordpress.com/2010/06/11/pelops-the-beautiful-cassandra-database-client-for-java/ > > > > Thanks for reading. > > Best, Dominic > > > > >
