this happened after several hours of operations and both nodes are started at the same time (clean start without any data). so it might not relate to Bootstrap.
In system.log I do not see any logs like "xxx node dead" or exceptions. and both nodes in test are alive. they serve read/write well, too. Below four connections between nodes are keep healthy from time to time. tcp 0 0 ::ffff:192.168.2.87:7000 ::ffff:192.168.2.88:58447 ESTABLISHED tcp 0 0 ::ffff:192.168.2.87:54986 ::ffff:192.168.2.88:7000 ESTABLISHED tcp 0 0 ::ffff:192.168.2.87:59138 ::ffff:192.168.2.88:7000 ESTABLISHED tcp 0 0 ::ffff:192.168.2.87:7000 ::ffff:192.168.2.88:39074 ESTABLISHED so connections end in CLOSE_WAIT should be newly created. (for streaming ?) This seems related to streaming issues we suffered recently: http://n2.nabble.com/busy-thread-on-IncomingStreamReader-td4908640.html I would like add some debug codes around opening and closing of socket to find out what happend. Could you give me some hint, about what classes I should take look ? On Tue, Apr 20, 2010 at 04:47, Jonathan Ellis <jbel...@gmail.com> wrote: > Is this after doing a bootstrap or other streaming operation? Or did > a node go down? > > The internal sockets are supposed to remain open, otherwise. > > On Mon, Apr 19, 2010 at 10:56 AM, Ingram Chen <ingramc...@gmail.com> > wrote: > > Thank your information. > > > > We do use connection pools with thrift client and ThriftAdress is on port > > 9160. > > > > Those problematic connections we found are all in port 7000, which is > > internal communications port between > > nodes. I guess this related to StreamingService. > > > > On Mon, Apr 19, 2010 at 23:46, Brandon Williams <dri...@gmail.com> > wrote: > >> > >> On Mon, Apr 19, 2010 at 10:27 AM, Ingram Chen <ingramc...@gmail.com> > >> wrote: > >>> > >>> Hi all, > >>> > >>> We have observed several connections between nodes in CLOSE_WAIT > >>> after several hours of operation: > >> > >> This is symptomatic of not pooling your client connections correctly. > Be > >> sure you're using one connection per thread, not one connection per > >> operation. > >> -Brandon > > > > > > -- > > Ingram Chen > > online share order: http://dinbendon.net > > blog: http://www.javaworld.com.tw/roller/page/ingramchen > > > -- Ingram Chen online share order: http://dinbendon.net blog: http://www.javaworld.com.tw/roller/page/ingramchen
