Re: too many CancelledKeyException throwed from ConnectionManager

2014-08-26 Thread yao
Wow, great job. We will take a look and try our application again with your patch. On Tue, Aug 26, 2014 at 5:31 AM, Kousuke Saruta wrote: > Hi Shengzhe > > I faced to same situation. > > I think, Connection and ConnectionManager have some race condition issues > and the error you mentioned may

Re: too many CancelledKeyException throwed from ConnectionManager

2014-08-26 Thread Kousuke Saruta
Hi Shengzhe I faced to same situation. I think, Connection and ConnectionManager have some race condition issues and the error you mentioned may be caused by the issues. Now I'm trying to resolve the issue in https://github.com/apache/spark/pull/2019. Please check it out. - Kousuke (2014/08/

too many CancelledKeyException throwed from ConnectionManager

2014-08-25 Thread yao
Hi Folks, We are testing our home-made KMeans algorithm using Spark on Yarn. Recently, we've found that the application failed frequently when doing clustering over 300,000,000 users (each user is represented by a feature vector and the whole data set is around 600,000,000). After digging into the