Any suggestion on consumer side? 在 2013-3-25 下午9:49，"Neha Narkhede" <neha.narkh...@gmail.com>写道：
> For Kafka 0.7 in production at Linkedin, we use a heap of size 3G, new gen > 256 MB, CMS collector with occupancy of 70%. > > Thanks, > Neha > > On Sunday, March 24, 2013, Yonghui Zhao wrote: > > > Hi Jun, > > > > I used kafka-server-start.sh to start kafka, there is only one jvm > setting > > "-Xmx512M“ > > > > Do you have some recommend GC setting? Usually our sever has 32GB or > 64GB > > RAM. > > > > 2013/3/22 Jun Rao <jun...@gmail.com> > > > > > A typical reason for many rebalancing is the consumer side GC. If so, > you > > > will see logs in the consume saying sth like "expired session" for ZK. > > > Occasional rebalances are fine. Too many rebalances can slow down the > > > consumption and you will need to tune your GC setting. > > > > > > Thanks, > > > > > > Jun > > > > > > On Thu, Mar 21, 2013 at 11:07 PM, Yonghui Zhao <zhaoyong...@gmail.com > > > >wrote: > > > > > > > Yes, before consumer exception: > > > > > > > > 2013/03/21 12:07:17.909 INFO [ZookeeperConsumerConnector] [] > > > > 0_lg-mc-db01.bj-1363784482043-f98c7868 *end rebalancing > > > > consumer*0_lg-mc-db01.bj-1363784482043-f98c7868 try #0 > > > > 2013/03/21 12:07:17.911 INFO [ZookeeperConsumerConnector] [] > > > > 0_lg-mc-db01.bj-1363784482043-f98c7868 *begin rebalancing > > > > consumer*0_lg-mc-db01.bj-1363784482043-f98c7868 try #0 > > > > 2013/03/21 12:07:17.934 INFO [FetcherRunnable] [] FetchRunnable-0 > start > > > > fetching topic: sms part: 0 offset: 43667888259 from 127.0.0.1:9093 > > > > 2013/03/21 12:07:17.940 INFO [SimpleConsumer] [] Reconnect in > > multifetch > > > > due to socket error: > > > > java.nio.channels.*ClosedByInterruptException* > > > > at java.nio.channels.spi.*AbstractInterruptibleChannel* > > > > .end(AbstractInterruptibleChannel.java:201) > > > > > > > > > > > > 2013/03/21 12:07:17.978 INFO [ZookeeperConsumerConnector] [] > > > > 0_lg-mc-db01.bj-1363784482043-f98c7868 *end rebalancing > > > > consumer*0_lg-mc-db01.bj-1363784482043-f98c7868 try #0 > > > > 2013/03/21 12:07:18.004 INFO [FetcherRunnable] [] FetchRunnable-0 > start > > > > fetching topic: sms part: 0 offset: 43667888259 from 127.0.0.1:9093 > > > > 2013/03/21 12:07:18.066 INFO [ZookeeperConsumerConnector] [] > > > > 0_lg-mc-db01.bj-1363784482043-f98c7868 *begin rebalancing consume*r > > > > 0_lg-mc-db01.bj-1363784482043-f98c7868 try #0 > > > > 2013/03/21 12:07:18.176 INFO [SimpleConsumer] [] Reconnect in > > multifetch > > > > due to socket error: > > > > java.nio.channels.*ClosedByInterruptException* > > > > at java.nio.channels.spi.*AbstractInterruptibleChannel* > > > > .end(AbstractInterruptibleChannel.java:201) > > > > > > > > > > > > So you think it is normal? How can we avoid this exception? > > > > > > > > I used 4 partitions in kafka, use only 1 partition？ > > > > > > > > > > > > > > > > 2013/3/22 Jun Rao <jun...@gmail.com> > > > > > > > > > Do you see any rebalances in the consumer? Each rebalance will > > > interrupt > > > > > existing fetcher threads first. > > > > > > > > > > Thanks, > > > > > > > > > > Jun > > > > > > > > > > On Thu, Mar 21, 2013 at 9:40 PM, Yonghui Zhao < > zhaoyong...@gmail.com > > > > > > > > wrote: > > > > > > > > > > > The application won't shut down the consumer connector. The > > > consumer > > > > is > > > > > > always alive. > > > > > > > > > > > > 2013/3/22 Jun Rao <jun...@gmail.com> > > > > > > > > > > > > > If you use the high level consumer, normally > > > > ClosedByInterruptException > > > > > > > happens because the application calls shutdown on the consumer > > > > > connector. > > > > > > > Is that the case? > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > > > Jun > > > > > > > > > > > > > > On Thu, Mar 21, 2013 at 8:38 PM, Yonghui Zhao < > > > zhaoyong...@gmail.com > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > No, I use java consumer connector, and set 10 seconds > timeout. > > > > > > > > > > >
