OutOfMemoryError during index mapping

2014-08-24 Thread pavel.zalu...@sematext.com
Hi, Sometimes we get: Caused by: java.lang.OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map0(Native Method) at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:885) ... 25 more (excuse me, we lost full stack trace, it starts from java.io.IOException which

Re: OutOfMemoryError during index mapping

2014-08-24 Thread Jay Kreps
Is it possible that this is a 32 bit machine and you have more than 2gb of index? -jay On Sunday, August 24, 2014, pavel.zalu...@sematext.com < pavel.zalu...@sematext.com> wrote: > Hi, > > Sometimes we get: > > Caused by: java.lang.OutOfMemoryError: Map failed > at sun.nio.ch.FileChannel

RE: read message use SimpleConsumer,can not start with assign offset

2014-08-24 Thread chenlax
the method getLastOffset as follow: public static long getLastOffset(SimpleConsumer consumer, String topic, int partition,long whichTime, String clientName) { long l_ret = kafka.api.OffsetRequest.EarliestTime(); System.out.println("whichTime = " + whichTime); TopicAnd

Re: OutOfMemoryError during index mapping

2014-08-24 Thread Jun Rao
In trunk, we can do parallel log loading to speed up recovery (see KAFKA-1414). Thanks, Jun On Sun, Aug 24, 2014 at 2:09 AM, pavel.zalu...@sematext.com < pavel.zalu...@sematext.com> wrote: > Hi, > > Sometimes we get: > > Caused by: java.lang.OutOfMemoryError: Map failed > at sun.nio.ch

Re: read message use SimpleConsumer,can not start with assign offset

2014-08-24 Thread Jun Rao
Is line 64 "return offsets[0]"? What value did you pass into whichTime? Thanks, Jun On Sun, Aug 24, 2014 at 7:24 PM, chenlax wrote: > the method getLastOffset as follow: > > public static long getLastOffset(SimpleConsumer consumer, String topic, > int partition,long whichTime, String clientNa

RE: read message use SimpleConsumer,can not start with assign offset

2014-08-24 Thread chenlax
yes, it's "return offsets[0]".whichTime is a long num, is message offset i cached,and the kafka keep message 7days. Thanks, Lax > Date: Sun, 24 Aug 2014 19:58:35 -0700 > Subject: Re: read message use SimpleConsumer,can not start with assign offset > From: jun...@gmail.com > To: users@kafka.apac

Re: read message use SimpleConsumer,can not start with assign offset

2014-08-24 Thread Jun Rao
If you want to get the latest offset, whichTime should be set to -1. Thanks, Jun On Sun, Aug 24, 2014 at 8:16 PM, chenlax wrote: > yes, it's "return offsets[0]".whichTime is a long num, is message offset i > cached,and the kafka keep message 7days. > > Thanks, > Lax > > > > Date: Sun, 24 Aug