Re: Error when processing messages in Windows

2013-07-10 Thread Jun Rao
My understanding is that the Sun engineers were concerned that a process may read data mapped by another process, if unmap is supported. Thanks, Jun On Wed, Jul 10, 2013 at 9:34 AM, Jay Kreps wrote: > Does anyone understand the discussion on that ticket Sriram posted? It > sounds like they ha

Re: Error when processing messages in Windows

2013-07-10 Thread Jay Kreps
Does anyone understand the discussion on that ticket Sriram posted? It sounds like they have an unmap call but they appear to be concerned about protecting threads from one another--i.e. if one thread unmapped the file and another mapped a different file it would show up in the old memory mapping.

Re: Error when processing messages in Windows

2013-07-10 Thread Denny Lee
Thanks very much for digging in! I was a tad concerned about that approach but in the process of testing that idea out along with some other more dramatic ideas ;). Will keep you updated - thanks again! On 7/10/13 7:55 AM, "Jun Rao" wrote: >From that link, one workaround is to set the buffer

Re: Error when processing messages in Windows

2013-07-10 Thread Jun Rao
>From that link, one workaround is to set the buffer to null and force a GC. Not sure if that's a good idea though. Thanks, Jun On Tue, Jul 9, 2013 at 10:13 PM, Sriram Subramanian < srsubraman...@linkedin.com> wrote: > As far as I am aware it is not possible to resize mapped buffer without > u

Re: Error when processing messages in Windows

2013-07-09 Thread Sriram Subramanian
As far as I am aware it is not possible to resize mapped buffer without unmapping in Windows. W.r.t Java the bug here gives more context on why it does not support synchronous unmap function. http://bugs.sun.com/view_bug.do?bug_id=4724038 On 7/9/13 9:54 PM, "Jay Kreps" wrote: >The problem app

Re: Error when processing messages in Windows

2013-07-09 Thread Jay Kreps
The problem appears to be that we are resizing a memory mapped file which it looks like windows does not allow (which is kind of sucky). The offending method is OffsetIndex.resize(). The most obvious fix would be to first unmap the file, then resize, then remap it. We can't do this though because

Re: Error when processing messages in Windows

2013-07-09 Thread Jun Rao
Hmm, not sure what the issue is. Any windows user wants to chime in? Thanks, Jun On Tue, Jul 9, 2013 at 9:00 AM, Denny Lee wrote: > Hey Jun, > > We've been running into this issue when running perf.Performance as per > http://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/. > W

Re: Error when processing messages in Windows

2013-07-09 Thread Denny Lee
Hey Jun, We've been running into this issue when running perf.Performance as per http://blog.liveramp.com/2013/04/08/kafka-0-8-producer-performance-2/. When running it using 100K messages, it works fine on Windows with about 20-30K msg/s. But when running it with 1M messages, then the broker fail

Re: Error when processing messages in Windows

2013-07-09 Thread Jun Rao
A couple of users seem to be able to get 0.8 working on Windows. Any thing special about your Windows environment? Are you using any jvm plugins? Thanks, Jun On Tue, Jul 9, 2013 at 12:59 AM, Timothy Chen wrote: > Hi all, > > I've tried pushing a large amount of messages into Kafka on Windows,

Error when processing messages in Windows

2013-07-09 Thread Timothy Chen
Hi all, I've tried pushing a large amount of messages into Kafka on Windows, and got the following error: Caused by: java.io.IOException: The requested operation cannot be performed on a file with a user-mapped section open at java.io.RandomAccessFile.setLength(Native Method) at