[
https://issues.apache.org/jira/browse/KAFKA-1008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13741035#comment-13741035
]
Jay Kreps commented on KAFKA-1008:
----------------------------------
Yeah, the problem is reads continue while the log is being rolled.
Here are a bunch of possible solutions I can think of
1. Lock reads
2. Delay the truncate until the larger mmap is collected
For (2) the problem is that it actually interferes with the recoverability of
the log. If we have old log segments with a bunch of unfilled bytes in their
index segment we need to recover those segments. But currently we only recover
from the last flush point.
So I think we need to lock read access. This isn't the end of the world but I
would prefer to do this only on windows. My suggestion on implementation would
be
1. Change from synchronized in OffsetIndex to Lock
2. Add an object called Os in kafka.utils that is something like
object Os {
private val osName = System.getProperty("os.name").toLowerCase
val isWindows = osName.startsWith("windows")
}
We can expand this later if we need more specific OS detection capabilities or
other OS-specific functionality.
Then elsewhere we can just do
if(Os.isWindows) lock.lock()
and the corresponding unlock.
Does that seem like it would work?
> Unmap before resizing
> ---------------------
>
> Key: KAFKA-1008
> URL: https://issues.apache.org/jira/browse/KAFKA-1008
> Project: Kafka
> Issue Type: Bug
> Components: core, log
> Affects Versions: 0.8
> Environment: Windows, Linux, Mac OS
> Reporter: Elizabeth Wei
> Assignee: Jay Kreps
> Labels: patch
> Fix For: 0.8
>
> Attachments: unmap-v4.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> While I was studying how MappedByteBuffer works, I saw a sharing runtime
> exception on Windows. I applied what I learned to generate a patch which uses
> an internal open JDK API to solve this problem.
> Following Jay's advice, I made a helper method called tryUnmap().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
