[
https://issues.apache.org/jira/browse/SOLR-8575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15143262#comment-15143262
]
Yonik Seeley commented on SOLR-8575:
------------------------------------
Here's an interesting exception I found logged:
{code}
2> 84141 ERROR (recoveryExecutor-132-thread-2-processing-n:127.0.0.1:44435_
x:collection1 s:shard5 c:collection1 r:core_node4) [n:127.0.0.1:44435_
c:collection1 s:shard5 r:core_node4 x:collection1] o.a.s.u.UpdateLog
java.io.EOFException
2> at
org.apache.solr.common.util.FastInputStream.readByte(FastInputStream.java:207)
2> at
org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:207)
2> at
org.apache.solr.update.HdfsTransactionLog$HDFSLogReader.next(HdfsTransactionLog.java:419)
2> at
org.apache.solr.update.UpdateLog$LogReplayer.doReplay(UpdateLog.java:1333)
2> at org.apache.solr.update.UpdateLog$LogReplayer.run(UpdateLog.java:1255)
{code}
An important part of this patch was recording the amount of data we've written
so far (as "sz"),
and then a new input stream is opened.
Does HDFS guarantee that all data written will be readable if we open the file
again (even if we haven't closed the file)?
And does read() make the same guarantees about reading at least a single byte
(or blocking) unless we've reached EOF?
> Fix HDFSLogReader replay status numbers and a performance bug where we can
> reopen FSDataInputStream too often.
> --------------------------------------------------------------------------------------------------------------
>
> Key: SOLR-8575
> URL: https://issues.apache.org/jira/browse/SOLR-8575
> Project: Solr
> Issue Type: Bug
> Reporter: Mark Miller
> Assignee: Mark Miller
> Fix For: master
>
> Attachments: SOLR-8575.patch
>
>
> [[email protected]] noticed some funny transaction log replay status
> logging a while back:
> active=true starting pos=444978 current pos=2855956 current size=16262 %
> read=17562
> active=true starting pos=444978 current pos=5748869 current size=16262 %
> read=35352
> 17562% read? Current size does not change as expected in this case?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
