[
https://issues.apache.org/jira/browse/SPARK-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16048650#comment-16048650
]
fangfengbin commented on SPARK-21078:
-------------------------------------
[~sowen], this actually cause a problem in practice, when applications number
is more than 10000, the applications show in JobHistory UI
is less than the event log number in hdfs.
Applications is modified by some threads when mergeApplicationListing, i have
test use a lock instead applications to synchronized can solve this problem.
> JobHistory applications synchronized is invalid
> -----------------------------------------------
>
> Key: SPARK-21078
> URL: https://issues.apache.org/jira/browse/SPARK-21078
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.1.1
> Reporter: fangfengbin
>
> FsHistoryProvider has some threads to mergeApplicationListing.
> mergeApplicationListing function has code like this:
> applications.synchronized {
> ...
> applications = mergedApps
> ...
> }
> applications object is changed, so the synchronized do not work.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
