[
https://issues.apache.org/jira/browse/SPARK-17243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15448732#comment-15448732
]
Steve Loughran commented on SPARK-17243:
----------------------------------------
One thing to consider here whether there are any ways to improve incremental
loading of histories; start at the most recent and work backwards.
There's also the fact that the entire history is loaded just to get the final
summary info (success/failure). Once parsed once, this could just be saved in a
summary file alongside the original. That'd reduce load time from O(files *
events) to O(files)
> Spark 2.0 history server summary page gets stuck at "loading history summary"
> with 10K+ application history
> -----------------------------------------------------------------------------------------------------------
>
> Key: SPARK-17243
> URL: https://issues.apache.org/jira/browse/SPARK-17243
> Project: Spark
> Issue Type: Bug
> Components: Web UI
> Affects Versions: 2.0.0
> Environment: Linux
> Reporter: Gang Wu
>
> The summary page of Spark 2.0 history server web UI keep displaying "Loading
> history summary..." all the time and crashes the browser when there are more
> than 10K application history event logs on HDFS.
> I did some investigation, "historypage.js" file sends a REST request to
> /api/v1/applications endpoint of history server REST endpoint and gets back
> json response. When there are more than 10K applications inside the event log
> directory it takes forever to parse them and render the page. When there are
> only hundreds or thousands of application history it is running fine.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
