[
https://issues.apache.org/jira/browse/SPARK-20582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhoukang updated SPARK-20582:
-----------------------------
Description:
In the current code of HistoryServer,jetty server will be started after
all the logs fetch from yarn has been replayed.However,when the number of logs
is becoming larger,the start time of jetty will be too long.
Here,we implement a solution which using ApplicationAttemptInfo
checkpointing to speed up the start of historyserver.When historyserver is
starting,it will load ApplicationAttemptInfo from checkpoint file first if
exists which is faster then replaying one by one.
was:
In the current code of HistoryServer,jetty server will be started after all the
logs fetch from yarn has been replayed.However,when the number of logs is
becoming larger,the start time of jetty will be too long.
Here,we implement a solution which using ApplicationAttemptInfo checkpointing
to speed up the start of historyserver.When historyserver is starting,it will
load ApplicationAttemptInfo from checkpoint file first if exists which is
faster then replaying one by one.
> Speed up the restart of HistoryServer using ApplicationAttemptInfo
> checkpointing
> --------------------------------------------------------------------------------
>
> Key: SPARK-20582
> URL: https://issues.apache.org/jira/browse/SPARK-20582
> Project: Spark
> Issue Type: Improvement
> Components: Deploy
> Affects Versions: 2.1.0
> Reporter: zhoukang
> Priority: Critical
>
> In the current code of HistoryServer,jetty server will be started
> after all the logs fetch from yarn has been replayed.However,when the number
> of logs is becoming larger,the start time of jetty will be too long.
> Here,we implement a solution which using ApplicationAttemptInfo
> checkpointing to speed up the start of historyserver.When historyserver is
> starting,it will load ApplicationAttemptInfo from checkpoint file first if
> exists which is faster then replaying one by one.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
