[
https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726053#comment-15726053
]
Marcelo Vanzin commented on SPARK-18085:
----------------------------------------
bq. Since SHS keeps this data in memory I don't see how these issues are
orthogonal to the memory issues in SHS
If you read my spec, I have a solution that doesn't keep data in memory and
doesn't need to change the log format. So yes, they are orthogonal issues.
And the even log is basically for "data interchange". It's not just meant for
the history server. There are other, third party tools that use it. Which is
why I intentionally avoided touching it in this project.
> Better History Server scalability for many / large applications
> ---------------------------------------------------------------
>
> Key: SPARK-18085
> URL: https://issues.apache.org/jira/browse/SPARK-18085
> Project: Spark
> Issue Type: Umbrella
> Components: Spark Core, Web UI
> Affects Versions: 2.0.0
> Reporter: Marcelo Vanzin
> Attachments: spark_hs_next_gen.pdf
>
>
> It's a known fact that the History Server currently has some annoying issues
> when serving lots of applications, and when serving large applications.
> I'm filing this umbrella to track work related to addressing those issues.
> I'll be attaching a document shortly describing the issues and suggesting a
> path to how to solve them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
