[
https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15727475#comment-15727475
]
Marcelo Vanzin commented on SPARK-18085:
----------------------------------------
I'm not trying to flame you. I'm trying to point out that the issues you
raised, while valid on their own, are not related to the problem described in
this bug, and trying to discuss those here is counter-productive. If you care
about those you should open separate bugs.
The SHS memory issues are not caused by the event log format nor by its size.
The SHS does not load the whole event log into memory, not does it keep any
JSON-formatted anything in memory. So the fact that the event logs are in JSON
is not relevant to how much memory the SHS is using.
> Better History Server scalability for many / large applications
> ---------------------------------------------------------------
>
> Key: SPARK-18085
> URL: https://issues.apache.org/jira/browse/SPARK-18085
> Project: Spark
> Issue Type: Umbrella
> Components: Spark Core, Web UI
> Affects Versions: 2.0.0
> Reporter: Marcelo Vanzin
> Attachments: spark_hs_next_gen.pdf
>
>
> It's a known fact that the History Server currently has some annoying issues
> when serving lots of applications, and when serving large applications.
> I'm filing this umbrella to track work related to addressing those issues.
> I'll be attaching a document shortly describing the issues and suggesting a
> path to how to solve them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
