[
https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15665618#comment-15665618
]
Marcelo Vanzin commented on SPARK-18085:
----------------------------------------
Hey all, any more feedback here?
I've made some progress in the M3 implementation, but I've been distracted by
more urgent issues, and it's also a pretty large milestone. But it still would
be good to know whether people have any concerns about the changes being
proposed before I spent too much time working on something that people just
don't want.
Just be aware that where the data is written to is not really the most
important things here; the main thing is that UI data would now live somewhere
that is not the Spark driver's memory, and most of the changes are towards
implementing the abstractions to support that. In fact, you could even still
store things in memory later on too, although I don't think that will be
necessary (nor desired).
> Better History Server scalability for many / large applications
> ---------------------------------------------------------------
>
> Key: SPARK-18085
> URL: https://issues.apache.org/jira/browse/SPARK-18085
> Project: Spark
> Issue Type: Umbrella
> Components: Spark Core, Web UI
> Affects Versions: 2.0.0
> Reporter: Marcelo Vanzin
> Attachments: spark_hs_next_gen.pdf
>
>
> It's a known fact that the History Server currently has some annoying issues
> when serving lots of applications, and when serving large applications.
> I'm filing this umbrella to track work related to addressing those issues.
> I'll be attaching a document shortly describing the issues and suggesting a
> path to how to solve them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
