[
https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Marcelo Vanzin updated SPARK-18085:
-----------------------------------
Attachment: spark_hs_next_gen.pdf
Here's an initial document to bootstrap the discussion of how to fix the
History Server. It discusses the current issues and proposes solutions to them,
providing an implementation path that will hopefully allow for finer-grained
reviews with as little disruption to functionality as possible while the work
is under review.
I could also share a google doc link if people prefer (although I believe the
ASF likes documents to be stored on their servers).
I think there are two points that might be a little controversial, so I'll list
them up front:
- use of a JNI library for data storage: this is not a requirement, but there
were no good pure Java alternatives I could find. The functionality is not
dependent on the actual library, though, so we could switch the library if
desired.
- move of Spark UI code to a separate module: I think this is good both as a
way to isolate these changes, and for future maintainability; it makes it
clearer that the UI is sort of a separate component, and that there should be a
clearer separation between the code in core and the ui.
> Scalability enhancements for the History Server
> -----------------------------------------------
>
> Key: SPARK-18085
> URL: https://issues.apache.org/jira/browse/SPARK-18085
> Project: Spark
> Issue Type: Umbrella
> Components: Spark Core, Web UI
> Affects Versions: 2.0.0
> Reporter: Marcelo Vanzin
> Attachments: spark_hs_next_gen.pdf
>
>
> It's a known fact that the History Server currently has some annoying issues
> when serving lots of applications, and when serving large applications.
> I'm filing this umbrella to track work related to addressing those issues.
> I'll be attaching a document shortly describing the issues and suggesting a
> path to how to solve them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
