[
https://issues.apache.org/jira/browse/SPARK-19802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958896#comment-15958896
]
Ben Barnard commented on SPARK-19802:
-------------------------------------
The scenario we're trying to support is running Spark applications in on a
cluster without HFS, with a scheduler such as Nomad or Kubernetes. Spark
applications are run by the scheduler w/o the need for a Spark master, but we'd
like a straightforward mechanism for the applications to be able to opt in to
publishing to a history server.
Yes, we were thinking of implementing an alternative
`ApplicationHistoryProvider` to receive events in the history server, and
something like `EventLoggingListener` in the driver that would send events to
the server.
We recognise that some refactoring may be required, but we are prepared to
contribute this.
> Remote History Server
> ---------------------
>
> Key: SPARK-19802
> URL: https://issues.apache.org/jira/browse/SPARK-19802
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 2.1.0
> Reporter: Ben Barnard
>
> Currently the history server expects to find history in a filesystem
> somewhere. It would be nice to have a history server that listens for
> application events on a TCP port, and have a EventLoggingListener that sends
> events to the listening history server instead of writing to a file. This
> would allow the history server to show up-to-date history for past and
> running jobs in a cluster environment that lacks a shared filesystem.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
