[
https://issues.apache.org/jira/browse/KAFKA-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14200095#comment-14200095
]
Steve Loughran commented on KAFKA-1754:
---------------------------------------
This is great! YARN is getting better and better at long-lived services; Kafka
should fit in nicely —and allocation of disks would be a very interesting
feature to add here.
# As arun says, your AM gets to choose whether to spawn a new container -and
where. You could try to ask for one back on the old machine, though if all data
was on in the container's workspace, you've lost it all.
# YARN will restart the AM, and if you ask it at launch time, it will keep the
containers up. Your AM has the task of rebuilding all its state. Hadoop 2.6+
adds sliding windows on restarts, so long lived apps aren't considered failures
if their AM fails once a fortnight.
# Security is an issue with long-lived apps. Have you tried this on secure
clusters yet?
> KOYA - Kafka on YARN
> --------------------
>
> Key: KAFKA-1754
> URL: https://issues.apache.org/jira/browse/KAFKA-1754
> Project: Kafka
> Issue Type: New Feature
> Reporter: Thomas Weise
> Attachments: DT-KOYA-Proposal- JIRA.pdf
>
>
> YARN (Hadoop 2.x) has enabled clusters to be used for a variety of workloads,
> emerging as distributed operating system for big data applications.
> Initiatives are on the way to bring long running services under the YARN
> umbrella, leveraging it for centralized resource management and operations
> ([YARN-896] and examples such as HBase, Accumulo or Memcached through
> Slider). This JIRA is to propose KOYA (Kafka On Yarn), a YARN application
> master to launch and manage Kafka clusters running on YARN. Brokers will use
> resources allocated through YARN with support for recovery, monitoring etc.
> Please see attached for more details.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
