[
https://issues.apache.org/jira/browse/SPARK-17815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15560684#comment-15560684
]
Cody Koeninger commented on SPARK-17815:
----------------------------------------
Regarding kafka consumer behavior, I'm not saying it's impossible, I'm
saying it A. Needs attention, especially since what's in master isn't
complete and B. May confuse users who expect the offsets in Kafka to
determine where the consumer starts.
Regarding wal, DStream checkpoints were in HDFS too. Being on HDFS doesn't
guarantee against getting screwed up.
And in any case, the boundary alignment of multiple different offset stores
still applies
> Report committed offsets
> ------------------------
>
> Key: SPARK-17815
> URL: https://issues.apache.org/jira/browse/SPARK-17815
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Reporter: Michael Armbrust
>
> Since we manage our own offsets, we have turned off auto-commit. However,
> this means that external tools are not able to report on how far behind a
> given streaming job is. When the user manually gives us a group.id, we
> should report back to it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
