[
https://issues.apache.org/jira/browse/SPARK-17815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15560837#comment-15560837
]
Cody Koeninger commented on SPARK-17815:
----------------------------------------
My personal concerns about complexity are because I'm the one who answers
nearly every email on the spark user list regarding the Kafka integration.
My business need relative to this ticket is that any Kafka integration I use
must give me a way to transactionally store offsets and results. If those
downstream offsets aren't used as at least a source of truth, then that goal
isn't met. If this ticket is implicitly saying WAL is the only source of
truth, that's a problem for me.
SIP is what it is or may not be, we're where we are now having the discussion
we're having now, and I'll leave it at that ;)
> Report committed offsets
> ------------------------
>
> Key: SPARK-17815
> URL: https://issues.apache.org/jira/browse/SPARK-17815
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Reporter: Michael Armbrust
>
> Since we manage our own offsets, we have turned off auto-commit. However,
> this means that external tools are not able to report on how far behind a
> given streaming job is. When the user manually gives us a group.id, we
> should report back to it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
