[
https://issues.apache.org/jira/browse/SPARK-17815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15560906#comment-15560906
]
Ofir Manor commented on SPARK-17815:
------------------------------------
I totally understand your concerns, and yes, I hope you will continue to
faithfully respond to all Kafka-related messages, and yes, you should insist on
fixing what you think broken or unclear or too complex.
I just don't see how it relates to this ticket, which is explicitly about
helping external monitoring tools that might rely on the Kafka consumer group
advancing its offsets.
You have some problems with the framework, specifically with its fault
tolerance semantics, not with some specific Kafka source implementation detail:
https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#fault-tolerance-semantics
Since Kafka might be the first or second "real" streaming source, it does make
a lot of sense to validate / improve the framework based on this feedback. I
think this ticket is the wrong place for this - it will have no visiblity.
Maybe that should go the dev list, or to the blocking ticket (which is about
the framework), or a new Structure Streaming JIRA (if there is a specific
suggestion), or maybe to a SEP/SPIP, or if no committer is willing to revisit
it or discuss it in any channel (as was with the Kafka source for many months),
you'll eventually have to face that this is how this project is unfortunately
managed.
Anyway, I think I maxed out my contribution to this thread. It should be
continued with the right committers, wherever they want, not with me.
> Report committed offsets
> ------------------------
>
> Key: SPARK-17815
> URL: https://issues.apache.org/jira/browse/SPARK-17815
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Reporter: Michael Armbrust
>
> Since we manage our own offsets, we have turned off auto-commit. However,
> this means that external tools are not able to report on how far behind a
> given streaming job is. When the user manually gives us a group.id, we
> should report back to it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
