[
https://issues.apache.org/jira/browse/SPARK-11045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15069506#comment-15069506
]
Dibyendu Bhattacharya commented on SPARK-11045:
-----------------------------------------------
Hi Sean,
Unfortunately existing implementation is not Fixable just because its uses the
Kafka High Level Consumer. Otherwise it would have been fixed by now. Only way
the existing Receiver based mode is fixable if we use Kafka Low Level consumer
and that is my consumer does. And I am very sure this is better implementation
and much stable and performant than what Spark has. I do not go by someone's
comment either . Things needs to be tested and measured by its performance and
stability and folks who using this has found that this perform much better than
what Spark Project have presently.
I think TD knows this very well that because of the High Level consumer issue
Kafka has rewritten their consumer API in 0.9. The problem is we just can not
move the existing receiver based consumer to 0.9 as Kafka 0.9 is recently
released and not stable yet. At the same time there are lot of people using
Kafka 0.8.x who are denied a stable Receiver consumer for Spark for quite a
long. The call is yours , I will obviously post it to forums and spark groups
as and when I release new features. But I am not convinced the reason for not
accepting it in Spark Project.
> Contributing Receiver based Low Level Kafka Consumer from Spark-Packages to
> Apache Spark Project
> ------------------------------------------------------------------------------------------------
>
> Key: SPARK-11045
> URL: https://issues.apache.org/jira/browse/SPARK-11045
> Project: Spark
> Issue Type: New Feature
> Components: Streaming
> Reporter: Dibyendu Bhattacharya
>
> This JIRA is to track the progress of making the Receiver based Low Level
> Kafka Consumer from spark-packages
> (http://spark-packages.org/package/dibbhatt/kafka-spark-consumer) to be
> contributed back to Apache Spark Project.
> This Kafka consumer has been around for more than year and has matured over
> the time . I see there are many adoptions of this package . I receive
> positive feedbacks that this consumer gives better performance and fault
> tolerant capabilities.
> This is the primary intent of this JIRA to give community a better
> alternative if they want to use Receiver Base model.
> If this consumer make it to Spark Core, it will definitely see more adoption
> and support from community and help many who still prefer the Receiver Based
> model of Kafka Consumer.
> I understand the Direct Stream is the consumer which can give Exact Once
> semantics and uses Kafka Low Level API , which is good . But Direct Stream
> has concerns around recovering checkpoint on driver code change . Application
> developer need to manage their own offset which complex . Even if some one
> does manages their own offset , it limits the parallelism Spark Streaming can
> achieve. If someone wants more parallelism and want
> spark.streaming.concurrentJobs more than 1 , you can no longer rely on
> storing offset externally as you have no control which batch will run in
> which sequence.
> Furthermore , the Direct Stream has higher latency , as it fetch messages
> form Kafka during RDD action . Also number of RDD partitions are limited to
> topic partition . So unless your Kafka topic does not have enough partitions,
> you have limited parallelism while RDD processing.
> Due to above mentioned concerns , many people who does not want Exactly Once
> semantics , still prefer Receiver based model. Unfortunately, when customer
> fall back to KafkaUtil.CreateStream approach, which use Kafka High Level
> Consumer, there are other issues around the reliability of Kafka High Level
> API. Kafka High Level API is buggy and has serious issue around Consumer
> Re-balance. Hence I do not think this is correct to advice people to use
> KafkaUtil.CreateStream in production .
> The better option presently is there is to use the Consumer from
> spark-packages . It is is using Kafka Low Level Consumer API , store offset
> in Zookeeper, and can recover from any failure . Below are few highlights of
> this consumer ..
> 1. It has a inbuilt PID Controller for dynamic rate limiting.
> 2. In this consumer , The Rate Limiting is done by modifying the size blocks
> by controlling the size of messages pulled from Kafka. Whereas , in Spark the
> Rate Limiting is done by controlling number of messages. The issue with
> throttling by number of message is, if message size various, block size will
> also vary . Let say your Kafka has messages for different sizes from 10KB to
> 500 KB. Thus throttling by number of message can never give any deterministic
> size of your block hence there is no guarantee that Memory Back-Pressure can
> really take affect.
> 3. This consumer is using Kafka low level API which gives better performance
> than KafkaUtils.createStream based High Level API.
> 4. This consumer can give end to end no data loss channel if enabled with WAL.
> By accepting this low level kafka consumer from spark packages to apache
> spark project , we will give community a better options for Kafka
> connectivity both for Receiver less and Receiver based model.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
