That PR hasn't been updated since the new kafka streaming stuff (including KafkaCluster) got merged to master, it will require more changes than what's in there currently.
On Tue, Feb 10, 2015 at 9:25 AM, Sean Owen <[email protected]> wrote: > Yes, did you see the PR for SPARK-2808? > https://github.com/apache/spark/pull/3631/files > > It requires more than just changing the version. > > On Tue, Feb 10, 2015 at 3:11 PM, Ted Yu <[email protected]> wrote: > > Compiling Spark master branch against Kafka 0.8.2, I got: > > > > [WARNING] > > > /home/hbase/spark/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/DirectKafkaInputDStream.scala:62: > > no valid targets for annotation on value ssc_ - it is discarded unused. > > You may specify targets with meta-annotations, e.g. @(transient @param) > > [WARNING] @transient ssc_ : StreamingContext, > > [WARNING] ^ > > [ERROR] > > > /home/hbase/spark/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaCluster.scala:273: > > type mismatch; > > found : > > Map[kafka.common.TopicAndPartition,kafka.common.OffsetMetadataAndError] > > required: > > > scala.collection.immutable.Map[kafka.common.TopicAndPartition,kafka.common.OffsetAndMetadata] > > [ERROR] val req = OffsetCommitRequest(groupId, metadata) > > [ERROR] ^ > > [ERROR] > > > /home/hbase/spark/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaCluster.scala:278: > > value requestInfo is not a member of kafka.api. > > OffsetCommitResponse > > [ERROR] val respMap = resp.requestInfo > > [ERROR] ^ > > > > FYI > > > > On Tue, Feb 10, 2015 at 3:23 AM, Sean Owen <[email protected]> wrote: > >> > >> I believe Kafka 0.8.2 is imminent for master / 1.4.0 -- see SPARK-2808. > >> Anecdotally I have used Spark 1.2 with Kafka 0.8.2 without problem > already. > >> > >> On Feb 10, 2015 10:53 AM, "critikaled" <[email protected]> wrote: > >>> > >>> When can we expect the latest kafka and scala 2.11 support in spark > >>> streaming? > >>> > >>> > >>> > >>> -- > >>> View this message in context: > >>> > http://apache-spark-user-list.1001560.n3.nabble.com/Kafka-Version-Update-0-8-2-status-tp21573.html > >>> Sent from the Apache Spark User List mailing list archive at > Nabble.com. > >>> > >>> --------------------------------------------------------------------- > >>> To unsubscribe, e-mail: [email protected] > >>> For additional commands, e-mail: [email protected] > >>> > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
