Thank you, But now I have this error:
java.lang.ClassCastException: java.lang.Integer cannot be cast to java.lang.Long
My offsets are actually not big enough to be long. If I put bigger values, I have no such exception.
For me looks like a bug. Any ideas for a workaround? Thank! On 05/02/2016 06:57 PM, Cody Koeninger wrote:
If you're confused about the type of an argument, you're probably better off looking at documentation that includes static types: http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.streaming.kafka.KafkaUtils$ createDirectStream's fromOffsets parameter takes a map from TopicAndPartition to Long. There is documentation for a python constructor for TopicAndPartition: http://spark.apache.org/docs/latest/api/python/_modules/pyspark/streaming/kafka.html#TopicAndPartition On Mon, May 2, 2016 at 5:54 AM, Tigran Avanesov <[email protected]> wrote:Hi, I'm trying to start consuming messages from a kafka topic (via direct stream) from a given offset. The documentation of createDirectStream says: :param fromOffsets: Per-topic/partition Kafka offsets defining the (inclusive) starting point of the stream. However it expects a dictionary of topics (not names...), as i tried to feed it something like { 'topic' : {0: 123, 1:234}}, and of course got an exception. How should I build this fromOffsets parameter? Documentation does not say anything about it. (In general, I think it would be better if the function accepted topic names) Thank you! Regards, Tigran --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
-- Tigran Avanesov | IT Architect phone: +352 261911 3562 email: [email protected] skype: tigran.avanesov.corporate post: Olamobile S.à.r.l. 2-4 rue Eugène Ruppert Bâtiment Vertigo-Polaris L-2453 Luxembourg Luxembourg web: www.olamobile.com --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
