Spark Streaming handling Kafka exceptions

2017-07-17 Thread Jean-Francois Gosselin
How can I handle an error with Kafka with my DirectStream (network issue, zookeeper or broker going down) ? For example when the consumer fails to connect with Kafka (at startup) I only get a DEBUG log (not even an ERROR) and no exception are thrown ... I'm using Spark 2.1.1 and spark-streaming-ka

From C* to DataFrames with JSON

2017-02-09 Thread Jean-Francois Gosselin
Hi all, I'm struggling (Spark / Scala newbie) to create a DataFrame from a C* table but also create a DataFrame from column with json . e.g. From C* table | id | jsonData | == | 1 | {"a": "123", "b": "xyz" } | +--+---+ | 2 |