Question about upgrading Kafka client version

2017-03-09 Thread Oscar Batori
Guys, To change the subject from meta-voting... We are doing Spark Streaming against a Kafka setup, everything is pretty standard, and pretty current. In particular we are using Spark 2.1, and Kafka 0.10.1, with batch windows that are quite large (5-10 minutes). The problem we are having is prett

Re: How to convert spark data-frame to datasets?

2016-08-18 Thread Oscar Batori
>From the docs , DataFrame is just Dataset[Row]. The are various converters for subtypes of Product if you want, using "as[T]", where T <: Product