Hello,
I am struggling with a task that should be super simple: I would like to
create a Spark DF of Type Dataset<Row> with one column from a single
String (or from a one-element List of Strings). The column header should
be "text".
SparkContext.parallelize() does not work, because it returns RDD<T> and
not Dataset<Row> and it takes a "ClassTag" as 3rd parameter.
I am able to convert a List of Strings to JavaRDD<Row> using this:
JavaSparkContext javaSparkContext = new
JavaSparkContext(sparkContext);
JavaRDD<String> javaRdd = javaSparkContext.parallelize(textList);
But then I am stuck with this javaRDD. Besides, it seems overly complex
having to create an intermediate representation.
There is also this SO post with a solution in Scala that I have not been
able to convert to Java, because the APIs differ:
https://stackoverflow.com/questions/44028677/how-to-create-a-dataframe-from-a-string
Basically, what I am looking for is something simple like:
Dataset<Row> myData = sparkSession.createDataFrame(textList,
"text");
Any hints? Thanks a lot.
Cheers,
Martin