The following fails for me in Spark 1.5:
https://gist.github.com/nitay/d08cb294ccf00b80c49a
Specifically, it returns 1 instead of 100 (in both versions).
When I print out the contents (i.e. collect()) I see all 100 items, yet the
count returns 1.
This works in 1.3 and 1.4.
Any ideas what's going
In Spark 1.2 I used to be able to do this:
scala>
org.apache.spark.sql.hive.HiveMetastoreTypes.toDataType("struct")
res30: org.apache.spark.sql.catalyst.types.DataType =
StructType(List(StructField(int,LongType,true)))
That is, the name of a column can be a keyword like "int". This is no
longer t
