I want to use custom function, but it doesn't work. Here are codes and outs
/// codes def unionByName(a: DataFrame, b:DataFrame): DataFrame = { val columns = a.dtypes.intersect(b.dtypes).map{case (c, _) => col(c)} a.select(columns: _*).unionAll(b.select(columns: _*)) } raw1.select(columns: _*).unionByName(raw3.select(columns: _*)).count() /// outputs unionByName: (a: org.apache.spark.sql.DataFrame, b: org.apache.spark.sql.DataFrame)org.apache.spark.sql.DataFrame <console>:141: error: value unionByName is not a member of org.apache.spark.sql.DataFrame raw1.select(columns: _*).unionByName(raw3.select(columns: _*)).count() p.s. Is there any function to check types of RDD or else? (e.g. dtypes for checking types of column)