I want to use custom function, but it doesn't work.

Here are codes and outs

/// codes
def unionByName(a: DataFrame, b:DataFrame): DataFrame = {
    val columns = a.dtypes.intersect(b.dtypes).map{case (c, _) => col(c)}
    a.select(columns: _*).unionAll(b.select(columns: _*))
}
raw1.select(columns: _*).unionByName(raw3.select(columns: _*)).count()

/// outputs
unionByName: (a: org.apache.spark.sql.DataFrame, b: 
org.apache.spark.sql.DataFrame)org.apache.spark.sql.DataFrame
<console>:141: error: value unionByName is not a member of 
org.apache.spark.sql.DataFrame
              raw1.select(columns: _*).unionByName(raw3.select(columns: 
_*)).count()


p.s. Is there any function to check types of RDD or else? (e.g. dtypes for 
checking types of column)

Reply via email to