I think that it is not related to Zeppelin. Because your function `unionByName`
is not method of `DataFrame`, you have to call it like following:
val unionized = unionByName(raw1.select(columns: _*), raw3.select(columns: _*))
Regards,
Chiwan Park
> On Jan 25, 2016, at 9:00 PM, 진상형 <jsh90...@gmail.com> wrote:
>
> I want to use custom function, but it doesn't work.
>
> Here are codes and outs
>
> /// codes
> def unionByName(a: DataFrame, b:DataFrame): DataFrame = {
> val columns = a.dtypes.intersect(b.dtypes).map{case (c, _) => col(c)}
> a.select(columns: _*).unionAll(b.select(columns: _*))
> }
> raw1.select(columns: _*).unionByName(raw3.select(columns: _*)).count()
>
> /// outputs
> unionByName: (a: org.apache.spark.sql.DataFrame, b:
> org.apache.spark.sql.DataFrame)org.apache.spark.sql.DataFrame
> <console>:141: error: value unionByName is not a member of
> org.apache.spark.sql.DataFrame
> raw1.select(columns: _*).unionByName(raw3.select(columns:
> _*)).count()
>
>
>
> p.s. Is there any function to check types of RDD or else? (e.g. dtypes for
> checking types of column)
