The UDF should return the result value you want, not a whole Row. In
Scala it figures out the schema of the UDF's result from the
signature.
On Thu, Aug 6, 2020 at 7:56 AM Amit Joshi <mailtojoshia...@gmail.com> wrote:
>
> Hi,
>
> I have a spark udf written in scala that takes couuple of columns and apply
> some logic and output InternalRow. There is spark schema of StructType also
> present. But when I try to return the InternalRow from UDF there is exception
>
> java.lang.UnsupportedOperationException: Schema for type
> org.apache.spark.sql.catalyst.GenericInternalRow is not supported
>
> val getData = (hash : String, type : String) => {
> val schema = hash match {
> case "people" =>
> peopleSchema
> case "empl" => emplSchema
> }
> getGenericInternalRow(schema)
> }
>
> val data = udf(getData)
>
> Spark Version : 2.4.5
>
>
> Please Help.
>
>
> Regards
>
> Amit Joshi
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org
