Dear Spark users,
Given a DataFrame df with a column named foo bar, I can call a Spark SQL
built-in function on it like so:
df.select(functions.max(df("foo bar")))
However, if I want to apply a Hive UDF named myCustomFunction, I need to
write
df.selectExpr("myCustomFunction(`foo bar`)")
which forces me to deal with escaping the name of the column so I can put
it inside a well-formed SQL query. Is there a programmatic way to invoke a
Hive function by name, so that I don’t have to worry about escaping?
Ideally, I’d like to do something like
val myCustomFunction = functions.udf("myCustomFunction")
df.select(myCustomFunction(df("foo bar")))
… but I couldn’t find any such API.
Regards,
Punya
