Thanks. I tried following versions. They both compiles: val colmap = map(idxMap.flatMap(en => Iterator(lit(en._1), lit(en._2))).toSeq: _*) val colmap = map(idxMap.flatMap(x => x._1 :: x._2 :: Nil).toSeq.map(lit _): _*)
However they fail on dataframe action like `show` with org.apache.spark.SparkException: Task not serializable error. I think it's an issue with interactive mode. I am using zeppelin. On Tue, Nov 15, 2016 at 12:38 AM, Takeshi Yamamuro <linguin....@gmail.com> wrote: > Hi, > > Literal cannot handle Tuple2. > Anyway, how about this? > > val rdd = sc.makeRDD(1 to 3).map(i => (i, 0)) > map(rdd.collect.flatMap(x => x._1 :: x._2 :: Nil).map(lit _): _*) > > // maropu > > On Tue, Nov 15, 2016 at 9:33 AM, Nirav Patel <npa...@xactlycorp.com> > wrote: > >> I am trying to use following API from Functions to convert a map into >> column so I can pass it to UDF. >> >> map(cols: Column >> <http://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/Column.html> >> *): Column >> <http://spark.apache.org/docs/latest/api/scala/org/apache/spark/sql/Column.html> >> >> "Creates a new map column. The input columns must be grouped as key-value >> pairs, e.g. (key1, value1, key2, value2, ...). The key columns must all >> have the same data type, and can't be null. The value columns must all have >> the same data type." >> >> >> final val idxMap = idxMapRdd.collectAsMap >> val colmap = map(idxMapA.map(lit _): _*) >> >> But getting following error: >> >> <console>:139: error: type mismatch; >> found : Iterable[org.apache.spark.sql.Column] >> required: Seq[org.apache.spark.sql.Column] >> val colmap = map(idxMapArr.map(lit _): _*) >> >> >> If I try: >> val colmap = map(idxMapArr.map(lit _).toSeq: _*) >> >> It says: >> >> java.lang.RuntimeException: Unsupported literal type class scala.Tuple2 >> (17.0,MBO) >> at org.apache.spark.sql.catalyst.expressions.Literal$.apply(lit >> erals.scala:57) >> at org.apache.spark.sql.functions$.lit(functions.scala:101) >> at $anonfun$1.apply(<console>:153) >> >> >> >> What is the correct usage of a `map` api to convert hashmap into column? >> >> >> >> >> >> >> >> [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/> >> >> <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn] >> <https://www.linkedin.com/company/xactly-corporation> [image: Twitter] >> <https://twitter.com/Xactly> [image: Facebook] >> <https://www.facebook.com/XactlyCorp> [image: YouTube] >> <http://www.youtube.com/xactlycorporation> > > > > > -- > --- > Takeshi Yamamuro > -- [image: What's New with Xactly] <http://www.xactlycorp.com/email-click/> <https://www.nyse.com/quote/XNYS:XTLY> [image: LinkedIn] <https://www.linkedin.com/company/xactly-corporation> [image: Twitter] <https://twitter.com/Xactly> [image: Facebook] <https://www.facebook.com/XactlyCorp> [image: YouTube] <http://www.youtube.com/xactlycorporation>
