Hi Anthony,
Hopefully that's what you wanted :)
case class Example(id:String, myMap: Map[String,Int])
> val myDF = sqlContext.createDataFrame(
> Seq(
> Example("a02d1fa5d87dce6a7", Map("Vans" -> 1, "Versace" ->2))
> )
> )
> myDF.
> explode("myMap","tempSeq")((x: Map[String,Int]) => x.toSeq).
> withColumn("brand_key",col("tempSeq")("_1")).
> withColumn("brand_count",col("tempSeq")("_2")).
> drop("tempSeq").
> show(false)
This gives us:
+-----------------+----------------------------+---------+-----------+
> |id |myMap |brand_key|brand_count|
> +-----------------+----------------------------+---------+-----------+
> |a02d1fa5d87dce6a7|Map(Vans -> 1, Versace -> 2)|Vans |1 |
> |a02d1fa5d87dce6a7|Map(Vans -> 1, Versace -> 2)|Versace |2 |
> +-----------------+----------------------------+---------+-----------+
Cheers,
Michael
On 25 February 2016 at 00:06, Anthony Brew <atb...@gmail.com> wrote:
> Hi,
> I have a Dataframe containing a column with a map Map[A,B] with
> multiple values. I want to explode the key,value pairs in the map into a
> new column, actually planing to create 2 new cols.
>
> My plan had been
>
> - explode "input": Map[K,V] to "temp":Iterable[Map[K,V]]
> - new col temp to temp.key
> - new col temp to temp.value
> - drop temp
>
> But I am failing at the first hurdle.
>
> For example my data looks a bit like like....
>
> scala> test.show()
> +----------------------------+------------------------------------------+
> | id | brand
> |
> +----------------------------+------------------------------------------+
> |a02d1fa5d87dce6a7...|Map(Vans -> 1, Versace ->2, ...|
>
>
> but I want to get to
>
> scala> test.show()
> +-----------------------------+------------------------------------------+
> | id | brand_key | brand_count |
> +-----------------------------+------------------------------------------+
> | a02d1fa5d87dce6a7...| Vans | 1 |
> | a02d1fa5d87dce6a7...| Versace | 2 |
>
>
> Any suggestions would be appreciated.
>
> Thanks,
> Anthony
>
>
>
