Hi How do you want your final DF to look like? Is it with all 5 value columns? Do you have a finite set of columns?
On Fri, Nov 15, 2019 at 4:50 AM anbutech <anbutec...@outlook.com> wrote: > Hello Sir, > > I have a scenario to flatten the different combinations of map type(key > value) in a column called eve_data like below: > > How do we flatten the map type into proper columns using pyspark > > > 1) Source Dataframe having 2 columns(event id,data) > > eve_id,eve_data > 001, "k1":"abc", > "k2":"xyz" > "k3":"10091" > > eve_id,eve_data > > 002, "k1":"12", > "k2":"jack", > "k3":"0.01", > "k4":"0998" > > eve_id,eve_data > > 003, "k1":"aaa", > "k2":"xxxx", > "k3":"device", > "k4":"endpoint", > "k5":"-" > > > Final output: > > (flatten the output of each event ids key values).The number of key values > will be different for each event id.so i want to flatten the records for > all > the map type(key values) as below > > eve_id k1 k2 k3 > 001 abc xyz 10091 > > eve_id, k1 k2 k3 k4 > 002, 12 jack 0.01 0998 > > eve_id, k1 k2 k3 k4 k5 > 003, aaa xxxx device endpoint - > > > Thanks > Anbu > > > > -- > Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > > -- Best Regards, Ayan Guha
