Hi If you want column wise distinct, you may need to define it. Will it be possible to demonstrate your problem with an example? Like what's the input and output. Maybe with few columns.. On 19 Sep 2016 20:36, "Abhishek Anand" <abhis.anan...@gmail.com> wrote:
> Hi Ayan, > > How will I get column wise distinct items using this approach ? > > On Mon, Sep 19, 2016 at 3:31 PM, ayan guha <guha.a...@gmail.com> wrote: > >> Create an array out of cilumns, convert to Dataframe, >> explode,distinct,write. >> On 19 Sep 2016 19:11, "Saurav Sinha" <sauravsinh...@gmail.com> wrote: >> >>> You can use distinct over you data frame or rdd >>> >>> rdd.distinct >>> >>> It will give you distinct across your row. >>> >>> On Mon, Sep 19, 2016 at 2:35 PM, Abhishek Anand <abhis.anan...@gmail.com >>> > wrote: >>> >>>> I have an rdd which contains 14 different columns. I need to find the >>>> distinct across all the columns of rdd and write it to hdfs. >>>> >>>> How can I acheive this ? >>>> >>>> Is there any distributed data structure that I can use and keep on >>>> updating it as I traverse the new rows ? >>>> >>>> Regards, >>>> Abhi >>>> >>> >>> >>> >>> -- >>> Thanks and Regards, >>> >>> Saurav Sinha >>> >>> Contact: 9742879062 >>> >> >
