I thought combiner comes from reduceByKey and not mapPartitions right...Let me dig deeper into the APIs
On Thu, Feb 19, 2015 at 8:29 AM, Daniel Siegmann <daniel.siegm...@velos.io> wrote: > I'm not sure what your use case is, but perhaps you could use > mapPartitions to reduce across the individual partitions and apply your > filtering. Then you can finish with a reduceByKey. > > On Thu, Feb 19, 2015 at 9:21 AM, Debasish Das <debasish.da...@gmail.com> > wrote: > >> Hi, >> >> Before I send out the keys for network shuffle, in reduceByKey after map >> + combine are done, I would like to filter the keys based on some >> threshold... >> >> Is there a way to get the key, value after map+combine stages so that I >> can run a filter on the keys ? >> >> Thanks. >> Deb >> > > > > -- > Daniel Siegmann, Software Developer > Velos > Accelerating Machine Learning > > 54 W 40th St, New York, NY 10018 > E: daniel.siegm...@velos.io W: www.velos.io >
