aDstream.transform(_.distinct()) will only make the elements of each RDD in the DStream distinct, not for the whole DStream globally. Is that what you're seeing?
Dean Wampler, Ph.D. Author: Programming Scala, 2nd Edition <http://shop.oreilly.com/product/0636920033073.do> (O'Reilly) Typesafe <http://typesafe.com> @deanwampler <http://twitter.com/deanwampler> http://polyglotprogramming.com On Fri, Mar 20, 2015 at 10:37 AM, Darren Hoo <darren....@gmail.com> wrote: > val aDstream = ... > > val distinctStream = aDstream.transform(_.distinct()) > > but the elements in distinctStream are not distinct. > > Did I use it wrong? >
