Hi Phil, AFAIK this is not supported by storm and you'll have to come up with your own solution.
Like MPI each task (i.e. instance of a Spout) has an id. In my personal opinion, best way is to partition the data upstream and avoid this at the storm level. Thanks, Supun.. On Tue, Dec 9, 2014 at 3:43 PM, Phillip Rhodes <[email protected]> wrote: > Storm users: > > I am just getting my feet wet with Storm, and have something of a > noobie question that I haven't found an answer to yet. I am wondering > if there is a canonical idiom/pattern for avoid duplicate tuples when > using parallelism in Spouts such that multiple Spout instances are > accessing the same data source. > > An example of what I mean: If I have a FileReaderSpout that reads > from some file, and parallelism set to "3", is there a clean way to > make sure that each Spout instance returns a unique set of tuples > relative to the input data? > > In the old days, when I was writing MPI code, we had a process ID in > each process that we could use to partition data. Is there any > analogous mechanism in Storm? Or a better way altogether? Or is it > best to partition the data "upstream" of Storm and avoid this at the > Spout level? > > > Thanks, > > > Phil > --- > This message optimized for indexing by NSA PRISM > -- Supun Kamburugamuva Member, Apache Software Foundation; http://www.apache.org E-mail: [email protected]; Mobile: +1 812 369 6762 Blog: http://supunk.blogspot.com
