Re: Input stream partition to task mapping

Hi Yan, Thanks for your quick reply. Since we don't need to change our rules, we decided to use the consumer to read the rules in init(), as you suggested. Works great - thanks so much for your help! Susan On Tue, May 12, 2015 at 5:32 PM, Yan Fang wrote: > Hi Susan, > > -- " I have a custom Sy

Re: Input stream partition to task mapping

Hi Susan, -- " I have a custom SystemStreamPartitionGrouper" -- this is a very good try! -- "However, it looks like messages from the Rules stream are going to only one task instance, and not to all of them like I hoped. I have them running in the same container. Is it because there is only one o

Re: Input stream partition to task mapping

Hi Yan, We are looking for a solution for a similar problem. Currently, we have 2 input streams, one transactions stream (5 partitions) and one rules stream (one partition). I have a custom SystemStreamPartitionGrouper that assigns the SSPs like so:. taskName0: Trx stream partition 0, Rules stre

Re: Input stream partition to task mapping

If the rule does not change, we can get the rule when starting up StreamTasks and then localize it. Cheers, Fang, Yan yanfang...@gmail.com On Tue, May 5, 2015 at 2:41 PM, Yan Fang wrote: > "If I understand it correctly the only viable solution at the moment is to > create a new stream for the

Re: Input stream partition to task mapping

"If I understand it correctly the only viable solution at the moment is to create a new stream for the rules messages with as many partitions as the data stream and write each rules update message to all partitions of the new rules stream." If the data is constantly changing, yes, AFAIK, this is

Re: Input stream partition to task mapping

Hi Yan, Thanks for your quick response. After I read the discussion on SAMZA-353 I think the best solution for my use case is a "shared state" store among StreamTasks described in SAMZA-402. To give you some background I have a stream with rules which are constantly changing and a data stream on

Re: Input stream partition to task mapping

Hi Ueli, This feature currently is not supported by Samza. There was some discussions in the JIRA - SAMZA-353 . But there are some workaround for this, depends on what you want to achieve. If you can specify what your requirement is, we can help th

Input stream partition to task mapping

Hi, Is it possible that multiple tasks read from the same input stream partition? example: task 0 stream A partition 0, stream B partition 0 task 1 stream A partition 1, stream B partition 0 task 2 stream A partition 3, stream B partition 0 In this example all messages in stream B partition 0 wo