Re: Partition assignment in kafka streams

2019-08-08 Thread Guozhang Wang
Hi Navneeth, In Streams you can only get your assignment at runtime via the API mentioned above, but cannot get it before the streams app starts up (I assume that is what you meant for "upfront"). So if you can 1) first start the streams app, and then 2) read the assignment, and then start writing

Re: Partition assignment in kafka streams

2019-08-07 Thread Navneeth Krishnan
Hi All, Any suggestions? Thanks On Thu, Aug 1, 2019 at 8:58 PM Navneeth Krishnan wrote: > Hi Guozhang, > > Thanks for the clarification. What I want to achieve is use of localized > data. We have much larger state which has to be used at a per instance > context. So if I can get the assignmen

Re: Partition assignment in kafka streams

2019-08-01 Thread Navneeth Krishnan
Hi Guozhang, Thanks for the clarification. What I want to achieve is use of localized data. We have much larger state which has to be used at a per instance context. So if I can get the assignment upfront I can basically write data to partitions in such a way that all data goes to that specific no

Re: Partition assignment in kafka streams

2019-08-01 Thread Guozhang Wang
Hello Navneeth, I may be misunderstanding your intent from the previous emails here, so just a quick summary: 1) if you just want to "know" which partitions are assigned to which instance, this can be retrieved in multiple ways (e.g. the one mentioned by Matthias, and also one can get this info f

Re: Partition assignment in kafka streams

2019-07-31 Thread Matthias J. Sax
You cannot hook into partition assignment, and I am not sure what you exactly want to do. You can get local assignment metadata via `KafkaStreams#localThreadMetadata()` though. Hope this helps. -Matthias On 7/29/19 11:29 PM, Navneeth Krishnan wrote: > Hi All, > > The main reason for knowing t

Re: Partition assignment in kafka streams

2019-07-29 Thread Navneeth Krishnan
Hi All, The main reason for knowing the partitions is to have a localized routing based on partitions assigned to set a stream tasks. This would really help in my use case. Thanks On Mon, Jul 29, 2019 at 8:58 PM Navneeth Krishnan wrote: > Hi, > > I'm using the processor topology for my use cas

Partition assignment in kafka streams

2019-07-29 Thread Navneeth Krishnan
Hi, I'm using the processor topology for my use case and I would like to get the partitions assigned to a particular stream instance. I looked at the addSouce function but I don't see a way to add a callback to get notified when partition assignment or reassignment happens. Please advise. Thank y