Hi Yi, Thank you for your quick response! Your suggestions make a lot of sense, and I will begin implementing them right away : )
Met vriendelijke groet / Kind regards, Robert Žuljević Software Developer Address: Trifkovicev trg 6, 21000 Novi Sad, Serbia Tel.: +31 20 6701 947 | +381 21 2155 500 Mobile: +381 64 428 28 46 Skype: robert.zuljevic Internet: www.levi9.com Chamber of commerce Levi9 Holding: 34221951 Chamber of commerce Levi9 IT Services BV: 34224746 This e-mail may contain confidential or privileged information. If you are not (one of) the intended recipient(s), please notify the sender immediately by reply e-mail and delete this message and any attachments permanently without retaining a copy. Any review, disclosure, copying, distribution or taking any action in reliance on the contents of this e-mail by persons or entities other than the intended recipient(s) is strictly prohibited and may be unlawful. The services of Levi9 are exclusively subject to its general terms and conditions. These general terms and conditions can be found on www.levi9.com and a copy will be promptly submitted to you on your request and free of charge. -----Original Message----- From: Yi Pan [mailto:nickpa...@gmail.com] Sent: Thursday, June 25, 2015 6:38 PM To: dev@samza.apache.org Subject: Re: [SAMZA-690] Changelog topic creation should not be in the container code Hi, Robert, Thanks for digging into this. I am embedding my answers below: On Thu, Jun 25, 2015 at 7:40 AM, Robert Zuljevic <r.zulje...@levi9.com> wrote: > 1. Is checkpoint topic referred to in the description coordinator > stream/topic? > In the master branch, checkpoint topic is deprecated (except for migration purpose). The checkpoints will be sent to coordinator stream. Hence, there is no need to change the creation of checkpoint topic any more. > 2. Changelog topic creation is handled by TaskStorageManager class > which is required by SamzaContainer. Would it be preferable to: > > a. Create TaskStoreManager instance(s) in JobRunner and pass them > to SamzaContainer > > b. Create changelog stream in JobRunner/JobCoordinator and skip it > in TaskStoreManager > I prefer option b in your above proposal, w/ a slight modification: 1. JobCoordinator now has a changelogManager which reads the changelog partition to task mapping from the coordinator stream 2. JobCoordinator also has access to the job config and will be able to figure out what are the changelog topics needed in the job 3. JobCoordinator start should have an additional step to create all the changelog topics needed in the job. If exists, validate the partition numbers 4. In TaskStoreManager, we should just get the changelog topic metadata and validate the partitions are correct. Does that sound reasonable? -Yi > > > > > Met vriendelijke groet / Kind regards, > > Robert Žuljević > > Software Developer > > [image: Title: Levi9 IT Services] > ------------------------------ > > Address: Trifkovicev trg 6, 21000 Novi Sad, Serbia > > Tel.: +31 20 6701 947 | +381 21 2155 500 > > Mobile: +381 64 428 28 46 > > Skype: robert.zuljevic > > Internet: www.levi9.com > > > > Chamber of commerce Levi9 Holding: 34221951 > > Chamber of commerce Levi9 IT Services BV: 34224746 > ------------------------------ > > This e-mail may contain confidential or privileged information. If you > are not (one of) the intended recipient(s), please notify the sender > immediately by reply e-mail and delete this message and any > attachments permanently without retaining a copy. Any review, > disclosure, copying, distribution or taking any action in reliance on > the contents of this e-mail by persons or entities other than the > intended recipient(s) is strictly prohibited and may be unlawful. > > The services of Levi9 are exclusively subject to its general terms and > conditions. These general terms and conditions can be found on > www.levi9.com and a copy will be promptly submitted to you on your > request and free of charge. > > >
