Hi, all. Greg has updated the overview doc according to our discussion with the MANO WG and the Doctor team. Please help review the following commit and see if you have further comments. We will try to get this accepted and move on to detail design. Thank you!
https://gerrit.opnfv.org/gerrit/#/c/33475/ -----邮件原件----- 发件人: mano-wg-boun...@lists.opnfv.org [mailto:mano-wg-boun...@lists.opnfv.org] 代表 Fu Qiao 发送时间: 2017年4月13日 11:10 收件人: 'Ryota Mibu'; opnfv-tech-discuss@lists.opnfv.org; mano...@lists.opnfv.org; 'Edgar StPierre'; 'Stefan Arntzen'; greg.wai...@windriver.com; 'Jolliffe, Ian'; 'Yue Yuan'; 'Yin Tony'; 'juan_qiu' 主题: [mano-wg] 答复: [opnfv-tech-discuss] [opnfv-tech-discussion][availability][doctor]Discussion of HA API with the doctor team Hi, all. We have had a good discussion with the Doctor team in our call. Below are the minutes. Thank you for Ryota and the doctor team to join the discussion and provide useful comments. • VM Heartbeating & Health-checking o libvirt watchdog and its integration / use with OpenStack https://blueprints.launchpad.net/nova/+spec/libvirt-watchdog https://wiki.openstack.org/wiki/LibvirtWatchdog#Notifications Need to review behavior of this capability as compared to the proposed VM Heartbeating/Health-checking I believe there is a bit of overlap, but believe that the VM Heartbeating / Health-checking provides a more complete solution I’ll update document with a NOTE on comparing the proposal with the libvirt watchdog o Update the architecture diagram to be consistent with the most current DOCTOR Architecture Diagrams e.g. Vitrage & Congress are not necessarily deployed at same time use DOCTOR terminology where applicable, e.g. “inspector modules” OPNFV DOCTOR includes patches/components of AODH, NOVA, NEUTRON … not just Vitrage and Congress, as shown in diagram • (although also had comment that should remove OPNFV DOCTOR outline from diagram as OPNFV DOCTOR is a requirement specification and not an implementation) indicate (possibly just in text below diagram) that the “Guest Heartbeat / Health-Check Server” on Controller Node is possibly not required, as the Vitrage data source interface can be remotely reached by “Guest Heartbeat / Health-Check Compute” on the Compute Node o In text, provide a little more detail on content of actual messaging, e.g. PDUs and rough message content o OVERALL: believe there was general agreement in the way that VM Heartbeating & Health-checking was integrated / inter-worked with OPNFV DOCTOR’s Vitrage/Congress and overall OPNFV fault reporting architecture key feedback was to understand and highlight the additional value of the VM Heartbeating & Health-checking functionality over the existing libvirt watchdog integration into OpenStack • Server Group Messaging o suggestion that Rabbit MQ pub/sub messaging could be an alternative for routing of messages this is an implementation detail though o general discussions on “HA use cases” for how this messaging could be leveraged e.g. split-brain avoidance, faster peer VM state change notifications, o Overall: agreement that the Server Group Messaging Architecture did NOT conflict with Doctor Architecture need further review with OPNFV MANO Team as to how they would position this functionality e.g. • position it as an alternative for various HA use cases ? • versus • mandating that this service group messaging be used for specific HA user cases ? -----邮件原件----- 发件人: opnfv-tech-discuss-boun...@lists.opnfv.org [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] 代表 Fu Qiao 发送时间: 2017年4月10日 11:22 收件人: 'Ryota Mibu'; opnfv-tech-discuss@lists.opnfv.org; mano...@lists.opnfv.org; 'Edgar StPierre'; 'Stefan Arntzen'; greg.wai...@windriver.com; 'Jolliffe, Ian'; 'Yue Yuan'; 'Yin Tony'; 'juan_qiu' 主题: [opnfv-tech-discuss] [opnfv-tech-discussion][availability][doctor]Discussion of HA API with the doctor team Hi, all. I invite the doctor team to join our next call. Below are the logistics. I suggest Greg continue the introduction of the HA API, and focus more on the coordination with Doctor. Hope we can finalize a coordinated scheme with the Doctor team through this call. Next Meeting on 13:00-14:00 UTC, Wednesday, Apr. 12nd https://global.gotomeeting.com/join/772448949 Toll free NO. USA: +1 (646) 749-3117 Meeting ID: 772-448-949 Agenda: coordination with doctor team about the HA API -----邮件原件----- 发件人: opnfv-tech-discuss-boun...@lists.opnfv.org [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] 代表 Fu Qiao 发送时间: 2017年4月5日 16:19 收件人: 'Ryota Mibu'; opnfv-tech-discuss@lists.opnfv.org; mano...@lists.opnfv.org; 'Edgar StPierre'; 'Stefan Arntzen'; greg.wai...@windriver.com; 'Jolliffe, Ian'; 'Yue Yuan'; 'Yin Tony'; 'juan_qiu' 主题: [opnfv-tech-discuss] 答复: [opnfv-tech-discussion][availability][doctor]Discussion of HA API with the doctor team Hi, Ryota. Thank you! I think we need to coordinate with Doctor because we need to utilize the notification scheme provided by doctor. However, I am not the expert on this. We can wait for Greg to introduce more during the call. -----邮件原件----- 发件人: Ryota Mibu [mailto:r-m...@cq.jp.nec.com] 发送时间: 2017年4月5日 15:50 收件人: Fu Qiao; opnfv-tech-discuss@lists.opnfv.org; mano...@lists.opnfv.org; 'Edgar StPierre'; 'Stefan Arntzen'; greg.wai...@windriver.com; 'Jolliffe, Ian'; 'Yue Yuan'; 'Yin Tony'; 'juan_qiu' 主题: RE: [opnfv-tech-discussion][availability][doctor]Discussion of HA API with the doctor team Hi Fu Qiao, Yes, I can join and also ask other Doctor members to join during our Doctor meeting the day before (April 11). My initial concern is that "Fault in VM" is not in the scope of Doctor, so I'm not clear whether we can contribute our insights. But, yes, we can join the discussion at the meeting. And, technically you can utilize the notification function implemented in OpenStack. Let's talk during the meeting. I'll also check the document by the meeting. BTW, the figure links are broken in the document you pointed. We can still see the figures in artifact server though [1,2]. [1] http://artifacts.opnfv.org/availability/docs/development_overview/_images/OPNFV_HA_Guest_APIs-Overview_HLD-Guest_Heartbeat-FIGURE-1.png [2] http://artifacts.opnfv.org/availability/docs/development_overview/_images/OPNFV_HA_Guest_APIs-Overview_HLD-Peer_Messaging-FIGURE-2.png BR, Ryota > -----Original Message----- > From: Fu Qiao [mailto:fuq...@chinamobile.com] > Sent: Wednesday, April 05, 2017 11:37 AM > To: Mibu Ryota(壬生 亮太) <r-m...@cq.jp.nec.com>; > opnfv-tech-discuss@lists.opnfv.org; mano...@lists.opnfv.org; 'Edgar > StPierre' <edgar.stpie...@dell.com>; 'Stefan Arntzen' > <stefan.arnt...@huawei.com>; greg.wai...@windriver.com; 'Jolliffe, Ian' > <ian.jolli...@windriver.com>; 'Yue Yuan' <yuan....@zte.com.cn>; 'Yin Tony' > <14_...@tongji.edu.cn>; 'juan_qiu' > <juan_...@tongji.edu.cn> > Subject: [opnfv-tech-discussion][availability][doctor]Discussion of HA > API with the doctor team > > Hi, Ryota. > As you may have noticed, the HA team has been working on the HA API > work during the D release. We had a proposal worked out for the > release. Since the scheme we proposed has close relationship with the doctor > scheme, we would like to discuss further with the doctor team to make sure we > can have a unified solution. > You can find detailed proposal as below. I wonder if your team is OK > to join the next HA project meeting on Apr. 12, at > 13:00-14:00 UTC? Thank you! > > http://docs.opnfv.org/en/stable-danube/submodules/availability/docs/de > velopment/overview/index.html > > > -----邮件原件----- > 发件人: opnfv-tech-discuss-boun...@lists.opnfv.org > [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] 代表 Fu Qiao > 发送时间: 2017年4月5日 10:28 > 收件人: greg.wai...@windriver.com; opnfv-tech-discuss@lists.opnfv.org; 'Yin > Tony'; 'juan_qiu' > 抄送: 'Edgar StPierre'; 'Hui Wang'; 'Yue Yuan'; mano...@lists.opnfv.org > 主题: [opnfv-tech-discuss] 答复: > [opnfv-tech-discussion][availability]project meeting on Mar. 29 > > Hi, all. Below are the minutes for our call last week. Sorry for the > delay due to the holiday in China. Thank you to everyone for the discussion. > > Minutes > HA API doc updates – Greg > The team goes through the update version. Some members of the MANO WG join > the discussion and provide useful suggestions. > Detailed suggestions are as follows: > 1) The team agrees to restructure the doc as a proposal, not a total solution. > 2) Michael, from the MANO WG, indicates that currently there are more > than one framework to improve the availability and reliability of the > OPNFV Platform, which could be a good thing. However coordination work is > needed in order to tell people how to behave in a unified way. > 3) priority should be defined among multiple schemes to avoid race > condition > 4) The team agrees to include the heart beat part and peer state notification > session for the D release. > 5) It is suggested to continue the discussion of the proposal with the > parameter team and doctor team. > > -----邮件原件----- > 发件人: opnfv-tech-discuss-boun...@lists.opnfv.org > [mailto:opnfv-tech-discuss-boun...@lists.opnfv.org] 代表 Fu Qiao > 发送时间: 2017年3月28日 14:41 > 收件人: greg.wai...@windriver.com; opnfv-tech-discuss@lists.opnfv.org; 'Yin > Tony'; 'juan_qiu' > 抄送: 'Edgar StPierre'; 'Hui Wang'; 'Yue Yuan' > 主题: [opnfv-tech-discuss] [opnfv-tech-discussion][availability]project > meeting on Mar. 29 > > Hi, Greg. Thank you for the update. > I arrange another meeting tomorrow on Mar. 29th to continue the > discussion. I suggest we go through the heartbeat section first and see if we > can reach an agreement. > > Meetings for High Availability Project Next Meeting on 13:00-14:00 > UTC, Wednesday, Mar. 29th > https://global.gotomeeting.com/join/772448949 > > Toll free NO. > USA: +1 (646) 749-3117 > > Meeting ID: 772-448-949 > Agenda > HA API doc updates – Greg > > -----邮件原件----- > 发件人: Greg Waines (OPNFV Code Review) [mailto:ger...@opnfv.org] > 发送时间: 2017年3月28日 0:36 > 抄送: Ulrich Kleber; Hui Wang; Stefan Arntzen; Qiao Fu; Ian Jolliffe; > Edgar StPierre; Yue Yuan > 主题: Change in availability[master]: Updating Commit based on third round of > review during OPNFV ... > > Greg Waines has posted comments on this change. ( > https://gerrit.opnfv.org/gerrit/26663 ) > > Change subject: Updating Commit based on third round of review during > OPNFV HA meetings, as well as submitted comments on the review. > ...................................................................... > > > Patch Set 4: > > (11 comments) > > Updated based on recent comments. > And added HTML and PDF versions of document for easier reading. > > https://gerrit.opnfv.org/gerrit/#/c/26663/4/Support_For_HA_Guest_APIs/ > OPNFV_HA_Guest_APIs-Overview_HLD.rst > File Support_For_HA_Guest_APIs/OPNFV_HA_Guest_APIs-Overview_HLD.rst: > > PS4, Line 15: bstract: This document provides an overview of a set of new > : optional capabilities where the OpenStack Cloud messages > : into the Guest VMs in order to provide improved Availability > : of the hosted VMs. These new capabilities include: enabling > : the detection of and recovery from internal VM faults, > enabling > : Guest VMs to gracefully handle and provide loss-of-service > : warnings to cloud adminstrative operations on the VM and > : providing a simple out-of-band messaging service to prevent > : scenarios such as split brain. > > Hi, Greg. You may find more info in the following link. > Ugh ... couldn't figure out how to do it from these instructions. > > I generated an HTML version and PDF version and included it in COMMIT along > with the .rst file. > > > PS4, Line 42: VM Heartbeating and Health Checking > : - VM Event Notification and Feedback > : - VM Peer State Notification and Messaging > > Good to hear. I think we should not merge this patch without this > > being doc > I added more explicit indication in the INTRODUCTION that VNFM interactions > were discussed in the specific sections. > > AND in the VM Event Notification and Feedback section, i added a note > at the beginning discussing the open item that this currently solicits > feedback from VM and notifies VM of upcoming command 'independent' of > VNFM. i.e. even for commands issued by the VNFM, this service > provides an optional check on the state of the VM and a Notification to the > VM for graceful handling of the command. An alternative approach would be to > go thru the VNFM to both check the state of the VM and notify the VM of the > event. More discussion is required here with the Management and > Orchestration OPNFV Team. > > > PS4, Line 51: lifecycle of the VM > > I think the host-to-guest message cannot be earlier than the VNFM > > having st > I changed the wording to say > "... is available very early after spawning the VM." > > There is not intent for host-to-guest VM messaging to be used for high-level > VM lifecycle management. > > The intent of the sentence is to say that the messaging mechanism is > quite simple and therefore can be available very early after booting the VM. > i.e. there are very few dependencies on the messaging mechanism. > > > PS4, Line 53: interaction with > : the Guest VM > > I think the relation of host-to-guest messaging to VNFM must be > > covered in > I added a more explicit statement in this introduction to indicate > that the interaction of these services with the VNFM are further explained in > the following sections. > > > PS4, Line 109: This would then enable VNF Managers listening to > : OPNFV's DOCTOR External Alarm Reporting through Telemetry's > AODH, to initiate > : any required fault recovery actions > > Happy to hear this same heartbeating will be reported via doctor. I > > didn't > I suspect being able to see the diagrams will help with this as well. > > > PS4, Line 121: A daemon within the Guest VM will register with the OpenStack > Guest > : Heartbeat Service on the compute node to initiate the > heartbeating. > > I think your first paragraph clarifies the main question. > I added this clarification > > > PS4, Line 128: final fault recovery actions are the > responsiblilty/authority of the VNF Managers > > Thanks, now I understand this part. Maybe mention this relation to > > vitrage > I updated this section. > > > PS4, Line 155: Host Deliverables > > Please clarify whether "host deliverable" means that you write > > OpenStack bl > Updated. > > > PS4, Line 169: Guest Deliverables > > OK. If this was a separate project proposal, this would be part of > > the deli > Updated > > > PS4, Line 218: VM Event Notifications > > I think with your answers above, things get clearer. I suggest to > > add that > Updated > > > PS4, Line 323: Server Group > > Please clarify what is a server group. Please use NFV terminology. > Updated with description of Server Group and Peer VM > > > -- > To view, visit https://gerrit.opnfv.org/gerrit/26663 > To unsubscribe, visit https://gerrit.opnfv.org/gerrit/settings > > Gerrit-MessageType: comment > Gerrit-Change-Id: I1e44efe6052dfa84cdffd9bbebe8be86e4992121 > Gerrit-PatchSet: 4 > Gerrit-Project: availability > Gerrit-Branch: master > Gerrit-Owner: Greg Waines <greg.wai...@windriver.com> > Gerrit-Reviewer: Edgar StPierre <edgar.stpie...@dell.com> > Gerrit-Reviewer: Greg Waines <greg.wai...@windriver.com> > Gerrit-Reviewer: Hui Wang <wanghu...@huawei.com> > Gerrit-Reviewer: Ian Jolliffe <ian.jolli...@windriver.com> > Gerrit-Reviewer: Qiao Fu <fuq...@chinamobile.com> > Gerrit-Reviewer: Stefan Arntzen <stefan.arnt...@huawei.com> > Gerrit-Reviewer: Ulrich Kleber <ulrich.kle...@huawei.com> > Gerrit-Reviewer: Yue Yuan <yuan....@zte.com.cn> > Gerrit-HasComments: Yes > > > > _______________________________________________ > opnfv-tech-discuss mailing list > opnfv-tech-discuss@lists.opnfv.org > https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss > > > > _______________________________________________ > opnfv-tech-discuss mailing list > opnfv-tech-discuss@lists.opnfv.org > https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss > > > _______________________________________________ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss _______________________________________________ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss _______________________________________________ mano-wg mailing list mano...@lists.opnfv.org https://lists.opnfv.org/mailman/listinfo/mano-wg _______________________________________________ opnfv-tech-discuss mailing list opnfv-tech-discuss@lists.opnfv.org https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
