Thanks for describing the OPNFV flow. That’s helpful. With regards to Option 1, I would suggest separating “where the tests are run” from “how the results are certified”. I don’t think having the provider upload the artifacts (Heat or CSAR) with their results necessarily constrains or bifurcates their testing experience. They can still run these tests in the lab and produce their results. In the infrastructure or functional testing world recreating the testing is prohibitive. I’m sure that was a large contributor to the certification scheme (independent review of the logs) used in OPNFV. In this case, we’re performing rather simple, repeatable scans of static artifacts. The provider can still develop and test externally producing their result files. However, for this portion of the results we don’t really have to rely on simple human inspection of forgeable text results – we can just retest them if they provide the associated artifact. The only change would be uploading the artifacts along with the results.
Now there might be other reasons we deem uploading the artifacts prohibitive, but in that example the actual testing experience has not changed. We’re just asking for one additional artifact during result submission. To elaborate on the checksum scenario in Option 2, the checksum is a unique, repeatable value generated from the contents of the artifact (all files). Any change to the artifact would generate a different checksum. If a provider submitted results and a checksum, they are stating that a specific artifact (identifiable by the checksum) passed these tests in the result file. The OVP portal or human reviewers can’t verify the checksum – they’re just looking at the result text files which can always be fabricated no matter what we do. However, the eventual consumer of the VNF can validate it. When they receive the artifact, they can run the validations independently. The results and checksums should match. If either is different, then either the results were doctored or they have provided a different artifact to the consumer (different checksum). Happy to discuss further on the next call. I’m sure we can work out a solution. Thanks, Trevor From: Lincoln Lavoie [mailto:[email protected]] Sent: Thursday, February 07, 2019 11:06 AM To: LOVETT, TREVOR J <[email protected]> Cc: [email protected]; [email protected]; [email protected]; [email protected]; Kanagaraj Manickam <[email protected]>; STARK, STEVEN <[email protected]>; WRIGHT, STEVEN A <[email protected]>; Katsaounis Molyvas Stamatios <[email protected]>; HALLAHAN, RYAN <[email protected]>; [email protected] Subject: Re: CVC Jonit Meeting (Feb 4th) Agenda Hi Trevor, The work flow is, user runs the dovetail tool, which produces the log output the user can upload to the portal. This allows users to run internal testing, debugging, etc. When they are ready to apply for certification, they upload the results to the portal, along with submitting an application (i.e. info about the VNF, like marketing name, etc.). That application and logs are reviewed by the reviewers (can't be from the same company). It takes two approvals from two different reviewers to pass that gate (i.e. 2 sets of eyes on the results). Once approved, the admin is able to publish the listing to the website, etc. Hopefully this clarifies things on the work flow. For option 2, the challenge is, couldn't they just paste over the checksum before they submit the results? Cheers, Lincoln On Thu, Feb 7, 2019 at 11:57 AM LOVETT, TREVOR J <[email protected]<mailto:[email protected]>> wrote: What was the solution in OPNFV? Can someone lay out the proposed experience and flow as well? Based on prior conversations, I thought the proposal for a short-term solution was a human-being from the VNF Provider/Lab to upload the result files to the web site. If there’s an alternate proposed flow (i.e. Dovetail directly uploading the results to the portal), then I think it would be helpful to start getting that written down somewhere and it has bearing on how the results are secured/certified. It’s not clear that all players are on the same page here at this juncture. In Option 2, I don’t fully understand your comment about a clever user overwriting what is in the logs. I did state that the report itself could still be fabricated so I’m not arguing that it couldn’t be. My point was that if they provide a checksum with their report, then the result could at least be challenged and verified. If a consumer of the VNF can’t reproduce the test results on the files with the same checksum then the fraud can be detected. Thanks, Trevor From: Lincoln Lavoie [mailto:[email protected]<mailto:[email protected]>] Sent: Thursday, February 07, 2019 10:22 AM To: LOVETT, TREVOR J <[email protected]<mailto:[email protected]>> Cc: [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]>; Kanagaraj Manickam <[email protected]<mailto:[email protected]>>; STARK, STEVEN <[email protected]<mailto:[email protected]>>; WRIGHT, STEVEN A <[email protected]<mailto:[email protected]>>; Katsaounis Molyvas Stamatios <[email protected]<mailto:[email protected]>>; HALLAHAN, RYAN <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: Re: CVC Jonit Meeting (Feb 4th) Agenda Hi Trevor, I think one concern I have with Option 1 is, it will only work for the VVP compliance testing, as soon as we move towards more advanced testing (i.e. live cycle testing, etc), it would cause a split path for VNF testing, where you have to do 1/2 of your testing through a web portal and then 1/2 on some type NFVI / MANO platform that is in your lab or another lab. I think this will just make for a complicated future. For option 2, this provides some protection, but a clever user could always just overwrite what is in the logs before submitting this. Many of these questions / points were already talked about a lot within the OPNFV program, when that was developed and there was a desire to protect the results as well. Cheers, Lincoln On Thu, Feb 7, 2019 at 10:42 AM LOVETT, TREVOR J <[email protected]<mailto:[email protected]>> wrote: During the call, one item that came up was how to lessen the likelihood of a fraudulent report being submitted as part of the certification/badging process. One suggestion was to include additional runtime, vnf-specific trace information in the report itself making it more difficult to fabricate. The VVP team discussed this, but we feel this would be non-trivial to implement and not truly prevent the core issue. We would not be in favor of pursuing this option. If we feel that we need to protect against fraudulent reports, then we are suggesting two alternatives that can be discussed in a future session. 1) Certification Portal Executes the validations directly using Dovetail: a. Since we are only scanning a set of static files, and the validation tools will be available in a container for execution, we could simply have the portal run the scan upon uploading of the artifacts. This eliminates a fraudulent report submission entirely, and avoids other issues such as ensuring the proper versions of the tools are executed externally. b. I do understand there may be sensitivity to uploading the proprietary artifacts, but there is no need for them to be stored or shared indefinitely. They would only need to be retained for the duration of the scan. After that we simply need to retain the checksum of the artifact that was submitted. c. If we simply don’t think VNF Providers would agree to that, then our back-up option is below 2) Display Artifact Checksum Along with Badge a. The VVP tool already computes an MD5 hash of all the file contents and stores that in the report b. If we retain this information and publish it on the certification site, then the ultimate consumer of the artifact can at least verify the artifact they receive from the vendor corresponds to the artifact submitted for certification. We should likely do this even in option #1 for the same reason. c. This doesn’t mean prevent the provider from submitting a fabricated report, but it does ensure that it can be challenged and verified after-the-fact if needed. d. I believe the CSAR similarly has an checksum that can be used for this purpose as well. We can discuss the topic further in one of our upcoming joint calls. Thanks, Trevor From: [email protected]<mailto:[email protected]> [mailto:[email protected]<mailto:[email protected]>] On Behalf Of Gaoweitao(Victor) Sent: Sunday, February 03, 2019 2:26 AM To: [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]> Cc: Kanagaraj Manickam <[email protected]<mailto:[email protected]>>; Lincoln Lavoie <[email protected]<mailto:[email protected]>> Subject: [onap-discuss] CVC Jonit Meeting (Feb 4th) Agenda Hi VNFSDK/VVP/Dovetail Developers, Here is the initial agenda for Feb 4th Joint meeting<https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.onap.org_display_DW_CVC-2BJoint-2BMeeting-2B02-2D04-2D2019&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=dIU_U39dl9FoORwHk72kyMcMyxm1s8RqOhr8grdzE2s&m=ZKQoo454X4U_lELjzO_WCevaKAV5R5GeOb-B0URqdcI&s=-YhkEGJSSKD46SqQByjR_bJvnYWC7lsI62elfSWGTQA&e=>, feel free to add your topics: 1. CVC one click deployment 2. Show marketplace capability and is it possible used for dovetail test script execution trigger? I will be absence due to Chinese new year and Kanagraj from VNFSDK Team is my proxy and host the meeting. The zoom bridge is still available during my absence: https://zoom.us/j/346625009<https://urldefense.proofpoint.com/v2/url?u=https-3A__zoom.us_j_346625009&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=dIU_U39dl9FoORwHk72kyMcMyxm1s8RqOhr8grdzE2s&m=ZKQoo454X4U_lELjzO_WCevaKAV5R5GeOb-B0URqdcI&s=-wmrOsLWyh45frQOsVJCAqGlh2DmMNJu15PMacF17Z0&e=> The previous meeting minutes is here: https://wiki.onap.org/display/DW/Joint+CVC+Meeting<https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.onap.org_display_DW_Joint-2BCVC-2BMeeting&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=dIU_U39dl9FoORwHk72kyMcMyxm1s8RqOhr8grdzE2s&m=ZKQoo454X4U_lELjzO_WCevaKAV5R5GeOb-B0URqdcI&s=mkSu7O2_kzmJipm2xlbfaaZGwXViIQJvQt12MmFifJo&e=> BR Victor _._,_._,_ ________________________________ Links: You receive all messages sent to this group. View/Reply Online (#15324)<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.onap.org_g_onap-2Ddiscuss_message_15324&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=dIU_U39dl9FoORwHk72kyMcMyxm1s8RqOhr8grdzE2s&m=ZKQoo454X4U_lELjzO_WCevaKAV5R5GeOb-B0URqdcI&s=2DIdFBAfxs43n6h4Lw6yH2FGyRmBqimlbz41vQNgGtw&e=> | Reply To Group<mailto:[email protected]?subject=Re:%20%5Bonap-discuss%5D%20CVC%20Jonit%20Meeting%20%28Feb%204th%29%20Agenda> | Reply To Sender<mailto:[email protected]?subject=Private:%20Re:%20%5Bonap-discuss%5D%20CVC%20Jonit%20Meeting%20%28Feb%204th%29%20Agenda> | Mute This Topic<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.onap.org_mt_29638769_1198157&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=dIU_U39dl9FoORwHk72kyMcMyxm1s8RqOhr8grdzE2s&m=ZKQoo454X4U_lELjzO_WCevaKAV5R5GeOb-B0URqdcI&s=LeTLkAhNSX8sD_rxDuVs1mIlWHYZESMZPmuIf18ic_A&e=> | New Topic<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.onap.org_g_onap-2Ddiscuss_post&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=dIU_U39dl9FoORwHk72kyMcMyxm1s8RqOhr8grdzE2s&m=ZKQoo454X4U_lELjzO_WCevaKAV5R5GeOb-B0URqdcI&s=rxFGHYc7sNu0tzbdyYovrN7Kpnj-T_bZ9Gu0niWSMT8&e=> Your Subscription<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.onap.org_g_onap-2Ddiscuss_editsub_1198157&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=dIU_U39dl9FoORwHk72kyMcMyxm1s8RqOhr8grdzE2s&m=ZKQoo454X4U_lELjzO_WCevaKAV5R5GeOb-B0URqdcI&s=6n1a-52slPnkgXiGGJySGaZzkxq_t5TrSTw_o9GtvqI&e=> | Contact Group Owner<mailto:[email protected]> | Unsubscribe<https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.onap.org_g_onap-2Ddiscuss_unsub&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=dIU_U39dl9FoORwHk72kyMcMyxm1s8RqOhr8grdzE2s&m=ZKQoo454X4U_lELjzO_WCevaKAV5R5GeOb-B0URqdcI&s=x3ICz0g1PsytMM03urA8OHzkY-WW_yA77jqC4tU5b7Q&e=> [[email protected]<mailto:[email protected]>] _._,_._,_ -- Lincoln Lavoie Senior Engineer, Broadband Technologies 21 Madbury Rd., Ste. 100, Durham, NH 03824 [email protected]<mailto:[email protected]> https://www.iol.unh.edu<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.iol.unh.edu&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=g9LhwjMTPM4AuoWvYyDmqA&m=UdvYRI0iWQVoM8OjcQ2mvJst7GbGwawBimVeis3IEm4&s=v7yOqkd93k2wkmp4FMkKZDTKNEblZrYoEpkpmHCrM5c&e=> +1-603-674-2755 (m) [http://homeautomation.lavoieholdings.com/_/rsrc/1390068882701/unh-iol-logo.png]<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.iol.unh.edu_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=g9LhwjMTPM4AuoWvYyDmqA&m=UdvYRI0iWQVoM8OjcQ2mvJst7GbGwawBimVeis3IEm4&s=M3cJfFEQvxci-Zj1_7zb_oIPoAuzKoUV4mXBTLeOcRc&e=> -- Lincoln Lavoie Senior Engineer, Broadband Technologies 21 Madbury Rd., Ste. 100, Durham, NH 03824 [email protected]<mailto:[email protected]> https://www.iol.unh.edu<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.iol.unh.edu&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=g9LhwjMTPM4AuoWvYyDmqA&m=ASzlsco1NQ_ij_9fuiLk-_y3RaYqHeyy9UrPQsEGMnk&s=yw5oCAAUBEh4sb3ycD5QfDXWKBYTquqHlCVFVwJyPWE&e=> +1-603-674-2755 (m) [http://homeautomation.lavoieholdings.com/_/rsrc/1390068882701/unh-iol-logo.png]<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.iol.unh.edu_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=g9LhwjMTPM4AuoWvYyDmqA&m=ASzlsco1NQ_ij_9fuiLk-_y3RaYqHeyy9UrPQsEGMnk&s=qs5TA4zmA7QSGWAfTZsKnUCecW27hp_JoxNCXNxkBNQ&e=>
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#22777): https://lists.opnfv.org/g/opnfv-tech-discuss/message/22777 Mute This Topic: https://lists.opnfv.org/mt/29667350/21656 Group Owner: [email protected] Unsubscribe: https://lists.opnfv.org/g/opnfv-tech-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
