The input to the tests is a directory containing the Heat Templates and any supporting files (environment files, scripts, etc) for a single VNF. there may be multiple VNF Components/Modules, but a single VNF.
Tests are generated based on their input or fixture. There may be multiple instances of a single test if its input is a template file. For example, there would be one test case executed per template file if the input fixture to the test is defined as heat_template. The input to each test case is found in the files attribute of each Test Result in the report.json. From: "Katsaounis Molyvas Stamatios" <[email protected]<mailto:[email protected]>> Date: Monday, January 21, 2019 at 7:03:53 AM To: "LOVETT, TREVOR J" <[email protected]<mailto:[email protected]>>, "[email protected]" <[email protected]<mailto:[email protected]>>, "[email protected]" <[email protected]<mailto:[email protected]>> Cc: "xudan (N)" <[email protected]<mailto:[email protected]>>, "STARK, STEVEN" <[email protected]<mailto:[email protected]>>, "WRIGHT, STEVEN A" <[email protected]<mailto:[email protected]>>, "HALLAHAN, RYAN" <[email protected]<mailto:[email protected]>>, "WEINSTOCK, ALAN M" <[email protected]<mailto:[email protected]>>, "[email protected]" <[email protected]<mailto:[email protected]>>, "Panagiotis Karalis" <[email protected]<mailto:[email protected]>> Subject: RE: [opnfv-tech-discuss] [Dovetail] Example VVP Output per Action Item from Developer Event Hi all, I would like to ask some questions concerning the results sample as well. >From my understanding after reading the wiki entry and the sample, the >script’s input is a directory which contains some HEAT templates. Then, for each template of the directory, the script runs every (or specified) test case of every (or specified?) test module with (some?) or without requirements. Is the above description correct? If yes, then directory == one and only xNF? Or can many xNFs co-exist inside the directory under validation? Best regards, Stamatis from Dovetail Stamatis Katsaounis Software Enginner Software Development Center ______________________________________ Intracom Telecom 19.7 km Markopoulou Ave., Peania, GR 19002 t: +30 2106677689 [email protected]<mailto:[email protected]> http://www.intracom-telecom.com/<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.intracom-2Dtelecom.com_&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=g9LhwjMTPM4AuoWvYyDmqA&m=ETD5YlkisKudx53a82hO0K7o0_VNLFuXXxQF_UyEMQE&s=8Svb3vsUsX9ZXjgPunKcDUd4adebI44jsVoiSpC2kGQ&e=> JOIN US Egypt Defence Expo 3-5 December Cairo, Egypt MWC Barcelona 25-28 February 2019 Barcelona, Spain MWC Shanghai 26-28 June 2019 Shanghai, China GITEX Technology Week 6-10 October 2019 Dubai, UAE MWC Los Angeles 22-24 October 2019 Los Angeles, USA Futurecom 28-31 October 2019 Sao Paulo, Brazil AfricaCom 12-14 November 2019 Cape Town, South Africa The information in this e-mail message and any attachments are intended only for the individual or entity to whom it is addressed and may be confidential. If you have received this transmission in error, and you are not an intended recipient, be aware that any copying, disclosure, distribution or use of this transmission or its contents is prohibited. Intracom Telecom and the sender accept no liability for any loss, disruption or damage to your data or computer system that may occur while using data contained in, or transmitted with, this email. Views or opinions expressed in this message may be those of the author and may not necessarily represent those of Intracom Telecom. From: xudan (N) [mailto:[email protected]] Sent: Monday, January 21, 2019 9:19 AM To: LOVETT, TREVOR J <[email protected]>; Kanagaraj Manickam <[email protected]>; [email protected] Cc: STARK, STEVEN <[email protected]>; WRIGHT, STEVEN A <[email protected]>; HALLAHAN, RYAN <[email protected]>; WEINSTOCK, ALAN M <[email protected]>; Gaoweitao (Victor, Cloudify Network OSDT) <[email protected]>; [email protected]; [email protected] Subject: RE: [opnfv-tech-discuss] [Dovetail] Example VVP Output per Action Item from Developer Event Hi Trevor, The results format looks much better and clearer. I have some further questions about the results. 1. The ‘result’ of each test case could be PASS/FAIL/SKIP. In which case one test case will SKIP? 2. For some test cases, there are ‘requirements’ which has 3 keys ‘id’, ‘text’ and ‘keyword’. Also there are some test cases with empty ‘requirements’ (Line 915). Why they are empty without ‘id’, ‘text’ and ‘keyword’? 3. From Line 10539 to the end, there is another different kind of data format. Why there are 2 totally different data formats? 4. The ‘outcome’ should be the final result of all test cases. How to get the outcome? It will be PASS if there is not any FAIL test cases or not any FAIL/SKIP test cases? Anyway, the results format is sufficient for both Dovetail tool and web portal. Regards, Dan From: LOVETT, TREVOR J [mailto:[email protected]] Sent: Monday, January 21, 2019 12:41 PM To: Kanagaraj Manickam <[email protected]<mailto:[email protected]>>; xudan (N) <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Cc: STARK, STEVEN <[email protected]<mailto:[email protected]>>; WRIGHT, STEVEN A <[email protected]<mailto:[email protected]>>; HALLAHAN, RYAN <[email protected]<mailto:[email protected]>>; WEINSTOCK, ALAN M <[email protected]<mailto:[email protected]>>; Gaoweitao (Victor, Cloudify Network OSDT) <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: RE: [opnfv-tech-discuss] [Dovetail] Example VVP Output per Action Item from Developer Event It still needs to be merged with master, but I’ve submitted a change to the validation-scripts that has a more comprehensive output. For more details, please see the file format specification here: https://wiki.onap.org/display/DW/Updated+JSON+Report+Output+for+Validation+Scripts<https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.onap.org_display_DW_Updated-2BJSON-2BReport-2BOutput-2Bfor-2BValidation-2BScripts&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=g9LhwjMTPM4AuoWvYyDmqA&m=ETD5YlkisKudx53a82hO0K7o0_VNLFuXXxQF_UyEMQE&s=AX6sRv3_sSHEoMfD8o07ewpx0IwnPrIBNVdV_DM233o&e=> I’ve attached a sample file here as well. I’ve also added a new option to the command line to direct output to a new directory ( --output-directory) The failures file still exists and con continue to be used if needed, but I would recommend the new file (report.json). Thanks, Trevor From: Kanagaraj Manickam [mailto:[email protected]] Sent: Friday, January 18, 2019 2:43 AM To: LOVETT, TREVOR J <[email protected]<mailto:[email protected]>>; xudan (N) <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Cc: STARK, STEVEN <[email protected]<mailto:[email protected]>>; WRIGHT, STEVEN A <[email protected]<mailto:[email protected]>>; HALLAHAN, RYAN <[email protected]<mailto:[email protected]>>; WEINSTOCK, ALAN M <[email protected]<mailto:[email protected]>>; Gaoweitao (Victor, Cloudify Network OSDT) <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: RE: [opnfv-tech-discuss] [Dovetail] Example VVP Output per Action Item from Developer Event Hi Trevor, Thank you for the inputs. I have integrated the VVP scripts in VTP and tested the results and needs to be refined further. Please find more details in-line response below. Thanks. Regards, Kanagaraj Manickam Senior System Architect P&S ONAP Huawei Technologies India Pvt. Ltd. Survey No. 37, Next to EPIP Area, Kundalahalli, Whitefield Bengaluru-560066, Karnataka Tel: + 91-80-49160700 ext 72410 Mob: 9945602938 [Company_logo] ________________________________ This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! From: LOVETT, TREVOR J [mailto:[email protected]] Sent: 16 January 2019 21:49 To: Kanagaraj Manickam <[email protected]<mailto:[email protected]>>; xudan (N) <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Cc: STARK, STEVEN <[email protected]<mailto:[email protected]>>; WRIGHT, STEVEN A <[email protected]<mailto:[email protected]>>; HALLAHAN, RYAN <[email protected]<mailto:[email protected]>>; WEINSTOCK, ALAN M <[email protected]<mailto:[email protected]>>; Gaoweitao (Victor, Cloudify Network OSDT) <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: RE: [opnfv-tech-discuss] [Dovetail] Example VVP Output per Action Item from Developer Event Here are the answers to your questions… 1. As we knew, ONAP Vendor Software Product (VSP) is zip containing the required HOT template with MANIFEST.json, Is VVP script validate this VSP as a whole including this MANIFEST.json or it only validates the HOT templates inside the VSP [Trevor] For a VNF created from Heat, the VSP is an artifact created in SDC itself so we are not uploading a VSP nor verifying the VSP as a whole – only the Heat. The Heat Orchestration Template package is a zip file with all files (Heat templates, environment files, and any supporting files/scripts) in the root directory of that zip file. This is what is uploaded into SDC. There is no MANIFEST.json file included per the Heat requirements. [Kanag] OK. I observed from the vFW demo that the artifact used for creating the VSP contains these files 1. HOT template 2. HOT env file 3. MANIFEST.json. so as part of CVC, when vendor submit their VNF package, I assumed MANIFEST also will be included. If this is not the case, I think we can ignore this :) 2. As VNFREQS defines list of HOT template requirements with MUST and MUST NOT criteria, a. How to validate given VSP against only MUST VNFREQS by using https://github.com/onap/vvp-validation-scripts<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_onap_vvp-2Dvalidation-2Dscripts&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=g9LhwjMTPM4AuoWvYyDmqA&m=X4a0XtgdvP5oIyxMXai6-iRAyDMRQnPHPkOjP0zP0so&s=xnL_aohPfyvDzBrOb3eDAjhbE9zNV98FEg9kkY7iObI&e=> ? [Trevor] The validation scripts already only validate MUST and MUST NOT requirements – not SHOULD and MAY. If you’re suggesting we need a way to filter further on testing just MUST and not MUST NOTS, please elaborate on why you think that is necessary and provide an example. All the tests in the validation-scripts section must execute and pass for Heat to be considered valid. [Kanag] OK. b. How to validate given VSP for a given set of VNFREQS by using these validation scripts NOTE: Here Assuming VVP scripts supports VSP as whole. [Trevor] Please provide an example or use case of what you mean here. The validation-scripts only validate the Heat requirements. The validations already bypass checks that are not necessary based on conditions. Why would you need further filtering of the execution? [Kanag] I was under assumption that we need to validate the VNF package w.r.t for the given set of VNFREQS to satisfy the CVC. But in this week VNFSDK meeting, it was decided to go with all those released checks as part of Casablanca. So this use case may not be required now. 3. What are the versions of OpenStack HOT template (heat_template_version), VVP scripts support? a. [Trevor] There are no restrictions placed on heat_template_version in the validation-scripts directly as the version of Open Stack is somewhat operator-specific. The validation-scripts support versions as early as 2013-05-23, but we haven’t extensively tested it against all versions. [Kanag] OK. I don’t have any issue with you all writing a wrapper. You can either take the output files we documented and convert the results to a compatible format or we can add an additional output format for consumption by Dovetail/OVP. If you would like us to produce an additional output format, then I would need some additional detail on the format described in your email. It’s relatively trivial to create a new output format. [Kanag] I tried to integrate the VVP scripts in VTP and followings are the responses: Success case: {"results" : [{}], "build_tag" : “build-1”, "criteria" : "PASS" } Failure case: it capture results as below by pulling details from the output/failures file by using entries vnfreqs, file (template) and messag.? {"results" : [ { "vnfreqs" : "[]", “file” : “xxx”, “message”: “xxx” }, { "vnfreqs" : "[]", “file” : “xxx”, “message”: “xxx” } ], "build_tag" : “build-1”, "criteria" : "FAILED" } Pls let me know if any changes required. Also I have attached the sample for reference I have made following observations while testing: 1. on consecutive execution of test cases, it appends the existing output/failures file. Is there a way to overwrite this file instead of appending? 2. Is there a way to set the output path while running the test case, currently the output folder is generated directly under the ice_validator folder. This would help to run test cases simultaneously from VTP or to run test cases for different HOT templates · In “results”, is there one entry per validation? [Kanag] yes. Similar to the output/failures file format. · In “results”, what does it look like if there’s a failure? [Kanag] as given in attachment · In “results”, are there any restrictions on length or formatting of the error messages? Some of the VVP information is multi-line. [Kanag] I think if the failures json file is proper json format, it should not be any issue, VTP will handle it smoothly · What does “build_tag” represent? [Kanag] Dovetail set this when it triggers the test cases on VTP. It can be ignored from the test case output as VTP takes care of it. · What does “criteria” refer to? Is it the overall result of the validation? If so, what are the valid values? [Kanag] Yes, PASS – testcase passed and no failures . otherwise it is FAILED. In VVP case. Thanks, Trevor From: Kanagaraj Manickam [mailto:[email protected]] Sent: Wednesday, January 16, 2019 3:10 AM To: xudan (N) <[email protected]<mailto:[email protected]>>; LOVETT, TREVOR J <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Cc: STARK, STEVEN <[email protected]<mailto:[email protected]>>; WRIGHT, STEVEN A <[email protected]<mailto:[email protected]>>; HALLAHAN, RYAN <[email protected]<mailto:[email protected]>>; WEINSTOCK, ALAN M <[email protected]<mailto:[email protected]>>; Gaoweitao (Victor, Cloudify Network OSDT) <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: RE: [opnfv-tech-discuss] [Dovetail] Example VVP Output per Action Item from Developer Event Hi Trevor, During Casablanca release, In VTP, we introduced test cases for validating the give VNF CSAR file weather it complaint to ETSI SOL004 and it produce the results with following details: 1. Passed/failed 2. Error details. Current result format is: { "results" : [ { "error" : "SUCCESS" } ], "build_tag" : “CVC”, "criteria" : "PASS" } } So this would help to verify weather given CSAR is compliant and if not, to let user knows the errors in the CSAR. Also we are enhancing this result with following details as its required for CVC 1. VNF is TOSCA or HOT based 2. VNF Template version I believe similar approach could be followed for validating the HOT VNF as well. In this aspect, could you please help to find answers for following queries, which would help to integrate VVP scripts in VTP as validation test case, as discussed in the PARIS CVC meeting last week: 1. As we knew, ONAP Vendor Software Product (VSP) is zip containing the required HOT template with MANIFEST.json, Is VVP script validate this VSP as a whole including this MANIFEST.json or it only validates the HOT templates inside the VSP ? 2. As VNFREQS defines list of HOT template requirements with MUST and MUST NOT criteria, a. How to validate given VSP against only MUST VNFREQS by using https://github.com/onap/vvp-validation-scripts<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_onap_vvp-2Dvalidation-2Dscripts&d=DwMFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=g9LhwjMTPM4AuoWvYyDmqA&m=X4a0XtgdvP5oIyxMXai6-iRAyDMRQnPHPkOjP0zP0so&s=xnL_aohPfyvDzBrOb3eDAjhbE9zNV98FEg9kkY7iObI&e=> ? b. How to validate given VSP for a given set of VNFREQS by using these validation scripts NOTE: Here Assuming VVP scripts supports VSP as whole. 3. What are the versions of OpenStack HOT template (heat_template_version), VVP scripts support? I assume that we need to introduce an wrapper test case over VVP scripts, which will run the VVP scripts and produce the result in the form as required by the OVP portal/ Dovetail. Pls let me know your inputs. Thank you. Regards Kanagaraj Manickam Senior System Architect P&S ONAP Huawei Technologies India Pvt. Ltd. Survey No. 37, Next to EPIP Area, Kundalahalli, Whitefield Bengaluru-560066, Karnataka Tel: + 91-80-49160700 ext 72410 Mob: 9945602938 [Company_logo] ________________________________ This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! From: xudan (N) Sent: 16 January 2019 12:14 To: LOVETT, TREVOR J <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Cc: STARK, STEVEN <[email protected]<mailto:[email protected]>>; WRIGHT, STEVEN A <[email protected]<mailto:[email protected]>>; HALLAHAN, RYAN <[email protected]<mailto:[email protected]>>; WEINSTOCK, ALAN M <[email protected]<mailto:[email protected]>>; Gaoweitao (Victor, Cloudify Network OSDT) <[email protected]<mailto:[email protected]>>; Kanagaraj Manickam <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]>; [email protected]<mailto:[email protected]> Subject: RE: [opnfv-tech-discuss] [Dovetail] Example VVP Output per Action Item from Developer Event Hi all, The results looks very friendly for users to debugging. But it will be better if there is an explicit value to show if it PASS or FAIL. The results seems to be running one or several test cases against several VNFs, but it’s difficult to find out the test case details (how many test cases there are and the results of each of them). When doing the compliance tests, it always tests a set of chosen test cases against one single VNF. And it should report that which test cases PASS and which FAIL (better with the failure reason for debugging). Please take these under consideration when VTP integrates these test cases. BR, Dan Xu From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of LOVETT, TREVOR J Sent: Friday, January 11, 2019 12:18 AM To: [email protected]<mailto:[email protected]> Cc: STARK, STEVEN <[email protected]<mailto:[email protected]>>; WRIGHT, STEVEN A <[email protected]<mailto:[email protected]>>; HALLAHAN, RYAN <[email protected]<mailto:[email protected]>>; WEINSTOCK, ALAN M <[email protected]<mailto:[email protected]>>; Gaoweitao (Victor, Cloudify Network OSDT) <[email protected]<mailto:[email protected]>> Subject: [opnfv-tech-discuss] [Dovetail] Example VVP Output per Action Item from Developer Event Per an action item out of the developer event in France, the ONAP VVP team is providing a sample of what the default output of the vvp/validation-scripts produce. This can be examined by the Dovetail team to determine if it can be consumed for the purpose of the LFN certification of Heat templates. The scripts can produce several different output formats, but I’m using the CSV output format as it’s the most machine readable. There are also HTML and Excel output formats. It’s fairly trivial to add additional output formats so if there’s an existing format that dovetail can consume, please provide the details and we can create that format if needed. The sample_template_with_errors.zip contains the Heat template files that produced these reports. Upon completion of a validation run, several files will be written to the outputs folder. The two key files are: · failures – This file contains information on the failures in JSON format. This file will not be present if no requirement violations are detected. · report.csv – Similar content to failures, but in CSV format and it also includes some additional data in a header section. This file will be present even if violations are not detected, but it will not have rows error section of the report. I’ve attached samples of both files in this email. I’ve also attached a copy of what the report.csv looks like when there are no errors (see report – SUCCESS.csv) The format of the files is fairly straight forward, but here’s some additional documentation of the fields in the file: report.csv file · Header – The first 9 rows of the csv file represent a header section that provides the following information o Row 1 – Static report header: “Validation Failures” o Row 2 - Blank o Row 3 – Profile Selected: Always ONAP o Row 4 – Tool Version: Semantic version ID Of the tool that produced the report o Row 5 – Report Generated At: Date and Time Stamp of when the report was generated (ex: 2018-12-18 12:34:19.064412 Eastern Standard Time) o Row 6 – Directory Validated: Shows the local, absolute directory that was scanned o Row 7 – Checksum: MD5 hash of all files in the directory that was scanned o Row 8 – Total Errors: The count of all errors encountered o Row 9 – Blank · Collection Failures – In the unlikely event there is setting up the test suite to run, then an optional section of the report will be written describing these errors. There will be 2 header rows, and then 1 row for each setup failure encountered. It is possible to have both collection failures and validation failures as the validation-scripts will execute any tests that did not fail setup. However, if this section exists it represents an unexpected error and invalid run of the tool. These issues should be referred to the VVP team for investigation. o Row 10 (if these errors are encountered): Start of Collection Failures will be denoted by a row containing “WARNING: The following unexpected errors…” in the first column of the row. o Row 11 (if these errors are encountered): Collection Failure heading columns: “Validation File”, “Test”, “Fixtures”, “Error” o Row 12-N (if these errors are encountered): Rows for each error. Each column is a string. · Validation Failures (Note the start row depends on whether or not collection failures are encountered) o Start of report – Static Report Header denoting the start of the individual errors. Always “Validation Failures” o Error Report Column Headings – “Input File”, “Test”, “Requirements”, “Resolution Steps”, “Error Message”, “Raw Test Output” o Following the heading - 0 or more individual error rows for each violation found Column Description Format Required/Optional Notes Input File List of files the error was detected in Semicolon delimited list Required Test Name of python module that detected the error String Required Requirements The full requirement text that the test validated. There can be multiple, related requirements listed here Multi-line String Optional Almost all requirements are mapped to 1 or more requirements, but there may be a few that are not yet mapped Resolution Steps Optional information to aid in resolution Multi-line String Optional Consider this deprecated Error Message Detailed error message produced by the validation test. This should provide specific information on why the test failed Multi-line String Required Raw Test Output Full output from pytest. This includes the test content itself and detailed error message Multi-line String Required failures file This file is similar to the report file, but in JSON format. There are a few notable differences beyond CSV vs. JSON: · The full requirement text is not available in this file – only the requirement IDs themselves. · Resolution Steps is also not available in this file · Collection failures are not included in this report The following mapping shows how the JSON elements map to the column above. - file -> Input File: Format is a JSON list - vnfrqts -> N/A: Format is a JSON list of the requirement IDs associated with the test - test -> N/A: Name of the test method - test_file -> test: Name of the python file containing the test - raw_output -> Raw Test Output - message -> Error Message Thanks, Trevor
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#22711): https://lists.opnfv.org/g/opnfv-tech-discuss/message/22711 Mute This Topic: https://lists.opnfv.org/mt/29006268/21656 Group Owner: [email protected] Unsubscribe: https://lists.opnfv.org/g/opnfv-tech-discuss/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
