Thanks Dan Xu for detailed response. I changed the pod.xml as u suggested and ran the ha test suite. I also enabled debug option while running the tests and noticed the outage times getting logged in the dovetail.log file.
Thanks Srikanth From: xudan (N) [mailto:[email protected]] Sent: Thursday, July 06, 2017 11:55 PM To: Srikanth Vavilapalli <[email protected]>; [email protected] Subject: Re: [opnfv-tech-discuss] [Dovetail] sdnvpn test cases with dovetail.cvp.0.1.0 throw an error Hi Srikanth, 1. About the pod.yaml file 1. Think it’s wrong. Switch to user “stack” first, then source under cloud rc file(named stackrc in my env), command “openstack server list” will show the platform nodes, mine is +---------------------------------------------------+------------------+-----------+-------------------------+-------------------+ | ID | Name | Status | Networks | Image Name | +---------------------------------------------------+------------------+-----------+-------------------------+-------------------+ | 8e4380ea-b348-49b7-a5e4-39dc18e8994b | compute-0 | ACTIVE | ctlplane=192.0.2.15 | overcloud-full | | 49a5f4f6-6b84-4ce3-9a69-654618511dfb | controller-2 | ACTIVE | ctlplane=192.0.2.10 | overcloud-full | | adcef3e3-f456-4c01-987a-1d72182b3f00 | compute-1 | ACTIVE | ctlplane=192.0.2.17 | overcloud-full | | f41552b9-a4e4-4445-bd07-e8b025bec9b6 | controller-0 | ACTIVE | ctlplane=192.0.2.14 | overcloud-full | | f622a698-5fa0-41ff-a0e9-566e3a5e45b7 | controller-1 | ACTIVE | ctlplane=192.0.2.13 | overcloud-full | +---------------------------------------------------+------------------+-----------+-------------------------+-------------------+ 1. Set the ${DOVETAIL_HOME}/pre_config/pod.yaml as follows nodes: - name: node1 role: Controller ip: 192.0.2.14 ——> node IP user: heat-admin ———> node login user key_filename: /root/.ssh/id_rsa ——> ssh key file - name: node2 role: Controller ip: 192.0.2.13 user: heat-admin key_filename: /root/.ssh/id_rsa - name: node3 role: Compute ip: 192.0.2.15 user: heat-admin key_filename: /root/.ssh/id_rsa The user “heat-admin” above is obtained from the installation guide document. You can check it according to your platform installation documents. I can “ssh [email protected]<mailto:[email protected]>” to verify that. If deployed in HA mode, since only process HA (no node HA) are tested for the current tests, you can only write one Controller “node1” information in the pod.yaml , it will work, i.e., if I delete node2 node3 info in the above file, it will also work. 1. Copy the user “stack” private key (the path usually is /home/stack/.ssh/id_rsa) to $DOVETAIL_HOME/pre_config/id_rsa 1. About the results data 1. In the result file, if the value of “sla_pass” is 1, the test passes, otherwise fails. Under linux vim env, suggest to use “:$!python -m json.tool” to transfer the result file into json to see more clearly. 1. For the outage time, it is not shown in the dovetail.ha.tc***.out file now. If you run “dovetail run --testsuite proposed_tests --testarea ha -d” (“-d” added for showing debug logs), you can find in results/dovetail.log, such as 2017-07-06 20:17:22,211 - container.Container - DEBUG - 2017-07-06 20:17:22,211 yardstick.benchmark.scenarios.availability.monitor.basemonitor basemonitor.py:155 DEBUG the monitor result:{'total_time': 10.886008977890015, 'outage_time': 1.1303958892822266, 'outage_count': 1, 'last_outage': 1499372232.053993, 'first_outage': 1499372230.923597, 'total_count': 8} Regards, Dan Xu 发件人: Srikanth Vavilapalli [mailto:[email protected]] 发送时间: 2017年7月7日 2:48 收件人: Srikanth Vavilapalli; xudan (N); [email protected]<mailto:[email protected]> 主题: RE: [opnfv-tech-discuss] [Dovetail] sdnvpn test cases with dovetail.cvp.0.1.0 throw an error Hi Just continuing my dovetail testing with HA test suite and have few questions: 1. I created the pod.yaml file as shown below. Are these correct settings for an apex based OPNFV deployment? root@r720-003 ~/dovetail/results $ cat ../pre_config/pod.yaml nodes: - name: node1 role: Controller ip: 192.168.122.140 <----- apex undercloud IP user: stack <----- apex undercloud login user key_filename: /root/.ssh/id_rsa <----- apex undercloud ssh key file 1. The test run has generated output files for each test case. What is the way to interpret this output? Which fields in that log indicate the outage time and recovery time? I expecting some outage for these tests because my backend opnfv deployment is running in non-HA mode. root@r720-003 ~/dovetail/results $ cat dovetail.ha.tc001.out {"status": 1, "result": [{"context_cfg": {"nodes": {"node1": {"ip": "192.168.122.140", "key_filename": "/root/.ssh/id_rsa", "role": "Controller", "name": "node1.LF-785e43fe", "user": "stack"}}}, "scenario_cfg": {"task_id": "785e43fe-9ef0-4a0f-a6e2-f813a468915f", "runner": {"object": "yardstick.benchmark.scenarios.availability.serviceha.ServiceHA", "type": "Iteration", "output_filename": "/home/opnfv/yardstick/results/dovetail.ha.tc001.out", "iterations": 1, "runner_id": 55}, "tc": "opnfv_yardstick_tc019", "options": {"wait_time": 10, "attackers": [{"fault_type": "kill-process", "host": "node1", "process_name": "nova-api"}], "monitors": [{"monitor_number": 3, "monitor_time": 10, "command_name": "openstack server list", "monitor_type": "openstack-cmd", "sla": {"max_outage_time": 5}}, {"process_name": "nova-api", "monitor_type": "process", "monitor_number": 3, "host": "node1", "monitor_time": 20, "sla": {"max_recover_time": 20}}]}, "nodes": {"node1": "node1.LF-785e43fe"}, "type": "ServiceHA", "sla": {"outage_time": 5, "action": "monitor"}}, "runner_id": 55}, {"benchmark": {"timestamp": 1499367559.269849, "errors": "", "data": {"sla_pass": 1}, "sequence": 1}, "runner_id": 55}]} root@r720-003 ~/dovetail/results $ cat dovetail.ha.tc002.out {"status": 1, "result": [{"context_cfg": {"nodes": {"node1": {"ip": "192.168.122.140", "key_filename": "/root/.ssh/id_rsa", "role": "Controller", "name": "node1.LF-1271a70f", "user": "stack"}}}, "scenario_cfg": {"task_id": "1271a70f-aa8a-44a7-83b9-f59705000483", "runner": {"duration": 1, "object": "yardstick.benchmark.scenarios.availability.serviceha.ServiceHA", "type": "Duration", "output_filename": "/home/opnfv/yardstick/results/dovetail.ha.tc002.out", "runner_id": 55}, "tc": "opnfv_yardstick_tc045", "options": {"attackers": [{"fault_type": "kill-process", "host": "node1", "process_name": "neutron-server"}], "monitors": [{"monitor_number": 3, "monitor_time": 10, "command_name": "openstack router list", "monitor_type": "openstack-cmd", "sla": {"max_outage_time": 5}}, {"process_name": "neutron-server", "monitor_type": "process", "monitor_number": 3, "host": "node1", "monitor_time": 20, "sla": {"max_recover_time": 20}}]}, "nodes": {"node1": "node1.LF-1271a70f"}, "type": "ServiceHA", "sla": {"outage_time": 5, "action": "monitor"}}, "runner_id": 55}, {"benchmark": {"timestamp": 1499367597.087087, "errors": "", "data": {"sla_pass": 1}, "sequence": 1}, "runner_id": 55}]} Thanks for your help. Thanks Srikanth From: [email protected]<mailto:[email protected]> [mailto:[email protected]] On Behalf Of Srikanth Vavilapalli Sent: Wednesday, July 05, 2017 4:34 PM To: xudan (N) <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: Re: [opnfv-tech-discuss] [Dovetail] sdnvpn test cases with dovetail.cvp.0.1.0 throw an error Thanks Dan Xu. I pulled cvp.0.2.0 version of dovetail and was able to verify the bgpvpn tests with this version. Thanks Srikanth From: xudan (N) [mailto:[email protected]] Sent: Wednesday, July 05, 2017 12:51 AM To: Srikanth Vavilapalli <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Cc: Jose Lausuch <[email protected]<mailto:[email protected]>> Subject: Re: [opnfv-tech-discuss] [Dovetail] sdnvpn test cases with dovetail.cvp.0.1.0 throw an error Hi Srikanth, The Functest WARNING logs you have mentioned can all be ignored, because they don’t have any matter with the test case itself. Dovetail passes “INSTALLER_TYPE=netvirt” and “DEPLOY_SCENARIO=bgpvpn” to Functest Container to ensure test case bgpvpn can be triggered. However, “INSTALLER_IP” has nothing to do with the Functest test cases themselves. So Dovetail doesn’t pass it to Functest. About the ERROR log “Unknown test case or tier 'bgpvpn', or not supported by the given scenario 'bgpvpn'”, we also met this problem during the test and that’s because opnfv/functest:danube.2.0 used by dovetail:cvp.0.1.0 disables bgpvpn with the config file testcases.yaml. Functest has tagged opnfv/functest:cvp.0.2.0 which has enabled bgpvpn and fixed some bugs about it. Following are related patches: https://gerrit.opnfv.org/gerrit/#/c/35051/2 https://gerrit.opnfv.org/gerrit/#/c/36777/3 https://gerrit.opnfv.org/gerrit/#/c/36785/ Dovetail cvp.0.2.0 will be available soon which will use opnfv/functest:cvp.0.2.0 instead of opnfv/functest:danube.2.0. Then you can have a try with this new version. You can go to the wiki page for the details about the version change logs: https://wiki.opnfv.org/display/dovetail/Running+history+for+the+dovetail+tool#Runninghistoryforthedovetailtool-5.Versionchangelogs Please feel free to contact me if you have any questions. Thanks, Dan Xu 发件人: Srikanth Vavilapalli [mailto:[email protected]] 发送时间: 2017年7月5日 11:31 收件人: Jose Lausuch 抄送: xudan (N); [email protected]<mailto:[email protected]> 主题: RE: [opnfv-tech-discuss] [Dovetail] sdnvpn test cases with dovetail.cvp.0.1.0 throw an error Hi Jose I have already tried setting these environment variables while launching dovetail docker container (as shown below), even then I have noticed these warning messages from functest. So looks like these env variables are not passed on to functest container. docker run --privileged=true -it -e DOVETAIL_HOME=$DOVETAIL_HOME -e INSTALLER_TYPE=apex -e INSTALLER_IP=192.168.37.11 -v $DOVETAIL_HOME:$DOVETAIL_HOME -v /var/run/docker.sock:/var/run/docker.sock opnfv/dovetail:cvp.0.1.0 /bin/bash Thanks Srikanth From: Jose Lausuch Sent: Monday, July 03, 2017 2:14 PM To: Srikanth Vavilapalli <[email protected]<mailto:[email protected]>> Cc: xudan (N) <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: Re: [opnfv-tech-discuss] [Dovetail] sdnvpn test cases with dovetail.cvp.0.1.0 throw an error Srikanth, You should set INSTALLER_TYPE=apex and also set INSTALLER_IP to the undercloud VM. You also need to set the DEPLOY_SCENARIO to the corresponding scenario you are deploying. If it’s a bgpvpn deployment, you can set it to os-odl_l2-bgpvpn-ha. That is at least from Functest side, not sure how Dovetail passes those env variables to Functest. Let me know if that fixes your problem. Regards, Jose On 3 Jul 2017, at 20:45, Srikanth Vavilapalli <[email protected]<mailto:[email protected]>> wrote: Thanks Dan Xu After fixing the DOVETAIL_HOME issue, I see the below WARNING logs in functest log, when I ran `dovetail run --testsuite proposed_tests --testarea sdnvpn`. Can I ignore these warning messages? I tried to set these environment variables while launching the dovetail container and also in env_config.sh file, but still I see those errors. 2017-07-03 18:49:09,172 - functest.utils.functest_utils - DEBUG - Executing command: 'python /home/opnfv/repos/functest/functest/ci/prepare_env.py start' 2017-07-03 18:49:10,232 - functest.ci.prepare_env - INFO - ######### Preparing Functest environment ######### 2017-07-03 18:49:10,232 - functest.ci.prepare_env - INFO - ============================================== 2017-07-03 18:49:10,232 - functest.ci.prepare_env - INFO - Checking environment variables... 2017-07-03 18:49:10,232 - functest.ci.prepare_env - WARNING - INSTALLER_TYPE=netvirt is not a valid OPNFV installer. Available OPNFV Installers are : ['apex', 'fuel', 'compass', 'joid', 'daisy']. Setting INSTALLER_TYPE=undefined. 2017-07-03 18:49:10,233 - functest.ci.prepare_env - WARNING - The env variable 'INSTALLER_IP' is not defined. It is needed to fetch the OpenStack credentials. If the credentials are not provided to the container as a volume, please add this env variable to the 'docker run' command. 2017-07-03 18:49:10,233 - functest.ci.prepare_env - INFO - DEPLOY_SCENARIO=bgpvpn 2017-07-03 18:49:10,233 - functest.ci.prepare_env - INFO - CI_DEBUG=true 2017-07-03 18:49:10,233 - functest.ci.prepare_env - INFO - NODE_NAME=master 2017-07-03 18:49:10,233 - functest.ci.prepare_env - INFO - BUILD_TAG=daily-master-a279734f-3d6f-4e7b-96f7-7e8e192471aa-dovetail.sdnvpn.tc001 2017-07-03 18:49:10,233 - functest.ci.prepare_env - INFO - IS_CI_RUN=True Also I see that the tests are ending with a failure log saying “Unknown test case or tier 'bgpvpn', or not supported by the given scenario 'bgpvpn'”. Attached the logs. Do I need to enable any specific config in my deployment for the sdnvpn testcases to pass? 2017-07-03 19:03:53,274 - functest.ci.run_tests - DEBUG - Sourcing the OpenStack RC file... 2017-07-03 19:03:53,275 - functest.ci.run_tests - ERROR - Unknown test case or tier 'bgpvpn', or not supported by the given scenario 'bgpvpn'. 2017-07-03 19:03:53,275 - functest.ci.run_tests - DEBUG - Available tiers are: Thanks for your help… Thanks Srikanth From: xudan (N) [mailto:[email protected]] Sent: Friday, June 30, 2017 9:17 PM To: Srikanth Vavilapalli <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: Re: [opnfv-tech-discuss] [Dovetail] sdnvpn test cases with dovetail.cvp.0.1.0 throw an error Hi Srikanth, The first step: Added ‘export DOVETAIL_HOME=$HOME/dovetail’ in ~/dovetail/pre_config/env_config.sh You need to replace $HOME with /root. Because in your host, $HOME is /root. But in Dovetail container $HOME is /home/opnfv. Dovetail tool will source this env_config.sh at the running time in the container and then it will reset the DOVETAIL_HOME to be /home/opnfv/dovetail. So then Dovetail tool will go to /home/opnfv/dovetail to look for all the files rather than /root/dovetail. Sorry for the mistake in Dovetail testing user guide and we will amend it soon. Thank you for your feedback. Thanks Dan Xu 发件人: Srikanth Vavilapalli [mailto:[email protected]] 发送时间: 2017年7月1日 1:58 收件人: xudan (N); [email protected]<mailto:[email protected]> 主题: RE: [opnfv-tech-discuss] [Dovetail] sdnvpn test cases with dovetail.cvp.0.1.0 throw an error Hi Dan Xu Here is the command I have used to launch the dovetail container * root@r720-003 ~ $ Added ‘export DOVETAIL_HOME=$HOME/dovetail’ in ~/dovetail/pre_config/env_config.sh * root@r720-003 ~ $ source $HOME/dovetail/pre_config/env_config.sh * root@r720-003 ~ $ sudo docker run --privileged=true -it -e DOVETAIL_HOME=$DOVETAIL_HOME -v $DOVETAIL_HOME:$DOVETAIL_HOME -v /var/run/docker.sock:/var/run/docker.sock opnfv/dovetail:cvp.0.1.0 /bin/bash I followed the instructions at http://artifacts.opnfv.org/dovetail/review/34285/testing_user_userguide/index.html to bringup the dovetail container. Thanks Srikanth From: xudan (N) [mailto:[email protected]] Sent: Thursday, June 29, 2017 6:29 PM To: Srikanth Vavilapalli <[email protected]<mailto:[email protected]>>; [email protected]<mailto:[email protected]> Subject: Re: [opnfv-tech-discuss] [Dovetail] sdnvpn test cases with dovetail.cvp.0.1.0 throw an error Hi Srikanth, We have tested dovetail:cvp.0.1.0 locally and have not found this ERROR. Maybe there are something wrong when you create the Dovetail container. Can you show us the command. According to the log, the right command for you is: sudo docker run -it --privileged=true -e DOVETAIL_HOME=/root/dovetail -v /root/dovetail: /root/dovetail -v /var/run/docker.sock:/var/run/docker.sock opnfv/dovetail:cvp.0.1.0 /bin/bash Thanks Dan Xu 发件人: [email protected]<mailto:[email protected]> [mailto:[email protected]] 代表 Srikanth Vavilapalli 发送时间: 2017年6月30日 7:03 收件人: [email protected]<mailto:[email protected]> 主题: [opnfv-tech-discuss] [Dovetail] sdnvpn test cases with dovetail.cvp.0.1.0 throw an error Hi When I run the `sdnvpn’ dovetail test cases, I see the below error for each run. I have ensured that the ubuntu cloudimg is present under $(DOVETAIL_HOME)/pre_config folder on the host before executing these tests. The failure is happening when the dovetail container is trying to copy the file from one location to other location in the functest container & this folder /home/opnfv/userconfig/pre_config/ does not exist in the functest container. Can any one plz help me with this error? root@98da04f3fb76:/usr/local/lib/python2.7/dist-packages/dovetail#<mailto:root@98da04f3fb76:/usr/local/lib/python2.7/dist-packages/dovetail> dovetail run --testsuite proposed_tests --testarea sdnvpn 2017-06-29 23:38:07,875 - run - INFO - ================================================ 2017-06-29 23:38:07,875 - run - INFO - Dovetail compliance: proposed_tests! 2017-06-29 23:38:07,875 - run - INFO - ================================================ 2017-06-29 23:38:07,875 - run - INFO - Build tag: daily-master-350ba85a-b2bf-4693-a107-ca25af60b925 2017-06-29 23:38:08,109 - run - INFO - >>[testcase]: dovetail.sdnvpn.tc001 2017-06-29 23:38:09,428 - container.Container - INFO - get hosts info 2017-06-29 23:38:09,947 - container.Container - ERROR - The command 'sudo docker exec 4e7f920152d3 /bin/bash -c "cp /home/opnfv/userconfig/pre_config/ubuntu-16.04-server-cloudimg-amd64-disk1.img /home/opnfv/functest/images"' failed. 2017-06-29 23:38:10,010 - container.Container - ERROR - The command 'sudo docker exec 4e7f920152d3 /bin/bash -c "cp /home/opnfv/userconfig/pre_config/sdnvpn_config_testcase1.yaml /home/opnfv/repos/sdnvpn/sdnvpn/test/functest/config.yaml"' failed. 2017-06-29 23:39:19,314 - run - ERROR - Fail to store results with file /root/dovetail/results/functest_results.txt. 2017-06-29 23:39:19,315 - report.FunctestCrawler - INFO - result file not found: /root/dovetail/results/functest_results.txt 2017-06-29 23:39:19,315 - run - INFO - >>[testcase]: dovetail.sdnvpn.tc002 2017-06-29 23:39:19,317 - container.Container - INFO - get hosts info 2017-06-29 23:39:19,788 - container.Container - ERROR - The command 'sudo docker exec 25805da3d12b /bin/bash -c "cp /home/opnfv/userconfig/pre_config/ubuntu-16.04-server-cloudimg-amd64-disk1.img /home/opnfv/functest/images"' failed. …. root@98da04f3fb76:/usr/local/lib/python2.7/dist-packages/dovetail#<mailto:root@98da04f3fb76:/usr/local/lib/python2.7/dist-packages/dovetail> ls /root/dovetail/pre_config/ env_config.sh sdnvpn_config_testcase3.yaml hosts.yaml sdnvpn_config_testcase4.yaml pod.yaml.sample sdnvpn_config_testcase8.yaml sdnvpn_config_testcase1.yaml ubuntu-16.04-server-cloudimg-amd64-disk1.img sdnvpn_config_testcase2.yaml Thanks Srikanth <dovetail.log><functest.log>_______________________________________________ opnfv-tech-discuss mailing list [email protected]<mailto:[email protected]> https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
_______________________________________________ opnfv-tech-discuss mailing list [email protected] https://lists.opnfv.org/mailman/listinfo/opnfv-tech-discuss
