I put it here https://etherpad.openstack.org/gate-tempest-devstack-vm-quantum-full
Ala 2013/6/19 Miguel Lavalle <mig...@mlavalle.com> > No, we haven't put this document in etherpad. But it's a great idea. This > way other folks can jump in with ideas. I'll put it up in eteherpad over > the next few days > > > On Wed, Jun 19, 2013 at 10:10 AM, Sean Dague <s...@dague.net> wrote: > >> This is great stuff! You guys have this up in etherpad somewhere? >> >> Really great to see people organizing around this. I'd reviewed a bunch >> of Jordan's code, so I knew he was in the mix, sorry for not realizing >> there were other folks working on this as well. >> >> -Sean >> >> >> On 06/19/2013 10:27 AM, Miguel Lavalle wrote: >> >>> Hi, >>> >>> As of last week, we have organized a team to deal with the Quantum in >>> full gate issues. There are 3 people working on this: Jordan Pittier, >>> Ala.Rezmerita and myself. Below you will find a document describing all >>> the identified issues and who is assigned to fix them. Of cource, we can >>> use more hands / heads. If anyone wants to help or has identified other >>> issues not listed below, please get in touch with us. >>> >>> Regards >>> >>> Fix Jenkins gate-tempest-devstack-vm-**quantum-full >>> >>> >>> This document lists all the Tempest tests that are failing on a Devstack >>> + Quantum Trunk setup. The goal is to make these tests pass for the >>> Havana Milestone 3. Alla, Miguel and Jordan are working on this. >>> >>> >>> For each test : >>> >>> * >>> >>> Provide the full path of the test. If the test has a double >>> interface (JSON and XML), it’s enough to provide only the JSON path. >>> >>> * >>> >>> Provide a small stacktrace or a link to a Paste service such as >>> http://paste.openstack.org/ >>> >>> * >>> >>> Would be great to provide the CURL URL that triggers the bug >>> >>> * >>> >>> Provide a small analysis of what the bug may be and the components >>> involved (Quantum, Quantum API on Nova’s side, Nova-network etc.) >>> >>> * >>> >>> As far as possible, provide the file and the line number where the >>> Python exception is raised or translated (“casted”) >>> >>> * >>> >>> If already filed, an URL to the bug report in Launchpad >>> >>> >>> Tests related to the Fixed IPs Compute API extension (Miguel) >>> >>> >>> 1) >>> tempest.api.compute.admin.**test_fixed_ips:**FixedIPsTestJson.test_list_ >>> **fixed_ip_details >>> >>> * >>> >>> Trace: >>> http://paste.openstack.org/**show/38363/<http://paste.openstack.org/show/38363/> >>> >>> * >>> >>> curl-H "X-Auth-Token:$TOKEN" -X GET >>> http://$IP:8774/v2/$TENANT_ID/**os-fixed-ips/10.0.0.3 < >>> http://10.0.0.3> >>> >>> * >>> >>> API: >>> http://api.openstack.org/api-**ref.html#ext-os-fixed-ips<http://api.openstack.org/api-ref.html#ext-os-fixed-ips> >>> >>> >>> * >>> >>> Explanation: Call to >>> nova.api.openstack.compute.**contrib.fixed_ips.** >>> FixedIPController::show() >>> This file doesn’t use the Quantum API nor the Nova-Network API. It >>> interacts directly with the DB, which is bad. >>> >>> * >>> >>> A possible fix would be to : >>> >>> 1. >>> >>> >>> Change >>> nova.api.openstack.compute.**contrib.fixed_ips.** >>> FixedIPController::show() >>> to use either nova.network.api.API::get_**fixed_ip() (for Nova >>> Network) or nova.network.quantumv2.api.**API:get_fixed_ip() (for >>> Nova Network) >>> >>> 2. >>> >>> >>> Implement nova.network.quantumv2.api.**API:get_fixed_ip() which >>> currently raises a NotImplementedError exception >>> >>> 2) >>> tempest.api.compute.admin.**test_fixed_ips:**FixedIPsTestJson.test_set_* >>> *reserve >>> >>> * >>> >>> Trace: >>> http://paste.openstack.org/**show/38372/<http://paste.openstack.org/show/38372/> >>> >>> * >>> >>> curl-H "X-Auth-Token:$TOKEN" -X POST >>> http://$IP:8774/v2/$TENANT_ID/**os-fixed-ips/10.0.0.3/action >>> <http://10.0.0.3/action> >>> >>> * >>> >>> API: >>> http://api.openstack.org/api-**ref.html#ext-os-fixed-ips<http://api.openstack.org/api-ref.html#ext-os-fixed-ips> >>> >>> >>> * >>> >>> Possible Fix: Should call >>> nova.api.openstack.compute.**contrib.fixed_ips.** >>> FixedIPController::_set_**reserved() >>> (once the result of db.fixed_ip_get_by_address() is made through the >>> API). Here the problem is that neither Nova-Network nor Quatum API >>> implement an equivalent of db.fixed_ip_update() >>> >>> 3) >>> tempest.api.compute.admin.**test_fixed_ips:**FixedIPsTestJson.test_set_* >>> *unreserve >>> >>> * >>> >>> Same as 2) >>> >>> >>> >>> Tests related to Quotas Admin(Miguel) >>> >>> >>> 4)tempest.api.compute.admin.**test_quotas:**QuotasAdminTestJSON.test_** >>> security_groups_exceed_limit >>> >>> 5)tempest.api.compute.admin.**test_quotas:**QuotasAdminTestJSON.test_** >>> security_groups_rules_exceed_**limit >>> >>> >>> Tests related to Floating IPs >>> >>> >>> 6)tempest.api.compute.**floating_ips.test_floating_**ips_actions:** >>> FloatingIPsTestJSON.test_**associate_ip_to_server_** >>> without_passing_floating_ip >>> >>> * >>> >>> Trace Tempest >>> http://paste.openstack.org/**show/38431/<http://paste.openstack.org/show/38431/> >>> >>> * >>> >>> Bug : >>> https://bugs.launchpad.net/**quantum/+bug/1190242<https://bugs.launchpad.net/quantum/+bug/1190242> >>> >>> * >>> >>> curl-H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X >>> POST http://$IP:8774/v2/$TENANT_ID/**servers/$SERVER_ID/action >>> <http://10.1.59.157:8774/v2/**de6c5fbc55a34cbcaa3d79eb6b21a7** >>> 84/servers/0b2ad3b6-c14a-4d89-**b2a0-c015f0a88a1f/action<http://10.1.59.157:8774/v2/de6c5fbc55a34cbcaa3d79eb6b21a784/servers/0b2ad3b6-c14a-4d89-b2a0-c015f0a88a1f/action> >>> >-d >>> ‘{"addFloatingIp": {"address": ""}}’ >>> >>> >>> * >>> >>> Notice that the address of the FloatingIP is empty. >>> >>> * >>> >>> API : >>> http://api.openstack.org/api-**ref.html#ext-floating-ips<http://api.openstack.org/api-ref.html#ext-floating-ips> >>> >>> >>> * >>> >>> Explanation : Mismatch in the 2 API. If there is only one IP in the >>> pool, Quantum allows the floating IP to be empty and returns the one >>> and only IP in the pool. Nova-network doesn’t allow this and returns >>> a 404 >>> >>> * >>> >>> Review >>> https://review.openstack.org/#**/c/32740/<https://review.openstack.org/#/c/32740/> >>> >>> >>> 7)tempest.api.compute.**floating_ips.test_floating_**ips_actions:** >>> FloatingIPsTestJSON.test_**delete_floating_ip >>> >>> * >>> >>> Trace Tempest : >>> http://paste.openstack.org/**show/38505/<http://paste.openstack.org/show/38505/> >>> >>> * >>> >>> Bug : >>> https://bugs.launchpad.net/**tempest/+bug/1160309(see<https://bugs.launchpad.net/tempest/+bug/1160309(see>comment >>> #2) >>> >>> >>> * >>> >>> This is related to bug 9) >>> >>> >>> 8)tempest.api.compute.**floating_ips.test_floating_**ips_actions:** >>> FloatingIPsTestJSON.test_**delete_nonexistant_floating_ip >>> >>> >>> * >>> >>> Related to 9) >>> >>> >>> 9)tempest.api.compute.**floating_ips.test_list_**floating_ips:** >>> FloatingIPDetailsTestJSON.**test_get_nonexistant_floating_**ip_details >>> >>> * >>> >>> Bug >>> :https://bugs.launchpad.net/**tempest/+bug/1160309<https://bugs.launchpad.net/tempest/+bug/1160309> >>> >>> * >>> >>> Trace >>> tempest:http://paste.**openstack.org/show/38430/<http://paste.openstack.org/show/38430/> >>> >>> * >>> >>> Trace nova: >>> http://paste.openstack.org/**show/38433/<http://paste.openstack.org/show/38433/> >>> >>> >>> * >>> >>> Curl:curl -H "X-Auth-Token:$TOKEN" -X GET >>> http://$IP:8774/v2/$PROJECT_**ID/os-floating-ips/**99987878787878 >>> >>> Proposed Fix: >>> https://review.openstack.org/**33024<https://review.openstack.org/33024> >>> >>> >>> Tests Related to Security Groups (Jordan) >>> >>> >>> 10)tempest.api.compute.**security_groups.test_security_**group_rules:** >>> SecurityGroupRulesTestJSON.**test_security_group_rules_** >>> create_with_invalid_id >>> >>> * >>> >>> TRACE: >>> http://paste.openstack.org/**show/38373/<http://paste.openstack.org/show/38373/> >>> >>> * >>> >>> curl-H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X >>> >>> POST http://$IP:8774/v2/$TENANT_ID/**os-security-group-rules -d >>> ‘{"security_group_rule": {"from_port": 22, "ip_protocol": "tcp", >>> "to_port": 22, "parent_group_id": "9991393497170", "cidr": null, >>> "group_id": null}}’ >>> >>> * >>> >>> Explanation: Notice that the parent_goup_id is a numerical ID and >>> not a UUID. Quantum has an additional check to validate that the ID >>> is an UUID (see >>> nova/network/security_group/**quantum_driver.py::validate_**id()) >>> >>> * >>> >>> API: http://api.openstack.org/api-**ref.html#ext-os-security-** >>> groups <http://api.openstack.org/api-ref.html#ext-os-security-groups> >>> >>> * >>> >>> Bug: >>> https://bugs.launchpad.net/**tempest/+bug/1182384<https://bugs.launchpad.net/tempest/+bug/1182384> >>> >>> >>> * >>> >>> Review: >>> https://review.openstack.org/#**/c/29899/<https://review.openstack.org/#/c/29899/> >>> >>> >>> 11)tempest.api.compute.**security_groups.test_security_**group_rules:** >>> SecurityGroupRulesTestJSON.**test_security_group_rules_** >>> delete_with_invalid_id >>> >>> * >>> >>> >>> http://paste.openstack.org/**show/38424/<http://paste.openstack.org/show/38424/> >>> >>> >>> * >>> >>> curl -H "X-Auth-Token:$TOKEN" -X DELETE >>> http://$IP:8774/v2$TENANT_ID/**os-security-group-rules/** >>> 9991407551273 >>> >>> * >>> >>> Explanation : Same bug as 10 >>> >>> * >>> >>> API: http://api.openstack.org/api-**ref.html#ext-os-security-** >>> groups <http://api.openstack.org/api-ref.html#ext-os-security-groups> >>> >>> >>> * >>> >>> Review: >>> https://review.openstack.org/#**/c/29899/<https://review.openstack.org/#/c/29899/> >>> >>> 12)tempest.api.compute.**security_groups.test_security_** >>> groups:SecurityGroupsTestJSON.**test_delete_nonexistant_**security_group >>> >>> * >>> >>> Same as bug 10 >>> >>> >>> * >>> >>> Review: >>> https://review.openstack.org/#**/c/29899/<https://review.openstack.org/#/c/29899/> >>> >>> >>> 13)tempest.api.compute.**security_groups.test_security_** >>> groups:SecurityGroupsTestJSON.**test_security_group_get_** >>> nonexistant_group >>> >>> * >>> >>> Same as bug 10 >>> >>> >>> * >>> >>> Review: >>> https://review.openstack.org/#**/c/29899/<https://review.openstack.org/#/c/29899/> >>> >>> >>> 14)tempest.api.compute.**security_groups.test_security_** >>> groups:SecurityGroupsTestJSON.**test_security_group_create_** >>> with_duplicate_name >>> >>> Security Group with duplicate name should not be created, but two groups >>> with the same name can be created in quantum. We have here the same >>> problem as in 15 and in 16. With Quantum, there is no validation that a >>> group with given name exists already or if the given SG name is empty >>> or is composed of white spaces or is more than 255 chars. >>> >>> >>> In the description of bug >>> https://bugs.launchpad.net/**nova/+bug/1161411this<https://bugs.launchpad.net/nova/+bug/1161411this>issue >>> is generally >>> >>> discussed. SecurityGroup API are based on the ID and not the names, >>> except for adding an instance to a security group. In order to solve >>> the last problem the bug https://bugs.launchpad.net/** >>> nova/+bug/1161473was <https://bugs.launchpad.net/nova/+bug/1161473was> >>> added. >>> >>> >>> The major question is if these 3 tests (14, 15, 16) : does the name of a >>> security group is really that important? If so, we must add some >>> validation methods. If not the test suit concerning this part must be >>> disable in tempest. What do you think Miguel? >>> >>> >>> >>> 15)tempest.api.compute.**security_groups.test_security_** >>> groups:SecurityGroupsTestJSON.**test_security_group_create_** >>> with_invalid_group_description >>> >>> 16)tempest.api.compute.**security_groups.test_security_** >>> groups:SecurityGroupsTestJSON.**test_security_group_create_** >>> with_invalid_group_name >>> >>> BUG:https://bugs.launchpad.**net/nova/+bug/1161411+<https://bugs.launchpad.net/nova/+bug/1161411+> >>> https://bugs.launchpad.net/**nova/+bug/1161473<https://bugs.launchpad.net/nova/+bug/1161473> >>> >>> Traceback(tempest) >>> :http://paste.openstack.org/**show/38423/<http://paste.openstack.org/show/38423/> >>> >>> The Security Group should not be created with group name an empty string >>> or with white spaces/chars more than 255 >>> >>> CURL:curl -H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" >>> -X POST http://$IP:8774/v2/$PROJECT_**ID/os-security-groups -d >>> '{"security_group": {"name": " ", "description": >>> "description-1554950088"}}' >>> >>> curl -H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X >>> POST http://$IP:8774/v2/$PROJECT_**ID/os-security-groups -d >>> '{"security_group": {"name": " ", "description": >>> "description-1554950088"}}' >>> >>> >>> 17)tempest.api.compute.**security_groups.test_security_** >>> groups:SecurityGroupsTestJSON.**test_server_security_groups >>> >>> * >>> >>> Tempest Trace : >>> http://paste.openstack.org/**show/38427/<http://paste.openstack.org/show/38427/> >>> >>> * >>> >>> Fixed by >>> https://review.openstack.org/#**/c/32288/(merged<https://review.openstack.org/#/c/32288/(merged>on >>> June,12th) >>> >>> >>> >>> >>> Tests related to servers(Ala) >>> >>> >>> 18)tempest.api.compute.**servers.test_list_server_**filters:** >>> ListServerFiltersTestXML.test_**list_servers_filtered_by_ip_**regex >>> >>> * >>> >>> BUG: >>> https://bugs.launchpad.net/**quantum/+bug/1182883<https://bugs.launchpad.net/quantum/+bug/1182883> >>> >>> * >>> >>> BP: >>> https://blueprints.launchpad.**net/quantum/+spec/like-op-list<https://blueprints.launchpad.net/quantum/+spec/like-op-list> >>> >>> >>> * >>> >>> CURL : GET http://$IP:8774/v2/$PROJECT_**ID/servers?ip=10. >>> >>> * >>> >>> Explanation: The regex search is not supported by Quantum. Thus >>> Quantum returns a 404 Not Found (0 server match) where Tempest >>> expects one server to be found. >>> >>> * >>> >>> Possible Fix : For "search port by IP with regex" feature, I think >>> the best place to hack it would be in file >>> quantum/db/db_base_plugin_v2.**py::_get_ports_query() >>> >>> >>> 19)tempest.api.compute.**servers.test_servers_negative:** >>> ServersNegativeTest.test_**create_with_nonexistent_**security_group >>> >>> FIXED : >>> https://review.openstack.org/#**/c/30271/<https://review.openstack.org/#/c/30271/> >>> >>> >>> 20)tempest.api.compute.**servers.test_virtual_**interfaces:** >>> VirtualInterfacesTestXML.test_**list_virtual_interfaces >>> >>> * >>> >>> TRACE (NOVA): >>> http://paste.openstack.org/**show/38371/<http://paste.openstack.org/show/38371/> >>> >>> * >>> >>> BUG: >>> https://bugs.launchpad.net/**tempest/+bug/1183436<https://bugs.launchpad.net/tempest/+bug/1183436> >>> >>> >>> * >>> >>> CURL: GET >>> http://$IP:8774/v2/$PROJECT_**ID/servers/$SERVER/os-virtual-** >>> interfaces >>> >>> * >>> >>> Explanation: This HTTP request calls the Quantum API >>> (nova/nova/network/quantumv2/**api.py) and specifically the >>> get_vifs_by_* methods which are not implemented (raise >>> NotImplementedError()) >>> >>> * >>> >>> Possible Fix: >>> >>> >>> o >>> >>> skip this test if Quantum is enabled as set in Tempest >>> configuration. Or >>> >>> o >>> >>> Implement the get_vifs_by_* methods >>> >>> PATCH: (still not approved) >>> https://review.openstack.org/#**/c/31755/<https://review.openstack.org/#/c/31755/> >>> >>> >>> On Tue, Jun 18, 2013 at 5:56 PM, Sean Dague <s...@dague.net >>> <mailto:s...@dague.net>> wrote: >>> >>> On 06/18/2013 03:32 PM, Monty Taylor wrote: >>> >>> >>> >>> On 06/18/2013 03:14 PM, David Ripton wrote: >>> >>> On 06/18/2013 12:43 PM, Martina Kollarova wrote: >>> >>> Jenkins keeps running all the tests, even if the basic >>> pep8 test fails, >>> and runs all of the (very slow) Tempest Quantum tests, >>> even though >>> almost all of them are failing. >>> >>> I propose that it should fail and stop all of the other >>> tests once there >>> is a failure in a voting test. For non-voting tests, it >>> should stop only >>> itself, not the others. >>> >>> This would decrease the feedback loop and we wouldn't >>> have to wait for >>> the non-voting Quantum tests to see that they failed as >>> always. >>> >>> >>> -1 >>> >>> In addition to the other objections, we currently get a lot >>> of false >>> positives (fail, retry, fail, retry, succeed), and it would >>> be harder to >>> debug these problems if the output was truncated differently >>> each time. >>> >>> Is anyone working on fixing the perma-failing Quantum test? >>> When the >>> Postgres test was perma-failing, one of the infrastructure >>> folks gave us >>> an ultimatum that if nobody fixed it soon, it would be >>> disabled. (Happy >>> ending: Mauro fixed it before it got disabled.) >>> >>> >>> That was brought up a little while ago, but we had already spent >>> so much >>> effort to get it working in the first place, none of us had the >>> heart to >>> put in such an ultimatum. But seriously- it might be time for an >>> all-hands-on-deck dogpile to figure out what's up the the >>> quantum gate. >>> >>> >>> The biggest cause of the Quantum vs. Full Tempest runs is that a lot >>> of the network api's in nova currently don't do translation of >>> errors. So under nova-network certain data validation and error >>> codes are returned, when quantum is used others are returned. >>> >>> This is a nova-api, so it needs to be consistent regardless of >>> backend (i.e. we don't return different API responses on different >>> databases). >>> >>> Issues like this one - >>> >>> https://bugs.launchpad.net/__**nova/+bug/1160309<https://bugs.launchpad.net/__nova/+bug/1160309> >>> >>> >>> <https://bugs.launchpad.net/**nova/+bug/1160309<https://bugs.launchpad.net/nova/+bug/1160309> >>> > >>> >>> Jordan Pittier has been working on some of these issues (he's the >>> only one I've seen working them from a Tempest / nova side), and got >>> to the crux of the problem. It could use more hands though to >>> organize the rest of those and get them banged out. >>> >>> I'm sure there are other issues once we get past this class. But >>> that would go a long way. >>> >>> -Sean >>> >>> -- >>> Sean Dague >>> http://dague.net >>> >>> ______________________________**___________________ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.**__org >>> >>> <mailto:OpenStack-dev@lists.**openstack.org<OpenStack-dev@lists.openstack.org> >>> > >>> http://lists.openstack.org/__**cgi-bin/mailman/listinfo/__** >>> openstack-dev<http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev>< >>> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev> >>> > >>> >>> >>> >>> >>> >>> ______________________________**_________________ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.**org <OpenStack-dev@lists.openstack.org> >>> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev> >>> >>> >> >> -- >> Sean Dague >> http://dague.net >> >> ______________________________**_________________ >> OpenStack-dev mailing list >> OpenStack-dev@lists.openstack.**org <OpenStack-dev@lists.openstack.org> >> http://lists.openstack.org/**cgi-bin/mailman/listinfo/**openstack-dev<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev> >> > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > -- Ala Rezmerita Ingénieur CloudWatt Tel : 06 77 43 23 91
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
