This is great stuff! You guys have this up in etherpad somewhere?
Really great to see people organizing around this. I'd reviewed a bunch
of Jordan's code, so I knew he was in the mix, sorry for not realizing
there were other folks working on this as well.
-Sean
On 06/19/2013 10:27 AM, Miguel Lavalle wrote:
Hi,
As of last week, we have organized a team to deal with the Quantum in
full gate issues. There are 3 people working on this: Jordan Pittier,
Ala.Rezmerita and myself. Below you will find a document describing all
the identified issues and who is assigned to fix them. Of cource, we can
use more hands / heads. If anyone wants to help or has identified other
issues not listed below, please get in touch with us.
Regards
Fix Jenkins gate-tempest-devstack-vm-quantum-full
This document lists all the Tempest tests that are failing on a Devstack
+ Quantum Trunk setup. The goal is to make these tests pass for the
Havana Milestone 3. Alla, Miguel and Jordan are working on this.
For each test :
*
Provide the full path of the test. If the test has a double
interface (JSON and XML), it’s enough to provide only the JSON path.
*
Provide a small stacktrace or a link to a Paste service such as
http://paste.openstack.org/
*
Would be great to provide the CURL URL that triggers the bug
*
Provide a small analysis of what the bug may be and the components
involved (Quantum, Quantum API on Nova’s side, Nova-network etc.)
*
As far as possible, provide the file and the line number where the
Python exception is raised or translated (“casted”)
*
If already filed, an URL to the bug report in Launchpad
Tests related to the Fixed IPs Compute API extension (Miguel)
1)
tempest.api.compute.admin.test_fixed_ips:FixedIPsTestJson.test_list_fixed_ip_details
*
Trace: http://paste.openstack.org/show/38363/
*
curl-H "X-Auth-Token:$TOKEN" -X GET
http://$IP:8774/v2/$TENANT_ID/os-fixed-ips/10.0.0.3 <http://10.0.0.3>
*
API: http://api.openstack.org/api-ref.html#ext-os-fixed-ips
*
Explanation: Call to
nova.api.openstack.compute.contrib.fixed_ips.FixedIPController::show()
This file doesn’t use the Quantum API nor the Nova-Network API. It
interacts directly with the DB, which is bad.
*
A possible fix would be to :
1.
Change
nova.api.openstack.compute.contrib.fixed_ips.FixedIPController::show()
to use either nova.network.api.API::get_fixed_ip() (for Nova
Network) or nova.network.quantumv2.api.API:get_fixed_ip() (for
Nova Network)
2.
Implement nova.network.quantumv2.api.API:get_fixed_ip() which
currently raises a NotImplementedError exception
2)
tempest.api.compute.admin.test_fixed_ips:FixedIPsTestJson.test_set_reserve
*
Trace: http://paste.openstack.org/show/38372/
*
curl-H "X-Auth-Token:$TOKEN" -X POST
http://$IP:8774/v2/$TENANT_ID/os-fixed-ips/10.0.0.3/action
<http://10.0.0.3/action>
*
API: http://api.openstack.org/api-ref.html#ext-os-fixed-ips
*
Possible Fix: Should call
nova.api.openstack.compute.contrib.fixed_ips.FixedIPController::_set_reserved()
(once the result of db.fixed_ip_get_by_address() is made through the
API). Here the problem is that neither Nova-Network nor Quatum API
implement an equivalent of db.fixed_ip_update()
3)
tempest.api.compute.admin.test_fixed_ips:FixedIPsTestJson.test_set_unreserve
*
Same as 2)
Tests related to Quotas Admin(Miguel)
4)tempest.api.compute.admin.test_quotas:QuotasAdminTestJSON.test_security_groups_exceed_limit
5)tempest.api.compute.admin.test_quotas:QuotasAdminTestJSON.test_security_groups_rules_exceed_limit
Tests related to Floating IPs
6)tempest.api.compute.floating_ips.test_floating_ips_actions:FloatingIPsTestJSON.test_associate_ip_to_server_without_passing_floating_ip
*
Trace Tempest http://paste.openstack.org/show/38431/
*
Bug : https://bugs.launchpad.net/quantum/+bug/1190242
*
curl-H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X
POST http://$IP:8774/v2/$TENANT_ID/servers/$SERVER_ID/action
<http://10.1.59.157:8774/v2/de6c5fbc55a34cbcaa3d79eb6b21a784/servers/0b2ad3b6-c14a-4d89-b2a0-c015f0a88a1f/action>-d
‘{"addFloatingIp": {"address": ""}}’
*
Notice that the address of the FloatingIP is empty.
*
API : http://api.openstack.org/api-ref.html#ext-floating-ips
*
Explanation : Mismatch in the 2 API. If there is only one IP in the
pool, Quantum allows the floating IP to be empty and returns the one
and only IP in the pool. Nova-network doesn’t allow this and returns
a 404
*
Review https://review.openstack.org/#/c/32740/
7)tempest.api.compute.floating_ips.test_floating_ips_actions:FloatingIPsTestJSON.test_delete_floating_ip
*
Trace Tempest : http://paste.openstack.org/show/38505/
*
Bug : https://bugs.launchpad.net/tempest/+bug/1160309(see comment #2)
*
This is related to bug 9)
8)tempest.api.compute.floating_ips.test_floating_ips_actions:FloatingIPsTestJSON.test_delete_nonexistant_floating_ip
*
Related to 9)
9)tempest.api.compute.floating_ips.test_list_floating_ips:FloatingIPDetailsTestJSON.test_get_nonexistant_floating_ip_details
*
Bug :https://bugs.launchpad.net/tempest/+bug/1160309
*
Trace tempest:http://paste.openstack.org/show/38430/
*
Trace nova: http://paste.openstack.org/show/38433/
*
Curl:curl -H "X-Auth-Token:$TOKEN" -X GET
http://$IP:8774/v2/$PROJECT_ID/os-floating-ips/99987878787878
Proposed Fix: https://review.openstack.org/33024
Tests Related to Security Groups (Jordan)
10)tempest.api.compute.security_groups.test_security_group_rules:SecurityGroupRulesTestJSON.test_security_group_rules_create_with_invalid_id
*
TRACE: http://paste.openstack.org/show/38373/
*
curl-H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X
POST http://$IP:8774/v2/$TENANT_ID/os-security-group-rules -d
‘{"security_group_rule": {"from_port": 22, "ip_protocol": "tcp",
"to_port": 22, "parent_group_id": "9991393497170", "cidr": null,
"group_id": null}}’
*
Explanation: Notice that the parent_goup_id is a numerical ID and
not a UUID. Quantum has an additional check to validate that the ID
is an UUID (see
nova/network/security_group/quantum_driver.py::validate_id())
*
API: http://api.openstack.org/api-ref.html#ext-os-security-groups
*
Bug: https://bugs.launchpad.net/tempest/+bug/1182384
*
Review: https://review.openstack.org/#/c/29899/
11)tempest.api.compute.security_groups.test_security_group_rules:SecurityGroupRulesTestJSON.test_security_group_rules_delete_with_invalid_id
*
http://paste.openstack.org/show/38424/
*
curl -H "X-Auth-Token:$TOKEN" -X DELETE
http://$IP:8774/v2$TENANT_ID/os-security-group-rules/9991407551273
*
Explanation : Same bug as 10
*
API: http://api.openstack.org/api-ref.html#ext-os-security-groups
*
Review: https://review.openstack.org/#/c/29899/
12)tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_delete_nonexistant_security_group
*
Same as bug 10
*
Review: https://review.openstack.org/#/c/29899/
13)tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_security_group_get_nonexistant_group
*
Same as bug 10
*
Review: https://review.openstack.org/#/c/29899/
14)tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_security_group_create_with_duplicate_name
Security Group with duplicate name should not be created, but two groups
with the same name can be created in quantum. We have here the same
problem as in 15 and in 16. With Quantum, there is no validation that a
group with given name exists already or if the given SG name is empty
or is composed of white spaces or is more than 255 chars.
In the description of bug
https://bugs.launchpad.net/nova/+bug/1161411this issue is generally
discussed. SecurityGroup API are based on the ID and not the names,
except for adding an instance to a security group. In order to solve
the last problem the bug https://bugs.launchpad.net/nova/+bug/1161473was
added.
The major question is if these 3 tests (14, 15, 16) : does the name of a
security group is really that important? If so, we must add some
validation methods. If not the test suit concerning this part must be
disable in tempest. What do you think Miguel?
15)tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_security_group_create_with_invalid_group_description
16)tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_security_group_create_with_invalid_group_name
BUG:https://bugs.launchpad.net/nova/+bug/1161411+
https://bugs.launchpad.net/nova/+bug/1161473
Traceback(tempest) :http://paste.openstack.org/show/38423/
The Security Group should not be created with group name an empty string
or with white spaces/chars more than 255
CURL:curl -H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN"
-X POST http://$IP:8774/v2/$PROJECT_ID/os-security-groups -d
'{"security_group": {"name": " ", "description":
"description-1554950088"}}'
curl -H "Content-Type: application/json" -H "X-Auth-Token:$TOKEN" -X
POST http://$IP:8774/v2/$PROJECT_ID/os-security-groups -d
'{"security_group": {"name": " ", "description":
"description-1554950088"}}'
17)tempest.api.compute.security_groups.test_security_groups:SecurityGroupsTestJSON.test_server_security_groups
*
Tempest Trace : http://paste.openstack.org/show/38427/
*
Fixed by https://review.openstack.org/#/c/32288/(merged on June,12th)
Tests related to servers(Ala)
18)tempest.api.compute.servers.test_list_server_filters:ListServerFiltersTestXML.test_list_servers_filtered_by_ip_regex
*
BUG: https://bugs.launchpad.net/quantum/+bug/1182883
*
BP: https://blueprints.launchpad.net/quantum/+spec/like-op-list
*
CURL : GET http://$IP:8774/v2/$PROJECT_ID/servers?ip=10.
*
Explanation: The regex search is not supported by Quantum. Thus
Quantum returns a 404 Not Found (0 server match) where Tempest
expects one server to be found.
*
Possible Fix : For "search port by IP with regex" feature, I think
the best place to hack it would be in file
quantum/db/db_base_plugin_v2.py::_get_ports_query()
19)tempest.api.compute.servers.test_servers_negative:ServersNegativeTest.test_create_with_nonexistent_security_group
FIXED : https://review.openstack.org/#/c/30271/
20)tempest.api.compute.servers.test_virtual_interfaces:VirtualInterfacesTestXML.test_list_virtual_interfaces
*
TRACE (NOVA): http://paste.openstack.org/show/38371/
*
BUG: https://bugs.launchpad.net/tempest/+bug/1183436
*
CURL: GET
http://$IP:8774/v2/$PROJECT_ID/servers/$SERVER/os-virtual-interfaces
*
Explanation: This HTTP request calls the Quantum API
(nova/nova/network/quantumv2/api.py) and specifically the
get_vifs_by_* methods which are not implemented (raise
NotImplementedError())
*
Possible Fix:
o
skip this test if Quantum is enabled as set in Tempest
configuration. Or
o
Implement the get_vifs_by_* methods
PATCH: (still not approved) https://review.openstack.org/#/c/31755/
On Tue, Jun 18, 2013 at 5:56 PM, Sean Dague <s...@dague.net
<mailto:s...@dague.net>> wrote:
On 06/18/2013 03:32 PM, Monty Taylor wrote:
On 06/18/2013 03:14 PM, David Ripton wrote:
On 06/18/2013 12:43 PM, Martina Kollarova wrote:
Jenkins keeps running all the tests, even if the basic
pep8 test fails,
and runs all of the (very slow) Tempest Quantum tests,
even though
almost all of them are failing.
I propose that it should fail and stop all of the other
tests once there
is a failure in a voting test. For non-voting tests, it
should stop only
itself, not the others.
This would decrease the feedback loop and we wouldn't
have to wait for
the non-voting Quantum tests to see that they failed as
always.
-1
In addition to the other objections, we currently get a lot
of false
positives (fail, retry, fail, retry, succeed), and it would
be harder to
debug these problems if the output was truncated differently
each time.
Is anyone working on fixing the perma-failing Quantum test?
When the
Postgres test was perma-failing, one of the infrastructure
folks gave us
an ultimatum that if nobody fixed it soon, it would be
disabled. (Happy
ending: Mauro fixed it before it got disabled.)
That was brought up a little while ago, but we had already spent
so much
effort to get it working in the first place, none of us had the
heart to
put in such an ultimatum. But seriously- it might be time for an
all-hands-on-deck dogpile to figure out what's up the the
quantum gate.
The biggest cause of the Quantum vs. Full Tempest runs is that a lot
of the network api's in nova currently don't do translation of
errors. So under nova-network certain data validation and error
codes are returned, when quantum is used others are returned.
This is a nova-api, so it needs to be consistent regardless of
backend (i.e. we don't return different API responses on different
databases).
Issues like this one -
https://bugs.launchpad.net/__nova/+bug/1160309
<https://bugs.launchpad.net/nova/+bug/1160309>
Jordan Pittier has been working on some of these issues (he's the
only one I've seen working them from a Tempest / nova side), and got
to the crux of the problem. It could use more hands though to
organize the rest of those and get them banged out.
I'm sure there are other issues once we get past this class. But
that would go a long way.
-Sean
--
Sean Dague
http://dague.net
_________________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.__org
<mailto:OpenStack-dev@lists.openstack.org>
http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev
<http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Sean Dague
http://dague.net
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
