Public bug reported: We have a multi-domain configuration in our private cloud that I've had to revert to using the Liberty policy.v3cloudsample.json file instead of Mitaka or master.
Horizon is generating the following trace when a domain admin is trying to look at projects/users: [pid: 22842|app: 0|req: 5/17] 10.38.202.12 () {46 vars in 907 bytes} [Thu Jun 2 07:17:24 2016] GET / => generated 0 bytes in 5 msecs (HTTP/1.1 302) 5 headers in 198 bytes (1 switches on core 1) Internal Server Error: /identity/ Traceback (most recent call last): File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/decorators.py", line 36, in dec return view_func(request, *args, **kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/decorators.py", line 52, in dec return view_func(request, *args, **kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/decorators.py", line 36, in dec return view_func(request, *args, **kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/django/views/generic/base.py", line 71, in view return self.dispatch(request, *args, **kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/django/views/generic/base.py", line 89, in dispatch return handler(request, *args, **kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 159, in get handled = self.construct_tables() File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 150, in construct_tables handled = self.handle_table(table) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 121, in handle_table data = self._get_data_dict() File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 187, in _get_data_dict self._data = {self.table_class._meta.name: self.get_data()} File "/opt/mhos/openstack/horizon/openstack_dashboard/dashboards/identity/projects/views.py", line 84, in get_data self.request): File "/opt/mhos/openstack/horizon/openstack_dashboard/policy.py", line 24, in check return policy_check(actions, request, target) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/openstack_auth/policy.py", line 155, in check enforcer[scope], action, target, domain_credentials) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/openstack_auth/policy.py", line 169, in _check_credentials if not enforcer_scope.enforce(action, target, credentials): File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/policy.py", line 578, in enforce result = self.rules[rule](target, creds, self) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/_checks.py", line 160, in __call__ if rule(target, cred, enforcer): File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/_checks.py", line 204, in __call__ return enforcer.rules[self.match](target, creds, enforcer) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/_checks.py", line 125, in __call__ if not rule(target, cred, enforcer): File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/_checks.py", line 160, in __call__ if rule(target, cred, enforcer): File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/_checks.py", line 311, in __call__ return self._find_in_dict(creds, path_segments, match) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/_checks.py", line 292, in _find_in_dict return self._find_in_dict(test_value, path_segments, match) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/_checks.py", line 283, in _find_in_dict test_value = test_value[key] TypeError: 'Token' object has no attribute '__getitem__' [pid: 22837|app: 0|req: 5/18] 10.38.202.12 () {46 vars in 925 bytes} [Thu Jun 2 07:17:24 2016] GET /identity/ => generated 375516 bytes in 251 msecs (HTTP/1.1 500) 4 headers in 145 bytes (2 switches on core 0) Or we will get another trace, as follows, which is a bit more understanding: [pid: 22623|app: 0|req: 17/76] 10.38.202.12 () {44 vars in 3206 bytes} [Thu Jun 2 07:05:15 2016] GET /i18n/js/horizon+openstack_dashboard+neutron_lbaas_dashboard+muranodashboard/ => generated 2372 bytes in 4 msecs (HTTP/1.1 200) 4 hea ders in 132 bytes (1 switches on core 1) Pure project admin doesn't have a domain token Internal Server Error: /identity/users/ Traceback (most recent call last): File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/decorators.py", line 36, in dec return view_func(request, *args, **kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/decorators.py", line 52, in dec return view_func(request, *args, **kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/decorators.py", line 36, in dec return view_func(request, *args, **kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/django/views/generic/base.py", line 71, in view return self.dispatch(request, *args, **kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/django/views/generic/base.py", line 89, in dispatch return handler(request, *args, **kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 159, in get handled = self.construct_tables() File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 150, in construct_tables handled = self.handle_table(table) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 121, in handle_table data = self._get_data_dict() File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 187, in _get_data_dict self._data = {self.table_class._meta.name: self.get_data()} File "/opt/mhos/openstack/horizon/openstack_dashboard/dashboards/identity/users/views.py", line 79, in get_data u.domain_name = domain_lookup.get(u.domain_id) AttributeError: 'NoneType' object has no attribute 'get' [pid: 22619|app: 0|req: 15/77] 10.38.202.12 () {46 vars in 3190 bytes} [Thu Jun 2 07:05:19 2016] GET /identity/users/ => generated 340688 bytes in 413 msecs (HTTP/1.1 500) 4 headers in 145 bytes (2 switches on core 0) The 2nd trace usually is associated with situations where the V2 policy is in effect. Horizon Config: Memcached backend for caching Session DB configured As a workaround, we reverted to the Liberty policy.v3cloudsample.json and multi-domain is working beautifully, and our domain admins are able to manage users,projects,roles of the domain. We believe the issue lies around the following line (pulled from master): https://github.com/openstack/keystone/blob/master/etc/policy.v3cloudsample.json#L3 And yes, the admin_domain_id was correctly set. We feel that this breaks domain functionality and I would like someone to take a look and let us know if this is a configuration problem and why this works without failure with the Liberty policy. ** Affects: keystone Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1588190 Title: policy.v3cloudsample.json broken in mitaka Status in OpenStack Identity (keystone): New Bug description: We have a multi-domain configuration in our private cloud that I've had to revert to using the Liberty policy.v3cloudsample.json file instead of Mitaka or master. Horizon is generating the following trace when a domain admin is trying to look at projects/users: [pid: 22842|app: 0|req: 5/17] 10.38.202.12 () {46 vars in 907 bytes} [Thu Jun 2 07:17:24 2016] GET / => generated 0 bytes in 5 msecs (HTTP/1.1 302) 5 headers in 198 bytes (1 switches on core 1) Internal Server Error: /identity/ Traceback (most recent call last): File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/decorators.py", line 36, in dec return view_func(request, *args, **kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/decorators.py", line 52, in dec return view_func(request, *args, **kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/decorators.py", line 36, in dec return view_func(request, *args, **kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/django/views/generic/base.py", line 71, in view return self.dispatch(request, *args, **kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/django/views/generic/base.py", line 89, in dispatch return handler(request, *args, **kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 159, in get handled = self.construct_tables() File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 150, in construct_tables handled = self.handle_table(table) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 121, in handle_table data = self._get_data_dict() File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 187, in _get_data_dict self._data = {self.table_class._meta.name: self.get_data()} File "/opt/mhos/openstack/horizon/openstack_dashboard/dashboards/identity/projects/views.py", line 84, in get_data self.request): File "/opt/mhos/openstack/horizon/openstack_dashboard/policy.py", line 24, in check return policy_check(actions, request, target) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/openstack_auth/policy.py", line 155, in check enforcer[scope], action, target, domain_credentials) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/openstack_auth/policy.py", line 169, in _check_credentials if not enforcer_scope.enforce(action, target, credentials): File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/policy.py", line 578, in enforce result = self.rules[rule](target, creds, self) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/_checks.py", line 160, in __call__ if rule(target, cred, enforcer): File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/_checks.py", line 204, in __call__ return enforcer.rules[self.match](target, creds, enforcer) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/_checks.py", line 125, in __call__ if not rule(target, cred, enforcer): File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/_checks.py", line 160, in __call__ if rule(target, cred, enforcer): File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/_checks.py", line 311, in __call__ return self._find_in_dict(creds, path_segments, match) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/_checks.py", line 292, in _find_in_dict return self._find_in_dict(test_value, path_segments, match) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/oslo_policy/_checks.py", line 283, in _find_in_dict test_value = test_value[key] TypeError: 'Token' object has no attribute '__getitem__' [pid: 22837|app: 0|req: 5/18] 10.38.202.12 () {46 vars in 925 bytes} [Thu Jun 2 07:17:24 2016] GET /identity/ => generated 375516 bytes in 251 msecs (HTTP/1.1 500) 4 headers in 145 bytes (2 switches on core 0) Or we will get another trace, as follows, which is a bit more understanding: [pid: 22623|app: 0|req: 17/76] 10.38.202.12 () {44 vars in 3206 bytes} [Thu Jun 2 07:05:15 2016] GET /i18n/js/horizon+openstack_dashboard+neutron_lbaas_dashboard+muranodashboard/ => generated 2372 bytes in 4 msecs (HTTP/1.1 200) 4 hea ders in 132 bytes (1 switches on core 1) Pure project admin doesn't have a domain token Internal Server Error: /identity/users/ Traceback (most recent call last): File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/decorators.py", line 36, in dec return view_func(request, *args, **kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/decorators.py", line 52, in dec return view_func(request, *args, **kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/decorators.py", line 36, in dec return view_func(request, *args, **kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/django/views/generic/base.py", line 71, in view return self.dispatch(request, *args, **kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/django/views/generic/base.py", line 89, in dispatch return handler(request, *args, **kwargs) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 159, in get handled = self.construct_tables() File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 150, in construct_tables handled = self.handle_table(table) File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 121, in handle_table data = self._get_data_dict() File "/opt/mhos/openstack/horizon/local/lib/python2.7/site-packages/horizon/tables/views.py", line 187, in _get_data_dict self._data = {self.table_class._meta.name: self.get_data()} File "/opt/mhos/openstack/horizon/openstack_dashboard/dashboards/identity/users/views.py", line 79, in get_data u.domain_name = domain_lookup.get(u.domain_id) AttributeError: 'NoneType' object has no attribute 'get' [pid: 22619|app: 0|req: 15/77] 10.38.202.12 () {46 vars in 3190 bytes} [Thu Jun 2 07:05:19 2016] GET /identity/users/ => generated 340688 bytes in 413 msecs (HTTP/1.1 500) 4 headers in 145 bytes (2 switches on core 0) The 2nd trace usually is associated with situations where the V2 policy is in effect. Horizon Config: Memcached backend for caching Session DB configured As a workaround, we reverted to the Liberty policy.v3cloudsample.json and multi-domain is working beautifully, and our domain admins are able to manage users,projects,roles of the domain. We believe the issue lies around the following line (pulled from master): https://github.com/openstack/keystone/blob/master/etc/policy.v3cloudsample.json#L3 And yes, the admin_domain_id was correctly set. We feel that this breaks domain functionality and I would like someone to take a look and let us know if this is a configuration problem and why this works without failure with the Liberty policy. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1588190/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp