On 01/05/2017 09:12 AM, Ronan-Alexandre Cherrueau wrote:
Hello,
TL;DR: We make a multi-regions deployment with Kolla. It requires to
patch the code a little bit, and you can find the diff on our
GitHub[1]. This patch is just a first attempt to support multi-regions
in Kolla and it raises questions. Some modifications are not done in
an idiomatic way and we do not expect this to be merged in Kolla. The
reminder of this mail explains our patch and states our questions.
At Inria/Discovery[2], we evaluate OpenStack at scale for the
Performance Working Group. So far, we focus on one single OpenStack
region deployment with hundreds of computes and we always go with
Kolla for our deployment. Over the last few days, we tried to achieve
a multi-regions OpenStack deployment with Kolla. We want to share with
you our current deployment workflow, patches we had to apply on Kolla
to support multi-regions, and also ask you if we do things correctly.
First of all, our multi-regions deployment follows the one described
by the OpenStack documentation[3].
I don't see an "Admin Region" as part of the OpenStack documentation for
multi-region deployment. I also see LDAP mentioned as the recommended
authentication/IdM store.
> Concretely, the deployment
considers /one/ Administrative Region (AR) that contains Keystone and
Horizon.
That's not a region. Those should be shared resources *across* regions.
> This is a Kolla-based deployment, so Keystone is hidden
behind an HAProxy, and has MariaDB and memcached as backend.
I thought at Inria, the Nova "MySQL DB has been replaced by the noSQL
system REDIS"? But here, you're using MariaDB -- a non-distributed
database -- for the Keystone component which is the very thing that is
the most highly distributed of all state storage in OpenStack.
So, you are replacing the Nova DB (which doesn't need to be distributed
at all, since it's a centralized control plane piece) within the regions
with a "distributed" NoSQL store (and throwing away transactional safety
I might add) but you're going with a non-distributed traditional RDBMS
for the very piece that needs to be shared, distributed, and
highly-available across OpenStack. I don't understand that.
> At the
same time, /n/ OpenStack Regions (OSR1, ..., OSRn) contain a full
OpenStack, except Keystone. We got something as follows at the end of
the deployment:
Admin Region (AR):
- control:
* Horizon
* HAProxy
* Keyston
* MariaDB
* memcached
Again, that's not a region. Those are merely shared services between
regions.
OpenStack Region x (OSRx):
- control:
* HAProxy
* nova-api/conductor/scheduler
* neutron-server/l3/dhcp/...
* glance-api/registry
* MariaDB
* RabbitMQ
- compute1:
* nova-compute
* neutron-agent
- compute2: ...
We do the deployment by running Kolla n+1 times. The first run deploys
the Administrative Region (AR) and the other runs deploy OpenStack
Regions (OSR). For each run, we fix the value of `openstack_region_name'
variable to the name of the current region.
In the context of multi-regions, Keystone (in the AR) should be
available to all OSRs. This means, there are as many Keystone
endpoints as regions. For instance, if we consider two OSRs, the
result of listing endpoints at the end of the AR deployment looks like
this:
$ openstack endpoint list
| Region | Serv Name | Serv Type | Interface | URL |
|--------+-----------+-----------+-----------+------------------------------|
| AR | keystone | identity | public | http://10.24.63.248:5000/v3 |
| AR | keystone | identity | internal | http://10.24.63.248:5000/v3 |
| AR | keystone | identity | admin | http://10.24.63.248:35357/v3 |
| OSR1 | keystone | identity | public | http://10.24.63.248:5000/v3 |
| OSR1 | keystone | identity | internal | http://10.24.63.248:5000/v3 |
| OSR1 | keystone | identity | admin | http://10.24.63.248:35357/v3 |
| OSR2 | keystone | identity | public | http://10.24.63.248:5000/v3 |
| OSR2 | keystone | identity | internal | http://10.24.63.248:5000/v3 |
| OSR2 | keystone | identity | admin | http://10.24.63.248:35357/v3 |
There shouldn't be an AR region. If the Keystone authentication domain
is indeed shared between OpenStack regions, then an administrative user
should be able to hit any Keystone endpoint in any OpenStack region and
add users/projects/roles, etc. to the shared Keystone data store (or if
using LDAP, the admin should be able to add a user to
ActiveDirectory/ApacheDS in any OpenStack region and have that user
information immediately show up in any of the other regions).
Best,
-jay
This requires patching the `keystone/tasks/register.yml' play[4] to
re-execute the `Creating admin project, user, role, service, and
endpoint' task for all regions we consider. An example of such a patch
is given on our GitHub[5]. In this example, the `openstack_regions'
variable is a list that contains the name of all regions (see [6]). As
a drawback, the patch implies to know in advance all OSR. A better
implementation would execute the `Creating admin project, user, role,
service, and endpoint' task every time a new OSR is going to be
deployed. But this requires to move the task somewhere else in the
Kolla workflow and we have no idea where this should be.
In the AR, we also have to change the Horizon configuration file to
handle multi-regions[7]. The modification could be done easily and
idiomatically by setting the `node_custome_config' variable to the
`multi-regions' directory[8] and benefits from Kolla merging config
system.
Also, deploying OSRs requires patching the kolla-toolbox as it seems
not region-aware. In particular, patch the `kolla_keystone_service.py'
module[9] that is responsible for contacting Keystone and creating a
new endpoint when we register a new OpenStack service.
73 for _endpoint in cloud.keystone_client.endpoints.list():
74 if _endpoint.service_id == service.id and \
75 _endpoint.interface == interface:
76 endpoint = _endpoint
77 if endpoint.url != url:
78 changed = True
79 cloud.keystone_client.endpoints.update(
80 endpoint, url=url)
81 break
82 else:
83 changed = True
84 cloud.keystone_client.endpoints.create(
85 service=service.id,
86 url=url,
87 interface=interface,
88 region=endpoint_region)
At some point, this module /create/ or /update/ a service endpoint. It
first tests if the service is already registered, and update the URL
if so (l. 74 to 81), or create the new endpoint in other cases (l.
82). Unfortunately, the test (l. 74 to 75) only looks at the service
(e.g., glance) and the interface (e.g., public). This makes the test
wrong because we deploy the same service many times, but into
different regions. We have to add an extra condition to not only test
the service but also its region.
74 if _endpoint.service_id == service.id and \
75 _endpoint.interface == interface and \
76 _endpoint.region == endpoint_region:
Finally, when we deployed OSRs, we override the value of
`keystone_admin_url', `keystone_internal_url' and `keystone_public_url'
to target Keystone in AR. We also have to change the
`[keystone_authtoken]' section of nova/neutron/glance conf[10] so that
it uses these variables instead of the canonical form with the
`kolla_internal_fqdn' variable[11]. In this regards, is it due to some
legacy code that configuration files use the `kolla_internal_fqdn'
variable instead of `keystone_*_url'?
That's almost all. As you can see, handle multi-regions implies a very
small number of modifications if you call Kolla multiple times.
So, thanks for the great job Kolla team! And waiting for your
feedback.
[1]
https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554
[2] https://beyondtheclouds.github.io/
[3] http://docs.openstack.org/arch-design/multi-site-architecture.html
[4]
https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/keystone/tasks/register.yml
[5]
https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-40be75c3a2237adfd1a05178f6f60006R1
[6]
https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-607e6e5925d0031dafa79eb80d198640R215
[7]
https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-6cbb63bb9c4843bcf8db4710e588475bR188
[8]
https://github.com/BeyondTheClouds/kolla/tree/7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions
[9]
https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-ba10dd4575f65e03d50d586febdccbadR72
[10]
https://github.com/BeyondTheClouds/kolla/blob/7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions/global.conf
[11]
https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/nova/templates/nova.conf.j2#L155
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
