Hello,
TL;DR: We make a multi-regions deployment with Kolla. It requires to patch the code a little bit, and you can find the diff on our GitHub[1]. This patch is just a first attempt to support multi-regions in Kolla and it raises questions. Some modifications are not done in an idiomatic way and we do not expect this to be merged in Kolla. The reminder of this mail explains our patch and states our questions. At Inria/Discovery[2], we evaluate OpenStack at scale for the Performance Working Group. So far, we focus on one single OpenStack region deployment with hundreds of computes and we always go with Kolla for our deployment. Over the last few days, we tried to achieve a multi-regions OpenStack deployment with Kolla. We want to share with you our current deployment workflow, patches we had to apply on Kolla to support multi-regions, and also ask you if we do things correctly. First of all, our multi-regions deployment follows the one described by the OpenStack documentation[3]. Concretely, the deployment considers /one/ Administrative Region (AR) that contains Keystone and Horizon. This is a Kolla-based deployment, so Keystone is hidden behind an HAProxy, and has MariaDB and memcached as backend. At the same time, /n/ OpenStack Regions (OSR1, ..., OSRn) contain a full OpenStack, except Keystone. We got something as follows at the end of the deployment: Admin Region (AR): - control: * Horizon * HAProxy * Keyston * MariaDB * memcached OpenStack Region x (OSRx): - control: * HAProxy * nova-api/conductor/scheduler * neutron-server/l3/dhcp/... * glance-api/registry * MariaDB * RabbitMQ - compute1: * nova-compute * neutron-agent - compute2: ... We do the deployment by running Kolla n+1 times. The first run deploys the Administrative Region (AR) and the other runs deploy OpenStack Regions (OSR). For each run, we fix the value of `openstack_region_name' variable to the name of the current region. In the context of multi-regions, Keystone (in the AR) should be available to all OSRs. This means, there are as many Keystone endpoints as regions. For instance, if we consider two OSRs, the result of listing endpoints at the end of the AR deployment looks like this: $ openstack endpoint list | Region | Serv Name | Serv Type | Interface | URL | |--------+-----------+-----------+-----------+------------------------------| | AR | keystone | identity | public | http://10.24.63.248:5000/v3 | | AR | keystone | identity | internal | http://10.24.63.248:5000/v3 | | AR | keystone | identity | admin | http://10.24.63.248:35357/v3 | | OSR1 | keystone | identity | public | http://10.24.63.248:5000/v3 | | OSR1 | keystone | identity | internal | http://10.24.63.248:5000/v3 | | OSR1 | keystone | identity | admin | http://10.24.63.248:35357/v3 | | OSR2 | keystone | identity | public | http://10.24.63.248:5000/v3 | | OSR2 | keystone | identity | internal | http://10.24.63.248:5000/v3 | | OSR2 | keystone | identity | admin | http://10.24.63.248:35357/v3 | This requires patching the `keystone/tasks/register.yml' play[4] to re-execute the `Creating admin project, user, role, service, and endpoint' task for all regions we consider. An example of such a patch is given on our GitHub[5]. In this example, the `openstack_regions' variable is a list that contains the name of all regions (see [6]). As a drawback, the patch implies to know in advance all OSR. A better implementation would execute the `Creating admin project, user, role, service, and endpoint' task every time a new OSR is going to be deployed. But this requires to move the task somewhere else in the Kolla workflow and we have no idea where this should be. In the AR, we also have to change the Horizon configuration file to handle multi-regions[7]. The modification could be done easily and idiomatically by setting the `node_custome_config' variable to the `multi-regions' directory[8] and benefits from Kolla merging config system. Also, deploying OSRs requires patching the kolla-toolbox as it seems not region-aware. In particular, patch the `kolla_keystone_service.py' module[9] that is responsible for contacting Keystone and creating a new endpoint when we register a new OpenStack service. 73 for _endpoint in cloud.keystone_client.endpoints.list(): 74 if _endpoint.service_id == service.id and \ 75 _endpoint.interface == interface: 76 endpoint = _endpoint 77 if endpoint.url != url: 78 changed = True 79 cloud.keystone_client.endpoints.update( 80 endpoint, url=url) 81 break 82 else: 83 changed = True 84 cloud.keystone_client.endpoints.create( 85 service=service.id, 86 url=url, 87 interface=interface, 88 region=endpoint_region) At some point, this module /create/ or /update/ a service endpoint. It first tests if the service is already registered, and update the URL if so (l. 74 to 81), or create the new endpoint in other cases (l. 82). Unfortunately, the test (l. 74 to 75) only looks at the service (e.g., glance) and the interface (e.g., public). This makes the test wrong because we deploy the same service many times, but into different regions. We have to add an extra condition to not only test the service but also its region. 74 if _endpoint.service_id == service.id and \ 75 _endpoint.interface == interface and \ 76 _endpoint.region == endpoint_region: Finally, when we deployed OSRs, we override the value of `keystone_admin_url', `keystone_internal_url' and `keystone_public_url' to target Keystone in AR. We also have to change the `[keystone_authtoken]' section of nova/neutron/glance conf[10] so that it uses these variables instead of the canonical form with the `kolla_internal_fqdn' variable[11]. In this regards, is it due to some legacy code that configuration files use the `kolla_internal_fqdn' variable instead of `keystone_*_url'? That's almost all. As you can see, handle multi-regions implies a very small number of modifications if you call Kolla multiple times. So, thanks for the great job Kolla team! And waiting for your feedback. [1] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554 [2] https://beyondtheclouds.github.io/ [3] http://docs.openstack.org/arch-design/multi-site-architecture.html [4] https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/keystone/tasks/register.yml [5] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-40be75c3a2237adfd1a05178f6f60006R1 [6] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-607e6e5925d0031dafa79eb80d198640R215 [7] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-6cbb63bb9c4843bcf8db4710e588475bR188 [8] https://github.com/BeyondTheClouds/kolla/tree/7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions [9] https://github.com/BeyondTheClouds/kolla/commit/7e5f7e6c7936b7cc3136dfc082935e2995a65554#diff-ba10dd4575f65e03d50d586febdccbadR72 [10] https://github.com/BeyondTheClouds/kolla/blob/7e5f7e6c7936b7cc3136dfc082935e2995a65554/multi-regions/global.conf [11] https://github.com/openstack/kolla/blob/11bfd9eb7518cc46ac441505a267f5cf974216ae/ansible/roles/nova/templates/nova.conf.j2#L155 -- Ronan-A. Cherrueau __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
