This is an automated email from the ASF dual-hosted git repository.
arvindsh pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/fluo-muchos.git
The following commit(s) were added to refs/heads/main by this push:
new 511216c Add support for AlmaLinux 8/9 and Rocky Linux 8 (#440)
511216c is described below
commit 511216ca7e74c953c10ba20ec3ed6533373e08ee
Author: Arvind Shyamsundar <[email protected]>
AuthorDate: Mon Nov 27 12:11:38 2023 -0800
Add support for AlmaLinux 8/9 and Rocky Linux 8 (#440)
* Add support for AlmaLinux and Rocky Linux in the Ansible playbooks.
* Switch the default Azure VM image to AlmaLinux 9.
* Add an Azure-specific cloud-init file for AlmaLinux 9 to install `rsync`
at first boot, as the image does not seem to install it.
* Add an Ansible task to stop and disable `firewalld` for AlmaLinux 9, to
allow RPC communication within the cluster.
* Use Python to install Ansible on the proxy for non-CentOS OS to ensure a
current version of Ansible is used. This avoids problems when the Ansible
package in the EL repo is outdated or requires a newer Python than is supported
by the given OS.
* While using Rocky Linux 9 is not blocked, there are some external issues
with the Azure Linux agent and this OS version.
* These changes have been tested on Azure with various OS images used for
the cluster nodes and the "proxy". Currently "known good" OS versions for Azure
include Fedora 39, CentOS 7.9, AlmaLinux 8/9, Rocky Linux 8.
* Update references to CentOS 7 in the context of EC2 as the default image
for EC2 had already been changed to a Fedora 35 AMI.
---
README.md | 16 ++------
ansible/roles/azure/files/cloud-init-alma9.yml | 24 ++++++++++++
ansible/roles/common/tasks/main.yml | 17 +++++---
ansible/scripts/install_ansible.sh | 18 ++++++---
conf/muchos.props.example | 9 ++---
docs/azure-image-reference.md | 54 ++++++++++++++++----------
docs/azure-ubuntu-1804.md | 14 +++----
lib/muchos/config/azure.py | 2 +-
lib/muchos/util.py | 17 ++++----
9 files changed, 105 insertions(+), 66 deletions(-)
diff --git a/README.md b/README.md
index 78144b3..9834908 100644
--- a/README.md
+++ b/README.md
@@ -110,13 +110,7 @@ set by the `cluster.username` property.
### Configuring the AMI
-You might also need to configure the `aws_ami` property in [muchos.props].
Muchos by default uses a free
-CentOS 7 image that is hosted in the AWS marketplace but managed by the
-CentOS organization. If you have never used this image in EC2 before, you will
need to go to the
-[CentOS 7 product page][centos7] to accept the software terms. If this is not
done, you will get an
-error when you try to launch your cluster. By default, the `aws_ami` property
is set to an AMI in `us-east-1`.
-You will need to changes this value if a newer image has been released or if
you are running in different region
-than `us-east-1`.
+You might also need to configure the `aws_ami` property in [muchos.props].
Muchos by default uses a Fedora 35 image for EC2. By default, the `aws_ami`
property is set to this Fedora 35 AMI in `us-east-1`. You will need to change
this value if a newer image has been released or if you are running in
different region than `us-east-1`.
### Launching the cluster
@@ -144,11 +138,8 @@ this file will be appended on each node to the
`~/.ssh/authorized_keys` file for
`cluster.username` property. You will also need to ensure you have
authenticated to Azure and set the target
subscription using the [Azure
CLI](https://docs.microsoft.com/en-us/cli/azure/manage-azure-subscriptions-azure-cli?view=azure-cli-latest).
-Muchos by default uses a CentOS 7 image that is hosted in the Azure
marketplace. The Azure Linux Agent is already
+Muchos by default uses an AlmaLinux 9 image that is hosted in the Azure
marketplace. The Azure Linux Agent is already
pre-installed on the Azure Marketplace images and is typically available from
the distribution's package repository.
-Azure requires that the publishers of the endorsed Linux distributions
regularly update their images in the Azure
-Marketplace with the latest patches and security fixes, at a quarterly or
faster cadence. Updated images in the Azure
-Marketplace are available automatically to customers as new versions of an
image SKU.
Edit the values in the sections within [muchos.props] as below
Under the `general` section, edit following values as per your configuration
@@ -168,7 +159,7 @@ Under the `azure` section, edit following values as per
your configuration:
* `azure_image_reference` allows you to specify the Azure image SKU in the
format as shown below.
```bash
offer|publisher|sku|version|image_id|
- Ex: CentOS|OpenLogic|7_9|latest||
+ Ex: almalinux-x86_64|almalinux|9-gen2|latest||
```
For more information on using other images, refer to [Azure
images](docs/azure-image-reference.md).
* `azure_proxy_image_reference` allows you to specify the Azure image SKU that
will be used for the optional proxy machine.
@@ -373,7 +364,6 @@ Muchos is powered by the following projects:
* [azure-cli] - The Azure CLI is a command-line tool for managing Azure
resources.
* [ansible-azure] - Ansible includes a suite of modules for interacting with
Azure Resource Manager.
-[centos7]: https://aws.amazon.com/marketplace/pp/B00O7WM7QW
[aws-config]:
http://docs.aws.amazon.com/cli/latest/userguide/cli-config-files.html
[awscli]: https://docs.aws.amazon.com/cli/latest/userguide/installing.html
[awscli-config]:
http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-quick-configuration
diff --git a/ansible/roles/azure/files/cloud-init-alma9.yml
b/ansible/roles/azure/files/cloud-init-alma9.yml
new file mode 100644
index 0000000..f4e5c53
--- /dev/null
+++ b/ansible/roles/azure/files/cloud-init-alma9.yml
@@ -0,0 +1,24 @@
+#cloud-config
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+#
+# NOTE: do not modify the first line in this file - it is a mandatory
+# header to designate the contents as a cloud-init configuration
+#
+# Install packages specific to AlmaLinux 9
+packages:
+ - rsync
diff --git a/ansible/roles/common/tasks/main.yml
b/ansible/roles/common/tasks/main.yml
index b844f63..431c4be 100644
--- a/ansible/roles/common/tasks/main.yml
+++ b/ansible/roles/common/tasks/main.yml
@@ -21,7 +21,7 @@
retries: 10
delay: 15
until: epelresult is not failed
- when: ansible_facts['distribution'] == "CentOS"
+ when: ansible_facts['distribution'] in ["CentOS", "AlmaLinux", "Rocky"]
- name: "install packages"
yum:
name:
@@ -50,7 +50,7 @@
delay: 15
until: yumresult_centos7 is not failed
when: (ansible_facts['distribution'] == "CentOS") and
(ansible_facts['distribution_major_version'] == "7")
-- name: "Install packages specific to CentOS 8"
+- name: "Install packages specific to AlmaLinux 8 / 9, Rocky Linux 8 / 9"
yum:
name:
- python3-policycoreutils
@@ -58,11 +58,11 @@
- collectd-write_http
- make
state: present
- register: yumresult_centos8
+ register: yumresult_os89
retries: 10
delay: 15
- until: yumresult_centos8 is not failed
- when: (ansible_facts['distribution'] == "CentOS") and
(ansible_facts['distribution_major_version'] == "8")
+ until: yumresult_os89 is not failed
+ when: (ansible_facts['distribution'] in ["AlmaLinux", "Rocky"]) and
(ansible_facts['distribution_major_version'] in ["8", "9"])
- name: "Install packages specific to Fedora"
yum:
name:
@@ -76,6 +76,13 @@
delay: 15
until: yumresult_fedora is not failed
when: ansible_facts['distribution'] == "Fedora"
+# Disable firewalld on AlmaLinux 9
+- name: "Stop and disable firewalld on AlmaLinux 9"
+ service:
+ name: "firewalld"
+ state: stopped
+ enabled: false
+ when: (ansible_facts['distribution'] == "AlmaLinux") and
(ansible_facts['distribution_major_version'] == "9")
- name: "get exact jdk folder path"
find:
file_type: directory
diff --git a/ansible/scripts/install_ansible.sh
b/ansible/scripts/install_ansible.sh
index d2d4b9a..b3c7e21 100755
--- a/ansible/scripts/install_ansible.sh
+++ b/ansible/scripts/install_ansible.sh
@@ -16,7 +16,6 @@
# limitations under the License.
#
-
bin=$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )
base_dir=$( cd "$( dirname "$bin" )" && pwd )
@@ -24,16 +23,23 @@ set -e
# enable yum epel repo
os_id=$(grep '^ID=' /etc/os-release | cut -d'=' -f2 | tr -d '"')
-if [[ $os_id = "centos" ]]; then
+if [[ $os_id = "centos" || $os_id = "almalinux" || $os_id = "rocky" ]]; then
is_installed_epel_release="rpm -q --quiet epel-release"
install_epel_release="sudo yum install -q -y epel-release"
for i in {1..10}; do ($is_installed_epel_release || $install_epel_release)
&& break || sleep 15; done
fi
# install ansible
-is_installed_ansible="rpm -q --quiet ansible"
-install_ansible="sudo yum install -q -y ansible"
-for i in {1..10}; do ($is_installed_ansible || $install_ansible) && break ||
sleep 15; done
+if [[ $os_id = "centos" ]]; then
+ is_installed_ansible="rpm -q --quiet ansible"
+ install_ansible="sudo yum install -q -y ansible"
+ for i in {1..10}; do ($is_installed_ansible || $install_ansible) && break ||
sleep 15; done
+fi
+
+if [[ $os_id = "fedora" || $os_id = "almalinux" || $os_id = "rocky" ]]; then
+ python3 -m ensurepip
+ pip3 install --quiet --user ansible
+fi
# setup user-specific ansible configuration
if [[ ! -h ~/.ansible.cfg ]]; then
@@ -44,6 +50,8 @@ fi
# setup ansible hosts
if [[ ! -h /etc/ansible/hosts ]]; then
+ # If Ansible was installed using Python, the /etc/ansible folder needs to be
created manually
+ sudo mkdir -p /etc/ansible
cd /etc/ansible
sudo rm -f hosts
sudo ln -s $base_dir/conf/hosts hosts
diff --git a/conf/muchos.props.example b/conf/muchos.props.example
index 6080cad..3851f2c 100644
--- a/conf/muchos.props.example
+++ b/conf/muchos.props.example
@@ -129,19 +129,18 @@ use_multiple_vmss = False
vmss_priority = None
# Azure image reference defined as a pipe-delimited string in the format
offer|publisher|sku|version|image_id|
# Please refer 'Launching an Azure cluster' section of the README before
making changes
-azure_image_reference = CentOS|OpenLogic|7_9|latest||
+azure_image_reference = almalinux-x86_64|almalinux|9-gen2|latest||
# Image payment plan information - values required only if the image requires
payment plan info.
# The format of this configuration line is plan_name|product|publisher|
azure_image_plan = |||
# Cloud init file to use when creating VMs with the above image reference.
-# Currently, a CentOS 7.9 specific cloud-init file is used. In the future,
different files can be used
-# as needed for different images like Alma / Rocky / Fedora.
-azure_image_cloud_init_file = cloud-init-centos79.yml
+# Currently, an Alma Linux 9 specific cloud-init file is used. Different files
can be used if needed.
+azure_image_cloud_init_file = cloud-init-alma9.yml
# Azure image reference defined as a pipe-delimited string in the format
offer|publisher|sku|version|image_id|
# This is the image that will be used for the proxy machine (if specified by
azure_proxy_host). If
# this is not set, then the value of azure_image_reference will be used on the
proxy.
# Please refer 'Launching an Azure cluster' section of the README before
making changes
-#azure_proxy_image_reference = CentOS|OpenLogic|7_9|latest||
+#azure_proxy_image_reference = almalinux-x86_64|almalinux|9-gen2|latest||
# Proxy image payment plan information - required only if the proxy image
requires payment plan info.
# If this is not set, the value of azure_image_plan will be used on the proxy.
# The format of this configuration line is plan_name|product|publisher|
diff --git a/docs/azure-image-reference.md b/docs/azure-image-reference.md
index 80dd61a..8d010b8 100644
--- a/docs/azure-image-reference.md
+++ b/docs/azure-image-reference.md
@@ -7,11 +7,22 @@ The sections below describe the various image related
configurations in `muchos.
## azure_image_reference
`azure_image_reference` is a pipe-delimited string in the format
`offer|publisher|sku|version|image_id|`. The trailing pipe character is
intentional.
-* For Azure Marketplace images, the values for the fields
`offer|publisher|sku|version` can be obtained from the Azure portal, or by
using the Azure CLI commands as shown later. For example, the CentOS 7.9 image
currently used as the default in Muchos is specified as:
+* For Azure Marketplace images, the values for the fields
`offer|publisher|sku|version` can be obtained from the Azure portal, or by
using the Azure CLI commands as shown later. For example, the AlmaLinux 9 image
which is used as the default for Azure clusters in Muchos is specified as:
- `CentOS|OpenLogic|7_9|latest||`
+ `azure_image_reference = almalinux-x86_64|almalinux|9-gen2|latest||`
- In the above case, since it's a marketplace image, the last value of
`image_id` is empty.
+ In the above case, since it's a marketplace image, the last value of
`image_id` is empty. In addition, an Alma Linux 9 specific cloud-init file is
specified. This cloud-init file installs `rsync` which seems to not be included
in Alma Linux 9 images:
+
+ `azure_image_cloud_init_file = cloud-init-alma9.yml`
+
+ The Rocky Linux images in Azure require plan information to be supplied.
Currently, the Rocky Linux 8 image has been verified to work correctly. For
using Rocky Linux 8 instead of Alma Linux, here is what you can use in
`muchos.props`. The image plan information is mandatory for this image:
+
+ * `azure_image_reference =
rockylinux|erockyenterprisesoftwarefoundationinc1653071250513|free|latest||`
+ * `azure_image_plan =
free|rockylinux|erockyenterprisesoftwarefoundationinc1653071250513|`
+
+ In addition, you might need to manually view and accept the terms of use for
the Rocky Linux image, before being able to use it within Muchos. See the
`azure_image_plan` section below for details.
+
+ Note: The Rocky Linux 9 image on Azure has an issue with data disks not
being correctly mapped to paths under /dev/disk/azure/scsi1. This is either an
image issue, or an issue with the Microsoft Azure Linux Guest Agent. Till the
issue is diagnosed and fixed, we do not recommend using the Azure Rocky Linux 9
image at this point in time.
* It is also possible to use a [custom Azure
image](https://learn.microsoft.com/en-us/azure/virtual-machines/linux/imaging)
with Muchos. For using an image from an Azure Compute Gallery ("Shared Image
Gallery"), the full resource ID of the image should be specified for `image_id`
and the other fields should not be specified. For example:
@@ -28,11 +39,13 @@ The sections below describe the various image related
configurations in `muchos.
## azure_image_plan
`azure_image_plan` is only needed when working with images which require
payment plan information to be supplied when a VM or VMSS is being created
using that image. The format of this configuration is
`plan_name|product|publisher|`. Plan information for the images published by a
given publisher can easily be queried by using the Azure CLI. For example, to
query the plan information for a Rocky Linux image in Azure:
- `az vm image show --urn
"erockyenterprisesoftwarefoundationinc1653071250513:rockylinux:free:latest"
--query "plan"`
+ `az vm image show --urn
"erockyenterprisesoftwarefoundationinc1653071250513:rockylinux:free:latest"
--query "plan"`
Then using that information, `azure_image_plan` can be configured as below in
muchos.props:
- `azure_image_plan =
free|rockylinux|erockyenterprisesoftwarefoundationinc1653071250513|`
+ `azure_image_plan =
free|rockylinux|erockyenterprisesoftwarefoundationinc1653071250513|`
+
+More information about purchase plans, and accepting terms, etc. is available
[here](https://learn.microsoft.com/en-us/azure/virtual-machines/linux/cli-ps-findimage#check-the-purchase-plan-information).
## azure_image_cloud_init_file
`azure_image_cloud_init_file` is used to optionally specify the name of a
cloud-init file to be used. Only specify the filename here, and make sure that
the file exists under the `ansible/roles/azure/files` directory in this repo.
@@ -48,21 +61,20 @@ By default, these configurations are commented out in the
muchos.props file, and
# Other useful commands
You can run the below Azure CLI command to determine the list of SKU's
available for a given product and publisher in a given region:
-```bash
-az vm image list-skus -l <region> -f AlmaLinux -p AlmaLinux -o table
-```
+ `az vm image list-skus -l <region> -f AlmaLinux -p AlmaLinux -o table`
+
For illustration, provided a sample output that displays the sku list from
`westus2` region. The sku name `8-gen1, 8-gen2` refer to [Azure VMs
generations](https://learn.microsoft.com/en-us/azure/virtual-machines/generation-2).
-```bash
-$ az vm image list-skus -l westus2 -f AlmaLinux -p AlmaLinux -o table
-Location Name
----------- --------
-westus2 8-gen1
-westus2 8-gen2
-westus2 8_4
-westus2 8_4-gen2
-westus2 8_5
-westus2 8_5-gen2
-westus2 9-gen1
-westus2 9-gen2
-```
+ ```bash
+ $ az vm image list-skus -l westus2 -f AlmaLinux -p AlmaLinux -o table
+ Location Name
+ ---------- --------
+ westus2 8-gen1
+ westus2 8-gen2
+ westus2 8_4
+ westus2 8_4-gen2
+ westus2 8_5
+ westus2 8_5-gen2
+ westus2 9-gen1
+ westus2 9-gen2
+ ```
diff --git a/docs/azure-ubuntu-1804.md b/docs/azure-ubuntu-1804.md
index 6b60a2d..17d75cf 100644
--- a/docs/azure-ubuntu-1804.md
+++ b/docs/azure-ubuntu-1804.md
@@ -1,9 +1,9 @@
Tips for running Muchos for Azure under Ubuntu 18.04
----------------------------------------------------
-Muchos sets up a Centos cluster by default, but it does not have to run in
Centos. If you
-wish to run Muchos under Ubuntu 18.04 and have it set up an Azure cluster, then
-the following steps can get you on your way.
+For Azure, Muchos sets up an AlmaLinux 9 based cluster by default, but Muchos
itself can
+be run on other flavors of Linux. If you wish to run Muchos under Ubuntu 18.04
and have it
+set up an Azure cluster, then the following steps can get you on your way.
```bash
# Install Azure CLI. See the Azure documentation.
@@ -17,10 +17,10 @@ sudo apt install python3-pip
sudo pip3 install -r lib/requirements.txt
# Current versions of Ansible separate out the Azure-specific modules into a
# separate "collection". To install that, and associated pre-requisites, a
helper
-# script has been provided. Please be sure to execute this script:
+# script has been provided. Please be sure to execute these scripts:
./scripts/install-ansible-for-azure
+./scripts/install-ansible-collections
```
-A virtual python environment is not needed in Ubuntu. The instructions that
-mention that are targeted for Centos 7. The version of Python 3 and pip3 that
-ship with Ubuntu 18.04 suffice.
+A virtual python environment is not needed in Ubuntu, but can be useful
considering
+the variety of dependencies installed locally for Azure.
diff --git a/lib/muchos/config/azure.py b/lib/muchos/config/azure.py
index 652636a..1c5fd20 100644
--- a/lib/muchos/config/azure.py
+++ b/lib/muchos/config/azure.py
@@ -202,7 +202,7 @@ class AzureDeployConfig(BaseConfig):
return self.get("azure", "azure_fileshare_password")
@ansible_host_var
- @default("CentOS|OpenLogic|7_9|latest|")
+ @default("almalinux-x86_64|almalinux|9-gen2|latest||")
def azure_image_reference(self):
return self.get("azure", "azure_image_reference")
diff --git a/lib/muchos/util.py b/lib/muchos/util.py
index 8022d77..a2d30c1 100644
--- a/lib/muchos/util.py
+++ b/lib/muchos/util.py
@@ -32,15 +32,14 @@ class EC2Type:
self.has_nvme = has_nvme
-AMI_HELP_MSG = """PLEASE NOTE - If you have accepted the software terms for
CentOS 7 and still get an error,
-this could be due to CentOS releasing new images of CentOS 7. When this
occurs, the old images
-are no longer available to new users. If you think this is the case, go to
the CentOS 7 product
-page on AWS Marketplace at the URL below to find the latest AMI:
-
- https://aws.amazon.com/marketplace/pp/B00O7WM7QW
-
-On the product page, find the latest AMI ID for your EC2 region. This should
be used to set the 'aws_ami'
-property in your muchos.props. After setting the 'aws_ami' property, run the
launch command again.
+AMI_HELP_MSG = """PLEASE NOTE - If you have accepted the software terms for
the selected AMI and still get an error,
+this could be due to the publisher releasing new images of that OS. When this
occurs, the old images
+are no longer available to new users. You will then need to visit the
publisher's page and find the latest AMI ID
+for your EC2 region. This should be used to set the 'aws_ami' property in your
muchos.props.
+After setting the 'aws_ami' property, run the launch command again.
+
+The Fedora Project also makes available AMIs for their Fedora Cloud releases.
The list of Fedora Cloud AMIs
+can be viewed at https://fedoraproject.org/cloud/download and clicking on the
AWS link."
""" # noqa
instance_types = {
