Enable AlmaLinux 8 and Rocky Linux 8 Calico CI with Python 3.12 bootstrap (#13593)

* Enable AlmaLinux 8 and Rocky Linux 8 Calico CI

Install Python 3.12 with yum before gathering facts when the interpreter
selected by Ansible is missing or older than Python 3.9, and add
bootstrap_os_install_python to control distribution-specific Python
bootstrap installation.

EL8 only ships the libselinux Python bindings for the distribution
Python 3.6, so ansible.posix.selinux fails once Kubespray runs with
Python 3.12. Keep using the module when the bindings are importable and
otherwise set the policy in /etc/selinux/config and apply the runtime
state with setenforce.

Enable rockylinux8-calico in regular PR CI, add almalinux8-calico as a
manual job, and document the RHEL 8 family lifecycle.

Signed-off-by: Kay Yan <kay.yan@daocloud.io>

* Fix EL8 containerd compatibility and SELinux runtime transitions

Signed-off-by: Kay Yan <kay.yan@daocloud.io>

---------

Signed-off-by: Kay Yan <kay.yan@daocloud.io>

.gitlab-ci/kubevirt.yml

@@ -45,6 +45,7 @@ pr:
           - fedora43-kube-router
           - opensuse-leap16-calico
           - rockylinux10-cilium
+          - rockylinux8-calico
           - rockylinux9-cilium
           - ubuntu22-calico-all-in-one
           - ubuntu22-calico-all-in-one-upgrade
@@ -114,13 +115,14 @@ pr_full:
           - ubuntu24-flannel-ha
           - ubuntu24-flannel-ha-once
 
-# Need an update of the container image to use schema v2
-# update: quay.io/kubespray/vm-amazon-linux-2:latest
+# Experimental OS coverage and images needing maintenance.
 manual:
   extends: pr_full
   parallel:
     matrix:
       - TESTCASE:
+          - almalinux8-calico
+          # Needs a schema v2 update to quay.io/kubespray/vm-amazon-linux-2:latest.
           - amazon-linux-2-all-in-one
   rules:
     - when: manual

docs/ansible/ansible.md

@@ -34,6 +34,10 @@ Based on the table below and the available python version for your ansible host
 |----------------------|-----------------------------|----------------------------|
 | >=2.21.0, <2.22.0    | 3.12-3.14                   | 3.9-3.14                   |
 
+### Target Python Bootstrap
+
+`bootstrap_os_install_python` defaults to `true` and controls Python installation for Debian/Ubuntu, Fedora, Fedora CoreOS, Flatcar, and distributions using the shared `centos.yml` bootstrap. Set it to `false` when a compatible interpreter is already provisioned, selecting it with `ansible_python_interpreter` if needed. This also skips Fedora CoreOS bootstrap preparation and reboot, but not general system package management. The shared CentOS bootstrap installs Python 3.12 using `yum` when the selected interpreter is missing or older than 3.9; configured repositories must provide that package. Other bootstrap paths retain their existing checks and package choices.
+
 ## Customize Ansible vars
 
 Kubespray expects users to use one of the following variables sources for settings and customization:

docs/developers/ci.md

@@ -6,6 +6,7 @@ To generate this Matrix run `./tests/scripts/md-table/main.py`
 
 | OS / CNI | calico | cilium | custom_cni | flannel | kube-ovn | kube-router |
 |---| --- | --- | --- | --- | --- | --- |
+almalinux8 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
 almalinux9 |  :white_check_mark: | :x: | :x: | :x: | :white_check_mark: | :x: |
 amazon |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
 debian12 |  :white_check_mark: | :white_check_mark: | :white_check_mark: | :x: | :x: | :x: |
@@ -17,6 +18,7 @@ openeuler24 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
 opensuse |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
 opensusemicroos |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
 rockylinux10 |  :white_check_mark: | :white_check_mark: | :x: | :x: | :x: | :x: |
+rockylinux8 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
 rockylinux9 |  :white_check_mark: | :white_check_mark: | :x: | :x: | :x: | :x: |
 ubuntu22 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
 ubuntu24 |  :white_check_mark: | :white_check_mark: | :x: | :white_check_mark: | :x: | :white_check_mark: |
@@ -26,6 +28,7 @@ ubuntu26 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
 
 | OS / CNI | calico | cilium | custom_cni | flannel | kube-ovn | kube-router |
 |---| --- | --- | --- | --- | --- | --- |
+almalinux8 |  :x: | :x: | :x: | :x: | :x: | :x: |
 almalinux9 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
 amazon |  :x: | :x: | :x: | :x: | :x: | :x: |
 debian12 |  :x: | :x: | :x: | :x: | :x: | :x: |
@@ -37,6 +40,7 @@ openeuler24 |  :x: | :x: | :x: | :x: | :x: | :x: |
 opensuse |  :x: | :x: | :x: | :x: | :x: | :x: |
 opensusemicroos |  :x: | :x: | :x: | :x: | :x: | :x: |
 rockylinux10 |  :x: | :x: | :x: | :x: | :x: | :x: |
+rockylinux8 |  :x: | :x: | :x: | :x: | :x: | :x: |
 rockylinux9 |  :x: | :x: | :x: | :x: | :x: | :x: |
 ubuntu22 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
 ubuntu24 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
@@ -46,6 +50,7 @@ ubuntu26 |  :x: | :x: | :x: | :x: | :x: | :x: |
 
 | OS / CNI | calico | cilium | custom_cni | flannel | kube-ovn | kube-router |
 |---| --- | --- | --- | --- | --- | --- |
+almalinux8 |  :x: | :x: | :x: | :x: | :x: | :x: |
 almalinux9 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
 amazon |  :x: | :x: | :x: | :x: | :x: | :x: |
 debian12 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
@@ -57,6 +62,7 @@ openeuler24 |  :x: | :x: | :x: | :x: | :x: | :x: |
 opensuse |  :x: | :x: | :x: | :x: | :x: | :x: |
 opensusemicroos |  :x: | :x: | :x: | :x: | :x: | :x: |
 rockylinux10 |  :x: | :x: | :x: | :x: | :x: | :x: |
+rockylinux8 |  :x: | :x: | :x: | :x: | :x: | :x: |
 rockylinux9 |  :x: | :x: | :x: | :x: | :x: | :x: |
 ubuntu22 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: |
 ubuntu24 |  :white_check_mark: | :x: | :x: | :x: | :x: | :x: |

docs/operating_systems/rhel.md

@@ -31,6 +31,26 @@ rh_subscription_sla: "Self-Support"
 
 If the RHEL hosts are already registered to a valid Red Hat support subscription via an alternative configuration management approach prior to the deployment of Kubespray, the successful RHEL `subscription-manager` status check will simply result in the RHEL subscription registration tasks being skipped.
 
+## RHEL 8 Family
+
+RHEL 8, AlmaLinux 8, and Rocky Linux 8 are not end-of-life, but they are no longer in full or active support. RHEL 8 remains in maintenance support, while AlmaLinux 8 and Rocky Linux 8 receive security maintenance, through May 31, 2029. Prefer a newer major release for new deployments; this guidance is intended primarily for existing RHEL 8 family environments. See the lifecycle documentation for [RHEL](https://access.redhat.com/support/policy/updates/errata), [AlmaLinux](https://wiki.almalinux.org/release-notes/), and [Rocky Linux](https://wiki.rockylinux.org/rocky/version/).
+
+With `bootstrap_os_install_python: true` (the default), the shared `centos.yml` bootstrap checks the selected interpreter before gathering facts, without restricting the distribution or OS version. If Python is missing or older than 3.9, it installs Python 3.12 using `yum`, preserving system Python for DNF. This requires repositories providing `python3.12`, available in AlmaLinux 8.10 and Rocky Linux 8.10. RHEL uses a separate bootstrap path and is unchanged.
+
+If the inventory pins an older interpreter, set:
+
+```yaml
+ansible_python_interpreter: /usr/bin/python3.12
+```
+
+The default containerd binaries require a newer glibc than EL8 provides. Use the static binaries on these systems:
+
+```yaml
+containerd_static_binary: true
+```
+
+Run bootstrap before `--check` on fresh hosts. Experimental CI includes `rockylinux8-calico` in the regular PR matrix and `almalinux8-calico` as a manual job. Both use static containerd binaries. SELinux, runtime, and kernel/cgroup compatibility still need validation; the kernel check and cgroup v1 exceptions are not production recommendations.
+
 ## Rocky Linux 10
 
 (Experimental in Kubespray CI)

roles/bootstrap_os/defaults/main.yml

@@ -43,6 +43,9 @@ fedora_coreos_packages:
   - containernetworking-plugins  # required by crio
 
 ## General
+# Install Python through the distribution-specific bootstrap tasks
+bootstrap_os_install_python: true
+
 # Set the hostname to inventory_hostname
 override_system_hostname: true
 

roles/bootstrap_os/tasks/centos.yml

@@ -1,4 +1,42 @@
 ---
+- name: Bootstrap Python
+  when: bootstrap_os_install_python
+  block:
+    - name: Check if Python bootstrap is needed
+      vars:
+        python_interpreter: "{{ lookup('config', 'INTERPRETER_PYTHON') | trim }}"
+        python_candidates: >-
+          {{ lookup('config', 'INTERPRETER_PYTHON_FALLBACK')
+             if python_interpreter in ['', 'auto', 'auto_silent'] else [python_interpreter] }}
+      raw: |
+        for python in {{ python_candidates | map('quote') | join(' ') }}; do
+          if command -v "$python" >/dev/null 2>&1; then
+            "$python" -c 'import sys; sys.exit(sys.version_info[:2] < (3, 9))'
+            exit $?
+          fi
+        done
+        exit 1
+      register: need_bootstrap
+      changed_when: false
+      failed_when: need_bootstrap.rc not in [0, 1]
+      check_mode: false
+
+    # Keep platform-python for the ansible-core DNF helper.
+    - name: Install Python 3.12
+      raw: >-
+        yum --assumeyes
+        {% if not skip_http_proxy_on_os_packages and http_proxy is defined %}
+        --setopt={{ ('proxy=' ~ http_proxy) | quote }}
+        {% endif %}
+        install python3.12
+      become: true
+      register: bootstrap_python
+      until: bootstrap_python is succeeded
+      retries: "{{ pkg_install_retries }}"
+      delay: "{{ retry_stagger | random + 3 }}"
+      timeout: "{{ pkg_install_timeout }}"
+      when: need_bootstrap.rc != 0
+
 - name: Gather host facts to get ansible_distribution_version ansible_distribution_major_version
   setup:
     gather_subset: '!all'
@@ -61,8 +99,6 @@
     - (ansible_facts['distribution_version'] | float) >= 7.6
     - (ansible_facts['distribution_version'] | float) < 9
 
-# CentOS ships with python installed
-
 - name: Check presence of fastestmirror.conf
   stat:
     path: /etc/yum/pluginconf.d/fastestmirror.conf

roles/bootstrap_os/tasks/debian.yml

@@ -49,4 +49,5 @@
     DEBIAN_FRONTEND=noninteractive apt-get install -y python3-minimal
   become: true
   when:
+    - bootstrap_os_install_python
     - need_bootstrap.rc != 0

roles/bootstrap_os/tasks/fedora-coreos.yml

@@ -12,24 +12,24 @@
   raw: "podman network rm podman"
   become: true
   ignore_errors: true  # noqa ignore-errors
-  when: need_bootstrap.rc != 0
+  when: bootstrap_os_install_python and need_bootstrap.rc != 0
 
 - name: Clean up possible pending packages on fedora coreos
   raw: "export http_proxy={{ http_proxy | default('') }};rpm-ostree cleanup -p }}"
   become: true
-  when: need_bootstrap.rc != 0
+  when: bootstrap_os_install_python and need_bootstrap.rc != 0
 
 - name: Install required packages on fedora coreos
   raw: "export http_proxy={{ http_proxy | default('') }};rpm-ostree install --allow-inactive {{ fedora_coreos_packages | join(' ') }}"
   become: true
-  when: need_bootstrap.rc != 0
+  when: bootstrap_os_install_python and need_bootstrap.rc != 0
 
 - name: Reboot immediately for updated ostree
   raw: "nohup bash -c 'sleep 5s && shutdown -r now'"
   become: true
   ignore_errors: true  # noqa ignore-errors
   ignore_unreachable: true
-  when: need_bootstrap.rc != 0
+  when: bootstrap_os_install_python and need_bootstrap.rc != 0
 
 - name: Wait for the reboot to complete
   wait_for_connection:
@@ -37,4 +37,4 @@
     connect_timeout: 20
     delay: 5
     sleep: 5
-  when: need_bootstrap.rc != 0
+  when: bootstrap_os_install_python and need_bootstrap.rc != 0

roles/bootstrap_os/tasks/fedora.yml

@@ -27,4 +27,5 @@
   raw: "dnf install --assumeyes python3 python3-dnf libselinux-python3"
   become: true
   when:
+    - bootstrap_os_install_python
     - need_bootstrap.rc != 0

roles/bootstrap_os/tasks/flatcar.yml

@@ -14,6 +14,7 @@
   become: true
   environment: "{{ proxy_env }}"
   when:
+    - bootstrap_os_install_python
     - need_bootstrap.rc != 0
 
 # Workaround ansible https://github.com/ansible/ansible/pull/82821

roles/kubernetes/preinstall/tasks/0079-selinux.yml

@@ -0,0 +1,60 @@
+---
+# The ansible.posix.selinux module needs the libselinux Python bindings, which
+# some distributions only package for their system Python. When Kubespray runs
+# with another interpreter (for example the Python 3.12 bootstrapped on EL8)
+# the bindings are missing, so fall back to editing the configuration file.
+- name: Check for the libselinux Python bindings
+  command: "{{ ansible_facts['python']['executable'] }} -c 'import selinux'"
+  register: selinux_bindings
+  changed_when: false
+  failed_when: false
+  check_mode: false
+
+- name: Set selinux policy
+  ansible.posix.selinux:
+    policy: targeted
+    state: "{{ selinux_state }}"
+  when: selinux_bindings.rc == 0
+
+- name: Set selinux policy without the libselinux Python bindings
+  when: selinux_bindings.rc != 0
+  block:
+    - name: Gather selinux facts
+      setup:
+        gather_subset: selinux
+
+    - name: Set selinux policy type
+      lineinfile:
+        path: /etc/selinux/config
+        regexp: '^SELINUXTYPE='
+        line: "SELINUXTYPE=targeted"
+        mode: "0644"
+        backup: "{{ leave_etc_backup_files }}"
+      when: selinux_state != 'disabled'
+
+    - name: Set selinux state
+      lineinfile:
+        path: /etc/selinux/config
+        regexp: '^SELINUX='
+        line: "SELINUX={{ selinux_state }}"
+        mode: "0644"
+        backup: "{{ leave_etc_backup_files }}"
+
+    # Disabling SELinux requires a reboot; use permissive mode until then.
+    - name: Update selinux runtime state
+      vars:
+        selinux_runtime_state: "{{ (selinux_state == 'enforcing') | ternary('enforcing', 'permissive') }}"
+      when:
+        - ansible_facts['selinux']['status'] == 'enabled'
+        - ansible_facts['selinux']['mode'] != selinux_runtime_state
+      block:
+        - name: Apply selinux state at runtime
+          command: setenforce {{ (selinux_runtime_state == 'enforcing') | ternary('1', '0') }}
+          changed_when: true
+          when: not ansible_check_mode
+
+        - name: Report selinux runtime change in check mode
+          debug:
+            msg: "SELinux runtime state would change to {{ selinux_runtime_state }}"
+          changed_when: true
+          when: ansible_check_mode

roles/kubernetes/preinstall/tasks/0080-system-configurations.yml

@@ -12,9 +12,9 @@
   register: slc
 
 - name: Set selinux policy
-  ansible.posix.selinux:
-    policy: targeted
-    state: "{{ preinstall_selinux_state }}"
+  include_tasks: 0079-selinux.yml
+  vars:
+    selinux_state: "{{ preinstall_selinux_state }}"
   when:
     - ansible_facts['os_family'] == "RedHat"
     - "'Amazon' not in ansible_facts['distribution']"

roles/kubernetes/preinstall/tasks/0081-ntp-configurations.yml

@@ -83,9 +83,9 @@
         - ansible_facts['os_family'] == "RedHat"
 
     - name: Put SELinux in permissive mode, logging actions that would be blocked.
-      ansible.posix.selinux:
-        policy: targeted
-        state: permissive
+      include_tasks: 0079-selinux.yml
+      vars:
+        selinux_state: permissive
       when:
         - ansible_facts['os_family'] == "RedHat"
         - ansible_facts.selinux.status == 'enabled'
@@ -96,9 +96,9 @@
         name: "{{ ntp_timezone }}"
 
     - name: Re-enable SELinux
-      ansible.posix.selinux:
-        policy: targeted
-        state: "{{ preinstall_selinux_state }}"
+      include_tasks: 0079-selinux.yml
+      vars:
+        selinux_state: "{{ preinstall_selinux_state }}"
       when:
         - ansible_facts['os_family'] == "RedHat"
         - ansible_facts.selinux.status == 'enabled'

tests/files/almalinux8-calico.yml

@@ -0,0 +1,15 @@
+---
+# Instance settings
+cloud_image: almalinux-8
+vm_memory: 3072
+
+# Exercise Python installation before gathering facts.
+bootstrap_os_install_python: true
+
+# EL8 glibc is older than required by the dynamically linked containerd binaries.
+containerd_static_binary: true
+
+# Experimental EL8 coverage with kernel 4.18 and cgroup v1.
+kubeadm_ignore_preflight_errors:
+  - SystemVerification
+kubelet_fail_cgroup_v1: false

tests/files/rockylinux8-calico.yml

@@ -0,0 +1,16 @@
+---
+# Instance settings
+cloud_image: rockylinux-8
+vm_memory: 3072
+
+# Kubespray settings
+bootstrap_os_install_python: true
+# EL8 glibc is older than required by the dynamically linked containerd binaries.
+containerd_static_binary: true
+metrics_server_enabled: true
+loadbalancer_apiserver_type: haproxy
+
+# Experimental EL8 coverage with kernel 4.18 and cgroup v1.
+kubeadm_ignore_preflight_errors:
+  - SystemVerification
+kubelet_fail_cgroup_v1: false