Commit Graph
100 Commits
Author SHA1 Message Date
Max Gautier a2a2dfa419 k8s/control-plane: cleanup excessive defaulting 2024-09-23 20:37:56 +02:00
Max Gautier baf0a331c9 Don't generate static tokens for nodes and control planes
Nodes to api-server relies by default certificates, and bootstrap
tokens, and there should be no need to generate tokens for every nodes,
even when enabling static token auth.
2024-09-23 16:58:42 +02:00
Max Gautier 1818993a8a CI: for upgrade testing, checkout old version before provisioning
Otherwise, modifying the test inventory + it's expected structure will
always fail.
2024-09-21 15:10:59 +02:00
Max Gautier 88b6f08e26 Documentation of k8s_cluster auto-defined
Also remove the group from the example inventory, since it should not be
needed anymore.
2024-09-21 14:35:36 +02:00
Max Gautier 7580e59bbf Define k8s_cluster dynamically
This allows inventories to not define the k8s_cluster group manually.
2024-09-21 14:35:35 +02:00
Max Gautier 2ec1c93897 Test group membership with group_names
Testing for group membership with group names makes Kubespray more
tolerant towards the structure of the inventory.
Where 'inventory_hostname in groups["some_group"] would fail if
"some_group" is not defined, '"some_group" in group_names' would not.
2024-09-21 14:09:09 +02:00
Max Gautier 76c42b4d3f CI: cleanup '-scale' tests infra (#11535)
There is actually no test using this since ad6fecefa8,
so there is no reason to keep that infra in our tests scripts.
2024-09-18 13:04:50 +01:00
Max Gautier b3b00775ea Remove unused test infrastructure (#11529)
These two files haven't been touched since 2016 and don't appear to be
referenced anywhere else
2024-09-18 13:04:44 +01:00
Max Gautier c3de25c782 Move the CRI endpoint setting to kubelet config (#11550)
The `--container-runtime-endpoint` kubelet argument is deprecated in
favor of the config file alternative.
2024-09-18 01:34:45 +01:00
Max Gautier 4bf2d7a2c2 Rework vagrant documentation
- Use proper syntax highlighting for config.rb examples
- Consistent shell style ($ as prompt)
- Use only one way to do things
- Remove OS specific details
2024-09-13 08:00:33 +02:00
Max Gautier 1e769b7260 Rework vagrant inventory handling + support for multiples inventories
The current way to handle a custom inventory in vagrant is a bit
hackish, copy files around and can break Vagrantfile parsing in
cornercase scenarios (removing vagrant inventories, or the inventory
copied into vagrant inventory).

Instead, simply pass additional inventories to the ansible-playbook
command lines as raw arguments with `-i`.
This also makes supporting multiples inventories trivial, so we add a
new `$inventories` variable for that purpose.
2024-09-13 08:00:32 +02:00
Max Gautier 8d8d063de4 Remove useless file (#11526)
This was somehow copied from vagrant docs, not sure why it's there...
2024-09-13 04:31:12 +01:00
Max Gautier 5ae433bf47 Remove refs to rhel7 in Vagrantfile (#11525) 2024-09-12 11:01:13 +01:00
Max Gautier d54356e113 Add testcase for kubeadm_patches 2024-09-12 10:31:10 +02:00
Max Gautier c87097fc35 Document how to use kubeadm patches 2024-09-12 10:31:09 +02:00
Max Gautier 8e254ec1e8 kubeadm: allow to provide patch inline in inventories
Specifying one directory for kubeadm patches is not ideal:
1. It does not allow working with multiples inventories easily
2. No ansible templating of the patch
3. Ansible path searching can sometimes be confusing

Instead, provide the patch directly in a variable, and add some quality
of life to handle components targeting and patch ordering more
explicitly (`target` and `type` which are translated to the kubeadm
scheme which is based on the file name)
2024-09-10 20:37:14 +02:00
Max Gautier a8b66fd207 Deduplicate kubeadm_patches tasks
kubernetes/control-plane and kubernetes/kubeadm roles both push kubeadm
patches in the same way.
Extract that code and make it a dependency of both.
This is safe because it's only configuration for kubeadm, which only
takes effect when kubeadm is run.
2024-09-10 17:39:31 +02:00
Max Gautier 230cb37626 download_hash: document usage 2024-09-08 14:27:03 +02:00
Max Gautier dec4e711d1 download_hash: cache request for 'multi-hash' files
This avoid re-downloading the same file for different arch and
re-parsing it
2024-09-08 14:27:03 +02:00
Max Gautier 1b1045c0e2 download_hash: use persistent session
This allows to reuse http connection and be more efficient.
From rough measuring it saves around 25-30% of execution time.
2024-09-08 14:27:02 +02:00
Max Gautier 86855be634 download_hash: document missing support 2024-09-08 14:27:02 +02:00
Max Gautier b2e64aed4b download_hash: support 'multi-hash' components 2024-09-08 14:27:01 +02:00
Max Gautier a2644c7a4f download_hash: add support for 'simple hash' components 2024-09-08 14:14:00 +02:00
Max Gautier e256f74f2a download_hash: propagate new patch versions to all archs 2024-09-08 14:11:47 +02:00
Max Gautier 2710e984c8 download_hash: argument handling with argparse
Allow the script to be called with a list of components, to only
download new versions checksums for those.
By default, we get new versions checksums for all supported (by the
script) components.
2024-09-08 11:13:25 +02:00
Max Gautier da0e445d69 download_hash.py: support for 'multi-hash' file + runc
runc upstream does not provide one hash file per assets in their
releases, but one file with all the hashes.
To handle this (and/or any arbitrary format from upstreams), add a
dictionary mapping the name of the download to a lambda function which
transform the file provided by upstream into a dictionary of hashes,
keyed by architecture.
2024-09-08 11:13:24 +02:00
Max Gautier a7616231a4 download_hash.py: generalized and data-driven
The script is currently limited to one hardcoded URL for kubernetes
related binaries, and a fixed set of architectures.

The solution is three-fold:
1. Use an url template dictionary for each download -> this allow to easily
   add support for new downloads.
2. Source the architectures to search from the existing data
3. Enumerate the existing versions in the data and start searching from
   the last one until no newer version is found (newer in the version
   order sense, irrespective of actual age)
2024-09-08 11:13:23 +02:00
Max Gautier fe60832a02 Remove kubelet_node_{custom_flags,config_extra_args}
There is no need to have an extra variables for this, just use different
values per host (using Ansible group_vars, for example)
2024-09-06 09:35:34 +02:00
Max Gautier 1bc61c9f35 Simplify kubelet-config template
Remove system|kube_master_<resource>_reserved variables.
Those variables are unnecessary because users can simply use the
variables in group_vars if they which to differentiate control plane
nodes from other nodes.

Set conservative defaults for ephemeral-storage and pids for both kube
and system reserved resources.
2024-09-06 09:33:07 +02:00
Max Gautier 872d717105 Add kube|system_reserved CI testing 2024-09-06 09:25:26 +02:00
Max Gautier 1533d40411 Fix kube_reserved_cgroups_for_service_slice
The default value is used across kubespray but only defined in
kubernetes/node.
Move it to kubespray-defaults
2024-09-06 09:25:23 +02:00
Max Gautier f9ebd45c74 boostrap-os: use import_tasks instead of symlinks (#11508)
Working symlinks are dependant on git configuration (when using the playbook as
a git repository, which is common), precisely `git config
core.symlinks`.

While this is enabled by default, some company policies will disable it.

Instead, use import_tasks which should avoid that class of bugs.
2024-09-05 08:24:49 +01:00
Max Gautier 7f527f6195 Drop support for RHEL 7 / CentOS 7 (#11246)
* Simplify docker systemd unit

systemd handles missing unit by ignoring the dependency so we don't need
to template them.

* Remove RHEL 7/CentOS 7 support

- remove ref in kubespray roles
- move CI from centos 7 to 8
- remove docs related to centos7

* Remove container-storage-setup

Only used for RHEL 7 and CentOS 7
2024-09-05 07:41:01 +01:00
Max Gautier e744a117d6 Remove systemd version + ostree check for docker TasksMax (#11493)
systemd ignores unknown keys (with a warning) so version checking is not
necessary.
There is no rationale for excluding it from ostree systems either.
2024-09-02 13:16:57 +01:00
Max Gautier b0be5f2dad Print the name of faulty jinja templates in pre-commit (#11484) 2024-08-30 06:43:30 +01:00
Max Gautier 5fe8714f05 Adding myself (VannTen) as approver (#11483) 2024-08-29 10:30:29 +01:00
Max Gautier 776b40a329 Adjust task name since we allow empty kube_node (#11474) 2024-08-28 06:35:02 +01:00
Max Gautier a3d0ba230d Remove kubeadm_version and use kube_version instead (#11473)
We explicitly check for equality so customizing kubeadm_version does not
work at the moment.

Use only one variable instead.
2024-08-28 06:34:56 +01:00
Max Gautier 5b057c7328 Update list of admission plugins with a config file 2024-08-26 14:53:20 +02:00
Max Gautier d3402736d4 Remove special case for PodNodeSelector
This is already handled by the previous task.
2024-08-26 14:43:00 +02:00
Max Gautier 47c3949477 Change plugins_needs_config list format
Makes easier diff when adding or removing plugins.
2024-08-26 14:42:04 +02:00
Max Gautier 821dfbfdba Switch back pre-commit hook misspell to upstream (#11280)
The pull request adding the pre-commit hook config was merged.
2024-06-26 05:14:21 -07:00
Max Gautier 3082fa3d0f Allow empty kube_node group (#11248)
While uncommon, provisioning only a control plane is a valid use case,
so don't block it.
2024-05-30 03:01:38 -07:00
Max Gautier 870049523f collection support: use manifest instead of excludes
The default for galaxy. `manifest` works well enough for our case, and
this avoids maintaining a blacklist.
2024-05-29 13:57:33 +02:00
Max Gautier 37d824fd2d Update pre-commit hooks 2024-05-28 13:28:03 +02:00
Max Gautier ff48144607 pre-commit: adjust mardownlint default, md fixes
Use a style file as recommended by upstream. This makes for only one
source of truth.
Conserve previous upstream default for MD007 (upstream default changed
here https://github.com/markdownlint/markdownlint/pull/373)
2024-05-28 13:26:49 +02:00
Max Gautier 0faa805525 Remove gitlab-ci job done in pre-commit 2024-05-28 13:26:47 +02:00
Max Gautier bc21433a05 Run pre-commit hooks in dynamic pipeline
Use gitlab dynamic child pipelines feature to have one source of truth
for the pre-commit jobs, the pre-commit config file.

Use one cache per pre-commit. This should reduce the "fetching cache"
time steps in gitlab-ci, since each job will have a separate cache with
only its hook installed.
2024-05-28 13:26:46 +02:00
Max Gautier 19851bb07c collection-build-install convert to pre-commit 2024-05-28 13:26:46 +02:00
Max Gautier 7f7b65d388 Convert check_typo to pre-commit + use maintained version
client9/misspell is unmaintained, and has been forked by the golangci
team, see https://github.com/client9/misspell/issues/197#issuecomment-1596318684.

They haven't yet added a pre-commit config, so use my fork with the
pre-commit hook config until the pull request is merged.
2024-05-28 13:26:45 +02:00
Max Gautier d50f61eae5 pre-commit: apply autofixes hooks and fix the rest manually
- markdownlint (manual fix)
- end-of-file-fixer
- requirements-txt-fixer
- trailing-whitespace
2024-05-28 13:26:44 +02:00
Max Gautier 77bfb53455 Fix ci-matrix pre-commit hook
- Remove dependency of pydblite which fails to setup on recent pythons
- Discard shell script and put everything into pre-commit
2024-05-28 13:26:44 +02:00
Max Gautier 0e449ca75e pre-commit: fix hooks dependencies
- ansible-syntax-check
- tox-inventory-builder
- jinja-syntax-check
2024-05-28 13:26:43 +02:00
Max Gautier f6d9ff4196 Switch to upstream ansible-lint pre-commit hook
This way, the hook is self contained and does not depend on a previous
virtualenv installation.
2024-05-28 13:26:42 +02:00
Max Gautier 21aba10e08 Use alternate self-sufficient shellcheck precommit
This pre-commit does not require prerequisite on the host, making it
easier to run in CI workflows.
2024-05-28 13:26:42 +02:00
Max Gautier 96bb0a3e12 sidebar_gen: force C locale
If we don't, find output can change depending on system locale, which
will produce inconsistent results.
2024-05-19 12:04:07 +02:00
Max Gautier 76dae63c69 Check that PRs have correctly ran the sidebar gen 2024-05-19 11:56:30 +02:00
Max Gautier fae41172ed gen_docs_sidebar: redirect output to sidebar.md as part of the script
The script should be self-sufficient (with no need to manually specify a
redirection).
This makes it easier to run it as part of pre-commit and CI.
2024-05-19 11:54:39 +02:00
Max Gautier 73c2722d00 rockylinux vagrant boxes only boot in UEFI mode (#11192)
https://forums.rockylinux.org/t/vagrant-box-rockylinux-8-v7-0-0-with-libvirt-provider-fails-to-boot/8212
2024-05-14 09:11:12 -07:00
Max Gautier 0b2533143f Don't force python interpreter on ubuntu (#11183)
This allows local override in tasks for using virtualenvs.
2024-05-13 19:51:11 -07:00
Max Gautier 3e4ea1065a Add the official vagrant boxes for debian 11/12 (#11135) 2024-05-13 18:43:10 -07:00
Max Gautier d8a4aea9bc Revert "support CoreDNS use host network and config dns port (#10617)" (#11185)
This reverts commit bc5b38a771.
2024-05-13 14:51:11 -07:00
Max Gautier d321e42d9e CI: Factorize ansible-playbook flags (#11173)
We have inconsistent sets of options passed to the playbooks during our
CI runs.

Don't run ansible-playbook directly, instead factorize the execution in
a bash function using all the common flags.
Also remove various ENABLE_* variables and instead directly test for the
relevant conditions at execution time, as this makes it more obvious and
does not force one to go back and forth in the script.
2024-05-13 03:20:48 -07:00
Max Gautier d870a3ee4e Avoid potential loop_var shadowing (#11162)
With CentOS, kubespray currently produces the following warning:

[WARNING]: TASK: bootstrap-os : Enable Oracle Linux repo: The loop variable
'item' is already in use. You should set the `loop_var` value in the
`loop_control` option for the task to something else to avoid variable
collisions and unexpected behavior.

This could bites us in nasty ways, so fix it.
2024-05-13 03:14:30 -07:00
Max Gautier 41036e3b53 Enable opensuse in CI (#11159)
It works, so let's ensure it continues to work.
2024-05-10 10:59:43 -07:00
Max Gautier 8c12dedf05 Fix amazon linux bootstrap (#11139)
c58497cde (Refactor bootstrap-os (#10983), 2024-03-27) refactored the
boostrap-os include but didn't adapt the amazon linux tasks to the
actual ID of amazon linux ('amzn')

Re-enable the CI so we can avoid that kind of breakage.
2024-05-08 02:15:23 -07:00
Max Gautier 21e8809186 Use official rockylinux vagrant boxes + add version 9 (#11160) 2024-05-05 19:59:02 -07:00
Max Gautier 4cb688d5e4 Fix CentOS typo (#11161)
The name reported in ansible_distribution is "CentOS", so this could
break some things.
2024-05-05 19:52:18 -07:00
Max Gautier 088b1b0cec Add enabled to pkgs to handle ipvs
Some packages requirements depends on inventory variables
(`kube_proxy_mode` in that case but it could apply to others).

As the case seems pretty rare, instead of adding complexity to pkgs, we
add an escape hatch to use jinja conditions.
That should be revisited if we find ourselves shoehorning lots of logic
in this later on.
2024-04-29 21:22:23 +02:00
Max Gautier 11f35e462c Convert common packages to use the new tooling
The empty dict means that packages will always be installed on the host.
2024-04-29 21:22:22 +02:00
Max Gautier da3ff1cc11 Convert OS specific packages to new format
Uses the logic introduced in the previous patch to convert all
kubernetes/preinstall/vars/* os specific files to the `pkgs`
dictionary.

Some niceties for devs:
- always validate the `pkgs` variable to catch mistakes in CI.
- ensure that `pkgs` is always sorted. This makes it easier to find the
  packages you're looking for.
2024-04-29 21:22:21 +02:00
Max Gautier 663fcd104c Filter packages installation by OS and by group
Adds infrastructure to install OS packages depending not only on OS
(family, versions, etc) but on groups.
All the informations related to a particular package should reside in
the `pkgs` dictionnary, which takes inspiration from the `downloads`
dictionary structure.
2024-04-29 21:22:20 +02:00
Max Gautier a2019c1c24 Add a JSON schema describing the packages install structure
Since the structure we're setting in place for installing packages has
some complexity, add a JSON schema to avoid frustrating errors when
modifying the informations (adding/removing packages install).
2024-04-29 21:22:19 +02:00
Max Gautier 3a43ac4506 Don't special case openssl for tumbleweed
openssl 1.1.1 is EOL since 12 September 2023, so just use the default
version on tumbleweed like we do on other distributions.
2024-04-29 21:22:18 +02:00
Max Gautier f91e00a61b preinstall: Move ipvs packages into defaults 2024-04-29 21:22:15 +02:00
Max Gautier 08a7010e80 Revert "Only download kubeadm images where needed (#10899)" (#11105)
This reverts commit 4b0a134bc9.

The mentionned PR break scale.yml. This goes back to the status quo
until a proper fix can be provided, at which point we'll reapply the
PR.
2024-04-29 01:59:51 -07:00
Max Gautier c6758fe544 Cleanup of kubernetes/preinstall (#11010)
* Move fedora ansible python install to bootstrap-os

* /bin/dir is set in bootstrap-os

* Removing ansible_os_family workarounds

Support for these distributions was merged in Ansible, no need to
override it ourselves now.
https://github.com/ansible/ansible/pull/69324 openEuler
https://github.com/ansible/ansible/pull/77275/ UnionTech OS Server 20
https://github.com/ansible/ansible/pull/78232/ Kylin

* Don't unconditionnaly set VARIANT_ID=coreos in os-release

WTF, this is so wrong.
Furthermore, is_fedora_coreos is already handled in boostrap-os

* Handle Clearlinux generically

Followup of 4eec302e86 (since we're using
package module anyway, let's get rid of the custom task)
2024-03-28 15:17:52 -07:00
Max Gautier c58497cde9 Refactor bootstrap-os (#10983)
* Remove leftover files for Coreos

Coreos was replaced by flatcar in 058438a25 but the file was copied
instead of moved.

* Remove workarounds for resolved ansible issues

* boostrap: Use first_found to include per distro

Using directly ID and VARIANT_ID with first_found allow for less manual
includes.
Distro "families" are simply handled by symlinks.

* boostrap: don't set ansible_python_interpreter

- Allows users to override the chosen python_interpreter with group_vars
  easily (group_vars have lesser precedence than facts)
- Allows us to use vars at the task scope to use a virtual env

Ansible python discovery has improved, so those workarounds should not
be necessary anymore.
Special workaround for Flatcar, due to upstream ansible not willing to
support it.
2024-03-27 05:58:53 -07:00
Max Gautier ef95eb078a Cleanup unused CI tooling (#11014) 2024-03-15 03:57:27 -07:00
Max Gautier 7ddc175b70 Show log on vagrant (#10958)
Should make it easier to understand what's going on when testing locally
and in CI.
2024-03-14 07:37:23 -07:00
Max Gautier 3305383873 Fix ansible python version range (#11009) 2024-03-14 05:54:31 -07:00
Max Gautier 7f6ca804a1 Upgrade ansible-core to 2.16.4 (#10984)
* upgrade ansible version

Needed for with_first_found to work correctly:
https://github.com/ansible/ansible/issues/70772 fixed in 2.16

* Remove unused google cloud cloud_playbook

* Fix dpkg_selection on non-existing packages

Needed since ansible-core>2.16, see:
https://github.com/ansible/ansible/commit/f10d11bcdc54c9b7edc0111eb38c59a88e396d0a
2024-03-14 02:12:45 -07:00
Max Gautier 71fa66c08d Delete old leftover script (#10996) 2024-03-11 13:28:00 -07:00
Max Gautier 12c8d0456f Move perma failing jobs in CI to manual run (#10886)
Should free CI resources
2024-02-19 10:00:39 -08:00
Max Gautier 667bb2c913 Remove holmsten from approver (contrib/terraform) (#10936)
As they're no longer in kubernetes-sigs org.
2024-02-19 09:46:29 -08:00
Max Gautier d40b073f97 Add extra_vars support to vagrant setup (#10932) 2024-02-19 02:58:20 -08:00
Max Gautier 4a259ee3f0 Remove mirwan from approvers (#10930)
Since they are no longer in the kubernetes-sigs organization, they can
no longer be assigned PR to approve.
2024-02-19 02:53:00 -08:00
Max Gautier c13b21e830 Explicit private/public nature of *ip vars (#10904) 2024-02-19 02:00:26 -08:00
Max Gautier 65b0604db7 download: Remove deleted kubeadm config field (#10931) 2024-02-16 05:08:43 -08:00
Max Gautier b60220c597 Update almalinux-8 base image to 8.9 (#10918)
Should fix GPG errors in CI
2024-02-16 01:19:22 -08:00
bf42ccee4e Fix ingress-nginx controller election (#10913)
Under the original code, leader election failed for ingress controllers
as a result of mismatch between election-id in the controller config,
and the resourceName in the relevant rule of role 'ingress-nginx'.
This appeared in the controller logs.

To fix the issue, a command-line option was added to container
execution (--election-id=...).

Now, the election-id agrees with the resourceName provided in
the role-ingress-nginx.yml file. A comment in that file was
changed to reflect the new logic.

Co-authored-by: Vasilis Samoladas <vsam@softnet.tuc.gr>
Co-authored-by: Mohamed Omar Zaian <mohamedzaian@gmail.com>
2024-02-12 02:58:45 -08:00
Max Gautier ffda3656d1 Enable containerd 'discard_unpacked_layers' by default (#10905)
* containerd: Remove redundant 'default' filters

* containerd: enable 'discard_unpacked_layers' by default

This should help with containerd disk usage
2024-02-09 06:33:16 -08:00
Max Gautier f5474ec6cc Don't try to set permissions recursively on cache+staging directory (#10900)
This should avoid permissions problems when the user creating the
directory and the user creating the content are different (when
containers images are saved by root for instances, because the user
can't use the container runtime).
2024-02-09 06:04:28 -08:00
Max Gautier ad9f194c24 Adjust galaxy.yml (#10906) 2024-02-09 05:40:19 -08:00
Max Gautier ef7197f925 Use run_once for checking ansible version instead of localhost (#10908)
The `assert` module does not connect to the host anyway, and this avoids
skipping the assert when running with --limit.
2024-02-09 02:34:28 -08:00
Max Gautier 9648300994 Add test for coredns_dual in CI (#10903) 2024-02-08 02:52:10 -08:00
Max Gautier 4b0a134bc9 Only download kubeadm images where needed (#10899)
* Refactor of kubeadm images listing

Instead of setting multiples facts, we directly create the dict we need from
kubeadm output.

* Remove useless 'default' filters in roles/download

* Only download kubeadm images where needed
2024-02-08 02:14:45 -08:00
Max Gautier 6f419aa18e Revert "implement download mirrors support (#8474)" (#10884)
This reverts commit c6e5314fab.

There is no user of the download mirrors support in kubespray, for a
long time.
2024-02-06 00:48:29 -08:00
Max Gautier de4d6a69ee Make more fields mandatory in bug report template (#10883)
-> to make people actually give us the information we need
2024-02-05 20:34:19 -08:00
Max Gautier f2bdd4bb2f Fix logical error when checking for boostrap-os (#10867)
Also remove some clutter along the way.
2024-02-05 07:58:55 -08:00