mirror of
https://github.com/kubernetes-sigs/kubespray.git
synced 2026-10-29 22:03:22 -02:30
Kay Yan
b7d99749b8
* preinstall: generalize minimum-value semantics for managed sysctls Move the hardcoded sysctl floor map out of 0080-system-configurations.yml into a default variable, and introduce additional_sysctl_min so users can declare their own sysctl floors. Floor semantics only apply a managed value when the node's current value is lower; pre-tuned images (Bottlerocket, EKS AMIs, OpenShift RHCOS, hardened distros) and newer kernels that already set higher values are left alone. Default floors are aligned with prior art from the OpenShift Node Tuning Operator (TuneD '=>' floor syntax in the openshift / openshift-node profiles) and kops. None of the floor keys are part of the kubelet protect-kernel-defaults validation list, so the floor mechanism cannot interfere with kubelet startup. Keys not present on the running kernel/distro are skipped without failing. additional_sysctl (exact-value semantics) is unchanged for backward compatibility. Signed-off-by: Kay Yan <kay.yan@daocloud.io> * preinstall: refine sysctl floor handling Keep built-in floor values internal while preserving user overrides through additional_sysctl_min. Use sysctl --ignore for unknown keys and filter empty results without masking other read errors. Signed-off-by: Kay Yan <kay.yan@daocloud.io> --------- Signed-off-by: Kay Yan <kay.yan@daocloud.io>
70 lines
3.0 KiB
Markdown
70 lines
3.0 KiB
Markdown
Large deployments of K8s
|
|
========================
|
|
|
|
For a large scaled deployments, consider the following configuration changes:
|
|
|
|
* Tune [ansible settings](https://docs.ansible.com/ansible/latest/intro_configuration.html)
|
|
for `forks` and `timeout` vars to fit large numbers of nodes being deployed.
|
|
|
|
* Override containers' `foo_image_repo` vars to point to intranet registry.
|
|
|
|
* Override the ``download_run_once: true`` and/or ``download_localhost: true``.
|
|
See [Downloading binaries and containers](/docs/advanced/downloads.md) for details.
|
|
|
|
* Adjust the `retry_stagger` global var as appropriate. It should provide sane
|
|
load on a delegate (the first K8s control plane node) then retrying failed
|
|
push or download operations.
|
|
|
|
* Tune parameters for DNS related applications
|
|
Those are ``dns_replicas``, ``dns_cpu_limit``,
|
|
``dns_cpu_requests``, ``dns_memory_limit``, ``dns_memory_requests``.
|
|
Please note that limits must always be greater than or equal to requests.
|
|
|
|
* Tune CPU/memory limits and requests. Those are located in roles' defaults
|
|
and named like ``foo_memory_limit``, ``foo_memory_requests`` and
|
|
``foo_cpu_limit``, ``foo_cpu_requests``. Note that 'Mi' memory units for K8s
|
|
will be submitted as 'M', if applied for ``docker run``, and cpu K8s units
|
|
will end up with the 'm' skipped for docker as well. This is required as
|
|
docker does not understand k8s units well.
|
|
|
|
* Tune ``kubelet_status_update_frequency`` to increase reliability of kubelet.
|
|
``kube_controller_node_monitor_grace_period``,
|
|
``kube_controller_node_monitor_period``,
|
|
``kube_apiserver_pod_eviction_not_ready_timeout_seconds`` &
|
|
``kube_apiserver_pod_eviction_unreachable_timeout_seconds`` for better Kubernetes reliability.
|
|
Check out [Kubernetes Reliability](/docs/advanced/kubernetes-reliability.md)
|
|
|
|
* Tune network prefix sizes. Those are ``kube_network_node_prefix``,
|
|
``kube_service_addresses`` and ``kube_pods_subnet``.
|
|
|
|
* Add calico_rr nodes if you are deploying with Calico or Canal. Nodes recover
|
|
from host/network interruption much quicker with calico_rr.
|
|
|
|
* Check out the
|
|
[Inventory](/docs/getting_started/getting-started.md#building-your-own-inventory)
|
|
section of the Getting started guide for tips on creating a large scale
|
|
Ansible inventory.
|
|
|
|
* Override the ``etcd_events_cluster_setup: true`` store events in a separate
|
|
dedicated etcd instance.
|
|
|
|
For example, when deploying 200 nodes, you may want to run ansible with
|
|
``--forks=50``, ``--timeout=600`` and define the ``retry_stagger: 60``.
|
|
|
|
Sysctl tuning with floor semantics
|
|
----------------------------------
|
|
|
|
Built-in sysctl floors are only applied when the node's current value is
|
|
lower, so pre-tuned images and newer kernels are left alone. Use
|
|
`additional_sysctl_min` to add or override floors (set a key to `0` to
|
|
disable a default); use `additional_sysctl` for exact-value semantics.
|
|
|
|
```yaml
|
|
additional_sysctl_min:
|
|
net.core.somaxconn: 65535
|
|
kernel.pid_max: 0
|
|
```
|
|
|
|
See `roles/kubespray_defaults/vars/main/main.yml` for the built-in floor
|
|
set.
|