Overhaul Cilium manifests to match the newer versions (#8717)

* [cilium] Separate templates for cilium, cilium-operator, and hubble installations

Signed-off-by: necatican <necaticanyildirim@gmail.com>

* [cilium] Update cilium-operator templates

Signed-off-by: necatican <necaticanyildirim@gmail.com>

* [cilium] Allow using custom args and mounting extra volumes for the Cilium Operator

Signed-off-by: necatican <necaticanyildirim@gmail.com>

* [cilium] Update the cilium configmap to filter out the deprecated variables, and add the new variables

Signed-off-by: necatican <necaticanyildirim@gmail.com>

* [cilium] Add an option to use Wireguard encryption on Cilium 1.10 and up

Signed-off-by: necatican <necaticanyildirim@gmail.com>

* [cilium] Update cilium-agent templates

Signed-off-by: necatican <necaticanyildirim@gmail.com>

* [cilium] Bump Cilium version to 1.11.3

Signed-off-by: necatican <necaticanyildirim@gmail.com>

docs/cilium.md

@@ -12,10 +12,51 @@ the external loadbalancer (even from a node in the control plane)
 and if there is no external load balancer It will ignore any local load
 balancer deployed by Kubespray and **only contacts the first master**.
 
+## Cilium Operator
+
+Unlike some operators, Cilium Operator does not exist for installation purposes.
+> The Cilium Operator is responsible for managing duties in the cluster which should logically be handled once for the entire cluster, rather than once for each node in the cluster.
+
+### Adding custom flags to the Cilium Operator
+
+You can set additional cilium-operator container arguments using `cilium_operator_custom_args`.
+This is an advanced option, and you should only use it if you know what you are doing.
+
+Accepts an array or a string.
+
+```yml
+cilium_operator_custom_args: ["--foo=bar", "--baz=qux"]
+```
+
+or
+
+```yml
+cilium_operator_custom_args: "--foo=bar"
+```
+
+You do not need to add a custom flag to enable debugging. Instead, feel free to use the `CILIUM_DEBUG` variable.
+
+### Adding extra volumes and mounting them
+
+You can use `cilium_operator_extra_volumes` to add extra volumes to the Cilium Operator, and use `cilium_operator_extra_volume_mounts` to mount those volumes.
+This is an advanced option, and you should only use it if you know what you are doing.
+
+```yml
+cilium_operator_extra_volumes:
+  - configMap:
+      name: foo
+    name: foo-mount-path
+
+cilium_operator_extra_volume_mounts:
+  - mountPath: /tmp/foo/bar
+    name: foo-mount-path
+    readOnly: true
+```
+
 ## Choose Cilium version
 
 ```yml
-cilium_version: v1.11.0
+cilium_version: v1.11.3
 ```
 
 ## Add variable to config
@@ -39,6 +80,47 @@ Cilium currently supports two different identity allocation modes:
   - These can be queried with `kubectl get ciliumid`
 - "kvstore" stores identities in an etcd kvstore.
 
+## Enable Transparent Encryption
+
+Cilium supports the transparent encryption of Cilium-managed host traffic and
+traffic between Cilium-managed endpoints either using IPsec or Wireguard.
+
+Wireguard option is only available in Cilium 1.10.0 and newer.
+
+### IPsec Encryption
+
+For further information, make sure to check the official [Cilium documentation.](https://docs.cilium.io/en/stable/gettingstarted/encryption-ipsec/)
+
+To enable IPsec encryption, you just need to set three variables.
+
+```yml
+cilium_encryption_enabled: true
+cilium_encryption_type: "ipsec"
+```
+
+The third variable is `cilium_ipsec_key.` You need to create a secret key string for this variable.
+Kubespray does not automate this process.
+Cilium documentation currently recommends creating a key using the following command:
+
+```shell
+echo "3 rfc4106(gcm(aes)) $(echo $(dd if=/dev/urandom count=20 bs=1 2> /dev/null | xxd -p -c 64)) 128"
+```
+
+Note that Kubespray handles secret creation. So you only need to pass the key as the `cilium_ipsec_key` variable.
+
+### Wireguard Encryption
+
+For further information, make sure to check the official [Cilium documentation.](https://docs.cilium.io/en/stable/gettingstarted/encryption-wireguard/)
+
+To enable Wireguard encryption, you just need to set two variables.
+
+```yml
+cilium_encryption_enabled: true
+cilium_encryption_type: "wireguard"
+```
+
+Kubespray currently supports Linux distributions with Wireguard Kernel mode on Linux 5.6 and newer.
+
 ## Install Cilium Hubble
 
 k8s-net-cilium.yml:
@@ -68,6 +150,6 @@ cilium_hubble_metrics:
   - flow
   - icmp
   - http
-```  
+```
 
 [More](https://docs.cilium.io/en/v1.9/operations/metrics/#hubble-exported-metrics)