Limit kubectl delete node to k8s nodes (#8101)

* Limit kubectl delete node to k8s nodes

This avoids the use of `kubectl delete node` when removing etcd nodes
which are not part of the cluser (separate etcd)

* Take errors into account when deleting node

There should not be error now that we're limiting the deletion to nodes
actually in the cluster

* Retrying on error

roles/remove-node/post-remove/tasks/main.yml

@@ -1,5 +1,10 @@
 ---
-- name: Delete node  # noqa 301 ignore-errors
+- name: Delete node
   command: "{{ bin_dir }}/kubectl delete node {{ kube_override_hostname|default(inventory_hostname) }}"
   delegate_to: "{{ groups['kube_control_plane']|first }}"
-  ignore_errors: true
+  when: inventory_hostname in groups['k8s_cluster']
+  retries: 10
+  # Sometimes the api-server can have a short window of indisponibility when we delete a master node
+  delay: 3
+  register: result
+  until: result is not failed