Merge pull request #938 from ansible/kubernetes_install_support

Kubernetes install support

INSTALL.md

@@ -13,24 +13,30 @@ This document provides a guide for installing AWX.
   - [Choose a deployment platform](#choose-a-deployment-platform)
   - [Official vs Building Images](#official-vs-building-images)
 - [OpenShift](#openshift)
-  - [Prerequisites](#prerequisites)
+  - [Prerequisites](#prerequisites-1)
     - [Deploying to Minishift](#deploying-to-minishift)
   - [Pre-build steps](#pre-build-steps)
   - [PostgreSQL](#postgresql)
   - [Start the build](#start-the-build)
   - [Post build](#post-build)
   - [Accessing AWX](#accessing-awx)
-- [Docker](#docker)
+- [Kubernetes](#kubernetes)
   - [Prerequisites](#prerequisites-2)
   - [Pre-build steps](#pre-build-steps-1)
+  - [Start the build](#start-the-build-1)
+  - [Accessingm AWX](#accessing-awx-1)
+  - [SSL Termination](#ssl-termination)
+- [Docker](#docker)
+  - [Prerequisites](#prerequisites-3)
+  - [Pre-build steps](#pre-build-steps-2)
     - [Deploying to a remote host](#deploying-to-a-remote-host)
     - [Inventory variables](#inventory-variables)
       - [Docker registry](#docker-registry)
       - [PostgreSQL](#postgresql-1)
       - [Proxy settings](#proxy-settings)
-  - [Start the build](#start-the-build-1)
+  - [Start the build](#start-the-build-2)
   - [Post build](#post-build-1)
-  - [Accessing AWX](#accessing-awx-1)
+  - [Accessing AWX](#accessing-awx-2)
 
 ## Getting started
 
@@ -63,7 +69,7 @@ The system that runs the AWX service will need to satisfy the following requirem
 - At leasts 4GB of memory
 - At least 2 cpu cores
 - At least 20GB of space
-- Running Docker or Openshift
+- Running Docker, Openshift, or Kubernetes
 
 ### AWX Tunables
 
@@ -75,7 +81,7 @@ We currently support running AWX as a containerized application using Docker ima
 
 The [installer](./installer) directory contains an [inventory](./installer/inventory) file, and a playbook, [install.yml](./installer/install.yml). You'll begin by setting variables in the inventory file according to the platform you wish to use, and then you'll start the image build and deployment process by running the playbook.
 
-In the sections below, you'll find deployment details and instructions for each platform. To deploy to Docker, view the [Docker section](#docker), and for OpenShift, view the [OpenShift section](#openshift).
+In the sections below, you'll find deployment details and instructions for each platform. To deploy to Docker, view the [Docker section](#docker), for OpenShift, view the [OpenShift section](#openshift), for Kubernetes, view the [Kubernetes section](#kubernetes).
 
 ### Official vs Building Images
 
@@ -140,7 +146,7 @@ Before starting the build process, review the [inventory](./installer/inventory)
 *docker_registry*
 
 > IP address and port, or URL, for accessing a registry that the OpenShift cluster can access. Defaults to *172.30.1.1:5000*, the internal registry delivered with Minishift. This is not needed if you are using official hosted images.
-n
+
 *docker_registry_repository*
 
 > Namespace to use when pushing and pulling images to and from the registry. Generally this will match the project name. It defaults to *awx*. This is not needed if you are using official hosted images.
@@ -267,6 +273,79 @@ The above example is taken from a Minishift instance. From a web browser, use `h
 
 Once you access the AWX server, you will be prompted with a login dialog. The default administrator username is `admin`, and the password is `password`.
 
+## Kubernetes
+
+### Prerequisites
+
+A Kubernetes deployment will require you to have access to a Kubernetes cluster as well as the following tools:
+
+- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
+- [helm](https://docs.helm.sh/using_helm/#quickstart-guide)
+
+The installation program will reference `kubectl` directly. `helm` is only necessary if you are letting the installer configure PostgreSQL for you.
+
+### Pre-build steps
+
+Before starting the build process, review the [inventory](./installer/inventory) file, and uncomment and provide values for the following variables found in the `[all:vars]` section uncommenting when necessary. Make sure the openshift and standalone docker sections are commented out:
+
+*kubernetes_context*
+
+> Prior to running the installer, make sure you've configured the context for the cluster you'll be installing to. This is how the installer knows which cluster to connect to and what authentication to use
+
+*awx_kubernetes_namespace*
+
+> Name of the Kubernetes namespace where the AWX resources will be installed. This will be created if it doesn't exist
+
+*docker_registry_*
+
+> These settings should be used if building your own base images. You'll need access to an external registry and are responsible for making sure your kube cluster can talk to it and use it. If these are undefined and the dockerhub_ configuration settings are uncommented then the images will be pulled from dockerhub instead
+
+### Start the build
+
+After making changes to the `inventory` file use `ansible-playbook` to begin the install
+
+```bash
+$ ansible-playbook -i inventory install.yml
+```
+
+### Post build
+
+After the playbook run completes, check the status of the deployment by running `kubectl get pods --namespace awx` (replace awx with the namespace you used):
+
+```bash
+# View the running pods, it may take a few minutes for everything to be marked in the Running state
+$ kubectl get pods --namespace awx
+NAME                             READY     STATUS    RESTARTS   AGE
+awx-2558692395-2r8ss             4/4       Running   0          29s
+awx-postgresql-355348841-kltkn   1/1       Running   0          1m
+```
+
+### Accessing AWX
+
+The AWX web interface is running in the AWX pod behind the `awx-web-svc` service:
+
+```bash
+# View available services
+$ kubectl get svc --namespace awx
+NAME             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
+awx-postgresql   ClusterIP   10.7.250.208   <none>        5432/TCP       2m
+awx-web-svc      NodePort    10.7.241.35    <none>        80:30177/TCP   1m
+```
+
+The deployment process creates an `Ingress` named `awx-web-svc` also. Some kubernetes cloud providers will automatically handle routing configuration when an Ingress is created others may require that you more explicitly configure it. You can see what kubernetes knows about things with:
+
+```bash
+ kubectl get ing --namespace awx
+NAME          HOSTS     ADDRESS          PORTS     AGE
+awx-web-svc   *         35.227.x.y       80        3m
+```
+
+If your provider is able to allocate an IP Address from the Ingress controller then you can navigate to the address and access the AWX interface. For some providers it can take a few minutes to allocate and make this accessible. For other providers it may require you to manually intervene.
+
+### SSL Termination
+
+Unlike Openshift's `Route` the Kubernetes `Ingress` doesn't yet handle SSL termination. As such the default configuration will only expose AWX through HTTP on port 80. You are responsible for configuring SSL support until support is added (either to Kubernetes or AWX itself).
+
 ## Docker
 
 ### Prerequisites

installer/install.yml

@@ -6,4 +6,5 @@
     - { role: check_vars }
     - { role: image_build, when: "dockerhub_base is not defined" }
     - { role: openshift, when: "openshift_host is defined" }
-    - { role: local_docker, when: "openshift_host is not defined" }
+    - { role: kubernetes, when: "kubernetes_context is defined" }
+    - { role: local_docker, when: "openshift_host is not defined and kubernetes_context is not defined" }

installer/inventory

@@ -26,6 +26,10 @@ awx_secret_key=awxsecret
 # openshift_user=developer
 # awx_node_port=30083
 
+# Kubernetes Install
+# kubernetes_context=test-cluster
+# awx_kubernetes_namespace=awx
+
 # Standalone Docker Install
 postgres_data_dir=/tmp/pgdocker
 host_port=80

installer/kubernetes/tasks/main.yml

@@ -0,0 +1,115 @@
+---
+- name: Set the Kubernetes Context
+  shell: "kubectl config set-context {{ kubernetes_context }}"
+
+- name: Get Namespace Detail
+  shell: "kubectl get namespace {{ awx_kubernetes_namespace }}"
+  register: namespace_details
+  ignore_errors: yes
+
+- name: Get Postgres Service Detail
+  shell: "kubectl describe svc awx-postgresql -n {{ awx_kubernetes_namespace }}"
+  register: postgres_svc_details
+  ignore_errors: yes
+  when: "pg_hostname is not defined or pg_hostname == ''"
+
+- name: Create AWX Kubernetes Project
+  shell: "kubectl create namespace {{ awx_kubernetes_namespace }}"
+  when: namespace_details.rc != 0
+
+# TODO: This is duplicated in the openshift role, probably needs to be moved to the image_build role
+- name: Manage AWX Container Images
+  block:
+    - name: Authenticate with Docker registry
+      docker_login:
+        registry: "{{ docker_registry }}"
+        username: "{{ docker_registry_username }}"
+        password: "{{ docker_registry_password }}"
+        reauthorize: yes
+      when: docker_registry is defined and docker_registry_password is defined
+      delegate_to: localhost
+
+    - name: Wait for Openshift
+      pause:
+        seconds: 30
+
+    - name: Tag and push web image to registry
+      docker_image:
+        name: "{{ awx_web_image }}"
+        repository: "{{ docker_registry }}/{{ docker_registry_repository }}/{{ awx_web_image }}"
+        tag: "{{ awx_version }}"
+        push: yes
+      when: docker_registry is defined
+      delegate_to: localhost
+
+    - name: Wait for the registry to settle
+      pause:
+        seconds: 10
+
+    - name: Tag and push task image to registry
+      docker_image:
+        name: "{{ awx_task_image }}"
+        repository: "{{ docker_registry }}/{{ docker_registry_repository }}/{{ awx_task_image }}"
+        tag: "{{ awx_version }}"
+        push: yes
+      when: docker_registry is defined
+      delegate_to: localhost
+
+    - name: Set full web image path
+      set_fact:
+        awx_web_kubernetes_image: "{{ awx_web_image }}:{{ awx_version }}"
+      when: awx_web_kubernetes_image is not defined
+
+    - name: Set full task image path
+      set_fact:
+        awx_task_kubernetes_image: "{{ awx_task_image }}:{{ awx_version }}"
+      when: awx_task_kubernetes_image is not defined
+  when: dockerhub_base is not defined
+
+- name: Set DockerHub Image Paths
+  set_fact:
+    awx_web_kubernetes_image: "{{ dockerhub_base }}/awx_web:{{ dockerhub_version }}"
+    awx_task_kubernetes_image: "{{ dockerhub_base }}/awx_task:{{ dockerhub_version }}"
+  when: dockerhub_base is defined
+
+- name: Deploy and Activate Postgres
+  shell: "helm install --name awx --namespace {{ awx_kubernetes_namespace }} --set postgresUser={{ pg_username }},postgresPassword={{ pg_password }},postgresDatabase={{ pg_database }},persistence.size={{ pg_volume_capacity|default('5')}}Gi stable/postgresql"
+  when: (pg_hostname is not defined or pg_hostname == '') and (postgres_svc_details is defined and postgres_svc_details.rc != 0)
+  register: kubernetes_pg_activate
+
+- name: Set postgresql hostname to helm package service
+  set_fact:
+    pg_hostname: awx-postgresql
+  when: pg_hostname is not defined or pg_hostname == ''
+
+- name: Wait for Postgres to activate
+  pause:
+    seconds: 60
+  when: kubernetes_pg_activate|changed
+
+- name: Set kubernetes base path
+  set_fact:
+    kubernetes_base_path: "{{ awx_local_base_config_path|default('/tmp') }}/awx-config"
+
+- name: Ensure directory exists
+  file:
+    path: "{{ kubernetes_base_path }}"
+    state: directory
+
+- name: Template Kubernetes AWX Config
+  template:
+    src: configmap.yml.j2
+    dest: "{{ kubernetes_base_path }}/configmap.yml"
+    mode: '0600'
+
+- name: Template Kubernetes AWX Deployment
+  template:
+    src: deployment.yml.j2
+    dest: "{{ kubernetes_base_path }}/deployment.yml"
+    mode: '0600'
+
+- name: Apply Configmap
+  shell: "kubectl apply -f {{ kubernetes_base_path }}/configmap.yml"
+
+- name: Apply Deployment
+  shell: "kubectl apply -f {{ kubernetes_base_path }}/deployment.yml"

installer/kubernetes/templates/configmap.yml.j2

@@ -0,0 +1,95 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: awx-config
+  namespace: {{ awx_kubernetes_namespace }}
+data:
+  secret_key: {{ awx_secret_key }}
+  awx_settings: |
+    import os
+    import socket
+    ADMINS = ()
+    
+    # Container environments don't like chroots
+    AWX_PROOT_ENABLED = False
+    
+    #Autoprovisioning should replace this
+    CLUSTER_HOST_ID = socket.gethostname()
+    SYSTEM_UUID = '00000000-0000-0000-0000-000000000000'
+    
+    CELERY_TASK_QUEUES += (Queue(CLUSTER_HOST_ID, Exchange(CLUSTER_HOST_ID), routing_key=CLUSTER_HOST_ID),)
+    CELERY_TASK_ROUTES['awx.main.tasks.cluster_node_heartbeat'] = {'queue': CLUSTER_HOST_ID, 'routing_key': CLUSTER_HOST_ID}
+    CELERY_TASK_ROUTES['awx.main.tasks.purge_old_stdout_files'] = {'queue': CLUSTER_HOST_ID, 'routing_key': CLUSTER_HOST_ID}
+    STATIC_ROOT = '/var/lib/awx/public/static'
+    PROJECTS_ROOT = '/var/lib/awx/projects'
+    JOBOUTPUT_ROOT = '/var/lib/awx/job_status'
+    SECRET_KEY = file('/etc/tower/SECRET_KEY', 'rb').read().strip()
+    ALLOWED_HOSTS = ['*']
+    INTERNAL_API_URL = 'http://127.0.0.1:8052'
+    AWX_TASK_ENV['HOME'] = '/var/lib/awx'
+    SERVER_EMAIL = 'root@localhost'
+    DEFAULT_FROM_EMAIL = 'webmaster@localhost'
+    EMAIL_SUBJECT_PREFIX = '[AWX] '
+    EMAIL_HOST = 'localhost'
+    EMAIL_PORT = 25
+    EMAIL_HOST_USER = ''
+    EMAIL_HOST_PASSWORD = ''
+    EMAIL_USE_TLS = False
+    
+    LOGGING['handlers']['console'] = {
+        '()': 'logging.StreamHandler',
+        'level': 'DEBUG',
+        'formatter': 'simple',
+    }
+    
+    LOGGING['loggers']['django.request']['handlers'] = ['console']
+    LOGGING['loggers']['rest_framework.request']['handlers'] = ['console']
+    LOGGING['loggers']['awx']['handlers'] = ['console']
+    LOGGING['loggers']['awx.main.commands.run_callback_receiver']['handlers'] = ['console']
+    LOGGING['loggers']['awx.main.commands.inventory_import']['handlers'] = ['console']
+    LOGGING['loggers']['awx.main.tasks']['handlers'] = ['console']
+    LOGGING['loggers']['awx.main.scheduler']['handlers'] = ['console']
+    LOGGING['loggers']['django_auth_ldap']['handlers'] = ['console']
+    LOGGING['loggers']['social']['handlers'] = ['console']
+    LOGGING['loggers']['system_tracking_migrations']['handlers'] = ['console']
+    LOGGING['loggers']['rbac_migrations']['handlers'] = ['console']
+    LOGGING['loggers']['awx.isolated.manager.playbooks']['handlers'] = ['console']
+    LOGGING['handlers']['callback_receiver'] = {'class': 'logging.NullHandler'}
+    LOGGING['handlers']['fact_receiver'] = {'class': 'logging.NullHandler'}
+    LOGGING['handlers']['task_system'] = {'class': 'logging.NullHandler'}
+    LOGGING['handlers']['tower_warnings'] = {'class': 'logging.NullHandler'}
+    LOGGING['handlers']['rbac_migrations'] = {'class': 'logging.NullHandler'}
+    LOGGING['handlers']['system_tracking_migrations'] = {'class': 'logging.NullHandler'}
+    LOGGING['handlers']['management_playbooks'] = {'class': 'logging.NullHandler'}
+    
+    DATABASES = {
+        'default': {
+            'ATOMIC_REQUESTS': True,
+            'ENGINE': 'django.db.backends.postgresql',
+            'NAME': "{{ pg_database }}",
+            'USER': "{{ pg_username }}",
+            'PASSWORD': "{{ pg_password }}",
+            'HOST': "{{ pg_hostname|default('postgresql') }}",
+            'PORT': "{{ pg_port }}",
+        }
+    }
+    CELERY_BROKER_URL = 'amqp://{}:{}@{}:{}/{}'.format(
+        "awx",
+        "abcdefg",
+        "localhost",
+        "5672",
+        "awx")
+    CHANNEL_LAYERS = {
+        'default': {'BACKEND': 'asgi_amqp.AMQPChannelLayer',
+                    'ROUTING': 'awx.main.routing.channel_routing',
+                    'CONFIG': {'url': CELERY_BROKER_URL}}
+    }
+    CACHES = {
+        'default': {
+            'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
+            'LOCATION': '{}:{}'.format("localhost", "11211")
+        },
+        'ephemeral': {
+            'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
+        },
+    }

installer/kubernetes/templates/deployment.yml.j2

@@ -0,0 +1,92 @@
+---
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+  name: awx
+  namespace: {{ awx_kubernetes_namespace }}
+spec:
+  replicas: 1
+  template:
+    metadata:
+      labels:
+        name: awx-web-deploy
+        service: django
+    spec:
+      containers:
+        - name: awx-web
+          image: {{ awx_web_kubernetes_image }}
+          ports:
+            - containerPort: 8052
+          volumeMounts:
+            - mountPath: /etc/tower
+              name: awx-application-config
+        - name: awx-celery
+          image: {{ awx_task_kubernetes_image }}
+          volumeMounts:
+            - mountPath: /etc/tower
+              name: awx-application-config
+          env:
+            - name: DATABASE_USER
+              value: {{ pg_username }}
+            - name: DATABASE_NAME
+              value: {{ pg_database }}
+            - name: DATABASE_HOST
+              value: {{ pg_hostname|default('postgresql') }}
+            - name: DATABASE_PORT
+              value: ({{ pg_port|default('5432') }})
+            - name: DATABASE_PASSWORD
+              value: {{ pg_password }}
+            - name: AWX_ADMIN_USER
+              value: {{ default_admin_user|default('admin') }}
+            - name: AWX_ADMIN_PASSWORD
+              value: {{ default_admin_password|default('password') }}
+        - name: awx-rabbit
+          image: rabbitmq:3
+          env:
+            - name: RABBITMQ_ERLANG_COOKIE
+              value: secretb
+            - name: RABBITMQ_NODENAME
+              value: rabbitmq
+            - name: RABBITMQ_DEFAULT_USER
+              value: awx
+            - name: RABBITMQ_DEFAULT_PASS
+              value: abcdefg
+            - name: RABBITMQ_DEFAULT_VHOST
+              value: awx
+        - name: awx-memcached
+          image: memcached
+      volumes:
+        - name: awx-application-config
+          configMap:
+            name: awx-config
+            items:
+              - key: awx_settings
+                path: settings.py
+              - key: secret_key
+                path: SECRET_KEY
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: awx-web-svc
+  namespace: {{ awx_kubernetes_namespace }}
+  labels:
+    name: awx-web-svc
+spec:
+  type: "NodePort"
+  ports:
+    - name: http
+      port: 80
+      targetPort: 8052
+  selector:
+    name: awx-web-deploy
+---
+apiVersion: extensions/v1beta1
+kind: Ingress
+metadata:
+  name: awx-web-svc
+  namespace: {{ awx_kubernetes_namespace }}
+spec:
+  backend:
+    serviceName: awx-web-svc
+    servicePort: 80

installer/openshift/tasks/main.yml

@@ -21,6 +21,7 @@
 - name: Mark Openshift User as Admin
   shell: "oc adm policy add-role-to-user admin {{ openshift_user }} -n {{ awx_openshift_project }}"
 
+# TODO: This is duplicated in the kubernetes role, probably needs to be moved to the image_build role
 - name: Manage AWX Container Images
   block:
     - name: Get docker registry password from oc if needed
@@ -55,7 +56,7 @@
       when: docker_registry is defined
       delegate_to: localhost
 
-    - name: Wait for openshift
+    - name: Wait for the registry to settle
       pause:
         seconds: 10
 
@@ -68,6 +69,7 @@
       when: docker_registry is defined
       delegate_to: localhost
 
+    # Note this is the one bit that is Openshift specific
     - name: Enable image stream lookups for awx images
       shell: "oc set image-lookup --all -n {{ awx_openshift_project }}"