Merge pull request #11187 from shanemcd/bump-19.4.0

AWX 19.4.0

.dockerignore

@@ -1,2 +1,2 @@
-.git
 awx/ui/node_modules
+Dockerfile

.env

@@ -1,3 +1,3 @@
 PYTHONUNBUFFERED=true
 SELENIUM_DOCKER_TAG=latest
-
+COMPOSE_PROJECT_NAME=tools

.github/CODEOWNERS

@@ -0,0 +1 @@
+workflows/e2e_test.yml @tiagodread @shanemcd @jakemcdermott

.github/ISSUE_TEMPLATE.md

@@ -23,15 +23,16 @@ https://www.ansible.com/security
 
 ##### ENVIRONMENT
 * AWX version: X.Y.Z
-* AWX install method: openshift, minishift, docker on linux, docker for mac, boot2docker
-* Ansible version:  X.Y.Z
+* AWX install method: operator, developer environment
+* AWX deployment target: openshift, kubernetes, minikube
 * Operating System:
 * Web Browser:
 
 ##### STEPS TO REPRODUCE
 
-<!-- For bugs, please show exactly how to reproduce the problem. For new
-features, show how the feature would be used.  -->
+<!-- For new features, show how the feature would be used. For bugs, please show
+exactly how to reproduce the problem. Ideally, provide all steps and data needed
+to recreate the bug from a new awx install. -->
 
 ##### EXPECTED RESULTS
 

.github/ISSUE_TEMPLATE/bug_report.md

@@ -1,35 +0,0 @@
----
-name: "\U0001F41B Bug report"
-about: Create a report to help us improve
-
----
-
-##### ISSUE TYPE
- - Bug Report
-
-##### SUMMARY
-<!-- Briefly describe the problem. -->
-
-##### ENVIRONMENT
-* AWX version: X.Y.Z
-* AWX install method: openshift, minishift, docker on linux, docker for mac, boot2docker
-* Ansible version:  X.Y.Z
-* Operating System:
-* Web Browser:
-
-##### STEPS TO REPRODUCE
-
-<!-- Please describe exactly how to reproduce the problem. -->
-
-##### EXPECTED RESULTS
-
-<!-- What did you expect to happen when running the steps above? -->
-
-##### ACTUAL RESULTS
-
-<!-- What actually happened? -->
-
-##### ADDITIONAL INFORMATION
-
-<!-- Include any links to sosreport, database dumps, screenshots or other
-information. -->

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -0,0 +1,135 @@
+---
+name: Bug Report
+description: Create a report to help us improve
+labels:
+  - bug
+body:
+  - type: markdown
+    attributes:
+      value: |
+        Issues are for **concrete, actionable bugs and feature requests** only. For debugging help or technical support, please use:
+        - The #ansible-awx channel on irc.libera.chat
+        - https://groups.google.com/forum/#!forum/awx-project
+
+  - type: checkboxes
+    id: terms
+    attributes:
+      label: Please confirm the following
+      options:
+        - label: I agree to follow this project's [code of conduct](http://docs.ansible.com/ansible/latest/community/code_of_conduct.html).
+          required: true
+        - label: I have checked the [current issues](https://github.com/ansible/awx/issues) for duplicates.
+          required: true
+        - label: I understand that AWX is open source software provided for free and that I am not entitled to status updates or other assurances.
+          required: true
+
+  - type: textarea
+    id: summary
+    attributes:
+      label: Summary
+      description: Briefly describe the problem.
+    validations:
+      required: false
+
+  - type: input
+    id: awx-version
+    attributes:
+      label: AWX version
+      description: What version of AWX are you running?
+    validations:
+      required: true
+
+  - type: dropdown
+    id: awx-install-method
+    attributes:
+      label: Installation method
+      description: How did you install AWX?
+      multiple: false
+      options:
+        - kubernetes
+        - minikube
+        - openshift
+        - minishift
+        - docker on linux
+        - docker for mac
+        - boot2docker
+    validations:
+      required: true
+
+  - type: dropdown
+    id: modified-architecture
+    attributes:
+      label: Modifications
+      description: >-
+       Have you modified the installation, deployment topology, or container images in any way? If yes, please
+       explain in the "additional information" field at the bottom of the form.
+      multiple: false
+      options:
+        - "no"
+        - "yes"
+    validations:
+      required: true
+
+  - type: input
+    id: ansible-version
+    attributes:
+      label: Ansible version
+      description: What version of Ansible are you running?
+    validations:
+      required: false
+
+  - type: input
+    id: operating-system
+    attributes:
+      label: Operating system
+      description: What operating system are you using?
+    validations:
+      required: false
+
+  - type: dropdown
+    id: browsers
+    attributes:
+      label: Web browser
+      description: Which browsers are affected?
+      multiple: true
+      options:
+        - Firefox
+        - Chrome
+        - Safari
+        - Edge
+    validations:
+      required: false
+
+  - type: textarea
+    id: steps-to-reproduce
+    attributes:
+      label: Steps to reproduce
+      description: >-
+        Starting from a new installation of the system, describe exactly how a developer or quality engineer can reproduce the bug
+        on infrastructure that isn't yours. Include any and all resources created, input values, test users, roles assigned, playbooks used, etc.
+    validations:
+      required: true
+
+  - type: textarea
+    id: expected-results
+    attributes:
+      label: Expected results
+      description: What did you expect to happpen when running the steps above?
+    validations:
+      required: true
+
+  - type: textarea
+    id: actual-results
+    attributes:
+      label: Actual results
+      description: What actually happened?
+    validations:
+      required: true
+
+  - type: textarea
+    id: additional-information
+    attributes:
+      label: Additional information
+      description: Include any relevant log output, links to sosreport, database dumps, screenshots, or other information.
+    validations:
+      required: false

.github/ISSUE_TEMPLATE/feature_request.md

@@ -3,6 +3,12 @@ name: "✨ Feature request"
 about: Suggest an idea for this project
 
 ---
+<!-- Issues are for **concrete, actionable bugs and feature requests** only - if you're just asking for debugging help or technical support, please use:
+
+- http://web.libera.chat/?channels=#ansible-awx
+- https://groups.google.com/forum/#!forum/awx-project
+
+We have to limit this because of limited volunteer time to respond to issues! -->
 
 ##### ISSUE TYPE
  - Feature Idea

.github/PULL_REQUEST_TEMPLATE.md

@@ -1,3 +1,11 @@
+<!--- changelog-entry
+# Fill in 'msg' below to have an entry automatically added to the next release changelog.
+# Leaving 'msg' blank will not generate a changelog entry for this PR.
+# Please ensure this is a simple (and readable) one-line string.
+---
+msg: ""
+-->
+
 ##### SUMMARY
 <!--- Describe the change, including rationale and design decisions -->
 
@@ -17,7 +25,6 @@ the change does.
 <!--- Name of the module/plugin/module/task -->
  - API
  - UI
- - Installer
 
 ##### AWX VERSION
 <!--- Paste verbatim output from `make VERSION` between quotes below -->

.github/workflows/ci.yml

@@ -0,0 +1,177 @@
+---
+name: CI
+env:
+  BRANCH: ${{ github.base_ref || 'devel' }}
+on:
+  pull_request:
+  push:
+    branches: [devel]
+jobs:
+  api-test:
+    runs-on: ubuntu-latest
+    permissions:
+      packages: write
+      contents: read
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Log in to registry
+        run: |
+          echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
+
+      - name: Pre-pull image to warm build cache
+        run: |
+          docker pull ghcr.io/${{ github.repository_owner }}/awx_devel:${{ env.BRANCH }}
+
+      - name: Build image
+        run: |
+          DEV_DOCKER_TAG_BASE=ghcr.io/${{ github.repository_owner }} COMPOSE_TAG=${{ env.BRANCH }} make docker-compose-build
+
+      - name: Run API Tests
+        run: |
+          docker run -u $(id -u) --rm -v ${{ github.workspace}}:/awx_devel/:Z \
+            --workdir=/awx_devel ghcr.io/${{ github.repository_owner }}/awx_devel:${{ env.BRANCH }} /start_tests.sh
+  api-lint:
+    runs-on: ubuntu-latest
+    permissions:
+      packages: write
+      contents: read
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Log in to registry
+        run: |
+          echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
+
+      - name: Pre-pull image to warm build cache
+        run: |
+          docker pull ghcr.io/${{ github.repository_owner }}/awx_devel:${{ env.BRANCH }}
+
+      - name: Build image
+        run: |
+          DEV_DOCKER_TAG_BASE=ghcr.io/${{ github.repository_owner }} COMPOSE_TAG=${{ env.BRANCH }} make docker-compose-build
+
+      - name: Run API Linters
+        run: |
+          docker run -u $(id -u) --rm -v ${{ github.workspace}}:/awx_devel/:Z \
+            --workdir=/awx_devel ghcr.io/${{ github.repository_owner }}/awx_devel:${{ env.BRANCH }} /var/lib/awx/venv/awx/bin/tox -e linters
+  api-swagger:
+    runs-on: ubuntu-latest
+    permissions:
+      packages: write
+      contents: read
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Log in to registry
+        run: |
+          echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
+
+      - name: Pre-pull image to warm build cache
+        run: |
+          docker pull ghcr.io/${{ github.repository_owner }}/awx_devel:${{ env.BRANCH }} || :
+
+      - name: Build image
+        run: |
+          DEV_DOCKER_TAG_BASE=ghcr.io/${{ github.repository_owner }} COMPOSE_TAG=${{ env.BRANCH }}  make docker-compose-build
+
+      - name: Generate API Reference
+        run: |
+          docker run -u $(id -u) --rm -v ${{ github.workspace}}:/awx_devel/:Z \
+            --workdir=/awx_devel ghcr.io/${{ github.repository_owner }}/awx_devel:${{ env.BRANCH }} /start_tests.sh swagger
+  awx-collection:
+    runs-on: ubuntu-latest
+    permissions:
+      packages: write
+      contents: read
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Log in to registry
+        run: |
+          echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
+
+      - name: Pre-pull image to warm build cache
+        run: |
+          docker pull ghcr.io/${{ github.repository_owner }}/awx_devel:${{ env.BRANCH }}
+
+      - name: Build image
+        run: |
+          DEV_DOCKER_TAG_BASE=ghcr.io/${{ github.repository_owner }} COMPOSE_TAG=${{ env.BRANCH }}  make docker-compose-build
+
+      - name: Run Collection Tests
+        run: |
+          docker run -u $(id -u) --rm -v ${{ github.workspace}}:/awx_devel/:Z \
+            --workdir=/awx_devel ghcr.io/${{ github.repository_owner }}/awx_devel:${{ env.BRANCH }} /start_tests.sh test_collection_all
+  api-schema:
+    runs-on: ubuntu-latest
+    permissions:
+      packages: write
+      contents: read
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Log in to registry
+        run: |
+          echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
+
+      - name: Pre-pull image to warm build cache
+        run: |
+          docker pull ghcr.io/${{ github.repository_owner }}/awx_devel:${{ env.BRANCH }}
+
+      - name: Build image
+        run: |
+          DEV_DOCKER_TAG_BASE=ghcr.io/${{ github.repository_owner }} COMPOSE_TAG=${{ env.BRANCH }}  make docker-compose-build
+
+      - name: Check API Schema
+        run: |
+          docker run -u $(id -u) --rm -v ${{ github.workspace}}:/awx_devel/:Z \
+            --workdir=/awx_devel ghcr.io/${{ github.repository_owner }}/awx_devel:${{ env.BRANCH }} /start_tests.sh detect-schema-change
+  ui-lint:
+    runs-on: ubuntu-latest
+    permissions:
+      packages: write
+      contents: read
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Log in to registry
+        run: |
+          echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
+
+      - name: Pre-pull image to warm build cache
+        run: |
+          docker pull ghcr.io/${{ github.repository_owner }}/awx_devel:${{ env.BRANCH }}
+
+      - name: Build image
+        run: |
+          DEV_DOCKER_TAG_BASE=ghcr.io/${{ github.repository_owner }} COMPOSE_TAG=${{ env.BRANCH }} make docker-compose-build
+
+      - name: Run UI Linters
+        run: |
+          docker run -u $(id -u) --rm -v ${{ github.workspace}}:/awx_devel/:Z \
+            --workdir=/awx_devel ghcr.io/${{ github.repository_owner }}/awx_devel:${{ env.BRANCH }} make ui-lint
+  ui-test:
+    runs-on: ubuntu-latest
+    permissions:
+      packages: write
+      contents: read
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Log in to registry
+        run: |
+          echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
+
+      - name: Pre-pull image to warm build cache
+        run: |
+          docker pull ghcr.io/${{ github.repository_owner }}/awx_devel:${{ env.BRANCH }}
+
+      - name: Build image
+        run: |
+          DEV_DOCKER_TAG_BASE=ghcr.io/${{ github.repository_owner }} COMPOSE_TAG=${{ env.BRANCH }} make docker-compose-build
+
+      - name: Run UI Tests
+        run: |
+          docker run -u $(id -u) --rm -v ${{ github.workspace}}:/awx_devel/:Z \
+            --workdir=/awx_devel ghcr.io/${{ github.repository_owner }}/awx_devel:${{ env.BRANCH }} make ui-test

.github/workflows/devel_image.yml

@@ -0,0 +1,30 @@
+---
+name: Push Development Image
+on:
+  push:
+    branches:
+      - devel
+jobs:
+  push:
+    runs-on: ubuntu-latest
+    permissions:
+      packages: write
+      contents: read
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Log in to registry
+        run: |
+          echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
+
+      - name: Pre-pull image to warm build cache
+        run: |
+          docker pull ghcr.io/${{ github.repository_owner }}/awx_devel:${GITHUB_REF##*/}
+
+      - name: Build image
+        run: |
+          DEV_DOCKER_TAG_BASE=ghcr.io/${{ github.repository_owner }} COMPOSE_TAG=${GITHUB_REF##*/} make docker-compose-build
+
+      - name: Push image
+        run: |
+          docker push ghcr.io/${{ github.repository_owner }}/awx_devel:${GITHUB_REF##*/}

.github/workflows/e2e_test.yml

@@ -0,0 +1,100 @@
+---
+name: E2E Tests
+on:
+  pull_request_target:
+    types: [labeled]
+jobs:   
+  e2e-test:
+    if: contains(github.event.pull_request.labels.*.name, 'qe:e2e')
+    runs-on: ubuntu-latest
+    timeout-minutes: 40
+    permissions:
+      packages: write
+      contents: read
+    strategy:
+      matrix:
+        job: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24]
+
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Install system deps
+        run: sudo apt-get install -y gettext
+
+      - name: Log in to registry
+        run: |
+          echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
+
+      - name: Pre-pull image to warm build cache
+        run: |
+          docker pull ghcr.io/${{ github.repository_owner }}/awx_devel:${{ github.base_ref }}
+
+      - name: Build UI
+        run: |
+          DEV_DOCKER_TAG_BASE=ghcr.io/${{ github.repository_owner }} COMPOSE_TAG=${{ github.base_ref }} make ui-devel
+
+      - name: Start AWX
+        run: |
+          DEV_DOCKER_TAG_BASE=ghcr.io/${{ github.repository_owner }} COMPOSE_TAG=${{ github.base_ref }} make docker-compose &> make-docker-compose-output.log &
+
+      - name: Pull awx_cypress_base image
+        run: |
+          docker pull quay.io/awx/awx_cypress_base:latest
+
+      - name: Checkout test project
+        uses: actions/checkout@v2
+        with:
+          repository: ${{ github.repository_owner }}/tower-qa
+          ssh-key: ${{ secrets.QA_REPO_KEY }}
+          path: tower-qa
+          ref: devel
+
+      - name: Build cypress
+        run: |
+          cd ${{ secrets.E2E_PROJECT }}/ui-tests/awx-pf-tests
+          docker build -t awx-pf-tests .
+
+      - name: Update default AWX password
+        run: |
+          while [[ "$(curl -s -o /dev/null -w ''%{http_code}'' -k https://localhost:8043/api/v2/ping/)" != "200" ]]
+          do
+          echo "Waiting for AWX..."
+          sleep 5;
+          done
+          echo "AWX is up, updating the password..."
+          docker exec -i tools_awx_1 sh <<-EOSH
+            awx-manage update_password --username=admin --password=password
+          EOSH
+
+      - name: Run E2E tests
+        env:
+          CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
+        run: |
+          export COMMIT_INFO_BRANCH=$GITHUB_HEAD_REF
+          export COMMIT_INFO_AUTHOR=$GITHUB_ACTOR
+          export COMMIT_INFO_SHA=$GITHUB_SHA
+          export COMMIT_INFO_REMOTE=$GITHUB_REPOSITORY_OWNER
+          cd ${{ secrets.E2E_PROJECT }}/ui-tests/awx-pf-tests
+          AWX_IP=$(docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' tools_awx_1)
+          printenv > .env
+          echo "Executing tests:"
+          docker run \
+          --network '_sources_default' \
+          --ipc=host \
+          --env-file=.env \
+          -e CYPRESS_baseUrl="https://$AWX_IP:8043" \
+          -e CYPRESS_AWX_E2E_USERNAME=admin \
+          -e CYPRESS_AWX_E2E_PASSWORD='password' \
+          -e COMMAND="npm run cypress-gha" \
+          -v /dev/shm:/dev/shm \
+          -v $PWD:/e2e \
+          -w /e2e \
+          awx-pf-tests run --project .
+
+      - name: Save AWX logs
+        uses: actions/upload-artifact@v2
+        with:
+          name: AWX-logs-${{ matrix.job }}
+          path: make-docker-compose-output.log
+
+

.github/workflows/release.yml

@@ -0,0 +1,56 @@
+name: Release AWX
+on:
+  workflow_dispatch:
+    inputs:
+      version:
+        description: 'Version'
+        required: true
+        default: ''
+      confirm:
+        description: 'Are you sure? Set this to yes.'
+        required: true
+        default: 'no'
+
+jobs:
+  release:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: "Verify inputs"
+        run: |
+          set -e
+
+          if [[ ${{ github.event.inputs.confirm }} != "yes" ]]; then
+            >&2 echo "Confirm must be 'yes'"
+            exit 1
+          fi
+
+          if [[ ${{ github.event.inputs.version }} == "" ]]; then
+            >&2 echo "Set version to continue."
+            exit 1
+          fi
+
+          exit 0
+
+      - name: Generate changelog
+        uses: shanemcd/simple-changelog-generator@v1
+        id: changelog
+        with:
+          repo: "${{ github.repository }}"
+
+      - name: Write changelog to file
+        run: |
+          cat << 'EOF' > /tmp/changelog
+          ${{ steps.changelog.outputs.changelog }}
+          EOF
+
+      - name: Release AWX
+        run: |
+          ansible-playbook -v tools/ansible/release.yml \
+            -e changelog_path=/tmp/changelog \
+            -e version=${{ github.event.inputs.version }} \
+            -e github_token=${{ secrets.GITHUB_TOKEN }} \
+            -e repo=${{ github.repository }}
+
+

.github/workflows/upload_schema.yml

@@ -0,0 +1,43 @@
+---
+name: Upload API Schema
+on:
+  push:
+    branches:
+      - devel
+jobs:
+  push:
+    runs-on: ubuntu-latest
+    permissions:
+      packages: write
+      contents: read
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Log in to registry
+        run: |
+          echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
+
+      - name: Pre-pull image to warm build cache
+        run: |
+          docker pull ghcr.io/${{ github.repository_owner }}/awx_devel:${GITHUB_REF##*/}
+
+      - name: Build image
+        run: |
+          DEV_DOCKER_TAG_BASE=ghcr.io/${{ github.repository_owner }} COMPOSE_TAG=${GITHUB_REF##*/} make docker-compose-build
+
+      - name: Generate API Schema
+        run: |
+          docker run -u $(id -u) --rm -v ${{ github.workspace }}:/awx_devel/:Z \
+            --workdir=/awx_devel ghcr.io/${{ github.repository_owner }}/awx_devel:${GITHUB_REF##*/} /start_tests.sh genschema
+
+      - name: Upload API Schema
+        env:
+          AWS_ACCESS_KEY: ${{ secrets.AWS_ACCESS_KEY }}
+          AWS_SECRET_KEY: ${{ secrets.AWS_SECRET_KEY }}
+          AWS_REGION: 'us-east-1'
+        run: |
+          ansible localhost -c local, -m command -a "{{ ansible_python_interpreter + ' -m pip install boto3'}}"
+          ansible localhost -c local -m aws_s3 \
+            -a 'src=${{ github.workspace }}/schema.json bucket=awx-public-ci-files object=schema.json mode=put permission=public-read'
+
+

.gitignore

@@ -28,12 +28,20 @@ awx/ui/build_test
 awx/ui/client/languages
 awx/ui/templates/ui/index.html
 awx/ui/templates/ui/installing.html
-awx/ui_next/node_modules/
-awx/ui_next/coverage/
-awx/ui_next/build/locales/_build
-/tower-license
-/tower-license/**
+awx/ui/node_modules/
+awx/ui/src/locales/*/messages.js
+awx/ui/coverage/
+awx/ui/build
+awx/ui/.env.local
+awx/ui/instrumented
+rsyslog.pid
 tools/prometheus/data
+tools/docker-compose/ansible/awx_dump.sql
+tools/docker-compose/Dockerfile
+tools/docker-compose/_build
+tools/docker-compose/_sources
+tools/docker-compose/overrides/
+tools/docker-compose-minikube/_sources
 
 # Tower setup playbook testing
 setup/test/roles/postgresql
@@ -55,14 +63,12 @@ __pycache__
 /Gruntfile.js
 /Brocfile.js
 /bower.json
-/package.json
 /testem.yml
 **/coverage
 /.istanbul.yml
 **/node_modules/**
 /tmp
 **/npm-debug.log*
-**/package-lock.json
 
 # UI build flag files
 awx/ui/.deps_built
@@ -84,6 +90,9 @@ awx/awx_test.sqlite3-journal
 # Mac OS X
 *.DS_Store
 
+# VSCode
+.vscode/
+
 # Editors
 *.sw[poj]
 *~
@@ -137,9 +146,14 @@ use_dev_supervisor.txt
 # Ansible module tests
 /awx_collection_test_venv/
 /awx_collection/*.tar.gz
-/awx_collection/galaxy.yml
 /sanity/
+/awx_collection_build/
 
 .idea/*
 *.unison.tmp
 *.#
+/awx/ui/.ui-built
+/Dockerfile
+/_build/
+/_build_kube_dev/
+/Dockerfile.kube-dev

.yamllint

@@ -1,5 +1,6 @@
 ---
 ignore: |
+  .github
   .tox
   awx/main/tests/data/inventory/plugins/**
   # vault files

API_STANDARDS.md

@@ -1,7 +1,7 @@
 Coding Standards and Practices
 ==============================
 
-This is not meant to be a style document so much as a practices document for ensuring performance and convention in the Ansible Tower API.
+This is not meant to be a style document so much as a practices document for ensuring performance and convention in the AWX API.
 
 Paginate Everything
 ===================

CHANGELOG.md

@@ -0,0 +1,7 @@
+# Changelog
+
+**Note:** This file is deprecated and will be removed at some point in a future release.
+
+Starting with AWX 20, release notes are published to [GitHub Releases](https://github.com/ansible/awx/releases).
+
+For older release notes, see https://github.com/ansible/awx/blob/19.3.0/CHANGELOG.md.

CONTRIBUTING.md

@@ -2,34 +2,25 @@
 
 Hi there! We're excited to have you as a contributor.
 
-Have questions about this document or anything not covered here? Come chat with us at `#ansible-awx` on irc.freenode.net, or submit your question to the [mailing list](https://groups.google.com/forum/#!forum/awx-project).
+Have questions about this document or anything not covered here? Come chat with us at `#ansible-awx` on irc.libera.chat, or submit your question to the [mailing list](https://groups.google.com/forum/#!forum/awx-project).
 
 ## Table of contents
 
-* [Things to know prior to submitting code](#things-to-know-prior-to-submitting-code)
-* [Setting up your development environment](#setting-up-your-development-environment)
-  * [Prerequisites](#prerequisites)
-    * [Docker](#docker)
-    * [Docker compose](#docker-compose)
-    * [Node and npm](#node-and-npm)
-  * [Build the environment](#build-the-environment)
-    * [Fork and clone the AWX repo](#fork-and-clone-the-awx-repo)
-    * [Create local settings](#create-local-settings)
-    * [Build the base image](#build-the-base-image)
-    * [Build the user interface](#build-the-user-interface)
-  * [Running the environment](#running-the-environment)
-    * [Start the containers](#start-the-containers)
-    * [Start from the container shell](#start-from-the-container-shell)
-  * [Post Build Steps](#post-build-steps)
-    * [Start a shell](#start-a-shell)
-    * [Create a superuser](#create-a-superuser)
-    * [Load the data](#load-the-data)
-    * [Building API Documentation](#build-api-documentation)
-  * [Accessing the AWX web interface](#accessing-the-awx-web-interface)
-  * [Purging containers and images](#purging-containers-and-images)
-* [What should I work on?](#what-should-i-work-on)
-* [Submitting Pull Requests](#submitting-pull-requests)
-* [Reporting Issues](#reporting-issues)
+- [Things to know prior to submitting code](#things-to-know-prior-to-submitting-code)
+- [Setting up your development environment](#setting-up-your-development-environment)
+  - [Prerequisites](#prerequisites)
+    - [Docker](#docker)
+    - [Docker compose](#docker-compose)
+    - [Frontend Development](#frontend-development)
+  - [Build and Run the Development Environment](#build-and-run-the-development-environment)
+    - [Fork and clone the AWX repo](#fork-and-clone-the-awx-repo)
+  - [Building API Documentation](#building-api-documentation)
+  - [Accessing the AWX web interface](#accessing-the-awx-web-interface)
+  - [Purging containers and images](#purging-containers-and-images)
+- [What should I work on?](#what-should-i-work-on)
+- [Submitting Pull Requests](#submitting-pull-requests)
+- [PR Checks run by Zuul](#pr-checks-run-by-zuul)
+- [Reporting Issues](#reporting-issues)
 
 ## Things to know prior to submitting code
 
@@ -37,12 +28,12 @@ Have questions about this document or anything not covered here? Come chat with
 - You must use `git commit --signoff` for any commit to be merged, and agree that usage of --signoff constitutes agreement with the terms of [DCO 1.1](./DCO_1_1.md).
 - Take care to make sure no merge commits are in the submission, and use `git rebase` vs `git merge` for this reason.
   - If collaborating with someone else on the same branch, consider using `--force-with-lease` instead of `--force`. This will prevent you from accidentally overwriting commits pushed by someone else. For more information, see https://git-scm.com/docs/git-push#git-push---force-with-leaseltrefnamegt
-- If submitting a large code change, it's a good idea to join the `#ansible-awx` channel on irc.freenode.net, and talk about what you would like to do or add first. This not only helps everyone know what's going on, it also helps save time and effort, if the community decides some changes are needed.
+- If submitting a large code change, it's a good idea to join the `#ansible-awx` channel on irc.libera.chat, and talk about what you would like to do or add first. This not only helps everyone know what's going on, it also helps save time and effort, if the community decides some changes are needed.
 - We ask all of our community members and contributors to adhere to the [Ansible code of conduct](http://docs.ansible.com/ansible/latest/community/code_of_conduct.html). If you have questions, or need assistance, please reach out to our community team at [codeofconduct@ansible.com](mailto:codeofconduct@ansible.com)
 
 ## Setting up your development environment
 
-The AWX development environment workflow and toolchain is based on Docker, and the docker-compose tool, to provide dependencies, services, and databases necessary to run all of the components. It also binds the local source tree into the development container, making it possible to observe and test changes in real time.
+The AWX development environment workflow and toolchain uses Docker and the docker-compose tool, to provide dependencies, services, and databases necessary to run all of the components. It also bind-mounts the local source tree into the development container, making it possible to observe and test changes in real time.
 
 ### Prerequisites
 
@@ -55,225 +46,47 @@ respectively.
 
 For Linux platforms, refer to the following from Docker:
 
-**Fedora**
+- **Fedora** - https://docs.docker.com/engine/installation/linux/docker-ce/fedora/
 
-> https://docs.docker.com/engine/installation/linux/docker-ce/fedora/
+- **CentOS** - https://docs.docker.com/engine/installation/linux/docker-ce/centos/
 
-**CentOS**
+- **Ubuntu** - https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/
 
-> https://docs.docker.com/engine/installation/linux/docker-ce/centos/
+- **Debian** - https://docs.docker.com/engine/installation/linux/docker-ce/debian/
 
-**Ubuntu**
+- **Arch** - https://wiki.archlinux.org/index.php/Docker
 
-> https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/
+#### Docker Compose
 
-**Debian**
-
-> https://docs.docker.com/engine/installation/linux/docker-ce/debian/
-
-**Arch**
-
-> https://wiki.archlinux.org/index.php/Docker
-
-#### Docker compose
-
-If you're not using Docker for Mac, or Docker for Windows, you may need, or choose to, install the Docker compose Python module separately, in which case you'll need to run the following:
+If you're not using Docker for Mac, or Docker for Windows, you may need, or choose to, install the `docker-compose` Python module separately.
 
 ```bash
-(host)$ pip install docker-compose
+(host)$ pip3 install docker-compose
 ```
 
 #### Frontend Development
 
-See [the ui development documentation](awx/ui/README.md).
-
-
-### Build the environment
+See [the ui development documentation](awx/ui/CONTRIBUTING.md).
 
 #### Fork and clone the AWX repo
 
 If you have not done so already, you'll need to fork the AWX repo on GitHub. For more on how to do this, see [Fork a Repo](https://help.github.com/articles/fork-a-repo/).
 
-#### Create local settings
+### Build and Run the Development Environment
 
-AWX will import the file `awx/settings/local_settings.py` and combine it with defaults in `awx/settings/defaults.py`. This file is required for starting the development environment and startup will fail if it's not provided.
+See the [README.md](./tools/docker-compose/README.md) for docs on how to build the awx_devel image and run the development environment.
 
-An example is provided. Make a copy of it, and edit as needed (the defaults are usually fine):
-
-```bash
-(host)$ cp awx/settings/local_settings.py.docker_compose awx/settings/local_settings.py
-```
-
-#### Build the base image
-
-The AWX base container image (defined in `tools/docker-compose/Dockerfile`) contains basic OS dependencies and symbolic links into the development environment that make running the services easy.
-
-Run the following to build the image:
-
-```bash
-(host)$ make docker-compose-build
-```
-
-**NOTE**
-
-> The image will need to be rebuilt, if the Python requirements or OS dependencies change.
-
-Once the build completes, you will have a `ansible/awx_devel` image in your local image cache. Use the `docker images` command to view it, as follows:
-
-```bash
-(host)$ docker images
-
-REPOSITORY                                   TAG                 IMAGE ID            CREATED             SIZE
-ansible/awx_devel                            latest              ba9ec3e8df74        26 minutes ago      1.42GB
-```
-
-#### Build the user interface
-
-Run the following to build the AWX UI:
-
-```bash
-(host) $ make ui-devel
-```
-See [the ui development documentation](awx/ui/README.md) for more information on using the frontend development, build, and test tooling.
-
-### Running the environment
-
-#### Start the containers
-
-Start the development containers by running the following:
-
-```bash
-(host)$ make docker-compose
-```
-
-The above utilizes the image built in the previous step, and will automatically start all required services and dependent containers. Once the containers launch, your session will be attached to the *awx* container, and you'll be able to watch log messages and events in real time. You will see messages from Django and the front end build process.
-
-If you start a second terminal session, you can take a look at the running containers using the `docker ps` command. For example:
-
-```bash
-# List running containers
-(host)$ docker ps
-
-$ docker ps
-CONTAINER ID        IMAGE                                              COMMAND                  CREATED             STATUS              PORTS                                                                                                                                      NAMES
-aa4a75d6d77b        gcr.io/ansible-tower-engineering/awx_devel:devel   "/tini -- /bin/sh ..."   23 seconds ago      Up 15 seconds       0.0.0.0:5555->5555/tcp, 0.0.0.0:7899-7999->7899-7999/tcp, 0.0.0.0:8013->8013/tcp, 0.0.0.0:8043->8043/tcp, 22/tcp, 0.0.0.0:8080->8080/tcp   tools_awx_1
-e4c0afeb548c        postgres:10                                       "docker-entrypoint..."   26 seconds ago      Up 23 seconds       5432/tcp                                                                                                                                   tools_postgres_1
-0089699d5afd        tools_logstash                                     "/docker-entrypoin..."   26 seconds ago      Up 25 seconds                                                                                                                                                  tools_logstash_1
-4d4ff0ced266        memcached:alpine                                   "docker-entrypoint..."   26 seconds ago      Up 25 seconds       0.0.0.0:11211->11211/tcp                                                                                                                   tools_memcached_1
-92842acd64cd        rabbitmq:3-management                              "docker-entrypoint..."   26 seconds ago      Up 24 seconds       4369/tcp, 5671-5672/tcp, 15671/tcp, 25672/tcp, 0.0.0.0:15672->15672/tcp                                                                    tools_rabbitmq_1
-```
-**NOTE**
-
-> The Makefile assumes that the image you built is tagged with your current branch. This allows you to build images for different contexts or branches. When starting the containers, you can choose a specific branch by setting `COMPOSE_TAG=<branch name>` in your environment.
-
-> For example, you might be working in a feature branch, but you want to run the containers using the `devel` image you built previously. To do that, start the containers using the following command: `$ COMPOSE_TAG=devel make docker-compose`
-
-##### Wait for migrations to complete
-
-The first time you start the environment, database migrations need to run in order to build the PostgreSQL database. It will take few moments, but eventually you will see output in your terminal session that looks like the following:
-
-```bash
-awx_1        | Operations to perform:
-awx_1        |   Synchronize unmigrated apps: solo, api, staticfiles, debug_toolbar, messages, channels, django_extensions, ui, rest_framework, polymorphic
-awx_1        |   Apply all migrations: sso, taggit, sessions, sites, kombu_transport_django, social_auth, contenttypes, auth, conf, main
-awx_1        | Synchronizing apps without migrations:
-awx_1        |   Creating tables...
-awx_1        |     Running deferred SQL...
-awx_1        |   Installing custom SQL...
-awx_1        | Running migrations:
-awx_1        |   Rendering model states... DONE
-awx_1        |   Applying contenttypes.0001_initial... OK
-awx_1        |   Applying contenttypes.0002_remove_content_type_name... OK
-awx_1        |   Applying auth.0001_initial... OK
-awx_1        |   Applying auth.0002_alter_permission_name_max_length... OK
-awx_1        |   Applying auth.0003_alter_user_email_max_length... OK
-awx_1        |   Applying auth.0004_alter_user_username_opts... OK
-awx_1        |   Applying auth.0005_alter_user_last_login_null... OK
-awx_1        |   Applying auth.0006_require_contenttypes_0002... OK
-awx_1        |   Applying taggit.0001_initial... OK
-awx_1        |   Applying taggit.0002_auto_20150616_2121... OK
-awx_1        |   Applying main.0001_initial... OK
-awx_1        |   Applying main.0002_squashed_v300_release... OK
-awx_1        |   Applying main.0003_squashed_v300_v303_updates... OK
-awx_1        |   Applying main.0004_squashed_v310_release... OK
-awx_1        |   Applying conf.0001_initial... OK
-awx_1        |   Applying conf.0002_v310_copy_tower_settings... OK
-...
-```
-
-Once migrations are completed, you can begin using AWX.
-
-#### Start from the container shell
-
-Often times you'll want to start the development environment without immediately starting all of the services in the *awx* container, and instead be taken directly to a shell. You can do this with the following:
-
-```bash
-(host)$ make docker-compose-test
-```
-
-Using `docker exec`, this will create a session in the running *awx* container, and place you at a command prompt, where you can run shell commands inside the container.
-
-If you want to start and use the development environment, you'll first need to bootstrap it by running the following command:
-
-```bash
-(container)# /bootstrap_development.sh
-```
-
-The above will do all the setup tasks, including running database migrations, so it may take a couple minutes.
-
-Now you can start each service individually, or start all services in a pre-configured tmux session like so:
-
-```bash
-(container)# cd /awx_devel
-(container)# make server
-```
-
-### Post Build Steps
-
-Before you can log in and use the system, you will need to create an admin user. Optionally, you may also want to load some demo data.
-
-##### Start a shell
-
-To create the admin user, and load demo data, you first need to start a shell session on the *awx* container. In a new terminal session, use the `docker exec` command as follows to start the shell session:
-
-```bash
-(host)$ docker exec -it tools_awx_1 bash
-```
-This creates a session in the *awx* containers, just as if you were using `ssh`, and allows you execute commands within the running container.
-
-##### Create an admin user
-
-Before you can log into AWX, you need to create an admin user. With this user you will be able to create more users, and begin configuring the server. From within the container shell, run the following command:
-
-```bash
-(container)# awx-manage createsuperuser
-```
-You will be prompted for a username, an email address, and a password, and you will be asked to confirm the password. The email address is not important, so just enter something that looks like an email address. Remember the username and password, as you will use them to log into the web interface for the first time.
-
-##### Load demo data
-
-You can optionally load some demo data. This will create a demo project, inventory, and job template. From within the container shell, run the following to load the data:
-
-```bash
-(container)# awx-manage create_preload_data
-```
-
-**NOTE**
-
-> This information will persist in the database running in the `tools_postgres_1` container, until the container is removed. You may periodically need to recreate
-this container, and thus the database, if the database schema changes in an upstream commit.
-
-##### Building API Documentation
+### Building API Documentation
 
 AWX includes support for building [Swagger/OpenAPI
-documentation](https://swagger.io).  To build the documentation locally, run:
+documentation](https://swagger.io). To build the documentation locally, run:
 
 ```bash
 (container)/awx_devel$ make swagger
 ```
 
 This will write a file named `swagger.json` that contains the API specification
-in OpenAPI format.  A variety of online tools are available for translating
+in OpenAPI format. A variety of online tools are available for translating
 this data into more consumable formats (such as HTML). http://editor.swagger.io
 is an example of one such service.
 
@@ -281,7 +94,7 @@ is an example of one such service.
 
 You can now log into the AWX web interface at [https://localhost:8043](https://localhost:8043), and access the API directly at [https://localhost:8043/api/](https://localhost:8043/api/).
 
-To log in use the admin user and password you created above in [Create an admin user](#create-an-admin-user).
+[Create an admin user](./tools/docker-compose/README.md#create-an-admin-user) if needed.
 
 ### Purging containers and images
 
@@ -301,7 +114,7 @@ Fixing bugs, adding translations, and updating the documentation are always appr
 
 **NOTE**
 
-> If you work in a part of the codebase that is going through active development, your changes may be rejected, or you may be asked to `rebase`. A good idea before starting work is to have a discussion with us in the `#ansible-awx` channel on irc.freenode.net, or on the [mailing list](https://groups.google.com/forum/#!forum/awx-project).
+> If you work in a part of the codebase that is going through active development, your changes may be rejected, or you may be asked to `rebase`. A good idea before starting work is to have a discussion with us in the `#ansible-awx` channel on irc.libera.chat, or on the [mailing list](https://groups.google.com/forum/#!forum/awx-project).
 
 **NOTE**
 
@@ -313,17 +126,17 @@ Fixes and Features for AWX will go through the Github pull request process. Subm
 
 Here are a few things you can do to help the visibility of your change, and increase the likelihood that it will be accepted:
 
-* No issues when running linters/code checkers
-  * Python: flake8: `(container)/awx_devel$ make flake8`
-  * Javascript: JsHint: `(container)/awx_devel$ make jshint`
-* No issues from unit tests
-  * Python: py.test: `(container)/awx_devel$ make test`
-  * JavaScript: Jasmine: `(container)/awx_devel$ make ui-test-ci`
-* Write tests for new functionality, update/add tests for bug fixes
-* Make the smallest change possible
-* Write good commit messages. See [How to write a Git commit message](https://chris.beams.io/posts/git-commit/).
+- No issues when running linters/code checkers
+  - Python: black: `(container)/awx_devel$ make black`
+  - Javascript: `(container)/awx_devel$ make ui-lint`
+- No issues from unit tests
+  - Python: py.test: `(container)/awx_devel$ make test`
+  - JavaScript: `(container)/awx_devel$ make ui-test`
+- Write tests for new functionality, update/add tests for bug fixes
+- Make the smallest change possible
+- Write good commit messages. See [How to write a Git commit message](https://chris.beams.io/posts/git-commit/).
 
-It's generally a good idea to discuss features with us first by engaging us in the `#ansible-awx` channel on irc.freenode.net, or on the [mailing list](https://groups.google.com/forum/#!forum/awx-project).
+It's generally a good idea to discuss features with us first by engaging us in the `#ansible-awx` channel on irc.libera.chat, or on the [mailing list](https://groups.google.com/forum/#!forum/awx-project).
 
 We like to keep our commit history clean, and will require resubmission of pull requests that contain merge commits. Use `git pull --rebase`, rather than
 `git pull`, and `git rebase`, rather than `git merge`.
@@ -332,22 +145,25 @@ Sometimes it might take us a while to fully review your PR. We try to keep the `
 
 All submitted PRs will have the linter and unit tests run against them via Zuul, and the status reported in the PR.
 
-## PR Checks ran by Zuul
+## PR Checks run by Zuul
+
 Zuul jobs for awx are defined in the [zuul-jobs](https://github.com/ansible/zuul-jobs) repo.
 
 Zuul runs the following checks that must pass:
-1) `tox-awx-api-lint`
-2) `tox-awx-ui-lint`
-3) `tox-awx-api`
-4) `tox-awx-ui`
-5) `tox-awx-swagger`
+
+1. `tox-awx-api-lint`
+2. `tox-awx-ui-lint`
+3. `tox-awx-api`
+4. `tox-awx-ui`
+5. `tox-awx-swagger`
 
 Zuul runs the following checks that are non-voting (can not pass but serve to inform PR reviewers):
-1) `tox-awx-detect-schema-change`
-    This check generates the schema and diffs it against a reference copy of the `devel` version of the schema.
-    Reviewers should inspect the `job-output.txt.gz` related to the check if their is a failure (grep for `diff -u -b` to find beginning of diff).
-    If the schema change is expected and makes sense in relation to the changes made by the PR, then you are good to go!
-    If not, the schema changes should be fixed, but this decision must be enforced by reviewers.
+
+1. `tox-awx-detect-schema-change`
+   This check generates the schema and diffs it against a reference copy of the `devel` version of the schema.
+   Reviewers should inspect the `job-output.txt.gz` related to the check if their is a failure (grep for `diff -u -b` to find beginning of diff).
+   If the schema change is expected and makes sense in relation to the changes made by the PR, then you are good to go!
+   If not, the schema changes should be fixed, but this decision must be enforced by reviewers.
 
 ## Reporting Issues
 

DATA_MIGRATION.md

@@ -2,96 +2,8 @@
 
 ## Introduction
 
-Upgrades using Django migrations are not expected to work in AWX.  As a result, to upgrade to a new version, it is necessary to export resources from the old AWX node and import them into a freshly-installed node with the new version.  The recommended way to do this is to use the tower-cli send/receive feature.
+Early versions of AWX did not support seamless upgrades between major versions and required the use of a backup and restore tool to perform upgrades.
 
-This tool does __not__ support export/import of the following:
-* Logs/history
-* Credential passwords
-* LDAP/AWX config
+Users who wish to upgrade modern AWX installations should follow the instructions at:
 
-### Install & Configure Tower-CLI
-
-In terminal, pip install tower-cli (if you do not have pip already, install [here](https://pip.pypa.io/en/stable/installing/)):
-```
-$ pip install --upgrade ansible-tower-cli
-```
-
-The AWX host URL, user, and password must be set for the AWX instance to be exported:
-```
-$ tower-cli config host http://<old-awx-host.example.com>
-$ tower-cli config username <user>
-$ tower-cli config password <pass>
-```
-
-For more information on installing tower-cli look [here](http://tower-cli.readthedocs.io/en/latest/quickstart.html).
-
-
-### Export Resources
-
-Export all objects
-
-```$ tower-cli receive --all > assets.json```
-
-
-
-### Teardown Old AWX
-
-Clean up remnants of the old AWX install:
-
-```docker rm -f $(docker ps -aq)```     # remove all old awx containers
-
-```make clean-ui```              # clean up ui artifacts
-
-
-### Install New AWX version
-
-If you are installing AWX as a dev container, pull down the latest code or version you want from GitHub, build
-the image locally, then start the container
-
-```
-git pull                                        # retrieve latest AWX changes from repository
-make docker-compose-build                       # build AWX image
-make docker-compose                             # run container
-```
-For other install methods, refer to the [Install.md](https://github.com/ansible/awx/blob/devel/INSTALL.md). 
- 
-
-### Import Resources
-
-
-Configure tower-cli for your new AWX host as shown earlier.  Import from a JSON file named assets.json
-
-```
-$ tower-cli config host http://<new-awx-host.example.com>
-$ tower-cli config username <user>
-$ tower-cli config password <pass>
-$ tower-cli send assets.json
-```
-
---------------------------------------------------------------------------------
-
-## Additional Info
-
-If you have two running AWX hosts, it is possible to copy all assets from one instance to another
-
-```$ tower-cli receive --tower-host old-awx-host.example.com --all | tower-cli send --tower-host new-awx-host.example.com```
-
-
-
-#### More Granular Exports:
-
-Export all credentials
-
-```$ tower-cli receive --credential all > credentials.json```
-> Note: This exports the credentials with blank strings for passwords and secrets
-
-Export a credential named "My Credential"
-
-```$ tower-cli receive --credential "My Credential"```
-
-#### More Granular Imports:
-
-
-You could import anything except an organization defined in a JSON file named assets.json
-
-```$ tower-cli send --prevent organization assets.json```
+https://github.com/ansible/awx/blob/devel/INSTALL.md#upgrading-from-previous-versions

INSTALL.md

@@ -1,636 +1,50 @@
+Table of Contents
+=================
+
+   * [Installing AWX](#installing-awx)
+      * [The AWX Operator](#the-awx-operator)
+   * [Installing the AWX CLI](#installing-the-awx-cli)
+      * [Building the CLI Documentation](#building-the-cli-documentation)
+
+
 # Installing AWX
 
-This document provides a guide for installing AWX.
+:warning: NOTE |
+--- |
+If you're installing an older release of AWX (prior to 18.0), these instructions have changed.  Take a look at your version specific instructions, e.g., for AWX 17.0.1, see: [https://github.com/ansible/awx/blob/17.0.1/INSTALL.md](https://github.com/ansible/awx/blob/17.0.1/INSTALL.md)
+If you're attempting to migrate an older Docker-based AWX installation, see: [Migrating Data from Local Docker](https://github.com/ansible/awx/blob/devel/tools/docker-compose/docs/data_migration.md) |
 
-## Table of contents
+## The AWX Operator
 
-- [Installing AWX](#installing-awx)
-  * [Getting started](#getting-started)
-    + [Clone the repo](#clone-the-repo)
-    + [AWX branding](#awx-branding)
-    + [Prerequisites](#prerequisites)
-    + [System Requirements](#system-requirements)
-    + [AWX Tunables](#awx-tunables)
-    + [Choose a deployment platform](#choose-a-deployment-platform)
-    + [Official vs Building Images](#official-vs-building-images)
-  * [Upgrading from previous versions](#upgrading-from-previous-versions)
-  * [OpenShift](#openshift)
-    + [Prerequisites](#prerequisites-1)
-    + [Pre-install steps](#pre-install-steps)
-      - [Deploying to Minishift](#deploying-to-minishift)
-      - [PostgreSQL](#postgresql)
-    + [Run the installer](#run-the-installer)
-    + [Post-install](#post-install)
-    + [Accessing AWX](#accessing-awx)
-  * [Kubernetes](#kubernetes)
-    + [Prerequisites](#prerequisites-2)
-    + [Pre-install steps](#pre-install-steps-1)
-    + [Configuring Helm](#configuring-helm)
-    + [Run the installer](#run-the-installer-1)
-    + [Post-install](#post-install-1)
-    + [Accessing AWX](#accessing-awx-1)
-    + [SSL Termination](#ssl-termination)
-  * [Docker-Compose](#docker-compose)
-    + [Prerequisites](#prerequisites-3)
-    + [Pre-install steps](#pre-install-steps-2)
-      - [Deploying to a remote host](#deploying-to-a-remote-host)
-      - [Inventory variables](#inventory-variables)
-      - [Docker registry](#docker-registry)
-      - [Proxy settings](#proxy-settings)
-      - [PostgreSQL](#postgresql-1)
-    + [Run the installer](#run-the-installer-2)
-    + [Post-install](#post-install-2)
-    + [Accessing AWX](#accessing-awx-2)
+Starting in version 18.0, the [AWX Operator](https://github.com/ansible/awx-operator) is the preferred way to install AWX. Please refer to the [AWX Operator](https://github.com/ansible/awx-operator) documentation.
 
-    
-## Getting started
+AWX can also alternatively be installed and [run in Docker](./tools/docker-compose/README.md), but this install path is only recommended for development/test-oriented deployments, and has no official published release.
 
-### Clone the repo
+# Installing the AWX CLI
 
-If you have not already done so, you will need to clone, or create a local copy, of the [AWX repo](https://github.com/ansible/awx). For more on how to clone the repo, view [git clone help](https://git-scm.com/docs/git-clone).
+`awx` is the official command-line client for AWX.  It:
 
-Once you have a local copy, run commands within the root of the project tree.
+* Uses naming and structure consistent with the AWX HTTP API
+* Provides consistent output formats with optional machine-parsable formats
+* To the extent possible, auto-detects API versions, available endpoints, and
+  feature support across multiple versions of AWX.
 
-### AWX branding
+Potential uses include:
 
-You can optionally install the AWX branding assets from the [awx-logos repo](https://github.com/ansible/awx-logos). Prior to installing, please review and agree to the [trademark guidelines](https://github.com/ansible/awx-logos/blob/master/TRADEMARKS.md).
+* Configuring and launching jobs/playbooks
+* Checking on the status and output of job runs
+* Managing objects like organizations, users, teams, etc...
 
-To install the assets, clone the `awx-logos` repo so that it is next to your `awx` clone. As you progress through the installation steps, you'll be setting variables in the [inventory](./installer/inventory) file. To include the assets in the build, set `awx_official=true`.
+The preferred way to install the AWX CLI is through pip directly from PyPI:
 
-### Prerequisites
+    pip3 install awxkit
+    awx --help
 
-Before you can run a deployment, you'll need the following installed in your local environment:
+## Building the CLI Documentation
 
-- [Ansible](http://docs.ansible.com/ansible/latest/intro_installation.html) Requires Version 2.8+
-- [Docker](https://docs.docker.com/engine/installation/)
-    + A recent version
-- [docker](https://pypi.org/project/docker/) Python module
-    + This is incompatible with `docker-py`. If you have previously installed `docker-py`, please uninstall it.
-    + We use this module instead of `docker-py` because it is what the `docker-compose` Python module requires.
-- [GNU Make](https://www.gnu.org/software/make/)
-- [Git](https://git-scm.com/) Requires Version 1.8.4+
-- [Node 10.x LTS version](https://nodejs.org/en/download/)
-- [NPM 6.x LTS](https://docs.npmjs.com/)
+To build the docs, spin up a real AWX server, `pip3 install sphinx sphinxcontrib-autoprogram`, and run:
 
-### System Requirements
-
-The system that runs the AWX service will need to satisfy the following requirements
-
-- At least 4GB of memory
-- At least 2 cpu cores
-- At least 20GB of space
-- Running Docker, Openshift, or Kubernetes
-- If you choose to use an external PostgreSQL database, please note that the minimum version is 9.6+.
-
-### AWX Tunables
-
-**TODO** add tunable bits
-
-### Choose a deployment platform
-
-We currently support running AWX as a containerized application using Docker images deployed to either an OpenShift cluster, a Kubernetes cluster, or docker-compose. The remainder of this document will walk you through the process of building the images, and deploying them to either platform.
-
-The [installer](./installer) directory contains an [inventory](./installer/inventory) file, and a playbook, [install.yml](./installer/install.yml). You'll begin by setting variables in the inventory file according to the platform you wish to use, and then you'll start the image build and deployment process by running the playbook.
-
-In the sections below, you'll find deployment details and instructions for each platform:
-- [OpenShift](#openshift)
-- [Kubernetes](#kubernetes)
-- [Docker Compose](#docker-compose).
-
-### Official vs Building Images
-
-When installing AWX you have the option of building your own images or using the images provided on DockerHub (see [awx_web](https://hub.docker.com/r/ansible/awx_web/) and [awx_task](https://hub.docker.com/r/ansible/awx_task/))
-
-This is controlled by the following variables in the `inventory` file
-
-```
-dockerhub_base=ansible
-dockerhub_version=latest
-```
-
-If these variables are present then all deployments will use these hosted images. If the variables are not present then the images will be built during the install.
-
-*dockerhub_base*
-
-> The base location on DockerHub where the images are hosted (by default this pulls container images named `ansible/awx_web:tag` and `ansible/awx_task:tag`)
-
-*dockerhub_version*
-
-> Multiple versions are provided. `latest` always pulls the most recent. You may also select version numbers at different granularities: 1, 1.0, 1.0.1, 1.0.0.123
-
-
-## Upgrading from previous versions
-
-Upgrading AWX involves rerunning the install playbook. Download a newer release from [https://github.com/ansible/awx/releases](https://github.com/ansible/awx/releases) and re-populate the inventory file with your customized variables.
-
-For convenience, you can create a file called `vars.yml`:
-
-```
-admin_password: 'adminpass'
-pg_password: 'pgpass'
-rabbitmq_password: 'rabbitpass'
-secret_key: 'mysupersecret'
-```
-
-And pass it to the installer:
-
-```
-$ ansible-playbook -i inventory install.yml -e @vars.yml
-```
-
-## OpenShift
-
-### Prerequisites
-
-To complete a deployment to OpenShift, you will obviously need access to an OpenShift cluster. For demo and testing purposes, you can use [Minishift](https://github.com/minishift/minishift) to create a single node cluster running inside a virtual machine.
-
-When using OpenShift for deploying AWX make sure you have correct privileges to add the security context 'privileged', otherwise the installation will fail. The privileged context is needed because of the use of [the bubblewrap tool](https://github.com/containers/bubblewrap) to add an additional layer of security when using containers.
-
-You will also need to have the `oc` command in your PATH. The `install.yml` playbook will call out to `oc` when logging into, and creating objects on the cluster.
-
-The default resource requests per-deployment requires:
-
-> Memory: 6GB
-> CPU: 3 cores
-
-This can be tuned by overriding the variables found in [/installer/roles/kubernetes/defaults/main.yml](/installer/roles/kubernetes/defaults/main.yml). Special care should be taken when doing this as undersized instances will experience crashes and resource exhaustion.
-
-For more detail on how resource requests are formed see: [https://docs.openshift.com/container-platform/latest/dev_guide/compute_resources.html#dev-compute-resources](https://docs.openshift.com/container-platform/latest/dev_guide/compute_resources.html#dev-compute-resources)
-
-### Pre-install steps
-
-Before starting the install, review the [inventory](./installer/inventory) file, and uncomment and provide values for the following variables found in the `[all:vars]` section:
-
-*openshift_host*
-
-> IP address or hostname of the OpenShift cluster. If you're using Minishift, this will be the value returned by `minishift ip`.
-
-
-*openshift_skip_tls_verify*
-
-> Boolean. Set to True if using self-signed certs.
-
-*openshift_project*
-
-> Name of the OpenShift project that will be created, and used as the namespace for the AWX app. Defaults to *awx*.
-
-*openshift_user*
-
-> Username of the OpenShift user that will create the project, and deploy the application. Defaults to *developer*.
-
-*openshift_pg_emptydir*
-
-> Boolean. Set to True to use an emptyDir volume when deploying the PostgreSQL pod. Note: This should only be used for demo and testing purposes.
-
-*docker_registry*
-
-> IP address and port, or URL, for accessing a registry that the OpenShift cluster can access. Defaults to *172.30.1.1:5000*, the internal registry delivered with Minishift. This is not needed if you are using official hosted images.
-
-*docker_registry_repository*
-
-> Namespace to use when pushing and pulling images to and from the registry. Generally this will match the project name. It defaults to *awx*. This is not needed if you are using official hosted images.
-
-*docker_registry_username*
-
-> Username of the user that will push images to the registry. Will generally match the *openshift_user* value. Defaults to *developer*. This is not needed if you are using official hosted images.
-
-#### Deploying to Minishift
-
-Install Minishift by following the [installation guide](https://docs.openshift.org/latest/minishift/getting-started/installing.html).
-
-The recommended minimum resources for your Minishift VM:
-
-```bash
-$ minishift start --cpus=4 --memory=8GB
-```
-
-The Minishift VM contains a Docker daemon, which you can use to build the AWX images. This is generally the approach you should take, and we recommend doing so. To use this instance, run the following command to setup your environment:
-
-```bash
-# Set DOCKER environment variable to point to the Minishift VM
-$ eval $(minishift docker-env)
-```
-
-**Note**
-
-> If you choose to not use the Docker instance running inside the VM, and build the images externally, you will have to enable the OpenShift cluster to access the images. This involves pushing the images to an external Docker registry, and granting the cluster access to it, or exposing the internal registry, and pushing the images into it.
-
-#### PostgreSQL
-
-By default, AWX will deploy a PostgreSQL pod inside of your cluster. You will need to create a [Persistent Volume Claim](https://docs.openshift.org/latest/dev_guide/persistent_volumes.html) which is named `postgresql` by default, and can be overridden by setting the `openshift_pg_pvc_name` variable. For testing and demo purposes, you may set `openshift_pg_emptydir=yes`.
-
-If you wish to use an external database, in the inventory file, set the value of `pg_hostname`, and update `pg_username`, `pg_password`, `pg_admin_password`, `pg_database`, and `pg_port` with the connection information. When setting `pg_hostname` the installer will assume you have configured the database in that location and will not launch the postgresql pod.
-
-### Run the installer
-
-To start the install, you will pass two *extra* variables on the command line. The first is *openshift_password*, which is the password for the *openshift_user*, and the second is *docker_registry_password*, which is the password associated with *docker_registry_username*.
-
-If you're using the OpenShift internal registry, then you'll pass an access token for the *docker_registry_password* value, rather than a password. The `oc whoami -t` command will generate the required token, as long as you're logged into the cluster via `oc cluster login`.
-
-Run the following command (docker_registry_password is optional if using official images):
-
-```bash
-# Start the install
-$ ansible-playbook -i inventory install.yml -e openshift_password=developer  -e docker_registry_password=$(oc whoami -t)
-```
-
-### Post-install
-
-After the playbook run completes, check the status of the deployment by running `oc get pods`:
-
-```bash
-# View the running pods
-$ oc get pods
-
-NAME                   READY     STATUS    RESTARTS   AGE
-awx-3886581826-5mv0l   4/4       Running   0          8s
-postgresql-1-l85fh     1/1       Running   0          20m
-
-```
-
-In the above example, the name of the AWX pod is `awx-3886581826-5mv0l`. Before accessing the AWX web interface, setup tasks and database migrations need to complete. These tasks are running in the `awx_task` container inside the AWX pod. To monitor their status, tail the container's STDOUT by running the following command, replacing the AWX pod name with the pod name from your environment:
-
-```bash
-# Follow the awx_task log output
-$ oc logs -f awx-3886581826-5mv0l -c awx-celery
-```
-
-You will see the following indicating that database migrations are running:
-
-```bash
-Using /etc/ansible/ansible.cfg as config file
-127.0.0.1 | SUCCESS => {
-    "changed": false,
-    "db": "awx"
-}
-Operations to perform:
-  Synchronize unmigrated apps: solo, api, staticfiles, messages, channels, django_extensions, ui, rest_framework, polymorphic
-  Apply all migrations: sso, taggit, sessions, sites, kombu_transport_django, social_auth, contenttypes, auth, conf, main
-Synchronizing apps without migrations:
-  Creating tables...
-    Running deferred SQL...
-  Installing custom SQL...
-Running migrations:
-  Rendering model states... DONE
-  Applying contenttypes.0001_initial... OK
-  Applying contenttypes.0002_remove_content_type_name... OK
-  Applying auth.0001_initial... OK
-  Applying auth.0002_alter_permission_name_max_length... OK
-  Applying auth.0003_alter_user_email_max_length... OK
-  Applying auth.0004_alter_user_username_opts... OK
-  Applying auth.0005_alter_user_last_login_null... OK
-  Applying auth.0006_require_contenttypes_0002... OK
-  Applying taggit.0001_initial... OK
-  Applying taggit.0002_auto_20150616_2121... OK
-  ...
-```
-
-When you see output similar to the following, you'll know that database migrations have completed, and you can access the web interface:
-
-```bash
-Python 2.7.5 (default, Nov  6 2016, 00:28:07)
-[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2
-Type "help", "copyright", "credits" or "license" for more information.
-(InteractiveConsole)
-
->>> <User: admin>
->>> Default organization added.
-Demo Credential, Inventory, and Job Template added.
-Successfully registered instance awx-3886581826-5mv0l
-(changed: True)
-Creating instance group tower
-Added instance awx-3886581826-5mv0l to tower
-```
-
-Once database migrations complete, the web interface will be accessible.
-
-### Accessing AWX
-
-The AWX web interface is running in the AWX pod, behind the `awx-web-svc` service. To view the service, and its port value, run the following command:
-
-```bash
-# View available services
-$ oc get services
-
-NAME          CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
-awx-web-svc   172.30.111.74   <nodes>       8052:30083/TCP   37m
-postgresql    172.30.102.9    <none>        5432/TCP         38m
-```
-
-The deployment process creates a route, `awx-web-svc`, to expose the service. How the ingres is actually created will vary depending on your environment, and how the cluster is configured. You can view the route, and the external IP address and hostname assigned to it, by running the following command:
-
-```bash
-# View available routes
-$ oc get routes
-
-NAME          HOST/PORT                             PATH      SERVICES      PORT      TERMINATION   WILDCARD
-awx-web-svc   awx-web-svc-awx.192.168.64.2.nip.io             awx-web-svc   http      edge/Allow    None
-```
-
-The above example is taken from a Minishift instance. From a web browser, use `https` to access the `HOST/PORT` value from your environment. Using the above example, the URL to access the server would be [https://awx-web-svc-awx.192.168.64.2.nip.io](https://awx-web-svc-awx.192.168.64.2.nip.io).
-
-Once you access the AWX server, you will be prompted with a login dialog. The default administrator username is `admin`, and the password is `password`.
-
-## Kubernetes
-
-### Prerequisites
-
-A Kubernetes deployment will require you to have access to a Kubernetes cluster as well as the following tools:
-
-- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
-- [helm](https://docs.helm.sh/using_helm/#quickstart-guide)
-
-The installation program will reference `kubectl` directly. `helm` is only necessary if you are letting the installer configure PostgreSQL for you.
-
-The default resource requests per-pod requires:
-
-> Memory: 6GB
-> CPU: 3 cores
-
-This can be tuned by overriding the variables found in [/installer/roles/kubernetes/defaults/main.yml](/installer/roles/kubernetes/defaults/main.yml). Special care should be taken when doing this as undersized instances will experience crashes and resource exhaustion.
-
-For more detail on how resource requests are formed see: [https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/)
-
-### Pre-install steps
-
-Before starting the install process, review the [inventory](./installer/inventory) file, and uncomment and provide values for the following variables found in the `[all:vars]` section uncommenting when necessary. Make sure the openshift and standalone docker sections are commented out:
-
-*kubernetes_context*
-
-> Prior to running the installer, make sure you've configured the context for the cluster you'll be installing to. This is how the installer knows which cluster to connect to and what authentication to use
-
-*kubernetes_namespace*
-
-> Name of the Kubernetes namespace where the AWX resources will be installed. This will be created if it doesn't exist
-
-*docker_registry_*
-
-> These settings should be used if building your own base images. You'll need access to an external registry and are responsible for making sure your kube cluster can talk to it and use it. If these are undefined and the dockerhub_ configuration settings are uncommented then the images will be pulled from dockerhub instead
-
-### Configuring Helm
-
-If you want the AWX installer to manage creating the database pod (rather than installing and configuring postgres on your own). Then you will need to have a working `helm` installation, you can find details here: [https://docs.helm.sh/using_helm/#quickstart-guide](https://docs.helm.sh/using_helm/#quickstart-guide).
-
-Newer Kubernetes clusters with RBAC enabled will need to make sure a service account is created, make sure to follow the instructions here [https://docs.helm.sh/using_helm/#role-based-access-control](https://docs.helm.sh/using_helm/#role-based-access-control)
-
-### Run the installer
-
-After making changes to the `inventory` file use `ansible-playbook` to begin the install
-
-```bash
-$ ansible-playbook -i inventory install.yml
-```
-
-### Post-install
-
-After the playbook run completes, check the status of the deployment by running `kubectl get pods --namespace awx` (replace awx with the namespace you used):
-
-```bash
-# View the running pods, it may take a few minutes for everything to be marked in the Running state
-$ kubectl get pods --namespace awx
-NAME                             READY     STATUS    RESTARTS   AGE
-awx-2558692395-2r8ss             4/4       Running   0          29s
-awx-postgresql-355348841-kltkn   1/1       Running   0          1m
-```
-
-### Accessing AWX
-
-The AWX web interface is running in the AWX pod behind the `awx-web-svc` service:
-
-```bash
-# View available services
-$ kubectl get svc --namespace awx
-NAME             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
-awx-postgresql   ClusterIP   10.7.250.208   <none>        5432/TCP       2m
-awx-web-svc      NodePort    10.7.241.35    <none>        80:30177/TCP   1m
-```
-
-The deployment process creates an `Ingress` named `awx-web-svc` also. Some kubernetes cloud providers will automatically handle routing configuration when an Ingress is created others may require that you more explicitly configure it. You can see what kubernetes knows about things with:
-
-```bash
- kubectl get ing --namespace awx
-NAME          HOSTS     ADDRESS          PORTS     AGE
-awx-web-svc   *         35.227.x.y       80        3m
-```
-
-If your provider is able to allocate an IP Address from the Ingress controller then you can navigate to the address and access the AWX interface. For some providers it can take a few minutes to allocate and make this accessible. For other providers it may require you to manually intervene.
-
-### SSL Termination
-
-Unlike Openshift's `Route` the Kubernetes `Ingress` doesn't yet handle SSL termination. As such the default configuration will only expose AWX through HTTP on port 80. You are responsible for configuring SSL support until support is added (either to Kubernetes or AWX itself).
-
-
-## Docker-Compose
-
-### Prerequisites
-
-- [Docker](https://docs.docker.com/engine/installation/) on the host where AWX will be deployed. After installing Docker, the Docker service must be started (depending on your OS, you may have to add the local user that uses Docker to the ``docker`` group, refer to the documentation for details)
-- [docker-compose](https://pypi.org/project/docker-compose/) Python module.
-    + This also installs the `docker` Python module, which is incompatible with `docker-py`. If you have previously installed `docker-py`, please uninstall it.
-- [Docker Compose](https://docs.docker.com/compose/install/).
-
-### Pre-install steps
-
-#### Deploying to a remote host
-
-By default, the delivered [installer/inventory](./installer/inventory) file will deploy AWX to the local host. It is possible, however, to deploy to a remote host. The [installer/install.yml](./installer/install.yml) playbook can be used to build images on the local host, and ship the built images to, and run deployment tasks on, a remote host. To do this, modify the [installer/inventory](./installer/inventory) file, by commenting out `localhost`, and adding the remote host.
-
-For example, suppose you wish to build images locally on your CI/CD host, and deploy them to a remote host named *awx-server*. To do this, add *awx-server* to the [installer/inventory](./installer/inventory) file, and comment out or remove `localhost`, as demonstrated by the following:
-
-```yaml
-# localhost ansible_connection=local
-awx-server
-
-[all:vars]
-...
-```
-
-In the above example, image build tasks will be delegated to `localhost`, which is typically where the clone of the AWX project exists. Built images will be archived, copied to remote host, and imported into the remote Docker image cache. Tasks to start the AWX containers will then execute on the remote host.
-
-If you choose to use the official images then the remote host will be the one to pull those images.
-
-**Note**
-
-> You may also want to set additional variables to control how Ansible connects to the host. For more information about this, view [Behavioral Inventory Parameters](http://docs.ansible.com/ansible/latest/intro_inventory.html#id12).
-
-> As mentioned above, in [Prerequisites](#prerequisites-1), the prerequisites are required on the remote host.
-
-> When deploying to a remote host, the playbook does not execute tasks with the `become` option. For this reason, make sure the user that connects to the remote host has privileges to run the `docker` command. This typically means that non-privileged users need to be part of the `docker` group.
-
-
-#### Inventory variables
-
-Before starting the install process, review the [inventory](./installer/inventory) file, and uncomment and provide values for the following variables found in the `[all:vars]` section:
-
-*postgres_data_dir*
-
-> If you're using the default PostgreSQL container (see [PostgreSQL](#postgresql-1) below), provide a path that can be mounted to the container, and where the database can be persisted.
-
-*host_port*
-
-> Provide a port number that can be mapped from the Docker daemon host to the web server running inside the AWX container. Defaults to *80*.
-
-*host_port_ssl*
-
-> Provide a port number that can be mapped from the Docker daemon host to the web server running inside the AWX container for SSL support. Defaults to *443*, only works if you also set `ssl_certificate` (see below).
-
-*ssl_certificate*
-
-> Optionally, provide the path to a file that contains a certificate and its private key.
-
-*docker_compose_dir*
-
-> When using docker-compose, the `docker-compose.yml` file will be created there (default `/tmp/awxcompose`).
-
-*custom_venv_dir*
-
-> Adds the custom venv environments from the local host to be passed into the containers at install.
-
-*ca_trust_dir*
-
-> If you're using a non trusted CA, provide a path where the untrusted Certs are stored on your Host.
-
-#### Docker registry
-
-If you wish to tag and push built images to a Docker registry, set the following variables in the inventory file:
-
-*docker_registry*
-
-> IP address and port, or URL, for accessing a registry.
-
-*docker_registry_repository*
-
-> Namespace to use when pushing and pulling images to and from the registry. Defaults to *awx*.
-
-*docker_registry_username*
-
-> Username of the user that will push images to the registry. Defaults to *developer*.
-
-*docker_remove_local_images*
-
-> Due to the way that the docker_image module behaves, images will not be pushed to a remote repository if they are present locally.  Set this to delete local versions of the images that will be pushed to the remote.  This will fail if containers are currently running from those images.
-
-**Note**
-
-> These settings are ignored if using official images
-
-
-#### Proxy settings
-
-*http_proxy*
-
-> IP address and port, or URL, for using an http_proxy.
-
-*https_proxy*
-
-> IP address and port, or URL, for using an https_proxy.
-
-*no_proxy*
-
-> Exclude IP address or URL from the proxy.
-
-#### PostgreSQL
-
-AWX requires access to a PostgreSQL database, and by default, one will be created and deployed in a container, and data will be persisted to a host volume. In this scenario, you must set the value of `postgres_data_dir` to a path that can be mounted to the container. When the container is stopped, the database files will still exist in the specified path.
-
-If you wish to use an external database, in the inventory file, set the value of `pg_hostname`, and update `pg_username`, `pg_password`, `pg_admin_password`, `pg_database`, and `pg_port` with the connection information.
-
-### Run the installer
-
-If you are not pushing images to a Docker registry, start the install by running the following:
-
-```bash
-# Set the working directory to installer
-$ cd installer
-
-# Run the Ansible playbook
-$ ansible-playbook -i inventory install.yml
-```
-
-If you're pushing built images to a repository, then use the `-e` option to pass the registry password as follows, replacing *password* with the password of the username assigned to `docker_registry_username` (note that you will also need to remove `dockerhub_base` and `dockerhub_version` from the inventory file):
-
-```bash
-# Set the working directory to installer
-$ cd installer
-
-# Run the Ansible playbook
-$ ansible-playbook -i inventory -e docker_registry_password=password install.yml
-```
-
-### Post-install
-
-After the playbook run completes, Docker will report up to 5 running containers. If you chose to use an existing PostgresSQL database, then it will report 4. You can view the running containers using the `docker ps` command, as follows:
-
-```bash
-CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                NAMES
-e240ed8209cd        awx_task:1.0.0.8    "/tini -- /bin/sh ..."   2 minutes ago       Up About a minute   8052/tcp                             awx_task
-1cfd02601690        awx_web:1.0.0.8     "/tini -- /bin/sh ..."   2 minutes ago       Up About a minute   0.0.0.0:443->8052/tcp                 awx_web
-55a552142bcd        memcached:alpine    "docker-entrypoint..."   2 minutes ago       Up 2 minutes        11211/tcp                            memcached
-84011c072aad        rabbitmq:3          "docker-entrypoint..."   2 minutes ago       Up 2 minutes        4369/tcp, 5671-5672/tcp, 25672/tcp   rabbitmq
-97e196120ab3        postgres:9.6        "docker-entrypoint..."   2 minutes ago       Up 2 minutes        5432/tcp                             postgres
-```
-
-If you're deploying using Docker Compose, container names will be prefixed by the name of the folder where the docker-compose.yml file is created (by default, `awx`).
-
-Immediately after the containers start, the *awx_task* container will perform required setup tasks, including database migrations. These tasks need to complete before the web interface can be accessed. To monitor the progress, you can follow the container's STDOUT by running the following:
-
-```bash
-# Tail the the awx_task log
-$ docker logs -f awx_task
-```
-
-You will see output similar to the following:
-
-```bash
-Using /etc/ansible/ansible.cfg as config file
-127.0.0.1 | SUCCESS => {
-    "changed": false,
-    "db": "awx"
-}
-Operations to perform:
-  Synchronize unmigrated apps: solo, api, staticfiles, messages, channels, django_extensions, ui, rest_framework, polymorphic
-  Apply all migrations: sso, taggit, sessions, sites, kombu_transport_django, social_auth, contenttypes, auth, conf, main
-Synchronizing apps without migrations:
-  Creating tables...
-    Running deferred SQL...
-  Installing custom SQL...
-Running migrations:
-  Rendering model states... DONE
-  Applying contenttypes.0001_initial... OK
-  Applying contenttypes.0002_remove_content_type_name... OK
-  Applying auth.0001_initial... OK
-  Applying auth.0002_alter_permission_name_max_length... OK
-  Applying auth.0003_alter_user_email_max_length... OK
-  Applying auth.0004_alter_user_username_opts... OK
-  Applying auth.0005_alter_user_last_login_null... OK
-  Applying auth.0006_require_contenttypes_0002... OK
-  Applying taggit.0001_initial... OK
-  Applying taggit.0002_auto_20150616_2121... OK
-  Applying main.0001_initial... OK
-...
-```
-
-Once migrations complete, you will see the following log output, indicating that migrations have completed:
-
-```bash
-Python 2.7.5 (default, Nov  6 2016, 00:28:07)
-[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2
-Type "help", "copyright", "credits" or "license" for more information.
-(InteractiveConsole)
-
->>> <User: admin>
->>> Default organization added.
-Demo Credential, Inventory, and Job Template added.
-Successfully registered instance awx
-(changed: True)
-Creating instance group tower
-Added instance awx to tower
-(changed: True)
-...
-```
-
-### Accessing AWX
-
-The AWX web server is accessible on the deployment host, using the *host_port* value set in the *inventory* file. The default URL is [http://localhost](http://localhost).
-
-You will prompted with a login dialog. The default administrator username is `admin`, and the password is `password`.
+    ~ cd awxkit/awxkit/cli/docs
+    ~ TOWER_HOST=https://awx.example.org TOWER_USERNAME=example TOWER_PASSWORD=secret make clean html
+    ~ cd build/html/ && python -m http.server
+    Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ..

ISSUES.md

@@ -5,7 +5,7 @@
 Use the GitHub [issue tracker](https://github.com/ansible/awx/issues) for filing bugs. In order to save time, and help us respond to issues quickly, make sure to fill out as much of the issue template
 as possible. Version information, and an accurate reproducing scenario are critical to helping us identify the problem.
 
-Please don't use the issue tracker as a way to ask how to do something. Instead, use the [mailing list](https://groups.google.com/forum/#!forum/awx-project) , and the `#ansible-awx` channel on irc.freenode.net to get help.
+Please don't use the issue tracker as a way to ask how to do something. Instead, use the [mailing list](https://groups.google.com/forum/#!forum/awx-project) , and the `#ansible-awx` channel on irc.libera.chat to get help.
 
 Before opening a new issue, please use the issue search feature to see if what you're experiencing has already been reported. If you have any extra detail to provide, please comment. Otherwise, rather than posting a "me too" comment, please consider giving it a ["thumbs up"](https://github.com/blog/2119-add-reactions-to-pull-requests-issues-and-comment) to give us an indication of the severity of the problem.
 
@@ -31,7 +31,7 @@ If your issue isn't considered high priority, then please be patient as it may t
 
 `state:needs_info` The issue needs more information. This could be more debug output, more specifics out the system such as version information. Any detail that is currently preventing this issue from moving forward. This should be considered a blocked state.
 
-`state:needs_review` The the issue/pull request needs to be reviewed by other maintainers and contributors. This is usually used when there is a question out to another maintainer or when a person is less familar with an area of the code base the issue is for.
+`state:needs_review` The issue/pull request needs to be reviewed by other maintainers and contributors. This is usually used when there is a question out to another maintainer or when a person is less familar with an area of the code base the issue is for.
 
 `state:needs_revision` More commonly used on pull requests, this state represents that there are changes that are being waited on.
 

MANIFEST.in

@@ -4,12 +4,13 @@ recursive-include awx *.mo
 recursive-include awx/static *
 recursive-include awx/templates *.html
 recursive-include awx/api/templates *.md *.html
-recursive-include awx/ui/templates *.html
-recursive-include awx/ui/static *
+recursive-include awx/ui/build *.html
+recursive-include awx/ui/build *
 recursive-include awx/playbooks *.yml
 recursive-include awx/lib/site-packages *
 recursive-include awx/plugins *.ps1
 recursive-include requirements *.txt
+recursive-include requirements *.yml
 recursive-include config *
 recursive-include docs/licenses *
 recursive-exclude awx devonly.py*
@@ -19,7 +20,7 @@ recursive-exclude awx/ui/client *
 recursive-exclude awx/settings local_settings.py*
 include tools/scripts/request_tower_configuration.sh
 include tools/scripts/request_tower_configuration.ps1
-include tools/scripts/ansible-tower-service
+include tools/scripts/automation-controller-service
 include tools/scripts/failure-event-handler
 include tools/scripts/awx-python
 include awx/playbooks/library/mkfifo.py

Makefile

@@ -1,85 +1,47 @@
-PYTHON ?= python3
+PYTHON ?= python3.8
 PYTHON_VERSION = $(shell $(PYTHON) -c "from distutils.sysconfig import get_python_version; print(get_python_version())")
-SITELIB=$(shell $(PYTHON) -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")
 OFFICIAL ?= no
-PACKER ?= packer
-PACKER_BUILD_OPTS ?= -var 'official=$(OFFICIAL)' -var 'aw_repo_url=$(AW_REPO_URL)'
 NODE ?= node
 NPM_BIN ?= npm
-DEPS_SCRIPT ?= packaging/bundle/deps.py
+CHROMIUM_BIN=/tmp/chrome-linux/chrome
 GIT_BRANCH ?= $(shell git rev-parse --abbrev-ref HEAD)
 MANAGEMENT_COMMAND ?= awx-manage
-IMAGE_REPOSITORY_AUTH ?=
-IMAGE_REPOSITORY_BASE ?= https://gcr.io
 VERSION := $(shell cat VERSION)
 
 # NOTE: This defaults the container image version to the branch that's active
 COMPOSE_TAG ?= $(GIT_BRANCH)
 COMPOSE_HOST ?= $(shell hostname)
+MAIN_NODE_TYPE ?= hybrid
 
-VENV_BASE ?= /venv
-COLLECTION_VENV ?= /awx_devel/awx_collection_test_venv
-SCL_PREFIX ?=
-CELERY_SCHEDULE_FILE ?= /var/lib/awx/beat.db
+VENV_BASE ?= /var/lib/awx/venv
+
+DEV_DOCKER_TAG_BASE ?= quay.io/awx
+DEVEL_IMAGE_NAME ?= $(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG)
 
-DEV_DOCKER_TAG_BASE ?= gcr.io/ansible-tower-engineering
 # Python packages to install only from source (not from binary wheels)
 # Comma separated list
 SRC_ONLY_PKGS ?= cffi,pycparser,psycopg2,twilio
-
-# Determine appropriate shasum command
-UNAME_S := $(shell uname -s)
-ifeq ($(UNAME_S),Linux)
-    SHASUM_BIN ?= sha256sum
-endif
-ifeq ($(UNAME_S),Darwin)
-    SHASUM_BIN ?= shasum -a 256
-endif
-
-# Get the branch information from git
-GIT_DATE := $(shell git log -n 1 --format="%ai")
-DATE := $(shell date -u +%Y%m%d%H%M)
+# These should be upgraded in the AWX and Ansible venv before attempting
+# to install the actual requirements
+VENV_BOOTSTRAP ?= pip==19.3.1 setuptools==41.6.0 wheel==0.36.2
 
 NAME ?= awx
-GIT_REMOTE_URL = $(shell git config --get remote.origin.url)
 
 # TAR build parameters
 SDIST_TAR_NAME=$(NAME)-$(VERSION)
-WHEEL_NAME=$(NAME)-$(VERSION)
 
 SDIST_COMMAND ?= sdist
-WHEEL_COMMAND ?= bdist_wheel
 SDIST_TAR_FILE ?= $(SDIST_TAR_NAME).tar.gz
-WHEEL_FILE ?= $(WHEEL_NAME)-py2-none-any.whl
-
-# UI flag files
-UI_DEPS_FLAG_FILE = awx/ui/.deps_built
-UI_RELEASE_DEPS_FLAG_FILE = awx/ui/.release_deps_built
-UI_RELEASE_FLAG_FILE = awx/ui/.release_built
 
 I18N_FLAG_FILE = .i18n_built
 
 .PHONY: awx-link clean clean-tmp clean-venv requirements requirements_dev \
-	develop refresh adduser migrate dbchange runserver \
+	develop refresh adduser migrate dbchange \
 	receiver test test_unit test_coverage coverage_html \
-	dev_build release_build release_clean sdist \
-	ui-docker-machine ui-docker ui-release ui-devel \
-	ui-test ui-deps ui-test-ci VERSION
-
-# remove ui build artifacts
-clean-ui: clean-languages
-	rm -rf awx/ui/static/
-	rm -rf awx/ui/node_modules/
-	rm -rf awx/ui/test/unit/reports/
-	rm -rf awx/ui/test/spec/reports/
-	rm -rf awx/ui/test/e2e/reports/
-	rm -rf awx/ui/client/languages/
-	rm -rf awx/ui_next/node_modules/
-	rm -rf awx/ui_next/coverage/
-	rm -rf awx/ui_next/build/locales/_build/
-	rm -f $(UI_DEPS_FLAG_FILE)
-	rm -f $(UI_RELEASE_DEPS_FLAG_FILE)
-	rm -f $(UI_RELEASE_FLAG_FILE)
+	dev_build release_build sdist \
+	ui-release ui-devel \
+	VERSION docker-compose-sources \
+	.git/hooks/pre-commit
 
 clean-tmp:
 	rm -rf tmp/
@@ -97,7 +59,7 @@ clean-schema:
 
 clean-languages:
 	rm -f $(I18N_FLAG_FILE)
-	find . -type f -regex ".*\.mo$$" -delete
+	find ./awx/locale/ -type f -regex ".*\.mo$" -delete
 
 # Remove temporary build files, compiled Python files.
 clean: clean-ui clean-api clean-awxkit clean-dist
@@ -119,7 +81,7 @@ clean-api:
 	rm -rf awx/projects
 
 clean-awxkit:
-	rm -rf awxkit/*.egg-info awxkit/.tox
+	rm -rf awxkit/*.egg-info awxkit/.tox awxkit/build/*
 
 # convenience target to assert environment variables are defined
 guard-%:
@@ -128,79 +90,38 @@ guard-%:
 	    exit 1; \
 	fi
 
-virtualenv: virtualenv_ansible virtualenv_awx
-
-virtualenv_ansible:
-	if [ "$(VENV_BASE)" ]; then \
-		if [ ! -d "$(VENV_BASE)" ]; then \
-			mkdir $(VENV_BASE); \
-		fi; \
-		if [ ! -d "$(VENV_BASE)/ansible" ]; then \
-			virtualenv -p python --system-site-packages $(VENV_BASE)/ansible && \
-			$(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) --ignore-installed six packaging appdirs && \
-			$(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) --ignore-installed setuptools==36.0.1 && \
-			$(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) --ignore-installed pip==9.0.1; \
-		fi; \
-	fi
-
-virtualenv_ansible_py3:
-	if [ "$(VENV_BASE)" ]; then \
-		if [ ! -d "$(VENV_BASE)" ]; then \
-			mkdir $(VENV_BASE); \
-		fi; \
-		if [ ! -d "$(VENV_BASE)/ansible" ]; then \
-			$(PYTHON) -m venv --system-site-packages $(VENV_BASE)/ansible; \
-		fi; \
-	fi
+virtualenv: virtualenv_awx
 
+# flit is needed for offline install of certain packages, specifically ptyprocess
+# it is needed for setup, but not always recognized as a setup dependency
+# similar to pip, setuptools, and wheel, these are all needed here as a bootstrapping issues
 virtualenv_awx:
 	if [ "$(VENV_BASE)" ]; then \
 		if [ ! -d "$(VENV_BASE)" ]; then \
 			mkdir $(VENV_BASE); \
 		fi; \
 		if [ ! -d "$(VENV_BASE)/awx" ]; then \
-			$(PYTHON) -m venv --system-site-packages $(VENV_BASE)/awx; \
-			$(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) --ignore-installed docutils==0.14; \
+			$(PYTHON) -m venv $(VENV_BASE)/awx; \
+			$(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) $(VENV_BOOTSTRAP); \
 		fi; \
 	fi
 
-requirements_ansible: virtualenv_ansible
-	if [[ "$(PIP_OPTIONS)" == *"--no-index"* ]]; then \
-	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_local.txt | $(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) --ignore-installed -r /dev/stdin ; \
-	else \
-	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_git.txt | $(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) --ignore-installed -r /dev/stdin ; \
-	fi
-	$(VENV_BASE)/ansible/bin/pip uninstall --yes -r requirements/requirements_ansible_uninstall.txt
-
-requirements_ansible_py3: virtualenv_ansible_py3
-	if [[ "$(PIP_OPTIONS)" == *"--no-index"* ]]; then \
-	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_local.txt | $(VENV_BASE)/ansible/bin/pip3 install $(PIP_OPTIONS) --ignore-installed -r /dev/stdin ; \
-	else \
-	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_git.txt | $(VENV_BASE)/ansible/bin/pip3 install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) --ignore-installed -r /dev/stdin ; \
-	fi
-	$(VENV_BASE)/ansible/bin/pip3 uninstall --yes -r requirements/requirements_ansible_uninstall.txt
-
-requirements_ansible_dev:
-	if [ "$(VENV_BASE)" ]; then \
-		$(VENV_BASE)/ansible/bin/pip install pytest mock; \
-	fi
-
 # Install third-party requirements needed for AWX's environment.
+# this does not use system site packages intentionally
 requirements_awx: virtualenv_awx
 	if [[ "$(PIP_OPTIONS)" == *"--no-index"* ]]; then \
-	    cat requirements/requirements.txt requirements/requirements_local.txt | $(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) --ignore-installed -r /dev/stdin ; \
+	    cat requirements/requirements.txt requirements/requirements_local.txt | $(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) -r /dev/stdin ; \
 	else \
-	    cat requirements/requirements.txt requirements/requirements_git.txt | $(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) --ignore-installed -r /dev/stdin ; \
+	    cat requirements/requirements.txt requirements/requirements_git.txt | $(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) -r /dev/stdin ; \
 	fi
-	echo "include-system-site-packages = true" >> $(VENV_BASE)/awx/lib/python$(PYTHON_VERSION)/pyvenv.cfg
 	$(VENV_BASE)/awx/bin/pip uninstall --yes -r requirements/requirements_tower_uninstall.txt
 
 requirements_awx_dev:
 	$(VENV_BASE)/awx/bin/pip install -r requirements/requirements_dev.txt
 
-requirements: requirements_ansible requirements_awx
+requirements: requirements_awx
 
-requirements_dev: requirements_awx requirements_ansible_py3 requirements_awx_dev requirements_ansible_dev
+requirements_dev: requirements_awx requirements_awx_dev
 
 requirements_test: requirements
 
@@ -219,7 +140,7 @@ version_file:
 	if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
 	fi; \
-	python -c "import awx; print(awx.__version__)" > /var/lib/awx/.awx_version; \
+	$(PYTHON) -c "import awx; print(awx.__version__)" > /var/lib/awx/.awx_version; \
 
 # Do any one-time init tasks.
 comma := ,
@@ -227,13 +148,17 @@ init:
 	if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
 	fi; \
-	$(MANAGEMENT_COMMAND) provision_instance --hostname=$(COMPOSE_HOST); \
-	$(MANAGEMENT_COMMAND) register_queue --queuename=tower --instance_percent=100;\
-	if [ "$(AWX_GROUP_QUEUES)" == "tower,thepentagon" ]; then \
-		$(MANAGEMENT_COMMAND) provision_instance --hostname=isolated; \
-		$(MANAGEMENT_COMMAND) register_queue --queuename='thepentagon' --hostnames=isolated --controller=tower; \
-		$(MANAGEMENT_COMMAND) generate_isolated_key > /awx_devel/awx/main/isolated/authorized_keys; \
-	fi;
+	$(MANAGEMENT_COMMAND) provision_instance --hostname=$(COMPOSE_HOST) --node_type=$(MAIN_NODE_TYPE); \
+	$(MANAGEMENT_COMMAND) register_queue --queuename=controlplane --instance_percent=100;\
+	$(MANAGEMENT_COMMAND) register_queue --queuename=default --instance_percent=100;
+	if [ ! -f /etc/receptor/certs/awx.key ]; then \
+		rm -f /etc/receptor/certs/*; \
+		receptor --cert-init commonname="AWX Test CA" bits=2048 outcert=/etc/receptor/certs/ca.crt outkey=/etc/receptor/certs/ca.key; \
+		for node in $(RECEPTOR_MUTUAL_TLS); do \
+			receptor --cert-makereq bits=2048 commonname="$$node test cert" dnsname=$$node nodeid=$$node outreq=/etc/receptor/certs/$$node.csr outkey=/etc/receptor/certs/$$node.key; \
+			receptor --cert-signreq req=/etc/receptor/certs/$$node.csr cacert=/etc/receptor/certs/ca.crt cakey=/etc/receptor/certs/ca.key outcert=/etc/receptor/certs/$$node.crt verify=yes; \
+		done; \
+	fi
 
 # Refresh development environment after pulling new code.
 refresh: clean requirements_dev version_file develop migrate
@@ -253,28 +178,6 @@ migrate:
 dbchange:
 	$(MANAGEMENT_COMMAND) makemigrations
 
-server_noattach:
-	tmux new-session -d -s awx 'exec make uwsgi'
-	tmux rename-window 'AWX'
-	tmux select-window -t awx:0
-	tmux split-window -v 'exec make dispatcher'
-	tmux new-window 'exec make daphne'
-	tmux select-window -t awx:1
-	tmux rename-window 'WebSockets'
-	tmux split-window -h 'exec make runworker'
-	tmux split-window -v 'exec make nginx'
-	tmux new-window 'exec make receiver'
-	tmux select-window -t awx:2
-	tmux rename-window 'Extra Services'
-	tmux select-window -t awx:0
-
-server: server_noattach
-	tmux -2 attach-session -t awx
-
-# Use with iterm2's native tmux protocol support
-servercc: server_noattach
-	tmux -2 -CC attach-session -t awx
-
 supervisor:
 	@if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
@@ -287,11 +190,27 @@ collectstatic:
 	fi; \
 	mkdir -p awx/public/static && $(PYTHON) manage.py collectstatic --clear --noinput > /dev/null 2>&1
 
+UWSGI_DEV_RELOAD_COMMAND ?= supervisorctl restart tower-processes:awx-dispatcher tower-processes:awx-receiver
+
 uwsgi: collectstatic
 	@if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
 	fi; \
-    uwsgi -b 32768 --socket 127.0.0.1:8050 --module=awx.wsgi:application --home=/venv/awx --chdir=/awx_devel/ --vacuum --processes=5 --harakiri=120 --master --no-orphans --py-autoreload 1 --max-requests=1000 --stats /tmp/stats.socket --lazy-apps --logformat "%(addr) %(method) %(uri) - %(proto) %(status)" --hook-accepting1="exec:supervisorctl restart tower-processes:awx-dispatcher tower-processes:awx-receiver"
+	uwsgi -b 32768 \
+	    --socket 127.0.0.1:8050 \
+	    --module=awx.wsgi:application \
+	    --home=/var/lib/awx/venv/awx \
+	    --chdir=/awx_devel/ \
+	    --vacuum \
+	    --processes=5 \
+	    --harakiri=120 --master \
+	    --no-orphans \
+	    --py-autoreload 1 \
+	    --max-requests=1000 \
+	    --stats /tmp/stats.socket \
+	    --lazy-apps \
+	    --logformat "%(addr) %(method) %(uri) - %(proto) %(status)" \
+	    --hook-accepting1="exec: $(UWSGI_DEV_RELOAD_COMMAND)"
 
 daphne:
 	@if [ "$(VENV_BASE)" ]; then \
@@ -299,18 +218,11 @@ daphne:
 	fi; \
 	daphne -b 127.0.0.1 -p 8051 awx.asgi:channel_layer
 
-runworker:
+wsbroadcast:
 	@if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
 	fi; \
-	$(PYTHON) manage.py runworker --only-channels websocket.*
-
-# Run the built-in development webserver (by default on http://localhost:8013).
-runserver:
-	@if [ "$(VENV_BASE)" ]; then \
-		. $(VENV_BASE)/awx/bin/activate; \
-	fi; \
-	$(PYTHON) manage.py runserver
+	$(PYTHON) manage.py run_wsbroadcast
 
 # Run to start the background task dispatcher for development.
 dispatcher:
@@ -339,20 +251,15 @@ jupyter:
 reports:
 	mkdir -p $@
 
-pep8: reports
-	@(set -o pipefail && $@ | tee reports/$@.report)
+black: reports
+	@command -v black >/dev/null 2>&1 || { echo "could not find black on your PATH, you may need to \`pip install black\`, or set AWX_IGNORE_BLACK=1" && exit 1; }
+	@(set -o pipefail && $@ $(BLACK_ARGS) awx awxkit awx_collection | tee reports/$@.report)
 
-flake8: reports
-	@if [ "$(VENV_BASE)" ]; then \
-		. $(VENV_BASE)/awx/bin/activate; \
-	fi; \
-	(set -o pipefail && $@ | tee reports/$@.report)
-
-pyflakes: reports
-	@(set -o pipefail && $@ | tee reports/$@.report)
-
-pylint: reports
-	@(set -o pipefail && $@ | reports/$@.report)
+.git/hooks/pre-commit:
+	@echo "if [ -x pre-commit.sh ]; then" > .git/hooks/pre-commit
+	@echo "    ./pre-commit.sh;" >> .git/hooks/pre-commit
+	@echo "fi" >> .git/hooks/pre-commit
+	@chmod +x .git/hooks/pre-commit
 
 genschema: reports
 	$(MAKE) swagger PYTEST_ARGS="--genschema --create-db "
@@ -364,54 +271,69 @@ swagger: reports
 	fi; \
 	(set -o pipefail && py.test $(PYTEST_ARGS) awx/conf/tests/functional awx/main/tests/functional/api awx/main/tests/docs --release=$(VERSION_TARGET) | tee reports/$@.report)
 
-check: flake8 pep8 # pyflakes pylint
+check: black
+
+api-lint:
+	BLACK_ARGS="--check" make black
+	flake8 awx
+	yamllint -s .
 
 awx-link:
-	cp -R /tmp/awx.egg-info /awx_devel/ || true
-	sed -i "s/placeholder/$(shell cat VERSION)/" /awx_devel/awx.egg-info/PKG-INFO
-	cp -f /tmp/awx.egg-link /venv/awx/lib/python$(PYTHON_VERSION)/site-packages/awx.egg-link
+	[ -d "/awx_devel/awx.egg-info" ] || $(PYTHON) /awx_devel/setup.py egg_info_dev
+	cp -f /tmp/awx.egg-link /var/lib/awx/venv/awx/lib/python$(PYTHON_VERSION)/site-packages/awx.egg-link
 
 TEST_DIRS ?= awx/main/tests/unit awx/main/tests/functional awx/conf/tests awx/sso/tests
 
 # Run all API unit tests.
 test:
-	@if [ "$(VENV_BASE)" ]; then \
+	if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
 	fi; \
 	PYTHONDONTWRITEBYTECODE=1 py.test -p no:cacheprovider -n auto $(TEST_DIRS)
-	cd awxkit && $(VENV_BASE)/awx/bin/tox -re py2,py3
-	awx-manage check_migrations --dry-run --check  -n 'vNNN_missing_migration_file'
-
-prepare_collection_venv:
-	rm -rf $(COLLECTION_VENV)
-	mkdir $(COLLECTION_VENV)
-	$(VENV_BASE)/awx/bin/pip install --target=$(COLLECTION_VENV) git+https://github.com/ansible/tower-cli.git
+	cmp VERSION awxkit/VERSION || "VERSION and awxkit/VERSION *must* match"
+	cd awxkit && $(VENV_BASE)/awx/bin/tox -re py3
+	awx-manage check_migrations --dry-run --check  -n 'missing_migration_file'
 
 COLLECTION_TEST_DIRS ?= awx_collection/test/awx
+COLLECTION_TEST_TARGET ?=
 COLLECTION_PACKAGE ?= awx
 COLLECTION_NAMESPACE ?= awx
+COLLECTION_INSTALL = ~/.ansible/collections/ansible_collections/$(COLLECTION_NAMESPACE)/$(COLLECTION_PACKAGE)
 
 test_collection:
-	@if [ "$(VENV_BASE)" ]; then \
+	rm -f $(shell ls -d $(VENV_BASE)/awx/lib/python* | head -n 1)/no-global-site-packages.txt
+	if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
-	fi; \
-	PYTHONPATH=$(COLLECTION_VENV):/awx_devel/awx_collection:$PYTHONPATH py.test $(COLLECTION_TEST_DIRS)
+	fi && \
+	pip install ansible-core && \
+	py.test $(COLLECTION_TEST_DIRS) -v
+	# The python path needs to be modified so that the tests can find Ansible within the container
+	# First we will use anything expility set as PYTHONPATH
+	# Second we will load any libraries out of the virtualenv (if it's unspecified that should be ok because python should not load out of an empty directory)
+	# Finally we will add the system path so that the tests can find the ansible libraries
 
-flake8_collection:
-	flake8 awx_collection/  # Different settings, in main exclude list
+test_collection_all: test_collection
 
-test_collection_all: prepare_collection_venv test_collection flake8_collection
-
-test_collection_sanity:
-	rm -rf sanity
-	mkdir -p sanity/ansible_collections/awx
-	cp -Ra awx_collection sanity/ansible_collections/awx/awx  # symlinks do not work
-	cd sanity/ansible_collections/awx/awx && git init && git add . # requires both this file structure and a git repo, so there you go
-	cd sanity/ansible_collections/awx/awx && ansible-test sanity
+# WARNING: symlinking a collection is fundamentally unstable
+# this is for rapid development iteration with playbooks, do not use with other test targets
+symlink_collection:
+	rm -rf $(COLLECTION_INSTALL)
+	mkdir -p ~/.ansible/collections/ansible_collections/$(COLLECTION_NAMESPACE)  # in case it does not exist
+	ln -s $(shell pwd)/awx_collection $(COLLECTION_INSTALL)
 
 build_collection:
-	ansible-playbook -i localhost, awx_collection/template_galaxy.yml -e collection_package=$(COLLECTION_PACKAGE) -e collection_namespace=$(COLLECTION_NAMESPACE) -e collection_version=$(VERSION)
-	ansible-galaxy collection build awx_collection --output-path=awx_collection
+	ansible-playbook -i localhost, awx_collection/tools/template_galaxy.yml -e collection_package=$(COLLECTION_PACKAGE) -e collection_namespace=$(COLLECTION_NAMESPACE) -e collection_version=$(VERSION) -e '{"awx_template_version":false}'
+	ansible-galaxy collection build awx_collection_build --force --output-path=awx_collection_build
+
+install_collection: build_collection
+	rm -rf $(COLLECTION_INSTALL)
+	ansible-galaxy collection install awx_collection_build/$(COLLECTION_NAMESPACE)-$(COLLECTION_PACKAGE)-$(VERSION).tar.gz
+
+test_collection_sanity: install_collection
+	cd $(COLLECTION_INSTALL) && ansible-test sanity
+
+test_collection_integration: install_collection
+	cd $(COLLECTION_INSTALL) && ansible-test integration $(COLLECTION_TEST_TARGET)
 
 test_unit:
 	@if [ "$(VENV_BASE)" ]; then \
@@ -442,147 +364,54 @@ bulk_data:
 	fi; \
 	$(PYTHON) tools/data_generators/rbac_dummy_data_generator.py --preset=$(DATA_GEN_PRESET)
 
-# l10n TASKS
-# --------------------------------------
-
-# check for UI po files
-HAVE_PO := $(shell ls awx/ui/po/*.po 2>/dev/null)
-check-po:
-ifdef HAVE_PO
-	# Should be 'Language: zh-CN' but not 'Language: zh_CN' in zh_CN.po
-	for po in awx/ui/po/*.po ; do \
-	    echo $$po; \
-	    mo="awx/ui/po/`basename $$po .po`.mo"; \
-	    msgfmt --check --verbose $$po -o $$mo; \
-	    if test "$$?" -ne 0 ; then \
-	        exit -1; \
-	    fi; \
-	    rm $$mo; \
-	    name=`echo "$$po" | grep '-'`; \
-	    if test "x$$name" != x ; then \
-	        right_name=`echo $$language | sed -e 's/-/_/'`; \
-	        echo "ERROR: WRONG $$name CORRECTION: $$right_name"; \
-	        exit -1; \
-	    fi; \
-	    language=`grep '^"Language:' "$$po" | grep '_'`; \
-	    if test "x$$language" != x ; then \
-	        right_language=`echo $$language | sed -e 's/_/-/'`; \
-	        echo "ERROR: WRONG $$language CORRECTION: $$right_language in $$po"; \
-	        exit -1; \
-	    fi; \
-	done;
-else
-	@echo No PO files
-endif
-
-# generate UI .pot
-pot: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run pot
-
-# generate django .pot .po
-LANG = "en-us"
-messages:
-	@if [ "$(VENV_BASE)" ]; then \
-		. $(VENV_BASE)/awx/bin/activate; \
-	fi; \
-	$(PYTHON) manage.py makemessages -l $(LANG) --keep-pot
-
-# generate l10n .json .mo
-languages: $(I18N_FLAG_FILE)
-
-$(I18N_FLAG_FILE): $(UI_RELEASE_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run languages
-	$(PYTHON) tools/scripts/compilemessages.py
-	touch $(I18N_FLAG_FILE)
-
-# End l10n TASKS
-# --------------------------------------
-
-# UI RELEASE TASKS
-# --------------------------------------
-ui-release: $(UI_RELEASE_FLAG_FILE)
-
-$(UI_RELEASE_FLAG_FILE): $(I18N_FLAG_FILE) $(UI_RELEASE_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run build-release
-	touch $(UI_RELEASE_FLAG_FILE)
-
-$(UI_RELEASE_DEPS_FLAG_FILE):
-	PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=1 $(NPM_BIN) --unsafe-perm --prefix awx/ui ci --no-save awx/ui
-	touch $(UI_RELEASE_DEPS_FLAG_FILE)
-
-# END UI RELEASE TASKS
-# --------------------------------------
 
 # UI TASKS
 # --------------------------------------
-ui-deps: $(UI_DEPS_FLAG_FILE)
 
-$(UI_DEPS_FLAG_FILE):
-	@if [ -f ${UI_RELEASE_DEPS_FLAG_FILE} ]; then \
-		rm -rf awx/ui/node_modules; \
-		rm -f ${UI_RELEASE_DEPS_FLAG_FILE}; \
-	fi; \
-	$(NPM_BIN) --unsafe-perm --prefix awx/ui ci --no-save awx/ui
-	touch $(UI_DEPS_FLAG_FILE)
+UI_BUILD_FLAG_FILE = awx/ui/.ui-built
 
-ui-docker-machine: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run ui-docker-machine -- $(MAKEFLAGS)
+clean-ui:
+	rm -rf node_modules
+	rm -rf awx/ui/node_modules
+	rm -rf awx/ui/build
+	rm -rf awx/ui/src/locales/_build
+	rm -rf $(UI_BUILD_FLAG_FILE)
 
-# Native docker. Builds UI and raises BrowserSync & filesystem polling.
-ui-docker: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run ui-docker -- $(MAKEFLAGS)
+awx/ui/node_modules:
+	NODE_OPTIONS=--max-old-space-size=4096 $(NPM_BIN) --prefix awx/ui --loglevel warn ci
 
-# Builds UI with development UI without raising browser-sync or filesystem polling.
-ui-devel: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run build-devel -- $(MAKEFLAGS)
+$(UI_BUILD_FLAG_FILE):
+	$(PYTHON) tools/scripts/compilemessages.py
+	$(NPM_BIN) --prefix awx/ui --loglevel warn run compile-strings
+	$(NPM_BIN) --prefix awx/ui --loglevel warn run build
+	mkdir -p awx/public/static/css
+	mkdir -p awx/public/static/js
+	mkdir -p awx/public/static/media
+	cp -r awx/ui/build/static/css/* awx/public/static/css
+	cp -r awx/ui/build/static/js/* awx/public/static/js
+	cp -r awx/ui/build/static/media/* awx/public/static/media
+	touch $@
 
-ui-test: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run test
+ui-release: awx/ui/node_modules $(UI_BUILD_FLAG_FILE)
 
-ui-lint: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) run --prefix awx/ui jshint
+ui-devel: awx/ui/node_modules
+	@$(MAKE) -B $(UI_BUILD_FLAG_FILE)
+
+ui-devel-instrumented: awx/ui/node_modules
+	$(NPM_BIN) --prefix awx/ui --loglevel warn run start-instrumented
+
+ui-devel-test: awx/ui/node_modules
+	$(NPM_BIN) --prefix awx/ui --loglevel warn run start
+
+ui-lint:
+	$(NPM_BIN) --prefix awx/ui install
 	$(NPM_BIN) run --prefix awx/ui lint
+	$(NPM_BIN) run --prefix awx/ui prettier-check
 
-# A standard go-to target for API developers to use building the frontend
-ui: clean-ui ui-devel
+ui-test:
+	$(NPM_BIN) --prefix awx/ui install
+	$(NPM_BIN) run --prefix awx/ui test -- --coverage --maxWorkers=4 --watchAll=false
 
-ui-test-ci: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run test:ci
-	$(NPM_BIN) --prefix awx/ui run unit
-
-jshint: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) run --prefix awx/ui jshint
-	$(NPM_BIN) run --prefix awx/ui lint
-
-ui-zuul-lint-and-test: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) run --prefix awx/ui jshint
-	$(NPM_BIN) run --prefix awx/ui lint
-	$(NPM_BIN) --prefix awx/ui run test:ci
-	$(NPM_BIN) --prefix awx/ui run unit
-
-# END UI TASKS
-# --------------------------------------
-
-# UI NEXT TASKS
-# --------------------------------------
-
-ui-next-lint:
-	$(NPM_BIN) --prefix awx/ui_next install
-	$(NPM_BIN) run --prefix awx/ui_next lint
-	$(NPM_BIN) run --prefix awx/ui_next prettier-check
-
-ui-next-test:
-	$(NPM_BIN) --prefix awx/ui_next install
-	$(NPM_BIN) run --prefix awx/ui_next test
-
-ui-next-zuul-lint-and-test:
-	$(NPM_BIN) --prefix awx/ui_next install
-	$(NPM_BIN) run --prefix awx/ui_next lint
-	$(NPM_BIN) run --prefix awx/ui_next prettier-check
-	$(NPM_BIN) run --prefix awx/ui_next test
-
-# END UI NEXT TASKS
-# --------------------------------------
 
 # Build a pip-installable package into dist/ with a timestamped version number.
 dev_build:
@@ -595,59 +424,52 @@ release_build:
 dist/$(SDIST_TAR_FILE): ui-release VERSION
 	$(PYTHON) setup.py $(SDIST_COMMAND)
 
-dist/$(WHEEL_FILE): ui-release
-	$(PYTHON) setup.py $(WHEEL_COMMAND)
-
 sdist: dist/$(SDIST_TAR_FILE)
 	@echo "#############################################"
 	@echo "Artifacts:"
 	@echo dist/$(SDIST_TAR_FILE)
 	@echo "#############################################"
 
-wheel: dist/$(WHEEL_FILE)
-	@echo "#############################################"
-	@echo "Artifacts:"
-	@echo dist/$(WHEEL_FILE)
-	@echo "#############################################"
-
-# Build setup bundle tarball
-setup-bundle-build:
-	mkdir -p $@
-
-docker-auth:
-	@if [ "$(IMAGE_REPOSITORY_AUTH)" ]; then \
-		echo "$(IMAGE_REPOSITORY_AUTH)" | docker login -u oauth2accesstoken --password-stdin $(IMAGE_REPOSITORY_BASE); \
-	fi;
-
 # This directory is bind-mounted inside of the development container and
 # needs to be pre-created for permissions to be set correctly. Otherwise,
 # Docker will create this directory as root.
 awx/projects:
 	@mkdir -p $@
 
-# Docker isolated rampart
-docker-compose-isolated: awx/projects
-	CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml -f tools/docker-isolated-override.yml up
+COMPOSE_UP_OPTS ?=
+COMPOSE_OPTS ?=
+CONTROL_PLANE_NODE_COUNT ?= 1
+EXECUTION_NODE_COUNT ?= 2
+MINIKUBE_CONTAINER_GROUP ?= false
 
-# Docker Compose Development environment
-docker-compose: docker-auth awx/projects
-	CURRENT_UID=$(shell id -u) OS="$(shell docker info | grep 'Operating System')" TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml up --no-recreate awx
+docker-compose-sources: .git/hooks/pre-commit
+	@if [ $(MINIKUBE_CONTAINER_GROUP) = true ]; then\
+	    ansible-playbook -i tools/docker-compose/inventory tools/docker-compose-minikube/deploy.yml; \
+	fi;
 
-docker-compose-cluster: docker-auth awx/projects
-	CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose-cluster.yml up
+	ansible-playbook -i tools/docker-compose/inventory tools/docker-compose/ansible/sources.yml \
+	    -e awx_image=$(DEV_DOCKER_TAG_BASE)/awx_devel \
+	    -e awx_image_tag=$(COMPOSE_TAG) \
+	    -e control_plane_node_count=$(CONTROL_PLANE_NODE_COUNT) \
+	    -e execution_node_count=$(EXECUTION_NODE_COUNT) \
+	    -e minikube_container_group=$(MINIKUBE_CONTAINER_GROUP)
 
-docker-compose-credential-plugins: docker-auth awx/projects
+
+docker-compose: docker-auth awx/projects docker-compose-sources
+	docker-compose -f tools/docker-compose/_sources/docker-compose.yml $(COMPOSE_OPTS) up $(COMPOSE_UP_OPTS) --remove-orphans
+
+docker-compose-credential-plugins: docker-auth awx/projects docker-compose-sources
 	echo -e "\033[0;31mTo generate a CyberArk Conjur API key: docker exec -it tools_conjur_1 conjurctl account create quick-start\033[0m"
-	CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml -f tools/docker-credential-plugins-override.yml up --no-recreate awx
+	docker-compose -f tools/docker-compose/_sources/docker-compose.yml -f tools/docker-credential-plugins-override.yml up --no-recreate awx_1 --remove-orphans
 
-docker-compose-test: docker-auth awx/projects
-	cd tools && CURRENT_UID=$(shell id -u) OS="$(shell docker info | grep 'Operating System')" TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm --service-ports awx /bin/bash
+docker-compose-test: docker-auth awx/projects docker-compose-sources
+	docker-compose -f tools/docker-compose/_sources/docker-compose.yml run --rm --service-ports awx_1 /bin/bash
 
-docker-compose-runtest: awx/projects
-	cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm --service-ports awx /start_tests.sh
+docker-compose-runtest: awx/projects docker-compose-sources
+	docker-compose -f tools/docker-compose/_sources/docker-compose.yml run --rm --service-ports awx_1 /start_tests.sh
 
-docker-compose-build-swagger: awx/projects
-	cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm --service-ports awx /start_tests.sh swagger
+docker-compose-build-swagger: awx/projects docker-compose-sources
+	docker-compose -f tools/docker-compose/_sources/docker-compose.yml run --rm --service-ports --no-deps awx_1 /start_tests.sh swagger
 
 detect-schema-change: genschema
 	curl https://s3.amazonaws.com/awx-public-ci-files/schema.json -o reference-schema.json
@@ -655,44 +477,44 @@ detect-schema-change: genschema
 	diff -u -b reference-schema.json schema.json
 
 docker-compose-clean: awx/projects
-	cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm -w /awx_devel --service-ports awx make clean
-	cd tools && TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose rm -sf
+	docker-compose -f tools/docker-compose/_sources/docker-compose.yml rm -sf
 
-docker-compose-build: awx-devel-build
+docker-compose-container-group-clean:
+	@if [ -f "tools/docker-compose-minikube/_sources/minikube" ]; then \
+	    tools/docker-compose-minikube/_sources/minikube delete; \
+	fi
+	rm -rf tools/docker-compose-minikube/_sources/
 
 # Base development image build
-awx-devel-build:
-	docker build -t ansible/awx_devel -f tools/docker-compose/Dockerfile \
-		--cache-from=$(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG) .
-	docker tag ansible/awx_devel $(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG)
-	#docker push $(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG)
+docker-compose-build:
+	ansible-playbook tools/ansible/dockerfile.yml -e build_dev=True
+	DOCKER_BUILDKIT=1 docker build -t $(DEVEL_IMAGE_NAME) \
+	    --build-arg BUILDKIT_INLINE_CACHE=1 \
+	    --cache-from=$(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG) .
 
-# For use when developing on "isolated" AWX deployments
-docker-compose-isolated-build: awx-devel-build
-	docker build -t ansible/awx_isolated -f tools/docker-isolated/Dockerfile .
-	docker tag ansible/awx_isolated $(DEV_DOCKER_TAG_BASE)/awx_isolated:$(COMPOSE_TAG)
-	#docker push $(DEV_DOCKER_TAG_BASE)/awx_isolated:$(COMPOSE_TAG)
-
-MACHINE?=default
 docker-clean:
-	eval $$(docker-machine env $(MACHINE))
-	$(foreach container_id,$(shell docker ps -f name=tools_awx -aq),docker stop $(container_id); docker rm -f $(container_id);)
-	-docker images | grep "awx_devel" | awk '{print $$1 ":" $$2}' | xargs docker rmi
+	$(foreach container_id,$(shell docker ps -f name=tools_awx -aq && docker ps -f name=tools_receptor -aq),docker stop $(container_id); docker rm -f $(container_id);)
+	if [ $(shell docker images | grep "awx_devel") ]; then \
+	  docker images | grep "awx_devel" | awk '{print $$3}' | xargs docker rmi --force; \
+	fi
+
+docker-clean-volumes: docker-compose-clean docker-compose-container-group-clean
+	docker volume rm tools_awx_db
 
 docker-refresh: docker-clean docker-compose
 
 # Docker Development Environment with Elastic Stack Connected
-docker-compose-elk: docker-auth awx/projects
-	CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml -f tools/elastic/docker-compose.logstash-link.yml -f tools/elastic/docker-compose.elastic-override.yml up --no-recreate
+docker-compose-elk: docker-auth awx/projects docker-compose-sources
+	docker-compose -f tools/docker-compose/_sources/docker-compose.yml -f tools/elastic/docker-compose.logstash-link.yml -f tools/elastic/docker-compose.elastic-override.yml up --no-recreate
 
-docker-compose-cluster-elk: docker-auth awx/projects
-	TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose-cluster.yml -f tools/elastic/docker-compose.logstash-link-cluster.yml -f tools/elastic/docker-compose.elastic-override.yml up --no-recreate
+docker-compose-cluster-elk: docker-auth awx/projects docker-compose-sources
+	docker-compose -f tools/docker-compose/_sources/docker-compose.yml -f tools/elastic/docker-compose.logstash-link-cluster.yml -f tools/elastic/docker-compose.elastic-override.yml up --no-recreate
 
 prometheus:
 	docker run -u0 --net=tools_default --link=`docker ps | egrep -o "tools_awx(_run)?_([^ ]+)?"`:awxweb --volume `pwd`/tools/prometheus:/prometheus --name prometheus -d -p 0.0.0.0:9090:9090 prom/prometheus --web.enable-lifecycle --config.file=/prometheus/prometheus.yml
 
-minishift-dev:
-	ansible-playbook -i localhost, -e devtree_directory=$(CURDIR) tools/clusterdevel/start_minishift_dev.yml
+docker-compose-container-group:
+	MINIKUBE_CONTAINER_GROUP=true make docker-compose
 
 clean-elk:
 	docker stop tools_kibana_1
@@ -703,7 +525,41 @@ clean-elk:
 	docker rm tools_kibana_1
 
 psql-container:
-	docker run -it --net tools_default --rm postgres:10 sh -c 'exec psql -h "postgres" -p "5432" -U postgres'
+	docker run -it --net tools_default --rm postgres:12 sh -c 'exec psql -h "postgres" -p "5432" -U postgres'
 
 VERSION:
 	@echo "awx: $(VERSION)"
+
+Dockerfile: tools/ansible/roles/dockerfile/templates/Dockerfile.j2
+	ansible-playbook tools/ansible/dockerfile.yml
+
+Dockerfile.kube-dev: tools/ansible/roles/dockerfile/templates/Dockerfile.j2
+	ansible-playbook tools/ansible/dockerfile.yml \
+	    -e dockerfile_name=Dockerfile.kube-dev \
+	    -e kube_dev=True \
+	    -e template_dest=_build_kube_dev
+
+awx-kube-dev-build: Dockerfile.kube-dev
+	docker build -f Dockerfile.kube-dev \
+	    --build-arg BUILDKIT_INLINE_CACHE=1 \
+	    -t $(DEV_DOCKER_TAG_BASE)/awx_kube_devel:$(COMPOSE_TAG) .
+
+
+# Translation TASKS
+# --------------------------------------
+
+# generate UI .pot file, an empty template of strings yet to be translated
+pot: $(UI_BUILD_FLAG_FILE)
+	$(NPM_BIN) --prefix awx/ui --loglevel warn run extract-template --clean
+
+# generate UI .po files for each locale (will update translated strings for `en`)
+po: $(UI_BUILD_FLAG_FILE)
+	$(NPM_BIN) --prefix awx/ui --loglevel warn run extract-strings -- --clean
+
+# generate API django .pot .po
+LANG = "en-us"
+messages:
+	@if [ "$(VENV_BASE)" ]; then \
+		. $(VENV_BASE)/awx/bin/activate; \
+	fi; \
+	$(PYTHON) manage.py makemessages -l $(LANG) --keep-pot

README.md

@@ -1,8 +1,9 @@
-[![Gated by Zuul](https://zuul-ci.org/gated.svg)](https://ansible.softwarefactory-project.io/zuul/status)
+[![CI](https://github.com/ansible/awx/actions/workflows/ci.yml/badge.svg?branch=devel)](https://github.com/ansible/awx/actions/workflows/ci.yml) [![Code of Conduct](https://img.shields.io/badge/code%20of%20conduct-Ansible-yellow.svg)](https://docs.ansible.com/ansible/latest/community/code_of_conduct.html) [![Apache v2 License](https://img.shields.io/badge/license-Apache%202.0-brightgreen.svg)](https://github.com/ansible/awx/blob/devel/LICENSE.md) [![AWX Mailing List](https://img.shields.io/badge/mailing%20list-AWX-orange.svg)](https://groups.google.com/g/awx-project)
+[![IRC Chat - #ansible-awx](https://img.shields.io/badge/IRC-%23ansible--awx-blueviolet.svg)](https://libera.chat)
 
 <img src="https://raw.githubusercontent.com/ansible/awx-logos/master/awx/ui/client/assets/logo-login.svg?sanitize=true" width=200 alt="AWX" />
 
-AWX provides a web-based user interface, REST API, and task engine built on top of [Ansible](https://github.com/ansible/ansible). It is the upstream project for [Tower](https://www.ansible.com/tower), a commercial derivative of AWX.  
+AWX provides a web-based user interface, REST API, and task engine built on top of [Ansible](https://github.com/ansible/ansible). It is one of the upstream projects for [Red Hat Ansible Automation Platform](https://www.ansible.com/products/automation-platform).
 
 To install AWX, please view the [Install guide](./INSTALL.md).
 
@@ -16,31 +17,25 @@ Contributing
 ------------
 
 - Refer to the [Contributing guide](./CONTRIBUTING.md) to get started developing, testing, and building AWX.
-- All code submissions are done through pull requests against the `devel` branch.
-- All contributors must use git commit --signoff for any commit to be merged, and agree that usage of --signoff constitutes agreement with the terms of [DCO 1.1](./DCO_1_1.md)
-- Take care to make sure no merge commits are in the submission, and use `git rebase` vs `git merge` for this reason.
-- If submitting a large code change, it's a good idea to join the `#ansible-awx` channel on irc.freenode.net, and talk about what you would like to do or add first. This not only helps everyone know what's going on, it also helps save time and effort, if the community decides some changes are needed.
+- All code submissions are made through pull requests against the `devel` branch.
+- All contributors must use git commit --signoff for any commit to be merged and agree that usage of --signoff constitutes agreement with the terms of [DCO 1.1](./DCO_1_1.md)
+- Take care to make sure no merge commits are in the submission, and use `git rebase` vs. `git merge` for this reason.
+- If submitting a large code change, it's a good idea to join the `#ansible-awx` channel on web.libera.chat and talk about what you would like to do or add first. This not only helps everyone know what's going on, but it also helps save time and effort if the community decides some changes are needed.
 
 Reporting Issues
 ----------------
 
-If you're experiencing a problem that you feel is a bug in AWX, or have ideas for how to improve AWX, we encourage you to open an issue, and share your feedback. But before opening a new issue, we ask that you please take a look at our [Issues guide](./ISSUES.md).
+If you're experiencing a problem that you feel is a bug in AWX or have ideas for improving AWX, we encourage you to open an issue and share your feedback. But before opening a new issue, we ask that you please take a look at our [Issues guide](./ISSUES.md).
 
 Code of Conduct
 ---------------
 
-We ask all of our community members and contributors to adhere to the [Ansible code of conduct](http://docs.ansible.com/ansible/latest/community/code_of_conduct.html). If you have questions, or need assistance, please reach out to our community team at [codeofconduct@ansible.com](mailto:codeofconduct@ansible.com)   
+We ask all of our community members and contributors to adhere to the [Ansible code of conduct](http://docs.ansible.com/ansible/latest/community/code_of_conduct.html). If you have questions or need assistance, please reach out to our community team at [codeofconduct@ansible.com](mailto:codeofconduct@ansible.com)   
 
 Get Involved
 ------------
 
 We welcome your feedback and ideas. Here's how to reach us with feedback and questions:
 
-- Join the `#ansible-awx` channel on irc.freenode.net
+- Join the `#ansible-awx` channel on irc.libera.chat
 - Join the [mailing list](https://groups.google.com/forum/#!forum/awx-project) 
-
-License
--------
-
-[Apache v2](./LICENSE.md)
-

VERSION

@@ -1 +1 @@
-9.1.0
+19.4.0

awx/__init__.py

@@ -15,43 +15,35 @@ __all__ = ['__version__']
 # Check for the presence/absence of "devonly" module to determine if running
 # from a source code checkout or release packaage.
 try:
-    import awx.devonly # noqa
+    import awx.devonly  # noqa
+
     MODE = 'development'
-except ImportError: # pragma: no cover
+except ImportError:  # pragma: no cover
     MODE = 'production'
 
 
 import hashlib
 
 try:
-    import django
-    from django.db.backends.base import schema
-    from django.db.backends.utils import names_digest
+    import django  # noqa: F401
+
     HAS_DJANGO = True
 except ImportError:
     HAS_DJANGO = False
+else:
+    from django.db.backends.base import schema
+    from django.db.models import indexes
+    from django.db.backends.utils import names_digest
+    from django.db import connection
 
 
 if HAS_DJANGO is True:
-    # This line exists to make sure we don't regress on FIPS support if we
-    # upgrade Django; if you're upgrading Django and see this error,
-    # update the version check below, and confirm that FIPS still works.
-    # If operating in a FIPS environment, `hashlib.md5()` will raise a `ValueError`,
-    # but will support the `usedforsecurity` keyword on RHEL and Centos systems.
-
-    # Keep an eye on https://code.djangoproject.com/ticket/28401
-    target_version = '2.2.4'
-    if django.__version__ != target_version:
-        raise RuntimeError(
-            "Django version other than {target} detected: {current}. "
-            "Overriding `names_digest` is known to work for Django {target} "
-            "and may not work in other Django versions.".format(target=target_version,
-                                                                current=django.__version__)
-        )
 
+    # See upgrade blocker note in requirements/README.md
     try:
         names_digest('foo', 'bar', 'baz', length=8)
     except ValueError:
+
         def names_digest(*args, length):
             """
             Generate a 32-bit digest of a set of arguments that can be used to shorten
@@ -63,6 +55,7 @@ if HAS_DJANGO is True:
             return h.hexdigest()[:length]
 
         schema.names_digest = names_digest
+        indexes.names_digest = names_digest
 
 
 def find_commands(management_dir):
@@ -75,7 +68,7 @@ def find_commands(management_dir):
                 continue
             elif f.endswith('.py') and f[:-3] not in commands:
                 commands.append(f[:-3])
-            elif f.endswith('.pyc') and f[:-4] not in commands: # pragma: no cover
+            elif f.endswith('.pyc') and f[:-4] not in commands:  # pragma: no cover
                 commands.append(f[:-4])
     except OSError:
         pass
@@ -86,13 +79,14 @@ def oauth2_getattribute(self, attr):
     # Custom method to override
     # oauth2_provider.settings.OAuth2ProviderSettings.__getattribute__
     from django.conf import settings
+
     val = None
     if 'migrate' not in sys.argv:
         # certain Django OAuth Toolkit migrations actually reference
         # setting lookups for references to model classes (e.g.,
         # oauth2_settings.REFRESH_TOKEN_MODEL)
         # If we're doing an OAuth2 setting lookup *while running* a migration,
-        # don't do our usual "Configure Tower in Tower" database setting lookup
+        # don't do our usual database settings lookup
         val = settings.OAUTH2_PROVIDER.get(attr)
     if val is None:
         val = object.__getattribute__(self, attr)
@@ -105,33 +99,38 @@ def prepare_env():
     # Hide DeprecationWarnings when running in production.  Need to first load
     # settings to apply our filter after Django's own warnings filter.
     from django.conf import settings
-    if not settings.DEBUG: # pragma: no cover
+
+    if not settings.DEBUG:  # pragma: no cover
         warnings.simplefilter('ignore', DeprecationWarning)
     # Monkeypatch Django find_commands to also work with .pyc files.
     import django.core.management
+
     django.core.management.find_commands = find_commands
 
     # Monkeypatch Oauth2 toolkit settings class to check for settings
     # in django.conf settings each time, not just once during import
     import oauth2_provider.settings
+
     oauth2_provider.settings.OAuth2ProviderSettings.__getattribute__ = oauth2_getattribute
 
     # Use the AWX_TEST_DATABASE_* environment variables to specify the test
     # database settings to use when management command is run as an external
     # program via unit tests.
-    for opt in ('ENGINE', 'NAME', 'USER', 'PASSWORD', 'HOST', 'PORT'): # pragma: no cover
+    for opt in ('ENGINE', 'NAME', 'USER', 'PASSWORD', 'HOST', 'PORT'):  # pragma: no cover
         if os.environ.get('AWX_TEST_DATABASE_%s' % opt, None):
             settings.DATABASES['default'][opt] = os.environ['AWX_TEST_DATABASE_%s' % opt]
     # Disable capturing all SQL queries in memory when in DEBUG mode.
     if settings.DEBUG and not getattr(settings, 'SQL_DEBUG', True):
         from django.db.backends.base.base import BaseDatabaseWrapper
         from django.db.backends.utils import CursorWrapper
+
         BaseDatabaseWrapper.make_debug_cursor = lambda self, cursor: CursorWrapper(cursor, self)
 
     # Use the default devserver addr/port defined in settings for runserver.
     default_addr = getattr(settings, 'DEVSERVER_DEFAULT_ADDR', '127.0.0.1')
     default_port = getattr(settings, 'DEVSERVER_DEFAULT_PORT', 8000)
     from django.core.management.commands import runserver as core_runserver
+
     original_handle = core_runserver.Command.handle
 
     def handle(self, *args, **options):
@@ -150,7 +149,14 @@ def manage():
     # Now run the command (or display the version).
     from django.conf import settings
     from django.core.management import execute_from_command_line
-    if len(sys.argv) >= 2 and sys.argv[1] in ('version', '--version'): # pragma: no cover
+
+    # enforce the postgres version is equal to 12. if not, then terminate program with exit code of 1
+    if not MODE == 'development':
+        if (connection.pg_version // 10000) < 12:
+            sys.stderr.write("Postgres version 12 is required\n")
+            sys.exit(1)
+
+    if len(sys.argv) >= 2 and sys.argv[1] in ('version', '--version'):  # pragma: no cover
         sys.stdout.write('%s\n' % __version__)
     # If running as a user without permission to read settings, display an
     # error message.  Allow --help to still work.

awx/api/authentication.py

@@ -18,7 +18,6 @@ logger = logging.getLogger('awx.api.authentication')
 
 
 class LoggedBasicAuthentication(authentication.BasicAuthentication):
-
     def authenticate(self, request):
         if not settings.AUTH_BASIC_ENABLED:
             return
@@ -35,22 +34,18 @@ class LoggedBasicAuthentication(authentication.BasicAuthentication):
 
 
 class SessionAuthentication(authentication.SessionAuthentication):
-    
     def authenticate_header(self, request):
         return 'Session'
 
 
 class LoggedOAuth2Authentication(OAuth2Authentication):
-
     def authenticate(self, request):
         ret = super(LoggedOAuth2Authentication, self).authenticate(request)
         if ret:
             user, token = ret
             username = user.username if user else '<none>'
-            logger.info(smart_text(
-                u"User {} performed a {} to {} through the API using OAuth 2 token {}.".format(
-                    username, request.method, request.path, token.pk
-                )
-            ))
+            logger.info(
+                smart_text(u"User {} performed a {} to {} through the API using OAuth 2 token {}.".format(username, request.method, request.path, token.pk))
+            )
             setattr(user, 'oauth_scopes', [x for x in token.scope.split() if x])
         return ret

awx/api/conf.py

@@ -1,8 +1,12 @@
 # Django
+from django.conf import settings
 from django.utils.translation import ugettext_lazy as _
 
+# Django REST Framework
+from rest_framework import serializers
+
 # AWX
-from awx.conf import fields, register
+from awx.conf import fields, register, register_validate
 from awx.api.fields import OAuth2ProviderField
 from oauth2_provider.settings import oauth2_settings
 
@@ -16,6 +20,7 @@ register(
     help_text=_('Number of seconds that a user is inactive before they will need to login again.'),
     category=_('Authentication'),
     category_slug='authentication',
+    unit=_('seconds'),
 )
 register(
     'SESSIONS_PER_USER',
@@ -26,6 +31,17 @@ register(
     category=_('Authentication'),
     category_slug='authentication',
 )
+register(
+    'DISABLE_LOCAL_AUTH',
+    field_class=fields.BooleanField,
+    label=_('Disable the built-in authentication system'),
+    help_text=_(
+        "Controls whether users are prevented from using the built-in authentication system. "
+        "You probably want to do this if you are using an LDAP or SAML integration."
+    ),
+    category=_('Authentication'),
+    category_slug='authentication',
+)
 register(
     'AUTH_BASIC_ENABLED',
     field_class=fields.BooleanField,
@@ -37,28 +53,35 @@ register(
 register(
     'OAUTH2_PROVIDER',
     field_class=OAuth2ProviderField,
-    default={'ACCESS_TOKEN_EXPIRE_SECONDS': oauth2_settings.ACCESS_TOKEN_EXPIRE_SECONDS,
-             'AUTHORIZATION_CODE_EXPIRE_SECONDS': oauth2_settings.AUTHORIZATION_CODE_EXPIRE_SECONDS,
-             'REFRESH_TOKEN_EXPIRE_SECONDS': oauth2_settings.REFRESH_TOKEN_EXPIRE_SECONDS},
+    default={
+        'ACCESS_TOKEN_EXPIRE_SECONDS': oauth2_settings.ACCESS_TOKEN_EXPIRE_SECONDS,
+        'AUTHORIZATION_CODE_EXPIRE_SECONDS': oauth2_settings.AUTHORIZATION_CODE_EXPIRE_SECONDS,
+        'REFRESH_TOKEN_EXPIRE_SECONDS': oauth2_settings.REFRESH_TOKEN_EXPIRE_SECONDS,
+    },
     label=_('OAuth 2 Timeout Settings'),
-    help_text=_('Dictionary for customizing OAuth 2 timeouts, available items are '
-                '`ACCESS_TOKEN_EXPIRE_SECONDS`, the duration of access tokens in the number '
-                'of seconds, `AUTHORIZATION_CODE_EXPIRE_SECONDS`, the duration of '
-                'authorization codes in the number of seconds, and `REFRESH_TOKEN_EXPIRE_SECONDS`, '
-                'the duration of refresh tokens, after expired access tokens, '
-                'in the number of seconds.'),
+    help_text=_(
+        'Dictionary for customizing OAuth 2 timeouts, available items are '
+        '`ACCESS_TOKEN_EXPIRE_SECONDS`, the duration of access tokens in the number '
+        'of seconds, `AUTHORIZATION_CODE_EXPIRE_SECONDS`, the duration of '
+        'authorization codes in the number of seconds, and `REFRESH_TOKEN_EXPIRE_SECONDS`, '
+        'the duration of refresh tokens, after expired access tokens, '
+        'in the number of seconds.'
+    ),
     category=_('Authentication'),
     category_slug='authentication',
+    unit=_('seconds'),
 )
 register(
     'ALLOW_OAUTH2_FOR_EXTERNAL_USERS',
     field_class=fields.BooleanField,
     default=False,
     label=_('Allow External Users to Create OAuth2 Tokens'),
-    help_text=_('For security reasons, users from external auth providers (LDAP, SAML, '
-                'SSO, Radius, and others) are not allowed to create OAuth2 tokens. '
-                'To change this behavior, enable this setting. Existing tokens will '
-                'not be deleted when this setting is toggled off.'),
+    help_text=_(
+        'For security reasons, users from external auth providers (LDAP, SAML, '
+        'SSO, Radius, and others) are not allowed to create OAuth2 tokens. '
+        'To change this behavior, enable this setting. Existing tokens will '
+        'not be deleted when this setting is toggled off.'
+    ),
     category=_('Authentication'),
     category_slug='authentication',
 )
@@ -67,9 +90,29 @@ register(
     field_class=fields.CharField,
     allow_blank=True,
     required=False,
+    default='',
     label=_('Login redirect override URL'),
-    help_text=_('URL to which unauthorized users will be redirected to log in. '
-                'If blank, users will be sent to the Tower login page.'),
+    help_text=_('URL to which unauthorized users will be redirected to log in.  If blank, users will be sent to the login page.'),
     category=_('Authentication'),
     category_slug='authentication',
 )
+
+
+def authentication_validate(serializer, attrs):
+    remote_auth_settings = [
+        'AUTH_LDAP_SERVER_URI',
+        'SOCIAL_AUTH_GOOGLE_OAUTH2_KEY',
+        'SOCIAL_AUTH_GITHUB_KEY',
+        'SOCIAL_AUTH_GITHUB_ORG_KEY',
+        'SOCIAL_AUTH_GITHUB_TEAM_KEY',
+        'SOCIAL_AUTH_SAML_ENABLED_IDPS',
+        'RADIUS_SERVER',
+        'TACACSPLUS_HOST',
+    ]
+    if attrs.get('DISABLE_LOCAL_AUTH', False):
+        if not any(getattr(settings, s, None) for s in remote_auth_settings):
+            raise serializers.ValidationError(_("There are no remote authentication systems configured."))
+    return attrs
+
+
+register_validate('authentication', authentication_validate)

awx/api/exceptions.py

@@ -16,7 +16,4 @@ class ActiveJobConflict(ValidationError):
         # turn everything in self.detail into string by using force_text.
         # Declare detail afterwards circumvent this behavior.
         super(ActiveJobConflict, self).__init__()
-        self.detail = {
-            "error": _("Resource is being used by running jobs."),
-            "active_jobs": active_jobs
-        }
+        self.detail = {"error": _("Resource is being used by running jobs."), "active_jobs": active_jobs}

awx/api/fields.py

@@ -16,10 +16,10 @@ __all__ = ['BooleanNullField', 'CharNullField', 'ChoiceNullField', 'VerbatimFiel
 
 
 class NullFieldMixin(object):
-    '''
+    """
     Mixin to prevent shortcutting validation when we want to allow null input,
     but coerce the resulting value to another type.
-    '''
+    """
 
     def validate_empty_values(self, data):
         (is_empty_value, data) = super(NullFieldMixin, self).validate_empty_values(data)
@@ -29,18 +29,18 @@ class NullFieldMixin(object):
 
 
 class BooleanNullField(NullFieldMixin, serializers.NullBooleanField):
-    '''
+    """
     Custom boolean field that allows null and empty string as False values.
-    '''
+    """
 
     def to_internal_value(self, data):
         return bool(super(BooleanNullField, self).to_internal_value(data))
 
 
 class CharNullField(NullFieldMixin, serializers.CharField):
-    '''
+    """
     Custom char field that allows null as input and coerces to an empty string.
-    '''
+    """
 
     def __init__(self, **kwargs):
         kwargs['allow_null'] = True
@@ -51,9 +51,9 @@ class CharNullField(NullFieldMixin, serializers.CharField):
 
 
 class ChoiceNullField(NullFieldMixin, serializers.ChoiceField):
-    '''
+    """
     Custom choice field that allows null as input and coerces to an empty string.
-    '''
+    """
 
     def __init__(self, **kwargs):
         kwargs['allow_null'] = True
@@ -64,9 +64,9 @@ class ChoiceNullField(NullFieldMixin, serializers.ChoiceField):
 
 
 class VerbatimField(serializers.Field):
-    '''
+    """
     Custom field that passes the value through without changes.
-    '''
+    """
 
     def to_internal_value(self, data):
         return data
@@ -77,22 +77,19 @@ class VerbatimField(serializers.Field):
 
 class OAuth2ProviderField(fields.DictField):
 
-    default_error_messages = {
-        'invalid_key_names': _('Invalid key names: {invalid_key_names}'),
-    }
+    default_error_messages = {'invalid_key_names': _('Invalid key names: {invalid_key_names}')}
     valid_key_names = {'ACCESS_TOKEN_EXPIRE_SECONDS', 'AUTHORIZATION_CODE_EXPIRE_SECONDS', 'REFRESH_TOKEN_EXPIRE_SECONDS'}
     child = fields.IntegerField(min_value=1)
 
     def to_internal_value(self, data):
         data = super(OAuth2ProviderField, self).to_internal_value(data)
-        invalid_flags = (set(data.keys()) - self.valid_key_names)
+        invalid_flags = set(data.keys()) - self.valid_key_names
         if invalid_flags:
             self.fail('invalid_key_names', invalid_key_names=', '.join(list(invalid_flags)))
         return data
 
 
 class DeprecatedCredentialField(serializers.IntegerField):
-
     def __init__(self, **kwargs):
         kwargs['allow_null'] = True
         kwargs['default'] = None

awx/api/filters.py

@@ -9,7 +9,7 @@ from functools import reduce
 # Django
 from django.core.exceptions import FieldError, ValidationError
 from django.db import models
-from django.db.models import Q
+from django.db.models import Q, CharField, IntegerField, BooleanField
 from django.db.models.fields import FieldDoesNotExist
 from django.db.models.fields.related import ForeignObjectRel, ManyToManyField, ForeignKey
 from django.contrib.contenttypes.models import ContentType
@@ -27,9 +27,9 @@ from awx.main.utils.db import get_all_field_names
 
 
 class TypeFilterBackend(BaseFilterBackend):
-    '''
+    """
     Filter on type field now returned with all objects.
-    '''
+    """
 
     def filter_queryset(self, request, queryset, view):
         try:
@@ -63,31 +63,28 @@ class TypeFilterBackend(BaseFilterBackend):
             raise ParseError(*e.args)
 
 
-def get_field_from_path(model, path):
-    '''
+def get_fields_from_path(model, path):
+    """
     Given a Django ORM lookup path (possibly over multiple models)
-    Returns the last field in the line, and also the revised lookup path
+    Returns the fields in the line, and also the revised lookup path
     ex., given
         model=Organization
         path='project__timeout'
-    returns tuple of field at the end of the line as well as a corrected
-    path, for special cases we do substitutions
-        (<IntegerField for timeout>, 'project__timeout')
-    '''
+    returns tuple of fields traversed as well and a corrected path,
+    for special cases we do substitutions
+        ([<IntegerField for timeout>], 'project__timeout')
+    """
     # Store of all the fields used to detect repeats
-    field_set = set([])
+    field_list = []
     new_parts = []
     for name in path.split('__'):
         if model is None:
             raise ParseError(_('No related model for field {}.').format(name))
         # HACK: Make project and inventory source filtering by old field names work for backwards compatibility.
         if model._meta.object_name in ('Project', 'InventorySource'):
-            name = {
-                'current_update': 'current_job',
-                'last_update': 'last_job',
-                'last_update_failed': 'last_job_failed',
-                'last_updated': 'last_job_run',
-            }.get(name, name)
+            name = {'current_update': 'current_job', 'last_update': 'last_job', 'last_update_failed': 'last_job_failed', 'last_updated': 'last_job_run'}.get(
+                name, name
+            )
 
         if name == 'type' and 'polymorphic_ctype' in get_all_field_names(model):
             name = 'polymorphic_ctype'
@@ -111,29 +108,58 @@ def get_field_from_path(model, path):
                 raise PermissionDenied(_('Filtering on %s is not allowed.' % name))
             elif getattr(field, '__prevent_search__', False):
                 raise PermissionDenied(_('Filtering on %s is not allowed.' % name))
-        if field in field_set:
+        if field in field_list:
             # Field traversed twice, could create infinite JOINs, DoSing Tower
             raise ParseError(_('Loops not allowed in filters, detected on field {}.').format(field.name))
-        field_set.add(field)
+        field_list.append(field)
         model = getattr(field, 'related_model', None)
 
-    return field, '__'.join(new_parts)
+    return field_list, '__'.join(new_parts)
+
+
+def get_field_from_path(model, path):
+    """
+    Given a Django ORM lookup path (possibly over multiple models)
+    Returns the last field in the line, and the revised lookup path
+    ex.
+        (<IntegerField for timeout>, 'project__timeout')
+    """
+    field_list, new_path = get_fields_from_path(model, path)
+    return (field_list[-1], new_path)
 
 
 class FieldLookupBackend(BaseFilterBackend):
-    '''
+    """
     Filter using field lookups provided via query string parameters.
-    '''
+    """
 
-    RESERVED_NAMES = ('page', 'page_size', 'format', 'order', 'order_by',
-                      'search', 'type', 'host_filter', 'count_disabled', 'no_truncate')
+    RESERVED_NAMES = ('page', 'page_size', 'format', 'order', 'order_by', 'search', 'type', 'host_filter', 'count_disabled', 'no_truncate', 'limit')
 
-    SUPPORTED_LOOKUPS = ('exact', 'iexact', 'contains', 'icontains',
-                         'startswith', 'istartswith', 'endswith', 'iendswith',
-                         'regex', 'iregex', 'gt', 'gte', 'lt', 'lte', 'in',
-                         'isnull', 'search')
+    SUPPORTED_LOOKUPS = (
+        'exact',
+        'iexact',
+        'contains',
+        'icontains',
+        'startswith',
+        'istartswith',
+        'endswith',
+        'iendswith',
+        'regex',
+        'iregex',
+        'gt',
+        'gte',
+        'lt',
+        'lte',
+        'in',
+        'isnull',
+        'search',
+    )
 
-    def get_field_from_lookup(self, model, lookup):
+    # A list of fields that we know can be filtered on without the possiblity
+    # of introducing duplicates
+    NO_DUPLICATES_ALLOW_LIST = (CharField, IntegerField, BooleanField)
+
+    def get_fields_from_lookup(self, model, lookup):
 
         if '__' in lookup and lookup.rsplit('__', 1)[-1] in self.SUPPORTED_LOOKUPS:
             path, suffix = lookup.rsplit('__', 1)
@@ -147,11 +173,16 @@ class FieldLookupBackend(BaseFilterBackend):
         # FIXME: Could build up a list of models used across relationships, use
         # those lookups combined with request.user.get_queryset(Model) to make
         # sure user cannot query using objects he could not view.
-        field, new_path = get_field_from_path(model, path)
+        field_list, new_path = get_fields_from_path(model, path)
 
         new_lookup = new_path
         new_lookup = '__'.join([new_path, suffix])
-        return field, new_lookup
+        return field_list, new_lookup
+
+    def get_field_from_lookup(self, model, lookup):
+        '''Method to match return type of single field, if needed.'''
+        field_list, new_lookup = self.get_fields_from_lookup(model, lookup)
+        return (field_list[-1], new_lookup)
 
     def to_python_related(self, value):
         value = force_text(value)
@@ -169,10 +200,7 @@ class FieldLookupBackend(BaseFilterBackend):
             try:
                 return self.to_python_related(value)
             except ValueError:
-                raise ParseError(_('Invalid {field_name} id: {field_id}').format(
-                    field_name=getattr(field, 'name', 'related field'),
-                    field_id=value)
-                )
+                raise ParseError(_('Invalid {field_name} id: {field_id}').format(field_name=getattr(field, 'name', 'related field'), field_id=value))
         else:
             return field.to_python(value)
 
@@ -182,13 +210,16 @@ class FieldLookupBackend(BaseFilterBackend):
         except UnicodeEncodeError:
             raise ValueError("%r is not an allowed field name. Must be ascii encodable." % lookup)
 
-        field, new_lookup = self.get_field_from_lookup(model, lookup)
+        field_list, new_lookup = self.get_fields_from_lookup(model, lookup)
+        field = field_list[-1]
+
+        needs_distinct = not all(isinstance(f, self.NO_DUPLICATES_ALLOW_LIST) for f in field_list)
 
         # Type names are stored without underscores internally, but are presented and
         # and serialized over the API containing underscores so we remove `_`
         # for polymorphic_ctype__model lookups.
         if new_lookup.startswith('polymorphic_ctype__model'):
-            value = value.replace('_','')
+            value = value.replace('_', '')
         elif new_lookup.endswith('__isnull'):
             value = to_python_boolean(value)
         elif new_lookup.endswith('__in'):
@@ -211,10 +242,10 @@ class FieldLookupBackend(BaseFilterBackend):
             for rm_field in related_model._meta.fields:
                 if rm_field.name in ('username', 'first_name', 'last_name', 'email', 'name', 'description', 'playbook'):
                     new_lookups.append('{}__{}__icontains'.format(new_lookup[:-8], rm_field.name))
-            return value, new_lookups
+            return value, new_lookups, needs_distinct
         else:
             value = self.value_to_python_for_field(field, value)
-        return value, new_lookup
+        return value, new_lookup, needs_distinct
 
     def filter_queryset(self, request, queryset, view):
         try:
@@ -225,6 +256,7 @@ class FieldLookupBackend(BaseFilterBackend):
             chain_filters = []
             role_filters = []
             search_filters = {}
+            needs_distinct = False
             # Can only have two values: 'AND', 'OR'
             # If 'AND' is used, an iterm must satisfy all condition to show up in the results.
             # If 'OR' is used, an item just need to satisfy one condition to appear in results.
@@ -233,6 +265,11 @@ class FieldLookupBackend(BaseFilterBackend):
                 if key in self.RESERVED_NAMES:
                     continue
 
+                # HACK: make `created` available via API for the Django User ORM model
+                # so it keep compatiblity with other objects which exposes the `created` attr.
+                if queryset.model._meta.object_name == 'User' and key.startswith('created'):
+                    key = key.replace('created', 'date_joined')
+
                 # HACK: Make job event filtering by host name mostly work even
                 # when not capturing job event hosts M2M.
                 if queryset.model._meta.object_name == 'JobEvent' and key.startswith('hosts__name'):
@@ -256,9 +293,12 @@ class FieldLookupBackend(BaseFilterBackend):
                         search_filter_relation = 'AND'
                         values = reduce(lambda list1, list2: list1 + list2, [i.split(',') for i in values])
                     for value in values:
-                        search_value, new_keys = self.value_to_python(queryset.model, key, force_text(value))
+                        search_value, new_keys, _ = self.value_to_python(queryset.model, key, force_text(value))
                         assert isinstance(new_keys, list)
                         search_filters[search_value] = new_keys
+                    # by definition, search *only* joins across relations,
+                    # so it _always_ needs a .distinct()
+                    needs_distinct = True
                     continue
 
                 # Custom chain__ and or__ filters, mutually exclusive (both can
@@ -282,7 +322,9 @@ class FieldLookupBackend(BaseFilterBackend):
                 for value in values:
                     if q_int:
                         value = int(value)
-                    value, new_key = self.value_to_python(queryset.model, key, value)
+                    value, new_key, distinct = self.value_to_python(queryset.model, key, value)
+                    if distinct:
+                        needs_distinct = True
                     if q_chain:
                         chain_filters.append((q_not, new_key, value))
                     elif q_or:
@@ -295,24 +337,20 @@ class FieldLookupBackend(BaseFilterBackend):
                 args = []
                 for n, k, v in and_filters:
                     if n:
-                        args.append(~Q(**{k:v}))
+                        args.append(~Q(**{k: v}))
                     else:
-                        args.append(Q(**{k:v}))
+                        args.append(Q(**{k: v}))
                 for role_name in role_filters:
                     if not hasattr(queryset.model, 'accessible_pk_qs'):
-                        raise ParseError(_(
-                            'Cannot apply role_level filter to this list because its model '
-                            'does not use roles for access control.'))
-                    args.append(
-                        Q(pk__in=queryset.model.accessible_pk_qs(request.user, role_name))
-                    )
+                        raise ParseError(_('Cannot apply role_level filter to this list because its model ' 'does not use roles for access control.'))
+                    args.append(Q(pk__in=queryset.model.accessible_pk_qs(request.user, role_name)))
                 if or_filters:
                     q = Q()
-                    for n,k,v in or_filters:
+                    for n, k, v in or_filters:
                         if n:
-                            q |= ~Q(**{k:v})
+                            q |= ~Q(**{k: v})
                         else:
-                            q |= Q(**{k:v})
+                            q |= Q(**{k: v})
                     args.append(q)
                 if search_filters and search_filter_relation == 'OR':
                     q = Q()
@@ -326,13 +364,15 @@ class FieldLookupBackend(BaseFilterBackend):
                         for constrain in constrains:
                             q_chain |= Q(**{constrain: term})
                         queryset = queryset.filter(q_chain)
-                for n,k,v in chain_filters:
+                for n, k, v in chain_filters:
                     if n:
-                        q = ~Q(**{k:v})
+                        q = ~Q(**{k: v})
                     else:
-                        q = Q(**{k:v})
+                        q = Q(**{k: v})
                     queryset = queryset.filter(q)
-                queryset = queryset.filter(*args).distinct()
+                queryset = queryset.filter(*args)
+                if needs_distinct:
+                    queryset = queryset.distinct()
             return queryset
         except (FieldError, FieldDoesNotExist, ValueError, TypeError) as e:
             raise ParseError(e.args[0])
@@ -341,9 +381,9 @@ class FieldLookupBackend(BaseFilterBackend):
 
 
 class OrderByBackend(BaseFilterBackend):
-    '''
+    """
     Filter to apply ordering based on query string parameters.
-    '''
+    """
 
     def filter_queryset(self, request, queryset, view):
         try:

awx/api/generics.py

@@ -5,10 +5,12 @@
 import inspect
 import logging
 import time
+import uuid
 import urllib.parse
 
 # Django
 from django.conf import settings
+from django.core.cache import cache
 from django.db import connection
 from django.db.models.fields import FieldDoesNotExist
 from django.db.models.fields.related import OneToOneRel
@@ -33,53 +35,51 @@ from rest_framework.negotiation import DefaultContentNegotiation
 
 # AWX
 from awx.api.filters import FieldLookupBackend
-from awx.main.models import (
-    UnifiedJob, UnifiedJobTemplate, User, Role, Credential,
-    WorkflowJobTemplateNode, WorkflowApprovalTemplate
-)
+from awx.main.models import UnifiedJob, UnifiedJobTemplate, User, Role, Credential, WorkflowJobTemplateNode, WorkflowApprovalTemplate
 from awx.main.access import access_registry
-from awx.main.utils import (
-    camelcase_to_underscore,
-    get_search_fields,
-    getattrd,
-    get_object_or_400,
-    decrypt_field
-)
+from awx.main.utils import camelcase_to_underscore, get_search_fields, getattrd, get_object_or_400, decrypt_field, get_awx_version
 from awx.main.utils.db import get_all_field_names
+from awx.main.utils.licensing import server_product_name
+from awx.main.views import ApiErrorView
 from awx.api.serializers import ResourceAccessListElementSerializer, CopySerializer, UserSerializer
 from awx.api.versioning import URLPathVersioning
 from awx.api.metadata import SublistAttachDetatchMetadata, Metadata
 
-__all__ = ['APIView', 'GenericAPIView', 'ListAPIView', 'SimpleListAPIView',
-           'ListCreateAPIView', 'SubListAPIView', 'SubListCreateAPIView',
-           'SubListDestroyAPIView',
-           'SubListCreateAttachDetachAPIView', 'RetrieveAPIView',
-           'RetrieveUpdateAPIView', 'RetrieveDestroyAPIView',
-           'RetrieveUpdateDestroyAPIView',
-           'SubDetailAPIView',
-           'ResourceAccessList',
-           'ParentMixin',
-           'DeleteLastUnattachLabelMixin',
-           'SubListAttachDetachAPIView',
-           'CopyAPIView', 'BaseUsersList',]
+__all__ = [
+    'APIView',
+    'GenericAPIView',
+    'ListAPIView',
+    'SimpleListAPIView',
+    'ListCreateAPIView',
+    'SubListAPIView',
+    'SubListCreateAPIView',
+    'SubListDestroyAPIView',
+    'SubListCreateAttachDetachAPIView',
+    'RetrieveAPIView',
+    'RetrieveUpdateAPIView',
+    'RetrieveDestroyAPIView',
+    'RetrieveUpdateDestroyAPIView',
+    'SubDetailAPIView',
+    'ResourceAccessList',
+    'ParentMixin',
+    'DeleteLastUnattachLabelMixin',
+    'SubListAttachDetachAPIView',
+    'CopyAPIView',
+    'BaseUsersList',
+]
 
 logger = logging.getLogger('awx.api.generics')
 analytics_logger = logging.getLogger('awx.analytics.performance')
 
 
 class LoggedLoginView(auth_views.LoginView):
-
     def get(self, request, *args, **kwargs):
         # The django.auth.contrib login form doesn't perform the content
         # negotiation we've come to expect from DRF; add in code to catch
         # situations where Accept != text/html (or */*) and reply with
         # an HTTP 406
         try:
-            DefaultContentNegotiation().select_renderer(
-                request,
-                [StaticHTMLRenderer],
-                'html'
-            )
+            DefaultContentNegotiation().select_renderer(request, [StaticHTMLRenderer], 'html')
         except NotAcceptable:
             resp = Response(status=status.HTTP_406_NOT_ACCEPTABLE)
             resp.accepted_renderer = StaticHTMLRenderer()
@@ -92,7 +92,7 @@ class LoggedLoginView(auth_views.LoginView):
         ret = super(LoggedLoginView, self).post(request, *args, **kwargs)
         current_user = getattr(request, 'user', None)
         if request.user.is_authenticated:
-            logger.info(smart_text(u"User {} logged in from {}".format(self.request.user.username,request.META.get('REMOTE_ADDR', None))))
+            logger.info(smart_text(u"User {} logged in from {}".format(self.request.user.username, request.META.get('REMOTE_ADDR', None))))
             ret.set_cookie('userLoggedIn', 'true')
             current_user = UserSerializer(self.request.user)
             current_user = smart_text(JSONRenderer().render(current_user.data))
@@ -102,29 +102,27 @@ class LoggedLoginView(auth_views.LoginView):
             return ret
         else:
             if 'username' in self.request.POST:
-                logger.warn(smart_text(u"Login failed for user {} from {}".format(self.request.POST.get('username'),request.META.get('REMOTE_ADDR', None))))
+                logger.warn(smart_text(u"Login failed for user {} from {}".format(self.request.POST.get('username'), request.META.get('REMOTE_ADDR', None))))
             ret.status_code = 401
             return ret
 
 
 class LoggedLogoutView(auth_views.LogoutView):
-
     def dispatch(self, request, *args, **kwargs):
         original_user = getattr(request, 'user', None)
         ret = super(LoggedLogoutView, self).dispatch(request, *args, **kwargs)
         current_user = getattr(request, 'user', None)
         ret.set_cookie('userLoggedIn', 'false')
-        if (not current_user or not getattr(current_user, 'pk', True)) \
-                and current_user != original_user:
+        if (not current_user or not getattr(current_user, 'pk', True)) and current_user != original_user:
             logger.info("User {} logged out.".format(original_user.username))
         return ret
 
 
 def get_view_description(view, html=False):
-    '''Wrapper around REST framework get_view_description() to continue
+    """Wrapper around REST framework get_view_description() to continue
     to support our historical div.
 
-    '''
+    """
     desc = views.get_view_description(view, html=html)
     if html:
         desc = '<div class="description">%s</div>' % desc
@@ -134,6 +132,7 @@ def get_view_description(view, html=False):
 def get_default_schema():
     if settings.SETTINGS_MODULE == 'awx.settings.development':
         from awx.api.swagger import AutoSchema
+
         return AutoSchema()
     else:
         return views.APIView.schema
@@ -145,21 +144,23 @@ class APIView(views.APIView):
     versioning_class = URLPathVersioning
 
     def initialize_request(self, request, *args, **kwargs):
-        '''
+        """
         Store the Django REST Framework Request object as an attribute on the
         normal Django request, store time the request started.
-        '''
+        """
         self.time_started = time.time()
         if getattr(settings, 'SQL_DEBUG', False):
             self.queries_before = len(connection.queries)
 
         # If there are any custom headers in REMOTE_HOST_HEADERS, make sure
-        # they respect the proxy whitelist
-        if all([
-            settings.PROXY_IP_WHITELIST,
-            request.environ.get('REMOTE_ADDR') not in settings.PROXY_IP_WHITELIST,
-            request.environ.get('REMOTE_HOST') not in settings.PROXY_IP_WHITELIST
-        ]):
+        # they respect the allowed proxy list
+        if all(
+            [
+                settings.PROXY_IP_ALLOWED_LIST,
+                request.environ.get('REMOTE_ADDR') not in settings.PROXY_IP_ALLOWED_LIST,
+                request.environ.get('REMOTE_HOST') not in settings.PROXY_IP_ALLOWED_LIST,
+            ]
+        ):
             for custom_header in settings.REMOTE_HOST_HEADERS:
                 if custom_header.startswith('HTTP_'):
                     request.environ.pop(custom_header, None)
@@ -174,27 +175,57 @@ class APIView(views.APIView):
             request.drf_request_user = None
             self.__init_request_error__ = exc
         except UnsupportedMediaType as exc:
-            exc.detail = _('You did not use correct Content-Type in your HTTP request. '
-                           'If you are using our REST API, the Content-Type must be application/json')
+            exc.detail = _(
+                'You did not use correct Content-Type in your HTTP request. ' 'If you are using our REST API, the Content-Type must be application/json'
+            )
             self.__init_request_error__ = exc
         return drf_request
 
     def finalize_response(self, request, response, *args, **kwargs):
-        '''
+        """
         Log warning for 400 requests.  Add header with elapsed time.
-        '''
+        """
+        #
+        # If the URL was rewritten, and we get a 404, we should entirely
+        # replace the view in the request context with an ApiErrorView()
+        # Without this change, there will be subtle differences in the BrowseableAPIRenderer
+        #
+        # These differences could provide contextual clues which would allow
+        # anonymous users to determine if usernames were valid or not
+        # (e.g., if an anonymous user visited `/api/v2/users/valid/`, and got a 404,
+        # but also saw that the page heading said "User Detail", they might notice
+        # that's a difference in behavior from a request to `/api/v2/users/not-valid/`, which
+        # would show a page header of "Not Found").  Changing the view here
+        # guarantees that the rendered response will look exactly like the response
+        # when you visit a URL that has no matching URL paths in `awx.api.urls`.
+        #
+        if response.status_code == 404 and 'awx.named_url_rewritten' in request.environ:
+            self.headers.pop('Allow', None)
+            response = super(APIView, self).finalize_response(request, response, *args, **kwargs)
+            view = ApiErrorView()
+            setattr(view, 'request', request)
+            response.renderer_context['view'] = view
+            return response
+
         if response.status_code >= 400:
-            status_msg = "status %s received by user %s attempting to access %s from %s" % \
-                         (response.status_code, request.user, request.path, request.META.get('REMOTE_ADDR', None))
+            status_msg = "status %s received by user %s attempting to access %s from %s" % (
+                response.status_code,
+                request.user,
+                request.path,
+                request.META.get('REMOTE_ADDR', None),
+            )
             if hasattr(self, '__init_request_error__'):
                 response = self.handle_exception(self.__init_request_error__)
             if response.status_code == 401:
-                response.data['detail'] += ' To establish a login session, visit /api/login/.'
+                response.data['detail'] += _(' To establish a login session, visit') + ' /api/login/.'
                 logger.info(status_msg)
             else:
-                logger.warn(status_msg)
+                logger.warning(status_msg)
         response = super(APIView, self).finalize_response(request, response, *args, **kwargs)
         time_started = getattr(self, 'time_started', None)
+        response['X-API-Product-Version'] = get_awx_version()
+        response['X-API-Product-Name'] = server_product_name()
+
         response['X-API-Node'] = settings.CLUSTER_HOST_ID
         if time_started:
             time_elapsed = time.time() - self.time_started
@@ -280,18 +311,12 @@ class APIView(views.APIView):
         return data
 
     def determine_version(self, request, *args, **kwargs):
-        return (
-            getattr(request, 'version', None),
-            getattr(request, 'versioning_scheme', None),
-        )
+        return (getattr(request, 'version', None), getattr(request, 'versioning_scheme', None))
 
     def dispatch(self, request, *args, **kwargs):
         if self.versioning_class is not None:
             scheme = self.versioning_class()
-            request.version, request.versioning_scheme = (
-                scheme.determine_version(request, *args, **kwargs),
-                scheme
-            )
+            request.version, request.versioning_scheme = (scheme.determine_version(request, *args, **kwargs), scheme)
             if 'version' in kwargs:
                 kwargs.pop('version')
         return super(APIView, self).dispatch(request, *args, **kwargs)
@@ -347,25 +372,22 @@ class GenericAPIView(generics.GenericAPIView, APIView):
         d = super(GenericAPIView, self).get_description_context()
         if hasattr(self.model, "_meta"):
             if hasattr(self.model._meta, "verbose_name"):
-                d.update({
-                    'model_verbose_name': smart_text(self.model._meta.verbose_name),
-                    'model_verbose_name_plural': smart_text(self.model._meta.verbose_name_plural),
-                })
+                d.update(
+                    {
+                        'model_verbose_name': smart_text(self.model._meta.verbose_name),
+                        'model_verbose_name_plural': smart_text(self.model._meta.verbose_name_plural),
+                    }
+                )
             serializer = self.get_serializer()
             metadata = self.metadata_class()
             metadata.request = self.request
-            for method, key in [
-                ('GET', 'serializer_fields'),
-                ('POST', 'serializer_create_fields'),
-                ('PUT', 'serializer_update_fields')
-            ]:
+            for method, key in [('GET', 'serializer_fields'), ('POST', 'serializer_create_fields'), ('PUT', 'serializer_update_fields')]:
                 d[key] = metadata.get_serializer_info(serializer, method=method)
         d['settings'] = settings
         return d
 
 
 class SimpleListAPIView(generics.ListAPIView, GenericAPIView):
-
     def get_queryset(self):
         return self.request.user.get_queryset(self.model)
 
@@ -382,9 +404,7 @@ class ListAPIView(generics.ListAPIView, GenericAPIView):
         else:
             order_field = 'name'
         d = super(ListAPIView, self).get_description_context()
-        d.update({
-            'order_field': order_field,
-        })
+        d.update({'order_field': order_field})
         return d
 
     @property
@@ -395,9 +415,13 @@ class ListAPIView(generics.ListAPIView, GenericAPIView):
     def related_search_fields(self):
         def skip_related_name(name):
             return (
-                name is None or name.endswith('_role') or name.startswith('_') or
-                name.startswith('deprecated_') or name.endswith('_set') or
-                name == 'polymorphic_ctype')
+                name is None
+                or name.endswith('_role')
+                or name.startswith('_')
+                or name.startswith('deprecated_')
+                or name.endswith('_set')
+                or name == 'polymorphic_ctype'
+            )
 
         fields = set([])
         for field in self.model._meta.fields:
@@ -451,9 +475,7 @@ class ParentMixin(object):
     def get_parent_object(self):
         if self.parent_object is not None:
             return self.parent_object
-        parent_filter = {
-            self.lookup_field: self.kwargs.get(self.lookup_field, None),
-        }
+        parent_filter = {self.lookup_field: self.kwargs.get(self.lookup_field, None)}
         self.parent_object = get_object_or_404(self.parent_model, **parent_filter)
         return self.parent_object
 
@@ -482,10 +504,12 @@ class SubListAPIView(ParentMixin, ListAPIView):
 
     def get_description_context(self):
         d = super(SubListAPIView, self).get_description_context()
-        d.update({
-            'parent_model_verbose_name': smart_text(self.parent_model._meta.verbose_name),
-            'parent_model_verbose_name_plural': smart_text(self.parent_model._meta.verbose_name_plural),
-        })
+        d.update(
+            {
+                'parent_model_verbose_name': smart_text(self.parent_model._meta.verbose_name),
+                'parent_model_verbose_name_plural': smart_text(self.parent_model._meta.verbose_name_plural),
+            }
+        )
         return d
 
     def get_queryset(self):
@@ -500,7 +524,6 @@ class SubListAPIView(ParentMixin, ListAPIView):
 
 
 class DestroyAPIView(generics.DestroyAPIView):
-
     def has_delete_permission(self, obj):
         return self.request.user.can_access(self.model, 'delete', obj)
 
@@ -514,12 +537,12 @@ class SubListDestroyAPIView(DestroyAPIView, SubListAPIView):
     """
     Concrete view for deleting everything related by `relationship`.
     """
+
     check_sub_obj_permission = True
 
     def destroy(self, request, *args, **kwargs):
         instance_list = self.get_queryset()
-        if (not self.check_sub_obj_permission and
-                not request.user.can_access(self.parent_model, 'delete', self.get_parent_object())):
+        if not self.check_sub_obj_permission and not request.user.can_access(self.parent_model, 'delete', self.get_parent_object()):
             raise PermissionDenied()
         self.perform_list_destroy(instance_list)
         return Response(status=status.HTTP_204_NO_CONTENT)
@@ -543,11 +566,18 @@ class SubListCreateAPIView(SubListAPIView, ListCreateAPIView):
 
     def get_description_context(self):
         d = super(SubListCreateAPIView, self).get_description_context()
-        d.update({
-            'parent_key': getattr(self, 'parent_key', None),
-        })
+        d.update({'parent_key': getattr(self, 'parent_key', None)})
         return d
 
+    def get_queryset(self):
+        if hasattr(self, 'parent_key'):
+            # Prefer this filtering because ForeignKey allows us more assumptions
+            parent = self.get_parent_object()
+            self.check_parent_access(parent)
+            qs = self.request.user.get_queryset(self.model)
+            return qs.filter(**{self.parent_key: parent})
+        return super(SubListCreateAPIView, self).get_queryset()
+
     def create(self, request, *args, **kwargs):
         # If the object ID was not specified, it probably doesn't exist in the
         # DB yet. We want to see if we can create it.  The URL may choose to
@@ -570,8 +600,7 @@ class SubListCreateAPIView(SubListAPIView, ListCreateAPIView):
         # attempt to deserialize the object
         serializer = self.get_serializer(data=data)
         if not serializer.is_valid():
-            return Response(serializer.errors,
-                            status=status.HTTP_400_BAD_REQUEST)
+            return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)
 
         # Verify we have permission to add the object as given.
         if not request.user.can_access(self.model, 'add', serializer.validated_data):
@@ -595,9 +624,7 @@ class SubListCreateAttachDetachAPIView(SubListCreateAPIView):
 
     def get_description_context(self):
         d = super(SubListCreateAttachDetachAPIView, self).get_description_context()
-        d.update({
-            "has_attach": True,
-        })
+        d.update({"has_attach": True})
         return d
 
     def attach_validate(self, request):
@@ -635,9 +662,7 @@ class SubListCreateAttachDetachAPIView(SubListCreateAPIView):
         sub = get_object_or_400(self.model, pk=sub_id)
 
         # Verify we have permission to attach.
-        if not request.user.can_access(self.parent_model, 'attach', parent, sub,
-                                       self.relationship, data,
-                                       skip_sub_obj_read_check=created):
+        if not request.user.can_access(self.parent_model, 'attach', parent, sub, self.relationship, data, skip_sub_obj_read_check=created):
             raise PermissionDenied()
 
         # Verify that the relationship to be added is valid.
@@ -676,8 +701,7 @@ class SubListCreateAttachDetachAPIView(SubListCreateAPIView):
         relationship = getattrd(parent, self.relationship)
         sub = get_object_or_400(self.model, pk=sub_id)
 
-        if not request.user.can_access(self.parent_model, 'unattach', parent,
-                                       sub, self.relationship, request.data):
+        if not request.user.can_access(self.parent_model, 'unattach', parent, sub, self.relationship, request.data):
             raise PermissionDenied()
 
         if parent_key:
@@ -695,28 +719,24 @@ class SubListCreateAttachDetachAPIView(SubListCreateAPIView):
 
     def post(self, request, *args, **kwargs):
         if not isinstance(request.data, dict):
-            return Response('invalid type for post data',
-                            status=status.HTTP_400_BAD_REQUEST)
+            return Response('invalid type for post data', status=status.HTTP_400_BAD_REQUEST)
         if 'disassociate' in request.data:
             return self.unattach(request, *args, **kwargs)
         else:
             return self.attach(request, *args, **kwargs)
 
 
-
 class SubListAttachDetachAPIView(SubListCreateAttachDetachAPIView):
-    '''
+    """
     Derived version of SubListCreateAttachDetachAPIView that prohibits creation
-    '''
+    """
+
     metadata_class = SublistAttachDetatchMetadata
 
     def post(self, request, *args, **kwargs):
         sub_id = request.data.get('id', None)
         if not sub_id:
-            return Response(
-                dict(msg=_("{} 'id' field is missing.".format(
-                    self.model._meta.verbose_name.title()))),
-                status=status.HTTP_400_BAD_REQUEST)
+            return Response(dict(msg=_("{} 'id' field is missing.".format(self.model._meta.verbose_name.title()))), status=status.HTTP_400_BAD_REQUEST)
         return super(SubListAttachDetachAPIView, self).post(request, *args, **kwargs)
 
     def update_raw_data(self, data):
@@ -728,11 +748,11 @@ class SubListAttachDetachAPIView(SubListCreateAttachDetachAPIView):
 
 
 class DeleteLastUnattachLabelMixin(object):
-    '''
+    """
     Models for which you want the last instance to be deleted from the database
     when the last disassociate is called should inherit from this class. Further,
     the model should implement is_detached()
-    '''
+    """
 
     def unattach(self, request, *args, **kwargs):
         (sub_id, res) = super(DeleteLastUnattachLabelMixin, self).unattach_validate(request)
@@ -758,7 +778,6 @@ class RetrieveAPIView(generics.RetrieveAPIView, GenericAPIView):
 
 
 class RetrieveUpdateAPIView(RetrieveAPIView, generics.RetrieveUpdateAPIView):
-
     def update(self, request, *args, **kwargs):
         self.update_filter(request, *args, **kwargs)
         return super(RetrieveUpdateAPIView, self).update(request, *args, **kwargs)
@@ -768,7 +787,7 @@ class RetrieveUpdateAPIView(RetrieveAPIView, generics.RetrieveUpdateAPIView):
         return super(RetrieveUpdateAPIView, self).partial_update(request, *args, **kwargs)
 
     def update_filter(self, request, *args, **kwargs):
-        ''' scrub any fields the user cannot/should not put/patch, based on user context.  This runs after read-only serialization filtering '''
+        '''scrub any fields the user cannot/should not put/patch, based on user context.  This runs after read-only serialization filtering'''
         pass
 
 
@@ -799,6 +818,7 @@ class ResourceAccessList(ParentMixin, ListAPIView):
 
 def trigger_delayed_deep_copy(*args, **kwargs):
     from awx.main.tasks import deep_copy_model_obj
+
     connection.on_commit(lambda: deep_copy_model_obj.delay(*args, **kwargs))
 
 
@@ -821,7 +841,7 @@ class CopyAPIView(GenericAPIView):
 
     @staticmethod
     def _decrypt_model_field_if_needed(obj, field_name, field_val):
-        if field_name in getattr(type(obj), 'REENCRYPTION_BLACKLIST_AT_COPY', []):
+        if field_name in getattr(type(obj), 'REENCRYPTION_BLOCKLIST_AT_COPY', []):
             return field_val
         if isinstance(obj, Credential) and field_name == 'inputs':
             for secret in obj.credential_type.secret_fields:
@@ -829,8 +849,7 @@ class CopyAPIView(GenericAPIView):
                     field_val[secret] = decrypt_field(obj, secret)
         elif isinstance(field_val, dict):
             for sub_field in field_val:
-                if isinstance(sub_field, str) \
-                        and isinstance(field_val[sub_field], str):
+                if isinstance(sub_field, str) and isinstance(field_val[sub_field], str):
                     field_val[sub_field] = decrypt_field(obj, field_name, sub_field)
         elif isinstance(field_val, str):
             try:
@@ -842,15 +861,11 @@ class CopyAPIView(GenericAPIView):
     def _build_create_dict(self, obj):
         ret = {}
         if self.copy_return_serializer_class:
-            all_fields = Metadata().get_serializer_info(
-                self._get_copy_return_serializer(), method='POST'
-            )
+            all_fields = Metadata().get_serializer_info(self._get_copy_return_serializer(), method='POST')
             for field_name, field_info in all_fields.items():
                 if not hasattr(obj, field_name) or field_info.get('read_only', True):
                     continue
-                ret[field_name] = CopyAPIView._decrypt_model_field_if_needed(
-                    obj, field_name, getattr(obj, field_name)
-                )
+                ret[field_name] = CopyAPIView._decrypt_model_field_if_needed(obj, field_name, getattr(obj, field_name))
         return ret
 
     @staticmethod
@@ -867,10 +882,12 @@ class CopyAPIView(GenericAPIView):
                 field_val = getattr(obj, field.name)
             except AttributeError:
                 continue
-            # Adjust copy blacklist fields here.
-            if field.name in fields_to_discard or field.name in [
-                'id', 'pk', 'polymorphic_ctype', 'unifiedjobtemplate_ptr', 'created_by', 'modified_by'
-            ] or field.name.endswith('_role'):
+            # Adjust copy blocked fields here.
+            if (
+                field.name in fields_to_discard
+                or field.name in ['id', 'pk', 'polymorphic_ctype', 'unifiedjobtemplate_ptr', 'created_by', 'modified_by']
+                or field.name.endswith('_role')
+            ):
                 create_kwargs.pop(field.name, None)
                 continue
             if field.one_to_many:
@@ -886,33 +903,24 @@ class CopyAPIView(GenericAPIView):
             elif field.name == 'name' and not old_parent:
                 create_kwargs[field.name] = copy_name or field_val + ' copy'
             elif field.name in fields_to_preserve:
-                create_kwargs[field.name] = CopyAPIView._decrypt_model_field_if_needed(
-                    obj, field.name, field_val
-                )
+                create_kwargs[field.name] = CopyAPIView._decrypt_model_field_if_needed(obj, field.name, field_val)
 
         # WorkflowJobTemplateNodes that represent an approval are *special*;
         # when we copy them, we actually want to *copy* the UJT they point at
         # rather than share the template reference between nodes in disparate
         # workflows
-        if (
-            isinstance(obj, WorkflowJobTemplateNode) and
-            isinstance(getattr(obj, 'unified_job_template'), WorkflowApprovalTemplate)
-        ):
-            new_approval_template, sub_objs = CopyAPIView.copy_model_obj(
-                None, None, WorkflowApprovalTemplate,
-                obj.unified_job_template, creater
-            )
+        if isinstance(obj, WorkflowJobTemplateNode) and isinstance(getattr(obj, 'unified_job_template'), WorkflowApprovalTemplate):
+            new_approval_template, sub_objs = CopyAPIView.copy_model_obj(None, None, WorkflowApprovalTemplate, obj.unified_job_template, creater)
             create_kwargs['unified_job_template'] = new_approval_template
 
         new_obj = model.objects.create(**create_kwargs)
-        logger.debug('Deep copy: Created new object {}({})'.format(
-            new_obj, model
-        ))
+        logger.debug('Deep copy: Created new object {}({})'.format(new_obj, model))
         # Need to save separatedly because Djang-crum get_current_user would
         # not work properly in non-request-response-cycle context.
         new_obj.created_by = creater
         new_obj.save()
         from awx.main.signals import disable_activity_stream
+
         with disable_activity_stream():
             for m2m in m2m_to_preserve:
                 for related_obj in m2m_to_preserve[m2m].all():
@@ -938,8 +946,7 @@ class CopyAPIView(GenericAPIView):
         for key in create_kwargs:
             create_kwargs[key] = getattr(create_kwargs[key], 'pk', None) or create_kwargs[key]
         try:
-            can_copy = request.user.can_access(self.model, 'add', create_kwargs) and \
-                request.user.can_access(self.model, 'copy_related', obj)
+            can_copy = request.user.can_access(self.model, 'add', create_kwargs) and request.user.can_access(self.model, 'copy_related', obj)
         except PermissionDenied:
             return Response({'can_copy': False})
         return Response({'can_copy': can_copy})
@@ -958,21 +965,21 @@ class CopyAPIView(GenericAPIView):
         if not serializer.is_valid():
             return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)
         new_obj, sub_objs = CopyAPIView.copy_model_obj(
-            None, None, self.model, obj, request.user, create_kwargs=create_kwargs,
-            copy_name=serializer.validated_data.get('name', '')
+            None, None, self.model, obj, request.user, create_kwargs=create_kwargs, copy_name=serializer.validated_data.get('name', '')
         )
         if hasattr(new_obj, 'admin_role') and request.user not in new_obj.admin_role.members.all():
             new_obj.admin_role.members.add(request.user)
         if sub_objs:
+            # store the copied object dict into cache, because it's
+            # often too large for postgres' notification bus
+            # (which has a default maximum message size of 8k)
+            key = 'deep-copy-{}'.format(str(uuid.uuid4()))
+            cache.set(key, sub_objs, timeout=3600)
             permission_check_func = None
             if hasattr(type(self), 'deep_copy_permission_check_func'):
-                permission_check_func = (
-                    type(self).__module__, type(self).__name__, 'deep_copy_permission_check_func'
-                )
+                permission_check_func = (type(self).__module__, type(self).__name__, 'deep_copy_permission_check_func')
             trigger_delayed_deep_copy(
-                self.model.__module__, self.model.__name__,
-                obj.pk, new_obj.pk, request.user.pk, sub_objs,
-                permission_check_func=permission_check_func
+                self.model.__module__, self.model.__name__, obj.pk, new_obj.pk, request.user.pk, key, permission_check_func=permission_check_func
             )
         serializer = self._get_copy_return_serializer(new_obj)
         headers = {'Location': new_obj.get_absolute_url(request=request)}
@@ -981,7 +988,7 @@ class CopyAPIView(GenericAPIView):
 
 class BaseUsersList(SubListCreateAttachDetachAPIView):
     def post(self, request, *args, **kwargs):
-        ret = super(BaseUsersList, self).post( request, *args, **kwargs)
+        ret = super(BaseUsersList, self).post(request, *args, **kwargs)
         if ret.status_code != 201:
             return ret
         try:

awx/api/metadata.py

@@ -2,6 +2,7 @@
 # All Rights Reserved.
 
 from collections import OrderedDict
+from uuid import UUID
 
 # Django
 from django.core.exceptions import PermissionDenied
@@ -20,24 +21,33 @@ from rest_framework.fields import JSONField as DRFJSONField
 from rest_framework.request import clone_request
 
 # AWX
+from awx.api.fields import ChoiceNullField
 from awx.main.fields import JSONField, ImplicitRoleField
-from awx.main.models import InventorySource, NotificationTemplate
-from awx.main.scheduler.kubernetes import PodManager
+from awx.main.models import NotificationTemplate
+from awx.main.utils.execution_environments import get_default_pod_spec
+
+# Polymorphic
+from polymorphic.models import PolymorphicModel
 
 
 class Metadata(metadata.SimpleMetadata):
-
     def get_field_info(self, field):
         field_info = OrderedDict()
         field_info['type'] = self.label_lookup[field]
         field_info['required'] = getattr(field, 'required', False)
 
         text_attrs = [
-            'read_only', 'label', 'help_text',
-            'min_length', 'max_length',
-            'min_value', 'max_value',
-            'category', 'category_slug',
-            'defined_in_file'
+            'read_only',
+            'label',
+            'help_text',
+            'min_length',
+            'max_length',
+            'min_value',
+            'max_value',
+            'category',
+            'category_slug',
+            'defined_in_file',
+            'unit',
         ]
 
         for attr in text_attrs:
@@ -59,7 +69,9 @@ class Metadata(metadata.SimpleMetadata):
                 'type': _('Data type for this {}.'),
                 'url': _('URL for this {}.'),
                 'related': _('Data structure with URLs of related resources.'),
-                'summary_fields': _('Data structure with name/description for related resources.'),
+                'summary_fields': _(
+                    'Data structure with name/description for related resources.  ' 'The output for some objects may be limited for performance reasons.'
+                ),
                 'created': _('Timestamp when this {} was created.'),
                 'modified': _('Timestamp when this {} was last modified.'),
             }
@@ -69,7 +81,9 @@ class Metadata(metadata.SimpleMetadata):
                 field_info['help_text'] = field_help_text[field.field_name].format(verbose_name)
 
             if field.field_name == 'type':
-                field_info['filterable'] = True
+                # Only include model classes with `type` field.
+                if issubclass(serializer.Meta.model, PolymorphicModel):
+                    field_info['filterable'] = True
             else:
                 for model_field in serializer.Meta.model._meta.fields:
                     if field.field_name == model_field.name:
@@ -84,6 +98,8 @@ class Metadata(metadata.SimpleMetadata):
         # FIXME: Still isn't showing all default values?
         try:
             default = field.get_default()
+            if type(default) is UUID:
+                default = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
             if field.field_name == 'TOWER_URL_BASE' and default == 'https://towerhost':
                 default = '{}://{}'.format(self.request.scheme, self.request.get_host())
             field_info['default'] = default
@@ -96,25 +112,18 @@ class Metadata(metadata.SimpleMetadata):
             field_info['children'] = self.get_serializer_info(field)
 
         if not isinstance(field, (RelatedField, ManyRelatedField)) and hasattr(field, 'choices'):
-            field_info['choices'] = [(choice_value, choice_name) for choice_value, choice_name in field.choices.items()]
+            choices = [(choice_value, choice_name) for choice_value, choice_name in field.choices.items()]
+            if not any(choice in ('', None) for choice, _ in choices):
+                if field.allow_blank:
+                    choices = [("", "---------")] + choices
+                if field.allow_null and not isinstance(field, ChoiceNullField):
+                    choices = [(None, "---------")] + choices
+            field_info['choices'] = choices
 
         # Indicate if a field is write-only.
         if getattr(field, 'write_only', False):
             field_info['write_only'] = True
 
-        # Special handling of inventory source_region choices that vary based on
-        # selected inventory source.
-        if field.field_name == 'source_regions':
-            for cp in ('azure_rm', 'ec2', 'gce'):
-                get_regions = getattr(InventorySource, 'get_%s_region_choices' % cp)
-                field_info['%s_region_choices' % cp] = get_regions()
-
-        # Special handling of group_by choices for EC2.
-        if field.field_name == 'group_by':
-            for cp in ('ec2',):
-                get_group_by_choices = getattr(InventorySource, 'get_%s_group_by_choices' % cp)
-                field_info['%s_group_by_choices' % cp] = get_group_by_choices()
-
         # Special handling of notification configuration where the required properties
         # are conditional on the type selected.
         if field.field_name == 'notification_configuration':
@@ -131,7 +140,6 @@ class Metadata(metadata.SimpleMetadata):
             for (notification_type_name, notification_tr_name, notification_type_class) in NotificationTemplate.NOTIFICATION_TYPES:
                 field_info[notification_type_name] = notification_type_class.default_messages
 
-
         # Update type of fields returned...
         model_field = None
         if serializer and hasattr(serializer, 'Meta') and hasattr(serializer.Meta, 'model'):
@@ -149,22 +157,19 @@ class Metadata(metadata.SimpleMetadata):
             field_info['type'] = 'integer'
         elif field.field_name in ('created', 'modified'):
             field_info['type'] = 'datetime'
-        elif (
-            RelatedField in field.__class__.__bases__ or
-            isinstance(model_field, ForeignKey)
-        ):
+        elif RelatedField in field.__class__.__bases__ or isinstance(model_field, ForeignKey):
             field_info['type'] = 'id'
         elif (
-            isinstance(field, JSONField) or
-            isinstance(model_field, JSONField) or
-            isinstance(field, DRFJSONField) or
-            isinstance(getattr(field, 'model_field', None), JSONField) or
-            field.field_name == 'credential_passwords'
+            isinstance(field, JSONField)
+            or isinstance(model_field, JSONField)
+            or isinstance(field, DRFJSONField)
+            or isinstance(getattr(field, 'model_field', None), JSONField)
+            or field.field_name == 'credential_passwords'
         ):
             field_info['type'] = 'json'
         elif (
-            isinstance(field, ManyRelatedField) and
-            field.field_name == 'credentials'
+            isinstance(field, ManyRelatedField)
+            and field.field_name == 'credentials'
             # launch-time credentials
         ):
             field_info['type'] = 'list_of_ids'
@@ -175,10 +180,7 @@ class Metadata(metadata.SimpleMetadata):
 
     def get_serializer_info(self, serializer, method=None):
         filterer = getattr(serializer, 'filter_field_metadata', lambda fields, method: fields)
-        return filterer(
-            super(Metadata, self).get_serializer_info(serializer),
-            method
-        )
+        return filterer(super(Metadata, self).get_serializer_info(serializer), method)
 
     def determine_actions(self, request, view):
         # Add field information for GET requests (so field names/labels are
@@ -209,7 +211,7 @@ class Metadata(metadata.SimpleMetadata):
                     continue
 
                 if field == "pod_spec_override":
-                    meta['default'] = PodManager().pod_definition
+                    meta['default'] = get_default_pod_spec()
 
                 # Add type choices if available from the serializer.
                 if field == 'type' and hasattr(serializer, 'get_type_choices'):
@@ -274,6 +276,7 @@ class Metadata(metadata.SimpleMetadata):
             metadata['object_roles'] = roles
 
         from rest_framework import generics
+
         if isinstance(view, generics.ListAPIView) and hasattr(view, 'paginator'):
             metadata['max_page_size'] = view.paginator.max_page_size
 
@@ -293,7 +296,6 @@ class RoleMetadata(Metadata):
 
 
 class SublistAttachDetatchMetadata(Metadata):
-
     def determine_actions(self, request, view):
         actions = super(SublistAttachDetatchMetadata, self).determine_actions(request, view)
         method = 'POST'

awx/api/metrics.py

@@ -3,13 +3,9 @@
 
 from django.conf.urls import url
 
-from awx.api.views import (
-    MetricsView
-)
+from awx.api.views import MetricsView
 
 
-urls = [
-    url(r'^$', MetricsView.as_view(), name='metrics_view'),
-]
+urls = [url(r'^$', MetricsView.as_view(), name='metrics_view')]
 
 __all__ = ['urls']

awx/api/pagination.py

@@ -1,16 +1,19 @@
 # Copyright (c) 2015 Ansible, Inc.
 # All Rights Reserved.
 
+from collections import OrderedDict
+
 # Django REST Framework
 from django.conf import settings
 from django.core.paginator import Paginator as DjangoPaginator
 from rest_framework import pagination
 from rest_framework.response import Response
 from rest_framework.utils.urls import replace_query_param
+from rest_framework.settings import api_settings
+from django.utils.translation import gettext_lazy as _
 
 
 class DisabledPaginator(DjangoPaginator):
-
     @property
     def num_pages(self):
         return 1
@@ -49,8 +52,7 @@ class Pagination(pagination.PageNumberPagination):
 
     def get_html_context(self):
         context = super().get_html_context()
-        context['page_links'] = [pl._replace(url=self.cap_page_size(pl.url))
-                                 for pl in context['page_links']]
+        context['page_links'] = [pl._replace(url=self.cap_page_size(pl.url)) for pl in context['page_links']]
 
         return context
 
@@ -67,3 +69,65 @@ class Pagination(pagination.PageNumberPagination):
         if self.count_disabled:
             return Response({'results': data})
         return super(Pagination, self).get_paginated_response(data)
+
+
+class LimitPagination(pagination.BasePagination):
+    default_limit = api_settings.PAGE_SIZE
+    limit_query_param = 'limit'
+    limit_query_description = _('Number of results to return per page.')
+    max_page_size = settings.MAX_PAGE_SIZE
+
+    def paginate_queryset(self, queryset, request, view=None):
+        self.limit = self.get_limit(request)
+        self.request = request
+
+        return list(queryset[0 : self.limit])
+
+    def get_paginated_response(self, data):
+        return Response(OrderedDict([('results', data)]))
+
+    def get_paginated_response_schema(self, schema):
+        return {
+            'type': 'object',
+            'properties': {
+                'results': schema,
+            },
+        }
+
+    def get_limit(self, request):
+        try:
+            return pagination._positive_int(request.query_params[self.limit_query_param], strict=True)
+        except (KeyError, ValueError):
+            pass
+
+        return self.default_limit
+
+
+class UnifiedJobEventPagination(Pagination):
+    """
+    By default, use Pagination for all operations.
+    If `limit` query parameter specified use LimitPagination
+    """
+
+    def __init__(self, *args, **kwargs):
+        self.use_limit_paginator = False
+        self.limit_pagination = LimitPagination()
+        return super().__init__(*args, **kwargs)
+
+    def paginate_queryset(self, queryset, request, view=None):
+        if 'limit' in request.query_params:
+            self.use_limit_paginator = True
+
+        if self.use_limit_paginator:
+            return self.limit_pagination.paginate_queryset(queryset, request, view=view)
+        return super().paginate_queryset(queryset, request, view=view)
+
+    def get_paginated_response(self, data):
+        if self.use_limit_paginator:
+            return self.limit_pagination.get_paginated_response(data)
+        return super().get_paginated_response(data)
+
+    def get_paginated_response_schema(self, schema):
+        if self.use_limit_paginator:
+            return self.limit_pagination.get_paginated_response_schema(schema)
+        return super().get_paginated_response_schema(schema)

awx/api/permissions.py

@@ -4,6 +4,8 @@
 # Python
 import logging
 
+from django.conf import settings
+
 # Django REST Framework
 from rest_framework.exceptions import MethodNotAllowed, PermissionDenied
 from rest_framework import permissions
@@ -15,16 +17,25 @@ from awx.main.utils import get_object_or_400
 
 logger = logging.getLogger('awx.api.permissions')
 
-__all__ = ['ModelAccessPermission', 'JobTemplateCallbackPermission', 'VariableDataPermission',
-           'TaskPermission', 'ProjectUpdatePermission', 'InventoryInventorySourcesUpdatePermission',
-           'UserPermission', 'IsSuperUser', 'InstanceGroupTowerPermission', 'WorkflowApprovalPermission']
+__all__ = [
+    'ModelAccessPermission',
+    'JobTemplateCallbackPermission',
+    'VariableDataPermission',
+    'TaskPermission',
+    'ProjectUpdatePermission',
+    'InventoryInventorySourcesUpdatePermission',
+    'UserPermission',
+    'IsSystemAdminOrAuditor',
+    'InstanceGroupTowerPermission',
+    'WorkflowApprovalPermission',
+]
 
 
 class ModelAccessPermission(permissions.BasePermission):
-    '''
+    """
     Default permissions class to check user access based on the model and
     request method, optionally verifying the request data.
-    '''
+    """
 
     def check_options_permissions(self, request, view, obj=None):
         return self.check_get_permissions(request, view, obj)
@@ -35,8 +46,7 @@ class ModelAccessPermission(permissions.BasePermission):
     def check_get_permissions(self, request, view, obj=None):
         if hasattr(view, 'parent_model'):
             parent_obj = view.get_parent_object()
-            if not check_user_access(request.user, view.parent_model, 'read',
-                                     parent_obj):
+            if not check_user_access(request.user, view.parent_model, 'read', parent_obj):
                 return False
         if not obj:
             return True
@@ -45,8 +55,7 @@ class ModelAccessPermission(permissions.BasePermission):
     def check_post_permissions(self, request, view, obj=None):
         if hasattr(view, 'parent_model'):
             parent_obj = view.get_parent_object()
-            if not check_user_access(request.user, view.parent_model, 'read',
-                                     parent_obj):
+            if not check_user_access(request.user, view.parent_model, 'read', parent_obj):
                 return False
             if hasattr(view, 'parent_key'):
                 if not check_user_access(request.user, view.model, 'add', {view.parent_key: parent_obj}):
@@ -60,10 +69,7 @@ class ModelAccessPermission(permissions.BasePermission):
             extra_kwargs = {}
             if view.obj_permission_type == 'admin':
                 extra_kwargs['data'] = {}
-            return check_user_access(
-                request.user, view.model, view.obj_permission_type, obj,
-                **extra_kwargs
-            )
+            return check_user_access(request.user, view.model, view.obj_permission_type, obj, **extra_kwargs)
         else:
             if obj:
                 return True
@@ -74,8 +80,7 @@ class ModelAccessPermission(permissions.BasePermission):
             # FIXME: For some reason this needs to return True
             # because it is first called with obj=None?
             return True
-        return check_user_access(request.user, view.model, 'change', obj,
-                                 request.data)
+        return check_user_access(request.user, view.model, 'change', obj, request.data)
 
     def check_patch_permissions(self, request, view, obj=None):
         return self.check_put_permissions(request, view, obj)
@@ -89,10 +94,10 @@ class ModelAccessPermission(permissions.BasePermission):
         return check_user_access(request.user, view.model, 'delete', obj)
 
     def check_permissions(self, request, view, obj=None):
-        '''
+        """
         Perform basic permissions checking before delegating to the appropriate
         method based on the request method.
-        '''
+        """
 
         # Don't allow anonymous users. 401, not 403, hence no raised exception.
         if not request.user or request.user.is_anonymous:
@@ -117,9 +122,7 @@ class ModelAccessPermission(permissions.BasePermission):
         return result
 
     def has_permission(self, request, view, obj=None):
-        logger.debug('has_permission(user=%s method=%s data=%r, %s, %r)',
-                     request.user, request.method, request.data,
-                     view.__class__.__name__, obj)
+        logger.debug('has_permission(user=%s method=%s data=%r, %s, %r)', request.user, request.method, request.data, view.__class__.__name__, obj)
         try:
             response = self.check_permissions(request, view, obj)
         except Exception as e:
@@ -134,10 +137,10 @@ class ModelAccessPermission(permissions.BasePermission):
 
 
 class JobTemplateCallbackPermission(ModelAccessPermission):
-    '''
+    """
     Permission check used by job template callback view for requests from
     empheral hosts.
-    '''
+    """
 
     def has_permission(self, request, view, obj=None):
         # If another authentication method was used and it's not a POST, return
@@ -160,18 +163,16 @@ class JobTemplateCallbackPermission(ModelAccessPermission):
 
 
 class VariableDataPermission(ModelAccessPermission):
-
     def check_put_permissions(self, request, view, obj=None):
         if not obj:
             return True
-        return check_user_access(request.user, view.model, 'change', obj,
-                                 dict(variables=request.data))
+        return check_user_access(request.user, view.model, 'change', obj, dict(variables=request.data))
 
 
 class TaskPermission(ModelAccessPermission):
-    '''
+    """
     Permission checks used for API callbacks from running a task.
-    '''
+    """
 
     def has_permission(self, request, view, obj=None):
         # If another authentication method was used other than the one for
@@ -182,8 +183,7 @@ class TaskPermission(ModelAccessPermission):
         # Verify that the ID present in the auth token is for a valid, active
         # unified job.
         try:
-            unified_job = UnifiedJob.objects.get(status='running',
-                                                 pk=int(request.auth.split('-')[0]))
+            unified_job = UnifiedJob.objects.get(status='running', pk=int(request.auth.split('-')[0]))
         except (UnifiedJob.DoesNotExist, TypeError):
             return False
 
@@ -197,10 +197,10 @@ class TaskPermission(ModelAccessPermission):
 
 
 class WorkflowApprovalPermission(ModelAccessPermission):
-    '''
+    """
     Permission check used by workflow `approval` and `deny` views to determine
     who has access to approve and deny paused workflow nodes
-    '''
+    """
 
     def check_post_permissions(self, request, view, obj=None):
         approval = get_object_or_400(view.model, pk=view.kwargs['pk'])
@@ -208,9 +208,10 @@ class WorkflowApprovalPermission(ModelAccessPermission):
 
 
 class ProjectUpdatePermission(ModelAccessPermission):
-    '''
+    """
     Permission check used by ProjectUpdateView to determine who can update projects
-    '''
+    """
+
     def check_get_permissions(self, request, view, obj=None):
         project = get_object_or_400(view.model, pk=view.kwargs['pk'])
         return check_user_access(request.user, view.model, 'read', project)
@@ -235,18 +236,23 @@ class UserPermission(ModelAccessPermission):
         raise PermissionDenied()
 
 
-class IsSuperUser(permissions.BasePermission):
+class IsSystemAdminOrAuditor(permissions.BasePermission):
     """
-    Allows access only to admin users.
+    Allows write access only to system admin users.
+    Allows read access only to system auditor users.
     """
 
     def has_permission(self, request, view):
-        return request.user and request.user.is_superuser
+        if not request.user:
+            return False
+        if request.method == 'GET':
+            return request.user.is_superuser or request.user.is_system_auditor
+        return request.user.is_superuser
 
 
 class InstanceGroupTowerPermission(ModelAccessPermission):
     def has_object_permission(self, request, view, obj):
-        if request.method == 'DELETE' and obj.name == "tower":
+        if request.method == 'DELETE' and obj.name in [settings.DEFAULT_EXECUTION_QUEUE_NAME, settings.DEFAULT_CONTROL_PLANE_QUEUE_NAME]:
             return False
         return super(InstanceGroupTowerPermission, self).has_object_permission(request, view, obj)
 

awx/api/renderers.py

@@ -7,12 +7,29 @@ from prometheus_client.parser import text_string_to_metric_families
 # Django REST Framework
 from rest_framework import renderers
 from rest_framework.request import override_method
+from rest_framework.utils import encoders
+
+
+class SurrogateEncoder(encoders.JSONEncoder):
+    def encode(self, obj):
+        ret = super(SurrogateEncoder, self).encode(obj)
+        try:
+            ret.encode()
+        except UnicodeEncodeError as e:
+            if 'surrogates not allowed' in e.reason:
+                ret = ret.encode('utf-8', 'replace').decode()
+        return ret
+
+
+class DefaultJSONRenderer(renderers.JSONRenderer):
+
+    encoder_class = SurrogateEncoder
 
 
 class BrowsableAPIRenderer(renderers.BrowsableAPIRenderer):
-    '''
+    """
     Customizations to the default browsable API renderer.
-    '''
+    """
 
     def get_default_renderer(self, view):
         renderer = super(BrowsableAPIRenderer, self).get_default_renderer(view)
@@ -30,9 +47,7 @@ class BrowsableAPIRenderer(renderers.BrowsableAPIRenderer):
             # see: https://github.com/ansible/awx/issues/3108
             # https://code.djangoproject.com/ticket/28121
             return data
-        return super(BrowsableAPIRenderer, self).get_content(renderer, data,
-                                                             accepted_media_type,
-                                                             renderer_context)
+        return super(BrowsableAPIRenderer, self).get_content(renderer, data, accepted_media_type, renderer_context)
 
     def get_context(self, data, accepted_media_type, renderer_context):
         # Store the associated response status to know how to populate the raw
@@ -107,18 +122,25 @@ class AnsiDownloadRenderer(PlainTextRenderer):
 
 
 class PrometheusJSONRenderer(renderers.JSONRenderer):
-
     def render(self, data, accepted_media_type=None, renderer_context=None):
         if isinstance(data, dict):
             # HTTP errors are {'detail': ErrorDetail(string='...', code=...)}
-            return super(PrometheusJSONRenderer, self).render(
-                data, accepted_media_type, renderer_context
-            )
+            return super(PrometheusJSONRenderer, self).render(data, accepted_media_type, renderer_context)
         parsed_metrics = text_string_to_metric_families(data)
         data = {}
         for family in parsed_metrics:
+            data[family.name] = {}
+            data[family.name]['help_text'] = family.documentation
+            data[family.name]['type'] = family.type
+            data[family.name]['samples'] = []
             for sample in family.samples:
-                data[sample[0]] = {"labels": sample[1], "value": sample[2]}
-        return super(PrometheusJSONRenderer, self).render(
-            data, accepted_media_type, renderer_context
-        )
+                sample_dict = {"labels": sample[1], "value": sample[2]}
+                if family.type == 'histogram':
+                    if sample[0].endswith("_sum"):
+                        sample_dict['sample_type'] = "sum"
+                    elif sample[0].endswith("_count"):
+                        sample_dict['sample_type'] = "count"
+                    elif sample[0].endswith("_bucket"):
+                        sample_dict['sample_type'] = "bucket"
+                data[family.name]['samples'].append(sample_dict)
+        return super(PrometheusJSONRenderer, self).render(data, accepted_media_type, renderer_context)

awx/api/swagger.py

@@ -14,7 +14,6 @@ from rest_framework_swagger import renderers
 
 
 class SuperUserSchemaGenerator(SchemaGenerator):
-
     def has_view_permissions(self, path, method, view):
         #
         # Generate the Swagger schema as if you were a superuser and
@@ -25,17 +24,17 @@ class SuperUserSchemaGenerator(SchemaGenerator):
 
 
 class AutoSchema(DRFAuthSchema):
-
     def get_link(self, path, method, base_url):
         link = super(AutoSchema, self).get_link(path, method, base_url)
         try:
             serializer = self.view.get_serializer()
         except Exception:
             serializer = None
-            warnings.warn('{}.get_serializer() raised an exception during '
-                          'schema generation. Serializer fields will not be '
-                          'generated for {} {}.'
-                          .format(self.view.__class__.__name__, method, path))
+            warnings.warn(
+                '{}.get_serializer() raised an exception during '
+                'schema generation. Serializer fields will not be '
+                'generated for {} {}.'.format(self.view.__class__.__name__, method, path)
+            )
 
         link.__dict__['deprecated'] = getattr(self.view, 'deprecated', False)
 
@@ -43,9 +42,7 @@ class AutoSchema(DRFAuthSchema):
         if hasattr(self.view, 'swagger_topic'):
             link.__dict__['topic'] = str(self.view.swagger_topic).title()
         elif serializer and hasattr(serializer, 'Meta'):
-            link.__dict__['topic'] = str(
-                serializer.Meta.model._meta.verbose_name_plural
-            ).title()
+            link.__dict__['topic'] = str(serializer.Meta.model._meta.verbose_name_plural).title()
         elif hasattr(self.view, 'model'):
             link.__dict__['topic'] = str(self.view.model._meta.verbose_name_plural).title()
         else:
@@ -62,18 +59,10 @@ class SwaggerSchemaView(APIView):
     _ignore_model_permissions = True
     exclude_from_schema = True
     permission_classes = [AllowAny]
-    renderer_classes = [
-        CoreJSONRenderer,
-        renderers.OpenAPIRenderer,
-        renderers.SwaggerUIRenderer
-    ]
+    renderer_classes = [CoreJSONRenderer, renderers.OpenAPIRenderer, renderers.SwaggerUIRenderer]
 
     def get(self, request):
-        generator = SuperUserSchemaGenerator(
-            title='Ansible Tower API',
-            patterns=None,
-            urlconf=None
-        )
+        generator = SuperUserSchemaGenerator(title='Ansible Automation Platform controller API', patterns=None, urlconf=None)
         schema = generator.get_schema(request=request)
         # python core-api doesn't support the deprecation yet, so track it
         # ourselves and return it in a response header
@@ -103,11 +92,6 @@ class SwaggerSchemaView(APIView):
                     schema._data[topic]._data[path] = node
 
         if not schema:
-            raise exceptions.ValidationError(
-                'The schema generator did not return a schema Document'
-            )
+            raise exceptions.ValidationError('The schema generator did not return a schema Document')
 
-        return Response(
-            schema,
-            headers={'X-Deprecated-Paths': json.dumps(_deprecated)}
-        )
+        return Response(schema, headers={'X-Deprecated-Paths': json.dumps(_deprecated)})

awx/api/templates/api/_schedule_detail.md

@@ -4,7 +4,6 @@ The following lists the expected format and details of our rrules:
 * DTSTART is expected to be in UTC
 * INTERVAL is required
 * SECONDLY is not supported
-* TZID is not supported
 * RRULE must precede the rule statements
 * BYDAY is supported but not BYDAY with a numerical prefix
 * BYYEARDAY and BYWEEKNO are not supported

awx/api/templates/api/ad_hoc_command_ad_hoc_command_events_list.md

@@ -0,0 +1 @@
+{% include "api/job_job_events_list.md" %}

awx/api/templates/api/dashboard_jobs_graph_view.md

@@ -8,7 +8,7 @@ The `period` of the data can be adjusted with:
 
     ?period=month
 
-Where `month` can be replaced with `week`, or `day`.  `month` is the default.
+Where `month` can be replaced with `week`, `two_weeks`, or `day`.  `month` is the default.
 
 The type of job can be filtered with:
 

awx/api/templates/api/instance_health_check.md

@@ -0,0 +1,33 @@
+{% ifmeth GET %}
+# Health Check Data
+
+Health checks are used to obtain important data about an instance.
+Instance fields affected by the health check are shown in this view.
+Fundamentally, health checks require running code on the machine in question.
+
+ - For instances with `node_type` of "control" or "hybrid", health checks are
+performed as part of a periodic task that runs in the background.
+ - For instances with `node_type` of "execution", health checks are done by submitting
+a work unit through the receptor mesh.
+
+If ran through the receptor mesh, the invoked command is:
+
+```
+ansible-runner worker --worker-info
+```
+
+For execution nodes, these checks are _not_ performed on a regular basis.
+Health checks against functional nodes will be ran when the node is first discovered.
+Health checks against nodes with errors will be repeated at a reduced frequency.
+
+{% endifmeth %}
+
+{% ifmeth POST %}
+# Manually Initiate a Health Check
+For purposes of error remediation or debugging, a health check can be
+manually initiated by making a POST request to this endpoint.
+
+This will submit the work unit to the target node through the receptor mesh and wait for it to finish.
+The model will be updated with the result.
+Up-to-date values of the fields will be returned in the response data.
+{% endifmeth %}

awx/api/templates/api/inventory_update_events_list.md

@@ -0,0 +1 @@
+{% include "api/job_job_events_list.md" %}

awx/api/templates/api/job_job_events_list.md

@@ -0,0 +1,21 @@
+{% include "api/sub_list_api_view.md" %}
+{% ifmeth GET %}
+## Special limit feature for event list views
+
+Use the `limit` query string parameter to opt out of the pagination keys.
+Doing this can improve response times for jobs that produce a large volume
+of outputs.
+
+    ?limit=25
+
+This will set the page size to 25 and the `previous` and `next` keys will be
+omitted from the response data. The data structure will look like this.
+
+    {
+        "results": [
+            ...
+        ]
+    }
+
+
+{% endifmeth %}

awx/api/templates/api/job_job_plays_list.md

@@ -1,25 +0,0 @@
-Make a GET request to retrieve the list of aggregated play data associated with a job
-
-## Filtering
-
-This endpoints supports a limited filtering subset:
-
-    ?event_id__in=1,2,3
-
-Will show only the given ids.
-
-    ?event_id__gt=1
-
-Will show ids greater than the given one.
-
-    ?event_id__lt=3
-
-Will show ids less than the given one.
-
-    ?failed=true
-
-Will show only failed plays.  Alternatively `false` may be used.
-
-    ?play__icontains=test
-
-Will filter plays matching the substring `test`

awx/api/templates/api/job_job_tasks_list.md

@@ -1,27 +0,0 @@
-Make a GET request to retrieve the list of aggregated task data associated with the play given by event_id.
-
-`event_id` is a required query parameter and must match the job event id of the parent play in order to receive the list of tasks associated with the play
-
-## Filtering
-
-This endpoints supports a limited filtering subset:
-
-    ?event_id__in=1,2,3
-
-Will show only the given task ids under the play given by `event_id`.
-
-    ?event_id__gt=1
-
-Will show ids greater than the given one.
-
-    ?event_id__lt=3
-
-Will show ids less than the given one.
-
-    ?failed=true
-
-Will show only failed plays.  Alternatively `false` may be used.
-
-    ?task__icontains=test
-
-Will filter tasks matching the substring `test`

awx/api/templates/api/metrics_view.md

@@ -0,0 +1 @@
+query params to filter response, e.g., ?subsystemonly=1&metric=callback_receiver_events_insert_db&node=awx-1

awx/api/templates/api/project_update_events_list.md

@@ -0,0 +1 @@
+{% include "api/job_job_events_list.md" %}

awx/api/templates/api/setting_logging_test.md

@@ -1 +1,2 @@
 # Test Logging Configuration
+

awx/api/templates/api/system_job_events_list.md

@@ -0,0 +1 @@
+{% include "api/job_job_events_list.md" %}

awx/api/urls/activity_stream.py

@@ -3,10 +3,7 @@
 
 from django.conf.urls import url
 
-from awx.api.views import (
-    ActivityStreamList,
-    ActivityStreamDetail,
-)
+from awx.api.views import ActivityStreamList, ActivityStreamDetail
 
 
 urls = [

awx/api/urls/ad_hoc_command_event.py

@@ -3,14 +3,10 @@
 
 from django.conf.urls import url
 
-from awx.api.views import (
-    AdHocCommandEventList,
-    AdHocCommandEventDetail,
-)
+from awx.api.views import AdHocCommandEventDetail
 
 
 urls = [
-    url(r'^$', AdHocCommandEventList.as_view(), name='ad_hoc_command_event_list'),
     url(r'^(?P<pk>[0-9]+)/$', AdHocCommandEventDetail.as_view(), name='ad_hoc_command_event_detail'),
 ]
 

awx/api/urls/credential_input_source.py

@@ -3,10 +3,7 @@
 
 from django.conf.urls import url
 
-from awx.api.views import (
-    CredentialInputSourceDetail,
-    CredentialInputSourceList,
-)
+from awx.api.views import CredentialInputSourceDetail, CredentialInputSourceList
 
 
 urls = [

awx/api/urls/credential_type.py

@@ -3,13 +3,7 @@
 
 from django.conf.urls import url
 
-from awx.api.views import (
-    CredentialTypeList,
-    CredentialTypeDetail,
-    CredentialTypeCredentialList,
-    CredentialTypeActivityStreamList,
-    CredentialTypeExternalTest,
-)
+from awx.api.views import CredentialTypeList, CredentialTypeDetail, CredentialTypeCredentialList, CredentialTypeActivityStreamList, CredentialTypeExternalTest
 
 
 urls = [

awx/api/urls/execution_environments.py

@@ -0,0 +1,20 @@
+from django.conf.urls import url
+
+from awx.api.views import (
+    ExecutionEnvironmentList,
+    ExecutionEnvironmentDetail,
+    ExecutionEnvironmentJobTemplateList,
+    ExecutionEnvironmentCopy,
+    ExecutionEnvironmentActivityStreamList,
+)
+
+
+urls = [
+    url(r'^$', ExecutionEnvironmentList.as_view(), name='execution_environment_list'),
+    url(r'^(?P<pk>[0-9]+)/$', ExecutionEnvironmentDetail.as_view(), name='execution_environment_detail'),
+    url(r'^(?P<pk>[0-9]+)/unified_job_templates/$', ExecutionEnvironmentJobTemplateList.as_view(), name='execution_environment_job_template_list'),
+    url(r'^(?P<pk>[0-9]+)/copy/$', ExecutionEnvironmentCopy.as_view(), name='execution_environment_copy'),
+    url(r'^(?P<pk>[0-9]+)/activity_stream/$', ExecutionEnvironmentActivityStreamList.as_view(), name='execution_environment_activity_stream_list'),
+]
+
+__all__ = ['urls']

awx/api/urls/host.py

@@ -16,7 +16,6 @@ from awx.api.views import (
     HostSmartInventoriesList,
     HostAdHocCommandsList,
     HostAdHocCommandEventsList,
-    HostInsights,
 )
 
 
@@ -33,7 +32,6 @@ urls = [
     url(r'^(?P<pk>[0-9]+)/smart_inventories/$', HostSmartInventoriesList.as_view(), name='host_smart_inventories_list'),
     url(r'^(?P<pk>[0-9]+)/ad_hoc_commands/$', HostAdHocCommandsList.as_view(), name='host_ad_hoc_commands_list'),
     url(r'^(?P<pk>[0-9]+)/ad_hoc_command_events/$', HostAdHocCommandEventsList.as_view(), name='host_ad_hoc_command_events_list'),
-    url(r'^(?P<pk>[0-9]+)/insights/$', HostInsights.as_view(), name='host_insights'),
 ]
 
 __all__ = ['urls']

awx/api/urls/instance.py

@@ -3,20 +3,15 @@
 
 from django.conf.urls import url
 
-from awx.api.views import (
-    InstanceList,
-    InstanceDetail,
-    InstanceUnifiedJobsList,
-    InstanceInstanceGroupsList,
-)
+from awx.api.views import InstanceList, InstanceDetail, InstanceUnifiedJobsList, InstanceInstanceGroupsList, InstanceHealthCheck
 
 
 urls = [
     url(r'^$', InstanceList.as_view(), name='instance_list'),
     url(r'^(?P<pk>[0-9]+)/$', InstanceDetail.as_view(), name='instance_detail'),
     url(r'^(?P<pk>[0-9]+)/jobs/$', InstanceUnifiedJobsList.as_view(), name='instance_unified_jobs_list'),
-    url(r'^(?P<pk>[0-9]+)/instance_groups/$', InstanceInstanceGroupsList.as_view(),
-        name='instance_instance_groups_list'),
+    url(r'^(?P<pk>[0-9]+)/instance_groups/$', InstanceInstanceGroupsList.as_view(), name='instance_instance_groups_list'),
+    url(r'^(?P<pk>[0-9]+)/health_check/$', InstanceHealthCheck.as_view(), name='instance_health_check'),
 ]
 
 __all__ = ['urls']

awx/api/urls/instance_group.py

@@ -3,12 +3,7 @@
 
 from django.conf.urls import url
 
-from awx.api.views import (
-    InstanceGroupList,
-    InstanceGroupDetail,
-    InstanceGroupUnifiedJobsList,
-    InstanceGroupInstanceList,
-)
+from awx.api.views import InstanceGroupList, InstanceGroupDetail, InstanceGroupUnifiedJobsList, InstanceGroupInstanceList
 
 
 urls = [

awx/api/urls/inventory_script.py

@@ -1,21 +0,0 @@
-# Copyright (c) 2017 Ansible, Inc.
-# All Rights Reserved.
-
-from django.conf.urls import url
-
-from awx.api.views import (
-    InventoryScriptList,
-    InventoryScriptDetail,
-    InventoryScriptObjectRolesList,
-    InventoryScriptCopy,
-)
-
-
-urls = [
-    url(r'^$', InventoryScriptList.as_view(), name='inventory_script_list'),
-    url(r'^(?P<pk>[0-9]+)/$', InventoryScriptDetail.as_view(), name='inventory_script_detail'),
-    url(r'^(?P<pk>[0-9]+)/object_roles/$', InventoryScriptObjectRolesList.as_view(), name='inventory_script_object_roles_list'),
-    url(r'^(?P<pk>[0-9]+)/copy/$', InventoryScriptCopy.as_view(), name='inventory_script_copy'),
-]
-
-__all__ = ['urls']

awx/api/urls/inventory_source.py

@@ -29,12 +29,21 @@ urls = [
     url(r'^(?P<pk>[0-9]+)/credentials/$', InventorySourceCredentialsList.as_view(), name='inventory_source_credentials_list'),
     url(r'^(?P<pk>[0-9]+)/groups/$', InventorySourceGroupsList.as_view(), name='inventory_source_groups_list'),
     url(r'^(?P<pk>[0-9]+)/hosts/$', InventorySourceHostsList.as_view(), name='inventory_source_hosts_list'),
-    url(r'^(?P<pk>[0-9]+)/notification_templates_started/$', InventorySourceNotificationTemplatesStartedList.as_view(),
-        name='inventory_source_notification_templates_started_list'),
-    url(r'^(?P<pk>[0-9]+)/notification_templates_error/$', InventorySourceNotificationTemplatesErrorList.as_view(),
-        name='inventory_source_notification_templates_error_list'),
-    url(r'^(?P<pk>[0-9]+)/notification_templates_success/$', InventorySourceNotificationTemplatesSuccessList.as_view(),
-        name='inventory_source_notification_templates_success_list'),
+    url(
+        r'^(?P<pk>[0-9]+)/notification_templates_started/$',
+        InventorySourceNotificationTemplatesStartedList.as_view(),
+        name='inventory_source_notification_templates_started_list',
+    ),
+    url(
+        r'^(?P<pk>[0-9]+)/notification_templates_error/$',
+        InventorySourceNotificationTemplatesErrorList.as_view(),
+        name='inventory_source_notification_templates_error_list',
+    ),
+    url(
+        r'^(?P<pk>[0-9]+)/notification_templates_success/$',
+        InventorySourceNotificationTemplatesSuccessList.as_view(),
+        name='inventory_source_notification_templates_success_list',
+    ),
 ]
 
 __all__ = ['urls']

awx/api/urls/job_event.py

@@ -3,19 +3,11 @@
 
 from django.conf.urls import url
 
-from awx.api.views import (
-    JobEventList,
-    JobEventDetail,
-    JobEventChildrenList,
-    JobEventHostsList,
-)
-
+from awx.api.views import JobEventDetail, JobEventChildrenList
 
 urls = [
-    url(r'^$', JobEventList.as_view(), name='job_event_list'),
     url(r'^(?P<pk>[0-9]+)/$', JobEventDetail.as_view(), name='job_event_detail'),
     url(r'^(?P<pk>[0-9]+)/children/$', JobEventChildrenList.as_view(), name='job_event_children_list'),
-    url(r'^(?P<pk>[0-9]+)/hosts/$', JobEventHostsList.as_view(), name='job_event_hosts_list'),
 ]
 
 __all__ = ['urls']

awx/api/urls/job_host_summary.py

@@ -3,13 +3,9 @@
 
 from django.conf.urls import url
 
-from awx.api.views import (
-    JobHostSummaryDetail,
-)
+from awx.api.views import JobHostSummaryDetail
 
 
-urls = [
-    url(r'^(?P<pk>[0-9]+)/$', JobHostSummaryDetail.as_view(), name='job_host_summary_detail'),
-]
+urls = [url(r'^(?P<pk>[0-9]+)/$', JobHostSummaryDetail.as_view(), name='job_host_summary_detail')]
 
 __all__ = ['urls']

awx/api/urls/job_template.py

@@ -34,12 +34,21 @@ urls = [
     url(r'^(?P<pk>[0-9]+)/schedules/$', JobTemplateSchedulesList.as_view(), name='job_template_schedules_list'),
     url(r'^(?P<pk>[0-9]+)/survey_spec/$', JobTemplateSurveySpec.as_view(), name='job_template_survey_spec'),
     url(r'^(?P<pk>[0-9]+)/activity_stream/$', JobTemplateActivityStreamList.as_view(), name='job_template_activity_stream_list'),
-    url(r'^(?P<pk>[0-9]+)/notification_templates_started/$', JobTemplateNotificationTemplatesStartedList.as_view(),
-        name='job_template_notification_templates_started_list'),
-    url(r'^(?P<pk>[0-9]+)/notification_templates_error/$', JobTemplateNotificationTemplatesErrorList.as_view(),
-        name='job_template_notification_templates_error_list'),
-    url(r'^(?P<pk>[0-9]+)/notification_templates_success/$', JobTemplateNotificationTemplatesSuccessList.as_view(),
-        name='job_template_notification_templates_success_list'),
+    url(
+        r'^(?P<pk>[0-9]+)/notification_templates_started/$',
+        JobTemplateNotificationTemplatesStartedList.as_view(),
+        name='job_template_notification_templates_started_list',
+    ),
+    url(
+        r'^(?P<pk>[0-9]+)/notification_templates_error/$',
+        JobTemplateNotificationTemplatesErrorList.as_view(),
+        name='job_template_notification_templates_error_list',
+    ),
+    url(
+        r'^(?P<pk>[0-9]+)/notification_templates_success/$',
+        JobTemplateNotificationTemplatesSuccessList.as_view(),
+        name='job_template_notification_templates_success_list',
+    ),
     url(r'^(?P<pk>[0-9]+)/instance_groups/$', JobTemplateInstanceGroupsList.as_view(), name='job_template_instance_groups_list'),
     url(r'^(?P<pk>[0-9]+)/access_list/$', JobTemplateAccessList.as_view(), name='job_template_access_list'),
     url(r'^(?P<pk>[0-9]+)/object_roles/$', JobTemplateObjectRolesList.as_view(), name='job_template_object_roles_list'),

awx/api/urls/label.py

@@ -3,15 +3,9 @@
 
 from django.conf.urls import url
 
-from awx.api.views import (
-    LabelList,
-    LabelDetail,
-)
+from awx.api.views import LabelList, LabelDetail
 
 
-urls = [
-    url(r'^$', LabelList.as_view(), name='label_list'),
-    url(r'^(?P<pk>[0-9]+)/$', LabelDetail.as_view(), name='label_detail'),
-]
+urls = [url(r'^$', LabelList.as_view(), name='label_list'), url(r'^(?P<pk>[0-9]+)/$', LabelDetail.as_view(), name='label_detail')]
 
 __all__ = ['urls']

awx/api/urls/notification.py

@@ -3,15 +3,9 @@
 
 from django.conf.urls import url
 
-from awx.api.views import (
-    NotificationList,
-    NotificationDetail,
-)
+from awx.api.views import NotificationList, NotificationDetail
 
 
-urls = [
-    url(r'^$', NotificationList.as_view(), name='notification_list'),
-    url(r'^(?P<pk>[0-9]+)/$', NotificationDetail.as_view(), name='notification_detail'),
-]
+urls = [url(r'^$', NotificationList.as_view(), name='notification_list'), url(r'^(?P<pk>[0-9]+)/$', NotificationDetail.as_view(), name='notification_detail')]
 
 __all__ = ['urls']

awx/api/urls/oauth2.py

@@ -16,32 +16,12 @@ from awx.api.views import (
 
 urls = [
     url(r'^applications/$', OAuth2ApplicationList.as_view(), name='o_auth2_application_list'),
-    url(
-        r'^applications/(?P<pk>[0-9]+)/$',
-        OAuth2ApplicationDetail.as_view(),
-        name='o_auth2_application_detail'
-    ),
-    url(
-        r'^applications/(?P<pk>[0-9]+)/tokens/$',
-        ApplicationOAuth2TokenList.as_view(),
-        name='o_auth2_application_token_list'
-    ),
-    url(
-        r'^applications/(?P<pk>[0-9]+)/activity_stream/$',
-        OAuth2ApplicationActivityStreamList.as_view(),
-        name='o_auth2_application_activity_stream_list'
-    ),
+    url(r'^applications/(?P<pk>[0-9]+)/$', OAuth2ApplicationDetail.as_view(), name='o_auth2_application_detail'),
+    url(r'^applications/(?P<pk>[0-9]+)/tokens/$', ApplicationOAuth2TokenList.as_view(), name='o_auth2_application_token_list'),
+    url(r'^applications/(?P<pk>[0-9]+)/activity_stream/$', OAuth2ApplicationActivityStreamList.as_view(), name='o_auth2_application_activity_stream_list'),
     url(r'^tokens/$', OAuth2TokenList.as_view(), name='o_auth2_token_list'),
-    url(
-        r'^tokens/(?P<pk>[0-9]+)/$',
-        OAuth2TokenDetail.as_view(),
-        name='o_auth2_token_detail'
-    ),
-    url(
-        r'^tokens/(?P<pk>[0-9]+)/activity_stream/$',
-        OAuth2TokenActivityStreamList.as_view(),
-        name='o_auth2_token_activity_stream_list'
-    ),   
+    url(r'^tokens/(?P<pk>[0-9]+)/$', OAuth2TokenDetail.as_view(), name='o_auth2_token_detail'),
+    url(r'^tokens/(?P<pk>[0-9]+)/activity_stream/$', OAuth2TokenActivityStreamList.as_view(), name='o_auth2_token_activity_stream_list'),
 ]
 
 __all__ = ['urls']

awx/api/urls/oauth2_root.py

@@ -1,19 +1,33 @@
 # Copyright (c) 2017 Ansible, Inc.
 # All Rights Reserved.
+from datetime import timedelta
 
+from django.utils.timezone import now
+from django.conf import settings
 from django.conf.urls import url
 
 from oauthlib import oauth2
 from oauth2_provider import views
 
-from awx.api.views import (
-    ApiOAuthAuthorizationRootView,
-)
+from awx.main.models import RefreshToken
+from awx.api.views import ApiOAuthAuthorizationRootView
 
 
 class TokenView(views.TokenView):
-
     def create_token_response(self, request):
+        # Django OAuth2 Toolkit has a bug whereby refresh tokens are *never*
+        # properly expired (ugh):
+        #
+        # https://github.com/jazzband/django-oauth-toolkit/issues/746
+        #
+        # This code detects and auto-expires them on refresh grant
+        # requests.
+        if request.POST.get('grant_type') == 'refresh_token' and 'refresh_token' in request.POST:
+            refresh_token = RefreshToken.objects.filter(token=request.POST['refresh_token']).first()
+            if refresh_token:
+                expire_seconds = settings.OAUTH2_PROVIDER.get('REFRESH_TOKEN_EXPIRE_SECONDS', 0)
+                if refresh_token.created + timedelta(seconds=expire_seconds) < now():
+                    return request.build_absolute_uri(), {}, 'The refresh token has expired.', '403'
         try:
             return super(TokenView, self).create_token_response(request)
         except oauth2.AccessDeniedError as e:

awx/api/urls/organization.py

@@ -9,7 +9,9 @@ from awx.api.views import (
     OrganizationUsersList,
     OrganizationAdminsList,
     OrganizationInventoriesList,
+    OrganizationExecutionEnvironmentsList,
     OrganizationProjectsList,
+    OrganizationJobTemplatesList,
     OrganizationWorkflowJobTemplatesList,
     OrganizationTeamsList,
     OrganizationCredentialList,
@@ -20,6 +22,7 @@ from awx.api.views import (
     OrganizationNotificationTemplatesSuccessList,
     OrganizationNotificationTemplatesApprovalList,
     OrganizationInstanceGroupsList,
+    OrganizationGalaxyCredentialsList,
     OrganizationObjectRolesList,
     OrganizationAccessList,
     OrganizationApplicationList,
@@ -32,21 +35,36 @@ urls = [
     url(r'^(?P<pk>[0-9]+)/users/$', OrganizationUsersList.as_view(), name='organization_users_list'),
     url(r'^(?P<pk>[0-9]+)/admins/$', OrganizationAdminsList.as_view(), name='organization_admins_list'),
     url(r'^(?P<pk>[0-9]+)/inventories/$', OrganizationInventoriesList.as_view(), name='organization_inventories_list'),
+    url(r'^(?P<pk>[0-9]+)/execution_environments/$', OrganizationExecutionEnvironmentsList.as_view(), name='organization_execution_environments_list'),
     url(r'^(?P<pk>[0-9]+)/projects/$', OrganizationProjectsList.as_view(), name='organization_projects_list'),
+    url(r'^(?P<pk>[0-9]+)/job_templates/$', OrganizationJobTemplatesList.as_view(), name='organization_job_templates_list'),
     url(r'^(?P<pk>[0-9]+)/workflow_job_templates/$', OrganizationWorkflowJobTemplatesList.as_view(), name='organization_workflow_job_templates_list'),
     url(r'^(?P<pk>[0-9]+)/teams/$', OrganizationTeamsList.as_view(), name='organization_teams_list'),
     url(r'^(?P<pk>[0-9]+)/credentials/$', OrganizationCredentialList.as_view(), name='organization_credential_list'),
     url(r'^(?P<pk>[0-9]+)/activity_stream/$', OrganizationActivityStreamList.as_view(), name='organization_activity_stream_list'),
     url(r'^(?P<pk>[0-9]+)/notification_templates/$', OrganizationNotificationTemplatesList.as_view(), name='organization_notification_templates_list'),
-    url(r'^(?P<pk>[0-9]+)/notification_templates_started/$', OrganizationNotificationTemplatesStartedList.as_view(),
-        name='organization_notification_templates_started_list'),
-    url(r'^(?P<pk>[0-9]+)/notification_templates_error/$', OrganizationNotificationTemplatesErrorList.as_view(),
-        name='organization_notification_templates_error_list'),
-    url(r'^(?P<pk>[0-9]+)/notification_templates_success/$', OrganizationNotificationTemplatesSuccessList.as_view(),
-        name='organization_notification_templates_success_list'),
-    url(r'^(?P<pk>[0-9]+)/notification_templates_approvals/$', OrganizationNotificationTemplatesApprovalList.as_view(),
-        name='organization_notification_templates_approvals_list'),
+    url(
+        r'^(?P<pk>[0-9]+)/notification_templates_started/$',
+        OrganizationNotificationTemplatesStartedList.as_view(),
+        name='organization_notification_templates_started_list',
+    ),
+    url(
+        r'^(?P<pk>[0-9]+)/notification_templates_error/$',
+        OrganizationNotificationTemplatesErrorList.as_view(),
+        name='organization_notification_templates_error_list',
+    ),
+    url(
+        r'^(?P<pk>[0-9]+)/notification_templates_success/$',
+        OrganizationNotificationTemplatesSuccessList.as_view(),
+        name='organization_notification_templates_success_list',
+    ),
+    url(
+        r'^(?P<pk>[0-9]+)/notification_templates_approvals/$',
+        OrganizationNotificationTemplatesApprovalList.as_view(),
+        name='organization_notification_templates_approvals_list',
+    ),
     url(r'^(?P<pk>[0-9]+)/instance_groups/$', OrganizationInstanceGroupsList.as_view(), name='organization_instance_groups_list'),
+    url(r'^(?P<pk>[0-9]+)/galaxy_credentials/$', OrganizationGalaxyCredentialsList.as_view(), name='organization_galaxy_credentials_list'),
     url(r'^(?P<pk>[0-9]+)/object_roles/$', OrganizationObjectRolesList.as_view(), name='organization_object_roles_list'),
     url(r'^(?P<pk>[0-9]+)/access_list/$', OrganizationAccessList.as_view(), name='organization_access_list'),
     url(r'^(?P<pk>[0-9]+)/applications/$', OrganizationApplicationList.as_view(), name='organization_applications_list'),

awx/api/urls/project.py

@@ -35,10 +35,16 @@ urls = [
     url(r'^(?P<pk>[0-9]+)/activity_stream/$', ProjectActivityStreamList.as_view(), name='project_activity_stream_list'),
     url(r'^(?P<pk>[0-9]+)/schedules/$', ProjectSchedulesList.as_view(), name='project_schedules_list'),
     url(r'^(?P<pk>[0-9]+)/notification_templates_error/$', ProjectNotificationTemplatesErrorList.as_view(), name='project_notification_templates_error_list'),
-    url(r'^(?P<pk>[0-9]+)/notification_templates_success/$', ProjectNotificationTemplatesSuccessList.as_view(),
-        name='project_notification_templates_success_list'),
-    url(r'^(?P<pk>[0-9]+)/notification_templates_started/$', ProjectNotificationTemplatesStartedList.as_view(),
-        name='project_notification_templates_started_list'),
+    url(
+        r'^(?P<pk>[0-9]+)/notification_templates_success/$',
+        ProjectNotificationTemplatesSuccessList.as_view(),
+        name='project_notification_templates_success_list',
+    ),
+    url(
+        r'^(?P<pk>[0-9]+)/notification_templates_started/$',
+        ProjectNotificationTemplatesStartedList.as_view(),
+        name='project_notification_templates_started_list',
+    ),
     url(r'^(?P<pk>[0-9]+)/object_roles/$', ProjectObjectRolesList.as_view(), name='project_object_roles_list'),
     url(r'^(?P<pk>[0-9]+)/access_list/$', ProjectAccessList.as_view(), name='project_access_list'),
     url(r'^(?P<pk>[0-9]+)/copy/$', ProjectCopy.as_view(), name='project_copy'),

awx/api/urls/role.py

@@ -3,14 +3,7 @@
 
 from django.conf.urls import url
 
-from awx.api.views import (
-    RoleList,
-    RoleDetail,
-    RoleUsersList,
-    RoleTeamsList,
-    RoleParentsList,
-    RoleChildrenList,
-)
+from awx.api.views import RoleList, RoleDetail, RoleUsersList, RoleTeamsList, RoleParentsList, RoleChildrenList
 
 
 urls = [

awx/api/urls/schedule.py

@@ -3,12 +3,7 @@
 
 from django.conf.urls import url
 
-from awx.api.views import (
-    ScheduleList,
-    ScheduleDetail,
-    ScheduleUnifiedJobsList,
-    ScheduleCredentialsList,
-)
+from awx.api.views import ScheduleList, ScheduleDetail, ScheduleUnifiedJobsList, ScheduleCredentialsList
 
 
 urls = [

awx/api/urls/system_job.py

@@ -3,13 +3,7 @@
 
 from django.conf.urls import url
 
-from awx.api.views import (
-    SystemJobList,
-    SystemJobDetail,
-    SystemJobCancel,
-    SystemJobNotificationsList,
-    SystemJobEventsList
-)
+from awx.api.views import SystemJobList, SystemJobDetail, SystemJobCancel, SystemJobNotificationsList, SystemJobEventsList
 
 
 urls = [

awx/api/urls/system_job_template.py

@@ -21,12 +21,21 @@ urls = [
     url(r'^(?P<pk>[0-9]+)/launch/$', SystemJobTemplateLaunch.as_view(), name='system_job_template_launch'),
     url(r'^(?P<pk>[0-9]+)/jobs/$', SystemJobTemplateJobsList.as_view(), name='system_job_template_jobs_list'),
     url(r'^(?P<pk>[0-9]+)/schedules/$', SystemJobTemplateSchedulesList.as_view(), name='system_job_template_schedules_list'),
-    url(r'^(?P<pk>[0-9]+)/notification_templates_started/$', SystemJobTemplateNotificationTemplatesStartedList.as_view(),
-        name='system_job_template_notification_templates_started_list'),
-    url(r'^(?P<pk>[0-9]+)/notification_templates_error/$', SystemJobTemplateNotificationTemplatesErrorList.as_view(),
-        name='system_job_template_notification_templates_error_list'),
-    url(r'^(?P<pk>[0-9]+)/notification_templates_success/$', SystemJobTemplateNotificationTemplatesSuccessList.as_view(),
-        name='system_job_template_notification_templates_success_list'),
+    url(
+        r'^(?P<pk>[0-9]+)/notification_templates_started/$',
+        SystemJobTemplateNotificationTemplatesStartedList.as_view(),
+        name='system_job_template_notification_templates_started_list',
+    ),
+    url(
+        r'^(?P<pk>[0-9]+)/notification_templates_error/$',
+        SystemJobTemplateNotificationTemplatesErrorList.as_view(),
+        name='system_job_template_notification_templates_error_list',
+    ),
+    url(
+        r'^(?P<pk>[0-9]+)/notification_templates_success/$',
+        SystemJobTemplateNotificationTemplatesSuccessList.as_view(),
+        name='system_job_template_notification_templates_success_list',
+    ),
 ]
 
 __all__ = ['urls']

awx/api/urls/urls.py

@@ -5,16 +5,14 @@ from __future__ import absolute_import, unicode_literals
 from django.conf import settings
 from django.conf.urls import include, url
 
-from awx.api.generics import (
-    LoggedLoginView,
-    LoggedLogoutView,
-)
+from awx.api.generics import LoggedLoginView, LoggedLogoutView
 from awx.api.views import (
     ApiRootView,
     ApiV2RootView,
     ApiV2PingView,
     ApiV2ConfigView,
     ApiV2SubscriptionView,
+    ApiV2AttachView,
     AuthView,
     UserMeList,
     DashboardView,
@@ -23,9 +21,7 @@ from awx.api.views import (
     UnifiedJobList,
     HostAnsibleFactsDetail,
     JobCredentialsList,
-    JobExtraCredentialsList,
     JobTemplateCredentialsList,
-    JobTemplateExtraCredentialsList,
     SchedulePreview,
     ScheduleZoneInfo,
     OAuth2ApplicationList,
@@ -41,12 +37,12 @@ from .user import urls as user_urls
 from .project import urls as project_urls
 from .project_update import urls as project_update_urls
 from .inventory import urls as inventory_urls
+from .execution_environments import urls as execution_environment_urls
 from .team import urls as team_urls
 from .host import urls as host_urls
 from .group import urls as group_urls
 from .inventory_source import urls as inventory_source_urls
 from .inventory_update import urls as inventory_update_urls
-from .inventory_script import urls as inventory_script_urls
 from .credential_type import urls as credential_type_urls
 from .credential import urls as credential_urls
 from .credential_input_source import urls as credential_input_source_urls
@@ -81,9 +77,7 @@ v2_urls = [
     url(r'^credential_types/', include(credential_type_urls)),
     url(r'^credential_input_sources/', include(credential_input_source_urls)),
     url(r'^hosts/(?P<pk>[0-9]+)/ansible_facts/$', HostAnsibleFactsDetail.as_view(), name='host_ansible_facts_detail'),
-    url(r'^jobs/(?P<pk>[0-9]+)/extra_credentials/$', JobExtraCredentialsList.as_view(), name='job_extra_credentials_list'),
     url(r'^jobs/(?P<pk>[0-9]+)/credentials/$', JobCredentialsList.as_view(), name='job_credentials_list'),
-    url(r'^job_templates/(?P<pk>[0-9]+)/extra_credentials/$', JobTemplateExtraCredentialsList.as_view(), name='job_template_extra_credentials_list'),
     url(r'^job_templates/(?P<pk>[0-9]+)/credentials/$', JobTemplateCredentialsList.as_view(), name='job_template_credentials_list'),
     url(r'^schedules/preview/$', SchedulePreview.as_view(), name='schedule_rrule'),
     url(r'^schedules/zoneinfo/$', ScheduleZoneInfo.as_view(), name='schedule_zoneinfo'),
@@ -96,6 +90,7 @@ v2_urls = [
     url(r'^ping/$', ApiV2PingView.as_view(), name='api_v2_ping_view'),
     url(r'^config/$', ApiV2ConfigView.as_view(), name='api_v2_config_view'),
     url(r'^config/subscriptions/$', ApiV2SubscriptionView.as_view(), name='api_v2_subscription_view'),
+    url(r'^config/attach/$', ApiV2AttachView.as_view(), name='api_v2_attach_view'),
     url(r'^auth/$', AuthView.as_view()),
     url(r'^me/$', UserMeList.as_view(), name='user_me_list'),
     url(r'^dashboard/$', DashboardView.as_view(), name='dashboard_view'),
@@ -106,6 +101,7 @@ v2_urls = [
     url(r'^schedules/', include(schedule_urls)),
     url(r'^organizations/', include(organization_urls)),
     url(r'^users/', include(user_urls)),
+    url(r'^execution_environments/', include(execution_environment_urls)),
     url(r'^projects/', include(project_urls)),
     url(r'^project_updates/', include(project_update_urls)),
     url(r'^teams/', include(team_urls)),
@@ -114,7 +110,6 @@ v2_urls = [
     url(r'^groups/', include(group_urls)),
     url(r'^inventory_sources/', include(inventory_source_urls)),
     url(r'^inventory_updates/', include(inventory_update_urls)),
-    url(r'^inventory_scripts/', include(inventory_script_urls)),
     url(r'^credentials/', include(credential_urls)),
     url(r'^roles/', include(role_urls)),
     url(r'^job_templates/', include(job_template_urls)),
@@ -144,17 +139,11 @@ app_name = 'api'
 urlpatterns = [
     url(r'^$', ApiRootView.as_view(), name='api_root_view'),
     url(r'^(?P<version>(v2))/', include(v2_urls)),
-    url(r'^login/$', LoggedLoginView.as_view(
-        template_name='rest_framework/login.html',
-        extra_context={'inside_login_context': True}
-    ), name='login'),
-    url(r'^logout/$', LoggedLogoutView.as_view(
-        next_page='/api/', redirect_field_name='next'
-    ), name='logout'),
+    url(r'^login/$', LoggedLoginView.as_view(template_name='rest_framework/login.html', extra_context={'inside_login_context': True}), name='login'),
+    url(r'^logout/$', LoggedLogoutView.as_view(next_page='/api/', redirect_field_name='next'), name='logout'),
     url(r'^o/', include(oauth2_root_urls)),
 ]
 if settings.SETTINGS_MODULE == 'awx.settings.development':
     from awx.api.swagger import SwaggerSchemaView
-    urlpatterns += [
-        url(r'^swagger/$', SwaggerSchemaView.as_view(), name='swagger_view'),
-    ]
+
+    urlpatterns += [url(r'^swagger/$', SwaggerSchemaView.as_view(), name='swagger_view')]

awx/api/urls/user.py

@@ -20,7 +20,7 @@ from awx.api.views import (
     UserAuthorizedTokenList,
 )
 
-urls = [ 
+urls = [
     url(r'^$', UserList.as_view(), name='user_list'),
     url(r'^(?P<pk>[0-9]+)/$', UserDetail.as_view(), name='user_detail'),
     url(r'^(?P<pk>[0-9]+)/teams/$', UserTeamsList.as_view(), name='user_teams_list'),
@@ -35,7 +35,6 @@ urls = [
     url(r'^(?P<pk>[0-9]+)/tokens/$', OAuth2UserTokenList.as_view(), name='o_auth2_token_list'),
     url(r'^(?P<pk>[0-9]+)/authorized_tokens/$', UserAuthorizedTokenList.as_view(), name='user_authorized_token_list'),
     url(r'^(?P<pk>[0-9]+)/personal_tokens/$', UserPersonalTokenList.as_view(), name='user_personal_token_list'),
-    
-] 
+]
 
 __all__ = ['urls']

awx/api/urls/webhooks.py

@@ -1,10 +1,6 @@
 from django.conf.urls import url
 
-from awx.api.views import (
-    WebhookKeyView,
-    GithubWebhookReceiver,
-    GitlabWebhookReceiver,
-)
+from awx.api.views import WebhookKeyView, GithubWebhookReceiver, GitlabWebhookReceiver
 
 
 urlpatterns = [

awx/api/urls/workflow_approval.py

@@ -3,12 +3,7 @@
 
 from django.conf.urls import url
 
-from awx.api.views import (
-    WorkflowApprovalList,
-    WorkflowApprovalDetail,
-    WorkflowApprovalApprove,
-    WorkflowApprovalDeny,
-)
+from awx.api.views import WorkflowApprovalList, WorkflowApprovalDetail, WorkflowApprovalApprove, WorkflowApprovalDeny
 
 
 urls = [

awx/api/urls/workflow_approval_template.py

@@ -3,10 +3,7 @@
 
 from django.conf.urls import url
 
-from awx.api.views import (
-    WorkflowApprovalTemplateDetail,
-    WorkflowApprovalTemplateJobsList,
-)
+from awx.api.views import WorkflowApprovalTemplateDetail, WorkflowApprovalTemplateJobsList
 
 
 urls = [

awx/api/urls/workflow_job_template.py

@@ -33,14 +33,26 @@ urls = [
     url(r'^(?P<pk>[0-9]+)/survey_spec/$', WorkflowJobTemplateSurveySpec.as_view(), name='workflow_job_template_survey_spec'),
     url(r'^(?P<pk>[0-9]+)/workflow_nodes/$', WorkflowJobTemplateWorkflowNodesList.as_view(), name='workflow_job_template_workflow_nodes_list'),
     url(r'^(?P<pk>[0-9]+)/activity_stream/$', WorkflowJobTemplateActivityStreamList.as_view(), name='workflow_job_template_activity_stream_list'),
-    url(r'^(?P<pk>[0-9]+)/notification_templates_started/$', WorkflowJobTemplateNotificationTemplatesStartedList.as_view(),
-        name='workflow_job_template_notification_templates_started_list'),
-    url(r'^(?P<pk>[0-9]+)/notification_templates_error/$', WorkflowJobTemplateNotificationTemplatesErrorList.as_view(),
-        name='workflow_job_template_notification_templates_error_list'),
-    url(r'^(?P<pk>[0-9]+)/notification_templates_success/$', WorkflowJobTemplateNotificationTemplatesSuccessList.as_view(),
-        name='workflow_job_template_notification_templates_success_list'),
-    url(r'^(?P<pk>[0-9]+)/notification_templates_approvals/$', WorkflowJobTemplateNotificationTemplatesApprovalList.as_view(),
-        name='workflow_job_template_notification_templates_approvals_list'),
+    url(
+        r'^(?P<pk>[0-9]+)/notification_templates_started/$',
+        WorkflowJobTemplateNotificationTemplatesStartedList.as_view(),
+        name='workflow_job_template_notification_templates_started_list',
+    ),
+    url(
+        r'^(?P<pk>[0-9]+)/notification_templates_error/$',
+        WorkflowJobTemplateNotificationTemplatesErrorList.as_view(),
+        name='workflow_job_template_notification_templates_error_list',
+    ),
+    url(
+        r'^(?P<pk>[0-9]+)/notification_templates_success/$',
+        WorkflowJobTemplateNotificationTemplatesSuccessList.as_view(),
+        name='workflow_job_template_notification_templates_success_list',
+    ),
+    url(
+        r'^(?P<pk>[0-9]+)/notification_templates_approvals/$',
+        WorkflowJobTemplateNotificationTemplatesApprovalList.as_view(),
+        name='workflow_job_template_notification_templates_approvals_list',
+    ),
     url(r'^(?P<pk>[0-9]+)/access_list/$', WorkflowJobTemplateAccessList.as_view(), name='workflow_job_template_access_list'),
     url(r'^(?P<pk>[0-9]+)/object_roles/$', WorkflowJobTemplateObjectRolesList.as_view(), name='workflow_job_template_object_roles_list'),
     url(r'^(?P<pk>[0-9]+)/labels/$', WorkflowJobTemplateLabelList.as_view(), name='workflow_job_template_label_list'),

awx/api/versioning.py

@@ -40,13 +40,10 @@ def reverse(viewname, args=None, kwargs=None, request=None, format=None, **extra
 
 
 class URLPathVersioning(BaseVersioning):
-
     def reverse(self, viewname, args=None, kwargs=None, request=None, format=None, **extra):
         if request.version is not None:
             kwargs = {} if (kwargs is None) else kwargs
             kwargs[self.version_param] = request.version
         request = None
 
-        return super(BaseVersioning, self).reverse(
-            viewname, args, kwargs, request, format, **extra
-        )
+        return super(BaseVersioning, self).reverse(viewname, args, kwargs, request, format, **extra)

awx/api/views/inventory.py

@@ -25,17 +25,8 @@ from awx.main.models import (
     InstanceGroup,
     InventoryUpdateEvent,
     InventoryUpdate,
-    InventorySource,
-    CustomInventoryScript,
-)
-from awx.api.generics import (
-    ListCreateAPIView,
-    RetrieveUpdateDestroyAPIView,
-    SubListAPIView,
-    SubListAttachDetachAPIView,
-    ResourceAccessList,
-    CopyAPIView,
 )
+from awx.api.generics import ListCreateAPIView, RetrieveUpdateDestroyAPIView, SubListAPIView, SubListAttachDetachAPIView, ResourceAccessList, CopyAPIView
 
 from awx.api.serializers import (
     InventorySerializer,
@@ -43,13 +34,12 @@ from awx.api.serializers import (
     RoleSerializer,
     InstanceGroupSerializer,
     InventoryUpdateEventSerializer,
-    CustomInventoryScriptSerializer,
     JobTemplateSerializer,
 )
-from awx.api.views.mixin import (
-    RelatedJobsPreventDeleteMixin,
-    ControlledByScmMixin,
-)
+from awx.api.views.mixin import RelatedJobsPreventDeleteMixin, ControlledByScmMixin
+
+from awx.api.pagination import UnifiedJobEventPagination
+
 
 logger = logging.getLogger('awx.api.views.organization')
 
@@ -62,61 +52,18 @@ class InventoryUpdateEventsList(SubListAPIView):
     relationship = 'inventory_update_events'
     name = _('Inventory Update Events List')
     search_fields = ('stdout',)
+    pagination_class = UnifiedJobEventPagination
+
+    def get_queryset(self):
+        iu = self.get_parent_object()
+        self.check_parent_access(iu)
+        return iu.get_event_queryset()
 
     def finalize_response(self, request, response, *args, **kwargs):
         response['X-UI-Max-Events'] = settings.MAX_UI_JOB_EVENTS
         return super(InventoryUpdateEventsList, self).finalize_response(request, response, *args, **kwargs)
 
 
-class InventoryScriptList(ListCreateAPIView):
-
-    deprecated = True
-
-    model = CustomInventoryScript
-    serializer_class = CustomInventoryScriptSerializer
-
-
-class InventoryScriptDetail(RetrieveUpdateDestroyAPIView):
-
-    deprecated = True
-
-    model = CustomInventoryScript
-    serializer_class = CustomInventoryScriptSerializer
-
-    def destroy(self, request, *args, **kwargs):
-        instance = self.get_object()
-        can_delete = request.user.can_access(self.model, 'delete', instance)
-        if not can_delete:
-            raise PermissionDenied(_("Cannot delete inventory script."))
-        for inv_src in InventorySource.objects.filter(source_script=instance):
-            inv_src.source_script = None
-            inv_src.save()
-        return super(InventoryScriptDetail, self).destroy(request, *args, **kwargs)
-
-
-class InventoryScriptObjectRolesList(SubListAPIView):
-
-    deprecated = True
-
-    model = Role
-    serializer_class = RoleSerializer
-    parent_model = CustomInventoryScript
-    search_fields = ('role_field', 'content_type__model',)
-
-    def get_queryset(self):
-        po = self.get_parent_object()
-        content_type = ContentType.objects.get_for_model(self.parent_model)
-        return Role.objects.filter(content_type=content_type, object_id=po.pk)
-
-
-class InventoryScriptCopy(CopyAPIView):
-
-    deprecated = True
-
-    model = CustomInventoryScript
-    copy_return_serializer_class = CustomInventoryScriptSerializer
-
-
 class InventoryList(ListCreateAPIView):
 
     model = Inventory
@@ -134,7 +81,7 @@ class InventoryDetail(RelatedJobsPreventDeleteMixin, ControlledByScmMixin, Retri
 
         # Do not allow changes to an Inventory kind.
         if kind is not None and obj.kind != kind:
-            return self.http_method_not_allowed(request, *args, **kwargs)
+            return Response(dict(error=_('You cannot turn a regular inventory into a "smart" inventory.')), status=status.HTTP_405_METHOD_NOT_ALLOWED)
         return super(InventoryDetail, self).update(request, *args, **kwargs)
 
     def destroy(self, request, *args, **kwargs):
@@ -174,7 +121,7 @@ class InventoryInstanceGroupsList(SubListAttachDetachAPIView):
 
 class InventoryAccessList(ResourceAccessList):
 
-    model = User # needs to be User for AccessLists's
+    model = User  # needs to be User for AccessLists's
     parent_model = Inventory
 
 
@@ -183,7 +130,7 @@ class InventoryObjectRolesList(SubListAPIView):
     model = Role
     serializer_class = RoleSerializer
     parent_model = Inventory
-    search_fields = ('role_field', 'content_type__model',)
+    search_fields = ('role_field', 'content_type__model')
 
     def get_queryset(self):
         po = self.get_parent_object()

awx/api/views/metrics.py

@@ -14,15 +14,14 @@ from rest_framework.exceptions import PermissionDenied
 
 # AWX
 # from awx.main.analytics import collectors
+import awx.main.analytics.subsystem_metrics as s_metrics
 from awx.main.analytics.metrics import metrics
 from awx.api import renderers
 
-from awx.api.generics import (
-    APIView,
-)
+from awx.api.generics import APIView
 
 
-logger = logging.getLogger('awx.main.analytics')
+logger = logging.getLogger('awx.analytics')
 
 
 class MetricsView(APIView):
@@ -30,12 +29,15 @@ class MetricsView(APIView):
     name = _('Metrics')
     swagger_topic = 'Metrics'
 
-    renderer_classes = [renderers.PlainTextRenderer,
-                        renderers.PrometheusJSONRenderer,
-                        renderers.BrowsableAPIRenderer,]
+    renderer_classes = [renderers.PlainTextRenderer, renderers.PrometheusJSONRenderer, renderers.BrowsableAPIRenderer]
 
     def get(self, request):
-        ''' Show Metrics Details '''
-        if (request.user.is_superuser or request.user.is_system_auditor):
-            return Response(metrics().decode('UTF-8'))
+        '''Show Metrics Details'''
+        if request.user.is_superuser or request.user.is_system_auditor:
+            metrics_to_show = ''
+            if not request.query_params.get('subsystemonly', "0") == "1":
+                metrics_to_show += metrics().decode('UTF-8')
+            if not request.query_params.get('dbonly', "0") == "1":
+                metrics_to_show += s_metrics.metrics(request)
+            return Response(metrics_to_show)
         raise PermissionDenied()

awx/api/views/mixin.py

@@ -4,10 +4,7 @@
 import dateutil
 import logging
 
-from django.db.models import (
-    Count,
-    F,
-)
+from django.db.models import Count
 from django.db import transaction
 from django.shortcuts import get_object_or_404
 from django.utils.timezone import now
@@ -19,14 +16,8 @@ from rest_framework.response import Response
 from rest_framework import status
 
 from awx.main.constants import ACTIVE_STATES
-from awx.main.utils import (
-    get_object_or_400,
-    parse_yaml_or_json,
-)
-from awx.main.models.ha import (
-    Instance,
-    InstanceGroup,
-)
+from awx.main.utils import get_object_or_400, parse_yaml_or_json
+from awx.main.models.ha import Instance, InstanceGroup
 from awx.main.models.organization import Team
 from awx.main.models.projects import Project
 from awx.main.models.inventory import Inventory
@@ -37,9 +28,10 @@ logger = logging.getLogger('awx.api.views.mixin')
 
 
 class UnifiedJobDeletionMixin(object):
-    '''
+    """
     Special handling when deleting a running unified job object.
-    '''
+    """
+
     def destroy(self, request, *args, **kwargs):
         obj = self.get_object()
         if not request.user.can_access(self.model, 'delete', obj):
@@ -56,29 +48,43 @@ class UnifiedJobDeletionMixin(object):
             # Prohibit deletion if job events are still coming in
             if obj.finished and now() < obj.finished + dateutil.relativedelta.relativedelta(minutes=1):
                 # less than 1 minute has passed since job finished and events are not in
-                return Response({"error": _("Job has not finished processing events.")},
-                                status=status.HTTP_400_BAD_REQUEST)
+                return Response({"error": _("Job has not finished processing events.")}, status=status.HTTP_400_BAD_REQUEST)
             else:
                 # if it has been > 1 minute, events are probably lost
-                logger.warning('Allowing deletion of {} through the API without all events '
-                               'processed.'.format(obj.log_format))
+                logger.warning('Allowing deletion of {} through the API without all events ' 'processed.'.format(obj.log_format))
+
+        # Manually cascade delete events if unpartitioned job
+        if obj.has_unpartitioned_events:
+            obj.get_event_queryset().delete()
+
         obj.delete()
         return Response(status=status.HTTP_204_NO_CONTENT)
 
 
 class InstanceGroupMembershipMixin(object):
-    '''
+    """
     This mixin overloads attach/detach so that it calls InstanceGroup.save(),
     triggering a background recalculation of policy-based instance group
     membership.
-    '''
+    """
+
+    def attach_validate(self, request):
+        parent = self.get_parent_object()
+        sub_id, res = super().attach_validate(request)
+        if res:  # handle an error
+            return sub_id, res
+        sub = get_object_or_400(self.model, pk=sub_id)
+        attach_errors = self.is_valid_relation(parent, sub)
+        if attach_errors:
+            return sub_id, Response(attach_errors, status=status.HTTP_400_BAD_REQUEST)
+        return sub_id, res
+
     def attach(self, request, *args, **kwargs):
         response = super(InstanceGroupMembershipMixin, self).attach(request, *args, **kwargs)
         sub_id, res = self.attach_validate(request)
         if status.is_success(response.status_code):
             if self.parent_model is Instance:
-                ig_obj = get_object_or_400(self.model, pk=sub_id)
-                inst_name = ig_obj.hostname
+                inst_name = self.get_parent_object().hostname
             else:
                 inst_name = get_object_or_400(self.model, pk=sub_id).hostname
             with transaction.atomic():
@@ -87,30 +93,20 @@ class InstanceGroupMembershipMixin(object):
                     ig_obj = get_object_or_400(ig_qs, pk=sub_id)
                 else:
                     # similar to get_parent_object, but selected for update
-                    parent_filter = {
-                        self.lookup_field: self.kwargs.get(self.lookup_field, None),
-                    }
+                    parent_filter = {self.lookup_field: self.kwargs.get(self.lookup_field, None)}
                     ig_obj = get_object_or_404(ig_qs, **parent_filter)
                 if inst_name not in ig_obj.policy_instance_list:
                     ig_obj.policy_instance_list.append(inst_name)
                     ig_obj.save(update_fields=['policy_instance_list'])
         return response
 
-    def is_valid_relation(self, parent, sub, created=False):
-        if sub.is_isolated():
-            return {'error': _('Isolated instances may not be added or removed from instances groups via the API.')}
-        if self.parent_model is InstanceGroup:
-            ig_obj = self.get_parent_object()
-            if ig_obj.controller_id is not None:
-                return {'error': _('Isolated instance group membership may not be managed via the API.')}
-        return None
-
     def unattach_validate(self, request):
+        parent = self.get_parent_object()
         (sub_id, res) = super(InstanceGroupMembershipMixin, self).unattach_validate(request)
         if res:
             return (sub_id, res)
         sub = get_object_or_400(self.model, pk=sub_id)
-        attach_errors = self.is_valid_relation(None, sub)
+        attach_errors = self.is_valid_relation(parent, sub)
         if attach_errors:
             return (sub_id, Response(attach_errors, status=status.HTTP_400_BAD_REQUEST))
         return (sub_id, res)
@@ -129,9 +125,7 @@ class InstanceGroupMembershipMixin(object):
                     ig_obj = get_object_or_400(ig_qs, pk=sub_id)
                 else:
                     # similar to get_parent_object, but selected for update
-                    parent_filter = {
-                        self.lookup_field: self.kwargs.get(self.lookup_field, None),
-                    }
+                    parent_filter = {self.lookup_field: self.kwargs.get(self.lookup_field, None)}
                     ig_obj = get_object_or_404(ig_qs, **parent_filter)
                 if inst_name in ig_obj.policy_instance_list:
                     ig_obj.policy_instance_list.pop(ig_obj.policy_instance_list.index(inst_name))
@@ -149,16 +143,13 @@ class RelatedJobsPreventDeleteMixin(object):
         if len(active_jobs) > 0:
             raise ActiveJobConflict(active_jobs)
         time_cutoff = now() - dateutil.relativedelta.relativedelta(minutes=1)
-        recent_jobs = obj._get_related_jobs().filter(finished__gte = time_cutoff)
+        recent_jobs = obj._get_related_jobs().filter(finished__gte=time_cutoff)
         for unified_job in recent_jobs.get_real_instances():
             if not unified_job.event_processing_finished:
-                raise PermissionDenied(_(
-                    'Related job {} is still processing events.'
-                ).format(unified_job.log_format))
+                raise PermissionDenied(_('Related job {} is still processing events.').format(unified_job.log_format))
 
 
 class OrganizationCountsMixin(object):
-
     def get_serializer_context(self, *args, **kwargs):
         full_context = super(OrganizationCountsMixin, self).get_serializer_context(*args, **kwargs)
 
@@ -175,48 +166,31 @@ class OrganizationCountsMixin(object):
 
         inv_qs = Inventory.accessible_objects(self.request.user, 'read_role')
         project_qs = Project.accessible_objects(self.request.user, 'read_role')
+        jt_qs = JobTemplate.accessible_objects(self.request.user, 'read_role')
 
         # Produce counts of Foreign Key relationships
-        db_results['inventories'] = inv_qs\
-            .values('organization').annotate(Count('organization')).order_by('organization')
+        db_results['inventories'] = inv_qs.values('organization').annotate(Count('organization')).order_by('organization')
 
-        db_results['teams'] = Team.accessible_objects(
-            self.request.user, 'read_role').values('organization').annotate(
-            Count('organization')).order_by('organization')
+        db_results['teams'] = (
+            Team.accessible_objects(self.request.user, 'read_role').values('organization').annotate(Count('organization')).order_by('organization')
+        )
 
-        JT_project_reference = 'project__organization'
-        JT_inventory_reference = 'inventory__organization'
-        db_results['job_templates_project'] = JobTemplate.accessible_objects(
-            self.request.user, 'read_role').exclude(
-            project__organization=F(JT_inventory_reference)).values(JT_project_reference).annotate(
-            Count(JT_project_reference)).order_by(JT_project_reference)
+        db_results['job_templates'] = jt_qs.values('organization').annotate(Count('organization')).order_by('organization')
 
-        db_results['job_templates_inventory'] = JobTemplate.accessible_objects(
-            self.request.user, 'read_role').values(JT_inventory_reference).annotate(
-            Count(JT_inventory_reference)).order_by(JT_inventory_reference)
-
-        db_results['projects'] = project_qs\
-            .values('organization').annotate(Count('organization')).order_by('organization')
+        db_results['projects'] = project_qs.values('organization').annotate(Count('organization')).order_by('organization')
 
         # Other members and admins of organization are always viewable
-        db_results['users'] = org_qs.annotate(
-            users=Count('member_role__members', distinct=True),
-            admins=Count('admin_role__members', distinct=True)
-        ).values('id', 'users', 'admins')
+        db_results['users'] = org_qs.annotate(users=Count('member_role__members', distinct=True), admins=Count('admin_role__members', distinct=True)).values(
+            'id', 'users', 'admins'
+        )
 
         count_context = {}
         for org in org_id_list:
             org_id = org['id']
-            count_context[org_id] = {
-                'inventories': 0, 'teams': 0, 'users': 0, 'job_templates': 0,
-                'admins': 0, 'projects': 0}
+            count_context[org_id] = {'inventories': 0, 'teams': 0, 'users': 0, 'job_templates': 0, 'admins': 0, 'projects': 0}
 
         for res, count_qs in db_results.items():
-            if res == 'job_templates_project':
-                org_reference = JT_project_reference
-            elif res == 'job_templates_inventory':
-                org_reference = JT_inventory_reference
-            elif res == 'users':
+            if res == 'users':
                 org_reference = 'id'
             else:
                 org_reference = 'organization'
@@ -229,35 +203,26 @@ class OrganizationCountsMixin(object):
                         continue
                     count_context[org_id][res] = entry['%s__count' % org_reference]
 
-        # Combine the counts for job templates by project and inventory
-        for org in org_id_list:
-            org_id = org['id']
-            count_context[org_id]['job_templates'] = 0
-            for related_path in ['job_templates_project', 'job_templates_inventory']:
-                if related_path in count_context[org_id]:
-                    count_context[org_id]['job_templates'] += count_context[org_id].pop(related_path)
-
         full_context['related_field_counts'] = count_context
 
         return full_context
 
 
 class ControlledByScmMixin(object):
-    '''
+    """
     Special method to reset SCM inventory commit hash
     if anything that it manages changes.
-    '''
+    """
 
     def _reset_inv_src_rev(self, obj):
         if self.request.method in SAFE_METHODS or not obj:
             return
-        project_following_sources = obj.inventory_sources.filter(
-            update_on_project_update=True, source='scm')
+        project_following_sources = obj.inventory_sources.filter(update_on_project_update=True, source='scm')
         if project_following_sources:
             # Allow inventory changes unrelated to variables
             if self.model == Inventory and (
-                    not self.request or not self.request.data or
-                    parse_yaml_or_json(self.request.data.get('variables', '')) == parse_yaml_or_json(obj.variables)):
+                not self.request or not self.request.data or parse_yaml_or_json(self.request.data.get('variables', '')) == parse_yaml_or_json(obj.variables)
+            ):
                 return
             project_following_sources.update(scm_last_revision='')
 

awx/api/views/organization.py

@@ -7,12 +7,15 @@ import logging
 # Django
 from django.db.models import Count
 from django.contrib.contenttypes.models import ContentType
+from django.utils.translation import ugettext_lazy as _
 
 # AWX
 from awx.main.models import (
     ActivityStream,
     Inventory,
+    Host,
     Project,
+    ExecutionEnvironment,
     JobTemplate,
     WorkflowJobTemplate,
     Organization,
@@ -21,6 +24,7 @@ from awx.main.models import (
     User,
     Team,
     InstanceGroup,
+    Credential,
 )
 from awx.api.generics import (
     ListCreateAPIView,
@@ -28,6 +32,7 @@ from awx.api.generics import (
     SubListAPIView,
     SubListCreateAttachDetachAPIView,
     SubListAttachDetachAPIView,
+    SubListCreateAPIView,
     ResourceAccessList,
     BaseUsersList,
 )
@@ -35,19 +40,19 @@ from awx.api.generics import (
 from awx.api.serializers import (
     OrganizationSerializer,
     InventorySerializer,
-    ProjectSerializer,
     UserSerializer,
     TeamSerializer,
     ActivityStreamSerializer,
     RoleSerializer,
     NotificationTemplateSerializer,
-    WorkflowJobTemplateSerializer,
     InstanceGroupSerializer,
+    ExecutionEnvironmentSerializer,
+    ProjectSerializer,
+    JobTemplateSerializer,
+    WorkflowJobTemplateSerializer,
+    CredentialSerializer,
 )
-from awx.api.views.mixin import (
-    RelatedJobsPreventDeleteMixin,
-    OrganizationCountsMixin,
-)
+from awx.api.views.mixin import RelatedJobsPreventDeleteMixin, OrganizationCountsMixin
 
 logger = logging.getLogger('awx.api.views.organization')
 
@@ -78,23 +83,21 @@ class OrganizationDetail(RelatedJobsPreventDeleteMixin, RetrieveUpdateDestroyAPI
 
         org_counts = {}
         access_kwargs = {'accessor': self.request.user, 'role_field': 'read_role'}
-        direct_counts = Organization.objects.filter(id=org_id).annotate(
-            users=Count('member_role__members', distinct=True),
-            admins=Count('admin_role__members', distinct=True)
-        ).values('users', 'admins')
+        direct_counts = (
+            Organization.objects.filter(id=org_id)
+            .annotate(users=Count('member_role__members', distinct=True), admins=Count('admin_role__members', distinct=True))
+            .values('users', 'admins')
+        )
 
         if not direct_counts:
             return full_context
 
         org_counts = direct_counts[0]
-        org_counts['inventories'] = Inventory.accessible_objects(**access_kwargs).filter(
-            organization__id=org_id).count()
-        org_counts['teams'] = Team.accessible_objects(**access_kwargs).filter(
-            organization__id=org_id).count()
-        org_counts['projects'] = Project.accessible_objects(**access_kwargs).filter(
-            organization__id=org_id).count()
-        org_counts['job_templates'] = JobTemplate.accessible_objects(**access_kwargs).filter(
-            project__organization__id=org_id).count()
+        org_counts['inventories'] = Inventory.accessible_objects(**access_kwargs).filter(organization__id=org_id).count()
+        org_counts['teams'] = Team.accessible_objects(**access_kwargs).filter(organization__id=org_id).count()
+        org_counts['projects'] = Project.accessible_objects(**access_kwargs).filter(organization__id=org_id).count()
+        org_counts['job_templates'] = JobTemplate.accessible_objects(**access_kwargs).filter(organization__id=org_id).count()
+        org_counts['hosts'] = Host.objects.org_active_count(org_id)
 
         full_context['related_field_counts'] = {}
         full_context['related_field_counts'][org_id] = org_counts
@@ -128,21 +131,37 @@ class OrganizationAdminsList(BaseUsersList):
     ordering = ('username',)
 
 
-class OrganizationProjectsList(SubListCreateAttachDetachAPIView):
+class OrganizationProjectsList(SubListCreateAPIView):
 
     model = Project
     serializer_class = ProjectSerializer
     parent_model = Organization
-    relationship = 'projects'
     parent_key = 'organization'
 
 
-class OrganizationWorkflowJobTemplatesList(SubListCreateAttachDetachAPIView):
+class OrganizationExecutionEnvironmentsList(SubListCreateAttachDetachAPIView):
+
+    model = ExecutionEnvironment
+    serializer_class = ExecutionEnvironmentSerializer
+    parent_model = Organization
+    relationship = 'executionenvironments'
+    parent_key = 'organization'
+    swagger_topic = "Execution Environments"
+
+
+class OrganizationJobTemplatesList(SubListCreateAPIView):
+
+    model = JobTemplate
+    serializer_class = JobTemplateSerializer
+    parent_model = Organization
+    parent_key = 'organization'
+
+
+class OrganizationWorkflowJobTemplatesList(SubListCreateAPIView):
 
     model = WorkflowJobTemplate
     serializer_class = WorkflowJobTemplateSerializer
     parent_model = Organization
-    relationship = 'workflows'
     parent_key = 'organization'
 
 
@@ -208,9 +227,21 @@ class OrganizationInstanceGroupsList(SubListAttachDetachAPIView):
     relationship = 'instance_groups'
 
 
+class OrganizationGalaxyCredentialsList(SubListAttachDetachAPIView):
+
+    model = Credential
+    serializer_class = CredentialSerializer
+    parent_model = Organization
+    relationship = 'galaxy_credentials'
+
+    def is_valid_relation(self, parent, sub, created=False):
+        if sub.kind != 'galaxy_api_token':
+            return {'msg': _(f"Credential must be a Galaxy credential, not {sub.credential_type.name}.")}
+
+
 class OrganizationAccessList(ResourceAccessList):
 
-    model = User # needs to be User for AccessLists's
+    model = User  # needs to be User for AccessLists's
     parent_model = Organization
 
 
@@ -219,7 +250,7 @@ class OrganizationObjectRolesList(SubListAPIView):
     model = Role
     serializer_class = RoleSerializer
     parent_model = Organization
-    search_fields = ('role_field', 'content_type__model',)
+    search_fields = ('role_field', 'content_type__model')
 
     def get_queryset(self):
         po = self.get_parent_object()

awx/api/views/root.py

@@ -1,9 +1,10 @@
 # Copyright (c) 2018 Ansible, Inc.
 # All Rights Reserved.
 
+import base64
+import json
 import logging
 import operator
-import json
 from collections import OrderedDict
 
 from django.conf import settings
@@ -20,23 +21,16 @@ from rest_framework import status
 import requests
 
 from awx.api.generics import APIView
+from awx.conf.registry import settings_registry
+from awx.main.analytics import all_collectors
 from awx.main.ha import is_ha_environment
-from awx.main.utils import (
-    get_awx_version,
-    get_ansible_version,
-    get_custom_venv_choices,
-    to_python_boolean,
-)
+from awx.main.utils import get_awx_version, get_custom_venv_choices
+from awx.main.utils.licensing import validate_entitlement_manifest
 from awx.api.versioning import reverse, drf_reverse
-from awx.conf.license import get_license
 from awx.main.constants import PRIVILEGE_ESCALATION_METHODS
-from awx.main.models import (
-    Project,
-    Organization,
-    Instance,
-    InstanceGroup,
-    JobTemplate,
-)
+from awx.main.models import Project, Organization, Instance, InstanceGroup, JobTemplate
+from awx.main.utils import set_environ
+from awx.main.utils.licensing import get_licenser
 
 logger = logging.getLogger('awx.api.views.root')
 
@@ -50,13 +44,13 @@ class ApiRootView(APIView):
 
     @method_decorator(ensure_csrf_cookie)
     def get(self, request, format=None):
-        ''' List supported API versions '''
+        '''List supported API versions'''
 
         v2 = reverse('api:api_v2_root_view', kwargs={'version': 'v2'})
         data = OrderedDict()
         data['description'] = _('AWX REST API')
         data['current_version'] = v2
-        data['available_versions'] = dict(v2 = v2)
+        data['available_versions'] = dict(v2=v2)
         data['oauth2'] = drf_reverse('api:oauth_authorization_root_view')
         data['custom_logo'] = settings.CUSTOM_LOGO
         data['custom_login_info'] = settings.CUSTOM_LOGIN_INFO
@@ -85,7 +79,7 @@ class ApiVersionRootView(APIView):
     swagger_topic = 'Versioning'
 
     def get(self, request, format=None):
-        ''' List top level resources '''
+        '''List top level resources'''
         data = OrderedDict()
         data['ping'] = reverse('api:api_v2_ping_view', request=request)
         data['instances'] = reverse('api:instance_list', request=request)
@@ -96,6 +90,7 @@ class ApiVersionRootView(APIView):
         data['dashboard'] = reverse('api:dashboard_view', request=request)
         data['organizations'] = reverse('api:organization_list', request=request)
         data['users'] = reverse('api:user_list', request=request)
+        data['execution_environments'] = reverse('api:execution_environment_list', request=request)
         data['projects'] = reverse('api:project_list', request=request)
         data['project_updates'] = reverse('api:project_update_list', request=request)
         data['teams'] = reverse('api:team_list', request=request)
@@ -106,14 +101,12 @@ class ApiVersionRootView(APIView):
         data['tokens'] = reverse('api:o_auth2_token_list', request=request)
         data['metrics'] = reverse('api:metrics_view', request=request)
         data['inventory'] = reverse('api:inventory_list', request=request)
-        data['inventory_scripts'] = reverse('api:inventory_script_list', request=request)
         data['inventory_sources'] = reverse('api:inventory_source_list', request=request)
         data['inventory_updates'] = reverse('api:inventory_update_list', request=request)
         data['groups'] = reverse('api:group_list', request=request)
         data['hosts'] = reverse('api:host_list', request=request)
         data['job_templates'] = reverse('api:job_template_list', request=request)
         data['jobs'] = reverse('api:job_list', request=request)
-        data['job_events'] = reverse('api:job_event_list', request=request)
         data['ad_hoc_commands'] = reverse('api:ad_hoc_command_list', request=request)
         data['system_job_templates'] = reverse('api:system_job_template_list', request=request)
         data['system_jobs'] = reverse('api:system_job_list', request=request)
@@ -141,6 +134,7 @@ class ApiV2PingView(APIView):
     """A simple view that reports very basic information about this
     instance, which is acceptable to be public information.
     """
+
     permission_classes = (AllowAny,)
     authentication_classes = ()
     name = _('Ping')
@@ -152,30 +146,34 @@ class ApiV2PingView(APIView):
         Everything returned here should be considered public / insecure, as
         this requires no auth and is intended for use by the installer process.
         """
-        response = {
-            'ha': is_ha_environment(),
-            'version': get_awx_version(),
-            'active_node': settings.CLUSTER_HOST_ID,
-            'install_uuid': settings.INSTALL_UUID,
-        }
+        response = {'ha': is_ha_environment(), 'version': get_awx_version(), 'active_node': settings.CLUSTER_HOST_ID, 'install_uuid': settings.INSTALL_UUID}
 
         response['instances'] = []
         for instance in Instance.objects.all():
-            response['instances'].append(dict(node=instance.hostname, uuid=instance.uuid, heartbeat=instance.modified,
-                                              capacity=instance.capacity, version=instance.version))
-            sorted(response['instances'], key=operator.itemgetter('node'))
+            response['instances'].append(
+                dict(
+                    node=instance.hostname,
+                    node_type=instance.node_type,
+                    uuid=instance.uuid,
+                    heartbeat=instance.modified,
+                    capacity=instance.capacity,
+                    version=instance.version,
+                )
+            )
+            response['instances'] = sorted(response['instances'], key=operator.itemgetter('node'))
         response['instance_groups'] = []
         for instance_group in InstanceGroup.objects.prefetch_related('instances'):
-            response['instance_groups'].append(dict(name=instance_group.name,
-                                                    capacity=instance_group.capacity,
-                                                    instances=[x.hostname for x in instance_group.instances.all()]))
+            response['instance_groups'].append(
+                dict(name=instance_group.name, capacity=instance_group.capacity, instances=[x.hostname for x in instance_group.instances.all()])
+            )
+            response['instance_groups'] = sorted(response['instance_groups'], key=lambda x: x['name'].lower())
         return Response(response)
 
 
 class ApiV2SubscriptionView(APIView):
 
     permission_classes = (IsAuthenticated,)
-    name = _('Configuration')
+    name = _('Subscriptions')
     swagger_topic = 'System Configuration'
 
     def check_permissions(self, request):
@@ -184,33 +182,80 @@ class ApiV2SubscriptionView(APIView):
             self.permission_denied(request)  # Raises PermissionDenied exception.
 
     def post(self, request):
-        from awx.main.utils.common import get_licenser
         data = request.data.copy()
-        if data.get('rh_password') == '$encrypted$':
-            data['rh_password'] = settings.REDHAT_PASSWORD
+        if data.get('subscriptions_password') == '$encrypted$':
+            data['subscriptions_password'] = settings.SUBSCRIPTIONS_PASSWORD
         try:
-            user, pw = data.get('rh_username'), data.get('rh_password')
-            validated = get_licenser().validate_rh(user, pw)
+            user, pw = data.get('subscriptions_username'), data.get('subscriptions_password')
+            with set_environ(**settings.AWX_TASK_ENV):
+                validated = get_licenser().validate_rh(user, pw)
             if user:
-                settings.REDHAT_USERNAME = data['rh_username']
+                settings.SUBSCRIPTIONS_USERNAME = data['subscriptions_username']
             if pw:
-                settings.REDHAT_PASSWORD = data['rh_password']
+                settings.SUBSCRIPTIONS_PASSWORD = data['subscriptions_password']
         except Exception as exc:
-            msg = _("Invalid License")
-            if (
-                isinstance(exc, requests.exceptions.HTTPError) and
-                getattr(getattr(exc, 'response', None), 'status_code', None) == 401
-            ):
+            msg = _("Invalid Subscription")
+            if isinstance(exc, requests.exceptions.HTTPError) and getattr(getattr(exc, 'response', None), 'status_code', None) == 401:
                 msg = _("The provided credentials are invalid (HTTP 401).")
-            if isinstance(exc, (ValueError, OSError)) and exc.args:
+            elif isinstance(exc, requests.exceptions.ProxyError):
+                msg = _("Unable to connect to proxy server.")
+            elif isinstance(exc, requests.exceptions.ConnectionError):
+                msg = _("Could not connect to subscription service.")
+            elif isinstance(exc, (ValueError, OSError)) and exc.args:
                 msg = exc.args[0]
-            logger.exception(smart_text(u"Invalid license submitted."),
-                             extra=dict(actor=request.user.username))
+            else:
+                logger.exception(smart_text(u"Invalid subscription submitted."), extra=dict(actor=request.user.username))
             return Response({"error": msg}, status=status.HTTP_400_BAD_REQUEST)
 
         return Response(validated)
 
 
+class ApiV2AttachView(APIView):
+
+    permission_classes = (IsAuthenticated,)
+    name = _('Attach Subscription')
+    swagger_topic = 'System Configuration'
+
+    def check_permissions(self, request):
+        super(ApiV2AttachView, self).check_permissions(request)
+        if not request.user.is_superuser and request.method.lower() not in {'options', 'head'}:
+            self.permission_denied(request)  # Raises PermissionDenied exception.
+
+    def post(self, request):
+        data = request.data.copy()
+        pool_id = data.get('pool_id', None)
+        if not pool_id:
+            return Response({"error": _("No subscription pool ID provided.")}, status=status.HTTP_400_BAD_REQUEST)
+        user = getattr(settings, 'SUBSCRIPTIONS_USERNAME', None)
+        pw = getattr(settings, 'SUBSCRIPTIONS_PASSWORD', None)
+        if pool_id and user and pw:
+
+            data = request.data.copy()
+            try:
+                with set_environ(**settings.AWX_TASK_ENV):
+                    validated = get_licenser().validate_rh(user, pw)
+            except Exception as exc:
+                msg = _("Invalid Subscription")
+                if isinstance(exc, requests.exceptions.HTTPError) and getattr(getattr(exc, 'response', None), 'status_code', None) == 401:
+                    msg = _("The provided credentials are invalid (HTTP 401).")
+                elif isinstance(exc, requests.exceptions.ProxyError):
+                    msg = _("Unable to connect to proxy server.")
+                elif isinstance(exc, requests.exceptions.ConnectionError):
+                    msg = _("Could not connect to subscription service.")
+                elif isinstance(exc, (ValueError, OSError)) and exc.args:
+                    msg = exc.args[0]
+                else:
+                    logger.exception(smart_text(u"Invalid subscription submitted."), extra=dict(actor=request.user.username))
+                return Response({"error": msg}, status=status.HTTP_400_BAD_REQUEST)
+        for sub in validated:
+            if sub['pool_id'] == pool_id:
+                sub['valid_key'] = True
+                settings.LICENSE = sub
+                return Response(sub)
+
+        return Response({"error": _("Error processing subscription metadata.")}, status=status.HTTP_400_BAD_REQUEST)
+
+
 class ApiV2ConfigView(APIView):
 
     permission_classes = (IsAuthenticated,)
@@ -225,15 +270,10 @@ class ApiV2ConfigView(APIView):
     def get(self, request, format=None):
         '''Return various sitewide configuration settings'''
 
-        if request.user.is_superuser or request.user.is_system_auditor:
-            license_data = get_license(show_key=True)
-        else:
-            license_data = get_license(show_key=False)
+        license_data = get_licenser().validate()
+
         if not license_data.get('valid_key', False):
             license_data = {}
-        if license_data and 'features' in license_data and 'activity_streams' in license_data['features']:
-            # FIXME: Make the final setting value dependent on the feature?
-            license_data['features']['activity_streams'] &= settings.ACTIVITY_STREAM_ENABLED
 
         pendo_state = settings.PENDO_TRACKING_STATE if settings.PENDO_TRACKING_STATE in ('off', 'anonymous', 'detailed') else 'off'
 
@@ -241,9 +281,9 @@ class ApiV2ConfigView(APIView):
             time_zone=settings.TIME_ZONE,
             license_info=license_data,
             version=get_awx_version(),
-            ansible_version=get_ansible_version(),
             eula=render_to_string("eula.md") if license_data.get('license_type', 'UNLICENSED') != 'open' else '',
             analytics_status=pendo_state,
+            analytics_collectors=all_collectors(),
             become_methods=PRIVILEGE_ESCALATION_METHODS,
         )
 
@@ -256,16 +296,20 @@ class ApiV2ConfigView(APIView):
             user_ldap_fields.extend(getattr(settings, 'AUTH_LDAP_USER_FLAGS_BY_GROUP', {}).keys())
             data['user_ldap_fields'] = user_ldap_fields
 
-        if request.user.is_superuser \
-                or request.user.is_system_auditor \
-                or Organization.accessible_objects(request.user, 'admin_role').exists() \
-                or Organization.accessible_objects(request.user, 'auditor_role').exists() \
-                or Organization.accessible_objects(request.user, 'project_admin_role').exists():
-            data.update(dict(
-                project_base_dir = settings.PROJECTS_ROOT,
-                project_local_paths = Project.get_local_path_choices(),
-                custom_virtualenvs = get_custom_venv_choices()
-            ))
+        if (
+            request.user.is_superuser
+            or request.user.is_system_auditor
+            or Organization.accessible_objects(request.user, 'admin_role').exists()
+            or Organization.accessible_objects(request.user, 'auditor_role').exists()
+            or Organization.accessible_objects(request.user, 'project_admin_role').exists()
+        ):
+            data.update(
+                dict(
+                    project_base_dir=settings.PROJECTS_ROOT,
+                    project_local_paths=Project.get_local_path_choices(),
+                    custom_virtualenvs=get_custom_venv_choices(),
+                )
+            )
         elif JobTemplate.accessible_objects(request.user, 'admin_role').exists():
             data['custom_virtualenvs'] = get_custom_venv_choices()
 
@@ -273,41 +317,47 @@ class ApiV2ConfigView(APIView):
 
     def post(self, request):
         if not isinstance(request.data, dict):
-            return Response({"error": _("Invalid license data")}, status=status.HTTP_400_BAD_REQUEST)
-        if "eula_accepted" not in request.data:
-            return Response({"error": _("Missing 'eula_accepted' property")}, status=status.HTTP_400_BAD_REQUEST)
-        try:
-            eula_accepted = to_python_boolean(request.data["eula_accepted"])
-        except ValueError:
-            return Response({"error": _("'eula_accepted' value is invalid")}, status=status.HTTP_400_BAD_REQUEST)
-
-        if not eula_accepted:
-            return Response({"error": _("'eula_accepted' must be True")}, status=status.HTTP_400_BAD_REQUEST)
-        request.data.pop("eula_accepted")
+            return Response({"error": _("Invalid subscription data")}, status=status.HTTP_400_BAD_REQUEST)
         try:
             data_actual = json.dumps(request.data)
         except Exception:
-            logger.info(smart_text(u"Invalid JSON submitted for license."),
-                        extra=dict(actor=request.user.username))
+            logger.info(smart_text(u"Invalid JSON submitted for license."), extra=dict(actor=request.user.username))
             return Response({"error": _("Invalid JSON")}, status=status.HTTP_400_BAD_REQUEST)
-        try:
-            from awx.main.utils.common import get_licenser
-            license_data = json.loads(data_actual)
-            license_data_validated = get_licenser(**license_data).validate()
-        except Exception:
-            logger.warning(smart_text(u"Invalid license submitted."),
-                           extra=dict(actor=request.user.username))
-            return Response({"error": _("Invalid License")}, status=status.HTTP_400_BAD_REQUEST)
+
+        license_data = json.loads(data_actual)
+        if 'license_key' in license_data:
+            return Response({"error": _('Legacy license submitted. A subscription manifest is now required.')}, status=status.HTTP_400_BAD_REQUEST)
+        if 'manifest' in license_data:
+            try:
+                json_actual = json.loads(base64.b64decode(license_data['manifest']))
+                if 'license_key' in json_actual:
+                    return Response({"error": _('Legacy license submitted. A subscription manifest is now required.')}, status=status.HTTP_400_BAD_REQUEST)
+            except Exception:
+                pass
+            try:
+                license_data = validate_entitlement_manifest(license_data['manifest'])
+            except ValueError as e:
+                return Response({"error": str(e)}, status=status.HTTP_400_BAD_REQUEST)
+            except Exception:
+                logger.exception('Invalid manifest submitted. {}')
+                return Response({"error": _('Invalid manifest submitted.')}, status=status.HTTP_400_BAD_REQUEST)
+
+            try:
+                license_data_validated = get_licenser().license_from_manifest(license_data)
+            except Exception:
+                logger.warning(smart_text(u"Invalid subscription submitted."), extra=dict(actor=request.user.username))
+                return Response({"error": _("Invalid License")}, status=status.HTTP_400_BAD_REQUEST)
+        else:
+            license_data_validated = get_licenser().validate()
 
         # If the license is valid, write it to the database.
         if license_data_validated['valid_key']:
-            settings.LICENSE = license_data
-            settings.TOWER_URL_BASE = "{}://{}".format(request.scheme, request.get_host())
+            if not settings_registry.is_setting_read_only('TOWER_URL_BASE'):
+                settings.TOWER_URL_BASE = "{}://{}".format(request.scheme, request.get_host())
             return Response(license_data_validated)
 
-        logger.warning(smart_text(u"Invalid license submitted."),
-                       extra=dict(actor=request.user.username))
-        return Response({"error": _("Invalid license")}, status=status.HTTP_400_BAD_REQUEST)
+        logger.warning(smart_text(u"Invalid subscription submitted."), extra=dict(actor=request.user.username))
+        return Response({"error": _("Invalid subscription")}, status=status.HTTP_400_BAD_REQUEST)
 
     def delete(self, request):
         try:

awx/api/views/webhooks.py

@@ -26,10 +26,7 @@ class WebhookKeyView(GenericAPIView):
     permission_classes = (WebhookKeyPermission,)
 
     def get_queryset(self):
-        qs_models = {
-            'job_templates': JobTemplate,
-            'workflow_job_templates': WorkflowJobTemplate,
-        }
+        qs_models = {'job_templates': JobTemplate, 'workflow_job_templates': WorkflowJobTemplate}
         self.model = qs_models.get(self.kwargs['model_kwarg'])
 
         return super().get_queryset()
@@ -57,10 +54,7 @@ class WebhookReceiverBase(APIView):
     ref_keys = {}
 
     def get_queryset(self):
-        qs_models = {
-            'job_templates': JobTemplate,
-            'workflow_job_templates': WorkflowJobTemplate,
-        }
+        qs_models = {'job_templates': JobTemplate, 'workflow_job_templates': WorkflowJobTemplate}
         model = qs_models.get(self.kwargs['model_kwarg'])
         if model is None:
             raise PermissionDenied
@@ -120,10 +114,7 @@ class WebhookReceiverBase(APIView):
         # Ensure that the full contents of the request are captured for multiple uses.
         request.body
 
-        logger.debug(
-            "headers: {}\n"
-            "data: {}\n".format(request.headers, request.data)
-        )
+        logger.debug("headers: {}\n" "data: {}\n".format(request.headers, request.data))
         obj = self.get_object()
         self.check_signature(obj)
 
@@ -132,16 +123,11 @@ class WebhookReceiverBase(APIView):
         event_ref = self.get_event_ref()
         status_api = self.get_event_status_api()
 
-        kwargs = {
-            'unified_job_template_id': obj.id,
-            'webhook_service': obj.webhook_service,
-            'webhook_guid': event_guid,
-        }
+        kwargs = {'unified_job_template_id': obj.id, 'webhook_service': obj.webhook_service, 'webhook_guid': event_guid}
         if WorkflowJob.objects.filter(**kwargs).exists() or Job.objects.filter(**kwargs).exists():
             # Short circuit if this webhook has already been received and acted upon.
             logger.debug("Webhook previously received, returning without action.")
-            return Response({'message': _("Webhook previously received, aborting.")},
-                            status=status.HTTP_202_ACCEPTED)
+            return Response({'message': _("Webhook previously received, aborting.")}, status=status.HTTP_202_ACCEPTED)
 
         kwargs = {
             '_eager_fields': {
@@ -156,7 +142,7 @@ class WebhookReceiverBase(APIView):
                 'tower_webhook_event_ref': event_ref,
                 'tower_webhook_status_api': status_api,
                 'tower_webhook_payload': request.data,
-            }
+            },
         }
 
         new_job = obj.create_unified_job(**kwargs)
@@ -205,11 +191,7 @@ class GithubWebhookReceiver(WebhookReceiverBase):
 class GitlabWebhookReceiver(WebhookReceiverBase):
     service = 'gitlab'
 
-    ref_keys = {
-        'Push Hook': 'checkout_sha',
-        'Tag Push Hook': 'checkout_sha',
-        'Merge Request Hook': 'object_attributes.last_commit.id',
-    }
+    ref_keys = {'Push Hook': 'checkout_sha', 'Tag Push Hook': 'checkout_sha', 'Merge Request Hook': 'object_attributes.last_commit.id'}
 
     def get_event_type(self):
         return self.request.META.get('HTTP_X_GITLAB_EVENT')
@@ -229,8 +211,7 @@ class GitlabWebhookReceiver(WebhookReceiverBase):
             return
         parsed = urllib.parse.urlparse(repo_url)
 
-        return "{}://{}/api/v4/projects/{}/statuses/{}".format(
-            parsed.scheme, parsed.netloc, project['id'], self.get_event_ref())
+        return "{}://{}/api/v4/projects/{}/statuses/{}".format(parsed.scheme, parsed.netloc, project['id'], self.get_event_ref())
 
     def get_signature(self):
         return force_bytes(self.request.META.get('HTTP_X_GITLAB_TOKEN') or '')

awx/asgi.py

@@ -2,14 +2,15 @@
 # All Rights Reserved.
 import os
 import logging
+import django
 from awx import __version__ as tower_version
 
 # Prepare the AWX environment.
 from awx import prepare_env, MODE
-prepare_env() # NOQA
+from channels.routing import get_default_application  # noqa
+
+prepare_env()  # NOQA
 
-from django.core.wsgi import get_wsgi_application  # NOQA
-from channels.asgi import get_channel_layer
 
 """
 ASGI config for AWX project.
@@ -25,13 +26,14 @@ if MODE == 'production':
     try:
         fd = open("/var/lib/awx/.tower_version", "r")
         if fd.read().strip() != tower_version:
-            raise Exception()
-    except Exception:
-        logger.error("Missing or incorrect metadata for Tower version.  Ensure Tower was installed using the setup playbook.")
-        raise Exception("Missing or incorrect metadata for Tower version.  Ensure Tower was installed using the setup playbook.")
+            raise ValueError()
+    except FileNotFoundError:
+        pass
+    except ValueError as e:
+        logger.error("Missing or incorrect metadata for controller version.  Ensure controller was installed using the setup playbook.")
+        raise Exception("Missing or incorrect metadata for controller version.  Ensure controller was installed using the setup playbook.") from e
 
 
 os.environ.setdefault("DJANGO_SETTINGS_MODULE", "awx.settings")
-
-
-channel_layer = get_channel_layer()
+django.setup()
+channel_layer = get_default_application()

awx/conf/__init__.py

@@ -4,7 +4,7 @@
 # Django
 from django.utils.module_loading import autodiscover_modules
 
-# Tower
+# AWX
 from .registry import settings_registry
 
 default_app_config = 'awx.conf.apps.ConfConfig'

awx/conf/access.py

@@ -4,18 +4,18 @@
 # Django
 from django.db.models import Q
 
-# Tower
+# AWX
 from awx.main.access import BaseAccess, register_access
 from awx.conf.models import Setting
 
 
 class SettingAccess(BaseAccess):
-    '''
+    """
     - I can see settings when I am a super user or system auditor.
     - I can edit settings when I am a super user.
     - I can clear settings when I am a super user.
     - I can always see/edit/clear my own user settings.
-    '''
+    """
 
     model = Setting
 

awx/conf/apps.py

@@ -1,5 +1,6 @@
 # Django
 from django.apps import AppConfig
+
 # from django.core import checks
 from django.utils.translation import ugettext_lazy as _
 
@@ -12,4 +13,5 @@ class ConfConfig(AppConfig):
     def ready(self):
         self.module.autodiscover()
         from .settings import SettingsWrapper
+
         SettingsWrapper.initialize()

awx/conf/conf.py

@@ -2,7 +2,7 @@
 from django.conf import settings
 from django.utils.translation import ugettext_lazy as _
 
-# Tower
+# AWX
 from awx.conf import fields, register
 from awx.conf import settings_registry
 

awx/conf/fields.py

@@ -10,10 +10,8 @@ from django.core.validators import URLValidator, _lazy_re_compile
 from django.utils.translation import ugettext_lazy as _
 
 # Django REST Framework
-from rest_framework.fields import (  # noqa
-    BooleanField, CharField, ChoiceField, DictField, EmailField,
-    IntegerField, ListField, NullBooleanField
-)
+from rest_framework.fields import BooleanField, CharField, ChoiceField, DictField, DateTimeField, EmailField, IntegerField, ListField, NullBooleanField  # noqa
+from rest_framework.serializers import PrimaryKeyRelatedField  # noqa
 
 logger = logging.getLogger('awx.conf.fields')
 
@@ -26,7 +24,6 @@ logger = logging.getLogger('awx.conf.fields')
 
 
 class CharField(CharField):
-
     def to_representation(self, value):
         # django_rest_frameworks' default CharField implementation casts `None`
         # to a string `"None"`:
@@ -38,7 +35,6 @@ class CharField(CharField):
 
 
 class IntegerField(IntegerField):
-
     def get_value(self, dictionary):
         ret = super(IntegerField, self).get_value(dictionary)
         # Handle UI corner case
@@ -59,9 +55,7 @@ class StringListField(ListField):
 
 class StringListBooleanField(ListField):
 
-    default_error_messages = {
-        'type_error': _('Expected None, True, False, a string or list of strings but got {input_type} instead.'),
-    }
+    default_error_messages = {'type_error': _('Expected None, True, False, a string or list of strings but got {input_type} instead.')}
     child = CharField()
 
     def to_representation(self, value):
@@ -100,10 +94,7 @@ class StringListBooleanField(ListField):
 
 class StringListPathField(StringListField):
 
-    default_error_messages = {
-        'type_error': _('Expected list of strings but got {input_type} instead.'),
-        'path_error': _('{path} is not a valid path choice.'),
-    }
+    default_error_messages = {'type_error': _('Expected list of strings but got {input_type} instead.'), 'path_error': _('{path} is not a valid path choice.')}
 
     def to_internal_value(self, paths):
         if isinstance(paths, (list, tuple)):
@@ -122,12 +113,12 @@ class URLField(CharField):
     # these lines set up a custom regex that allow numbers in the
     # top-level domain
     tld_re = (
-        r'\.'                                              # dot
-        r'(?!-)'                                           # can't start with a dash
-        r'(?:[a-z' + URLValidator.ul + r'0-9' + '-]{2,63}' # domain label, this line was changed from the original URLValidator
-        r'|xn--[a-z0-9]{1,59})'                            # or punycode label
-        r'(?<!-)'                                          # can't end with a dash
-        r'\.?'                                             # may have a trailing dot
+        r'\.'  # dot
+        r'(?!-)'  # can't start with a dash
+        r'(?:[a-z' + URLValidator.ul + r'0-9' + '-]{2,63}'  # domain label, this line was changed from the original URLValidator
+        r'|xn--[a-z0-9]{1,59})'  # or punycode label
+        r'(?<!-)'  # can't end with a dash
+        r'\.?'  # may have a trailing dot
     )
 
     host_re = '(' + URLValidator.hostname_re + URLValidator.domain_re + tld_re + '|localhost)'
@@ -138,7 +129,9 @@ class URLField(CharField):
         r'(?:' + URLValidator.ipv4_re + '|' + URLValidator.ipv6_re + '|' + host_re + ')'
         r'(?::\d{2,5})?'  # port
         r'(?:[/?#][^\s]*)?'  # resource path
-        r'\Z', re.IGNORECASE)
+        r'\Z',
+        re.IGNORECASE,
+    )
 
     def __init__(self, **kwargs):
         schemes = kwargs.pop('schemes', None)
@@ -172,9 +165,9 @@ class URLField(CharField):
                         netloc = '{}:{}'.format(netloc, url_parts.port)
                     if url_parts.username:
                         if url_parts.password:
-                            netloc = '{}:{}@{}' % (url_parts.username, url_parts.password, netloc)
+                            netloc = '{}:{}@{}'.format(url_parts.username, url_parts.password, netloc)
                         else:
-                            netloc = '{}@{}' % (url_parts.username, netloc)
+                            netloc = '{}@{}'.format(url_parts.username, netloc)
                     value = urlparse.urlunsplit([url_parts.scheme, netloc, url_parts.path, url_parts.query, url_parts.fragment])
             except Exception:
                 raise  # If something fails here, just fall through and let the validators check it.
@@ -183,9 +176,7 @@ class URLField(CharField):
 
 class KeyValueField(DictField):
     child = CharField()
-    default_error_messages = {
-        'invalid_child': _('"{input}" is not a valid string.')
-    }
+    default_error_messages = {'invalid_child': _('"{input}" is not a valid string.')}
 
     def to_internal_value(self, data):
         ret = super(KeyValueField, self).to_internal_value(data)
@@ -198,9 +189,7 @@ class KeyValueField(DictField):
 
 
 class ListTuplesField(ListField):
-    default_error_messages = {
-        'type_error': _('Expected a list of tuples of max length 2 but got {input_type} instead.'),
-    }
+    default_error_messages = {'type_error': _('Expected a list of tuples of max length 2 but got {input_type} instead.')}
 
     def to_representation(self, value):
         if isinstance(value, (list, tuple)):

awx/conf/license.py

@@ -1,18 +1,15 @@
 # Copyright (c) 2016 Ansible, Inc.
 # All Rights Reserved.
 
-
 __all__ = ['get_license']
 
 
 def _get_validated_license_data():
-    from awx.main.utils.common import get_licenser
+    from awx.main.utils import get_licenser
+
     return get_licenser().validate()
 
 
-def get_license(show_key=False):
-    """Return a dictionary representing the active license on this Tower instance."""
-    license_data = _get_validated_license_data()
-    if not show_key:
-        license_data.pop('license_key', None)
-    return license_data
+def get_license():
+    """Return a dictionary representing the active license on this instance."""
+    return _get_validated_license_data()

awx/conf/migrations/0001_initial.py

@@ -8,9 +8,7 @@ from django.conf import settings
 
 class Migration(migrations.Migration):
 
-    dependencies = [
-        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
-    ]
+    dependencies = [migrations.swappable_dependency(settings.AUTH_USER_MODEL)]
 
     operations = [
         migrations.CreateModel(
@@ -21,11 +19,11 @@ class Migration(migrations.Migration):
                 ('modified', models.DateTimeField(default=None, editable=False)),
                 ('key', models.CharField(max_length=255)),
                 ('value', jsonfield.fields.JSONField(null=True)),
-                ('user', models.ForeignKey(related_name='settings', default=None, editable=False,
-                                           to=settings.AUTH_USER_MODEL, on_delete=models.CASCADE, null=True)),
+                (
+                    'user',
+                    models.ForeignKey(related_name='settings', default=None, editable=False, to=settings.AUTH_USER_MODEL, on_delete=models.CASCADE, null=True),
+                ),
             ],
-            options={
-                'abstract': False,
-            },
-        ),
+            options={'abstract': False},
+        )
     ]

awx/conf/migrations/0002_v310_copy_tower_settings.py

@@ -15,11 +15,7 @@ def copy_tower_settings(apps, schema_editor):
             if tower_setting.key == 'LICENSE':
                 value = json.loads(value)
             setting, created = Setting.objects.get_or_create(
-                key=tower_setting.key,
-                user=tower_setting.user,
-                created=tower_setting.created,
-                modified=tower_setting.modified,
-                defaults=dict(value=value),
+                key=tower_setting.key, user=tower_setting.user, created=tower_setting.created, modified=tower_setting.modified, defaults=dict(value=value)
             )
             if not created and setting.value != value:
                 setting.value = value
@@ -36,18 +32,9 @@ def revert_tower_settings(apps, schema_editor):
         # LICENSE is stored as a JSON object; convert it back to a string.
         if setting.key == 'LICENSE':
             value = json.dumps(value)
-        defaults = dict(
-            value=value,
-            value_type='string',
-            description='',
-            category='',
-        )
+        defaults = dict(value=value, value_type='string', description='', category='')
         try:
-            tower_setting, created = TowerSettings.objects.get_or_create(
-                key=setting.key,
-                user=setting.user,
-                defaults=defaults,
-            )
+            tower_setting, created = TowerSettings.objects.get_or_create(key=setting.key, user=setting.user, defaults=defaults)
             if not created:
                 update_fields = []
                 for k, v in defaults.items():
@@ -62,15 +49,8 @@ def revert_tower_settings(apps, schema_editor):
 
 class Migration(migrations.Migration):
 
-    dependencies = [
-        ('conf', '0001_initial'),
-        ('main', '0004_squashed_v310_release'),
-    ]
+    dependencies = [('conf', '0001_initial'), ('main', '0004_squashed_v310_release')]
 
-    run_before = [
-        ('main', '0005_squashed_v310_v313_updates'),
-    ]
+    run_before = [('main', '0005_squashed_v310_v313_updates')]
 
-    operations = [
-        migrations.RunPython(copy_tower_settings, revert_tower_settings),
-    ]
+    operations = [migrations.RunPython(copy_tower_settings, revert_tower_settings)]

awx/conf/migrations/0003_v310_JSONField_changes.py

@@ -7,14 +7,6 @@ import awx.main.fields
 
 class Migration(migrations.Migration):
 
-    dependencies = [
-        ('conf', '0002_v310_copy_tower_settings'),
-    ]
+    dependencies = [('conf', '0002_v310_copy_tower_settings')]
 
-    operations = [
-        migrations.AlterField(
-            model_name='setting',
-            name='value',
-            field=awx.main.fields.JSONField(null=True),
-        ),
-    ]
+    operations = [migrations.AlterField(model_name='setting', name='value', field=awx.main.fields.JSONField(null=True))]

awx/conf/migrations/0004_v320_reencrypt.py

@@ -6,9 +6,7 @@ from django.db import migrations
 
 class Migration(migrations.Migration):
 
-    dependencies = [
-        ('conf', '0003_v310_JSONField_changes'),
-    ]
+    dependencies = [('conf', '0003_v310_JSONField_changes')]
 
     operations = [
         # This list is intentionally empty.

awx/conf/migrations/0005_v330_rename_two_session_settings.py

@@ -2,8 +2,8 @@
 from __future__ import unicode_literals
 from django.db import migrations
 from awx.conf.migrations import _rename_setting
-    
-    
+
+
 def copy_session_settings(apps, schema_editor):
     _rename_setting.rename_setting(apps, schema_editor, old_key='AUTH_TOKEN_PER_USER', new_key='SESSIONS_PER_USER')
     _rename_setting.rename_setting(apps, schema_editor, old_key='AUTH_TOKEN_EXPIRATION', new_key='SESSION_COOKIE_AGE')
@@ -16,11 +16,6 @@ def reverse_copy_session_settings(apps, schema_editor):
 
 class Migration(migrations.Migration):
 
-    dependencies = [
-        ('conf', '0004_v320_reencrypt'),
-    ]
+    dependencies = [('conf', '0004_v320_reencrypt')]
 
-    operations = [
-        migrations.RunPython(copy_session_settings, reverse_copy_session_settings),
-    ]
-    
+    operations = [migrations.RunPython(copy_session_settings, reverse_copy_session_settings)]