Merge pull request #7420 from elyezer/update-credential-creation

Update awxkit credential creation Reviewed-by: https://github.com/apps/softwarefactory-project-zuul
Merge pull request #7416 from jakemcdermott/fix-cred-user-team-org-save
2026-02-06 12:04:44 -03:30 · 2020-06-23 18:31:50 +00:00 · 2020-06-23 16:49:21 +00:00 · 2020-06-23 12:32:51 -04:00 · 2020-06-23 16:01:26 +00:00 · 2020-06-23 15:55:57 +00:00
1582 changed files with 145612 additions and 91795 deletions
--- a/.github/ISSUE_TEMPLATE.md
+++ b/.github/ISSUE_TEMPLATE.md
@@ -30,8 +30,9 @@ https://www.ansible.com/security

 ##### STEPS TO REPRODUCE

-<!-- For bugs, please show exactly how to reproduce the problem. For new
-features, show how the feature would be used.  -->
+<!-- For new features, show how the feature would be used. For bugs, please show
+exactly how to reproduce the problem. Ideally, provide all steps and data needed
+to recreate the bug from a new awx install. -->

 ##### EXPECTED RESULTS

--- a/.gitignore
+++ b/.gitignore
@@ -31,9 +31,11 @@ awx/ui/templates/ui/installing.html
 awx/ui_next/node_modules/
 awx/ui_next/coverage/
 awx/ui_next/build/locales/_build
+rsyslog.pid
 /tower-license
 /tower-license/**
 tools/prometheus/data
+tools/docker-compose/Dockerfile

 # Tower setup playbook testing
 setup/test/roles/postgresql
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,119 @@

 This is a list of high-level changes for each release of AWX. A full list of commits can be found at `https://github.com/ansible/awx/releases/tag/<version>`.

+## 13.0.0 (Jun 23, 2020)
+- Added import and export subcommands to the awx-cli tool, replacing send and receive from the old tower-cli (https://github.com/ansible/awx/pull/6125).
+- Removed scripts as a means of running inventory updates of built-in types (https://github.com/ansible/awx/pull/6911)
+- Ansible 2.8 is now partially unsupported; some inventory source types are known to no longer work.
+- Fixed an issue where the vmware inventory source ssl_verify source variable was not recognized (https://github.com/ansible/awx/pull/7360)
+- Fixed a bug that caused redis' listen socket to have too-permissive file permissions (https://github.com/ansible/awx/pull/7317)
+- Fixed a bug that caused rsyslogd's configuration file to have world-readable file permissions, potentially leaking secrets (CVE-2020-10782)
+
+## 12.0.0 (Jun 9, 2020)
+- Removed memcached as a dependency of AWX (https://github.com/ansible/awx/pull/7240) 
+- Moved to a single container image build instead of separate awx_web and awx_task images. The container image is just `awx` (https://github.com/ansible/awx/pull/7228)
+- Official AWX container image builds now use a two-stage container build process that notably reduces the size of our published images (https://github.com/ansible/awx/pull/7017)
+- Removed support for HipChat notifications ([EoL announcement](https://www.atlassian.com/partnerships/slack/faq#faq-98b17ca3-247f-423b-9a78-70a91681eff0)); all previously-created HipChat notification templates will be deleted due to this removal.
+- Fixed a bug which broke AWX installations with oc version 4.3 (https://github.com/ansible/awx/pull/6948/files)
+- Fixed a performance issue that caused notable delay of stdout processing for playbooks run against large numbers of hosts (https://github.com/ansible/awx/issues/6991)
+- Fixed a bug that caused CyberArk AIM credential plugin looks to hang forever in some environments (https://github.com/ansible/awx/issues/6986)
+- Fixed a bug that caused ANY/ALL converage settings not to properly save when editing approval nodes in the UI (https://github.com/ansible/awx/issues/6998)
+- Fixed a bug that broke support for the satellite6_group_prefix source variable (https://github.com/ansible/awx/issues/7031)
+- Fixed a bug that prevented changes to workflow node convergence settings when approval nodes were in use (https://github.com/ansible/awx/issues/7063)
+- Fixed a bug that caused notifications to fail on newer version of Mattermost (https://github.com/ansible/awx/issues/7264)
+- Fixed a bug (by upgrading to 0.8.1 of the foreman collection) that prevented host_filters from working properly with Foreman-based inventory (https://github.com/ansible/awx/issues/7225)
+- Fixed a bug that prevented the usage of the Conjur credential plugin with secrets that contain spaces (https://github.com/ansible/awx/issues/7191)
+- Fixed a bug in awx-manage run_wsbroadcast --status in kubernetes (https://github.com/ansible/awx/pull/7009)
+- Fixed a bug that broke notification toggles for system jobs in the UI (https://github.com/ansible/awx/pull/7042)
+- Fixed a bug that broke local pip installs of awxkit (https://github.com/ansible/awx/issues/7107)
+- Fixed a bug that prevented PagerDuty notifications from sending for workflow job template approvals (https://github.com/ansible/awx/issues/7094)
+- Fixed a bug that broke external log aggregation support for URL paths that include the = character (such as the tokens for SumoLogic) (https://github.com/ansible/awx/issues/7139)
+- Fixed a bug that prevented organization admins from removing labels from workflow job templates (https://github.com/ansible/awx/pull/7143)
+
+## 11.2.0 (Apr 29, 2020)
+
+- Inventory updates now use collection-based plugins by default (in Ansible 2.9+):
+    - amazon.aws.aws_ec2
+    - community.vmware.vmware_vm_inventory
+    - azure.azcollection.azure_rm
+    - google.cloud.gcp_compute
+    - theforeman.foreman.foreman
+    - openstack.cloud.openstack
+    - ovirt.ovirt_collection.ovirt
+    - awx.awx.tower
+- Added support for Approle and LDAP/AD mechanisms to the Hashicorp Vault credential plugin (https://github.com/ansible/awx/issues/5076)
+- Added Project (Domain Name) support for the OpenStack Keystone v3 API (https://github.com/ansible/awx/issues/6831)
+- Added a new setting for raising log verbosity for rsyslogd (https://github.com/ansible/awx/pull/6818)
+- Added the ability to monitor stdout in the CLI for running jobs and workflow jobs (https://github.com/ansible/awx/issues/6165)
+- Fixed a bug which prevented the AWX CLI from properly installing with newer versions of pip (https://github.com/ansible/awx/issues/6870)
+- Fixed a bug which broke AWX's external logging support when configured with HTTPS endpoints that utilize self-signed certificates (https://github.com/ansible/awx/issues/6851)
+- Fixed a local docker installer bug that mistakenly attempted to upgrade PostgreSQL when an external pg_hostname is specified (https://github.com/ansible/awx/pull/5398)
+- Fixed a race condition that caused task container crashes when pods are quickly brought down and back up (https://github.com/ansible/awx/issues/6750)
+- Fixed a bug that caused 404 errors when attempting to view the second page of the workflow approvals view (https://github.com/ansible/awx/issues/6803)
+- Fixed a bug that prevented the use of ANSIBLE_SSH_ARGS for ad-hoc-commands (https://github.com/ansible/awx/pull/6811)
+- Fixed a bug that broke AWX installs/upgrades on Red Hat OpenShift (https://github.com/ansible/awx/issues/6791)
+
+
+## 11.1.0 (Apr 22, 2020)
+- Changed rsyslogd to persist queued events to disk (to prevent a risk of out-of-memory errors) (https://github.com/ansible/awx/issues/6746)
+- Added the ability to configure the destination and maximum disk size of rsyslogd spool (in the event of a log aggregator outage) (https://github.com/ansible/awx/pull/6763)
+- Added the ability to discover playbooks in project clones from symlinked directories (https://github.com/ansible/awx/pull/6773)
+- Fixed a bug that caused certain log aggregator settings to break logging integration (https://github.com/ansible/awx/issues/6760)
+- Fixed a bug that caused playbook execution in container groups to sometimes unexpectedly deadlock (https://github.com/ansible/awx/issues/6692)
+- Improved stability of the new redis clustering implementation (https://github.com/ansible/awx/pull/6739 https://github.com/ansible/awx/pull/6720)
+- Improved stability of the new rsyslogd-based logging implementation (https://github.com/ansible/awx/pull/6796)
+
+## 11.0.0 (Apr 16, 2020)
+- As of AWX 11.0.0, Kubernetes-based deployments use a Deployment rather than a StatefulSet.
+- Reimplemented external logging support using rsyslogd to improve reliability and address a number of issues (https://github.com/ansible/awx/issues/5155)
+- Changed activity stream logs to include summary fields for related objects (https://github.com/ansible/awx/issues/1761)
+- Added code to more gracefully attempt to reconnect to redis if it restarts/becomes unavailable (https://github.com/ansible/awx/pull/6670)
+- Fixed a bug that caused REFRESH_TOKEN_EXPIRE_SECONDS to not properly be respected for OAuth2.0 refresh tokens generated by AWX (https://github.com/ansible/awx/issues/6630)
+- Fixed a bug that broke schedules containing RRULES with very old DTSTART dates (https://github.com/ansible/awx/pull/6550)
+- Fixed a bug that broke installs on older versions of Ansible packaged with certain Linux distributions (https://github.com/ansible/awx/issues/5501)
+- Fixed a bug that caused the activity stream to sometimes report the incorrect actor when associating user membership on SAML login (https://github.com/ansible/awx/pull/6525)
+- Fixed a bug in AWX's Grafana notification support when annotation tags are omitted (https://github.com/ansible/awx/issues/6580)
+- Fixed a bug that prevented some users from searching for Source Control credentials in the AWX user interface (https://github.com/ansible/awx/issues/6600)
+- Fixed a bug that prevented disassociating orphaned users from credentials (https://github.com/ansible/awx/pull/6554)
+- Updated Twisted to address CVE-2020-10108 and CVE-2020-10109.
+
+## 10.0.0 (Mar 30, 2020)
+- As of AWX 10.0.0, the official AWX CLI no longer supports Python 2 (it requires at least Python 3.6) (https://github.com/ansible/awx/pull/6327)
+- AWX no longer relies on RabbitMQ; Redis is added as a new dependency (https://github.com/ansible/awx/issues/5443)
+- Altered AWX's event tables to allow more than ~2 billion total events (https://github.com/ansible/awx/issues/6010)
+- Improved the performance (time to execute, and memory consumption) of the periodic job cleanup system job (https://github.com/ansible/awx/pull/6166)
+- Updated Job Templates so they now have an explicit Organization field (it is no longer inferred from the associated Project) (https://github.com/ansible/awx/issues/3903)
+- Updated social-auth-core to address an upcoming GitHub API deprecation (https://github.com/ansible/awx/issues/5970)
+- Updated to ansible-runner 1.4.6 to address various bugs.
+- Updated Django to address CVE-2020-9402
+- Updated pyyaml version to address CVE-2017-18342
+- Fixed a bug which prevented the new `scm_branch` field from being used in custom notification templates (https://github.com/ansible/awx/issues/6258)
+- Fixed a race condition that sometimes causes success/failure notifications to include an incomplete list of hosts (https://github.com/ansible/awx/pull/6290)
+- Fixed a bug that can cause certain setting pages to lose unsaved form edits when a playbook is launched (https://github.com/ansible/awx/issues/5265)
+- Fixed a bug that can prevent the "Use TLS/SSL" field from properly saving when editing email notification templates (https://github.com/ansible/awx/issues/6383)
+- Fixed a race condition that sometimes broke event/stdout processing for jobs launched in container groups (https://github.com/ansible/awx/issues/6280)
+
+## 9.3.0 (Mar 12, 2020)
+- Added the ability to specify an OAuth2 token description in the AWX CLI (https://github.com/ansible/awx/issues/6122)
+- Added support for K8S service account annotations to the installer (https://github.com/ansible/awx/pull/6007)
+- Added support for K8S imagePullSecrets to the installer (https://github.com/ansible/awx/pull/5989)
+- Launching jobs (and workflows) using the --monitor flag in the AWX CLI now returns a non-zero exit code on job failure (https://github.com/ansible/awx/issues/5920)
+- Improved UI performance for various job views when many simultaneous users are logged into AWX (https://github.com/ansible/awx/issues/5883)
+- Updated to the latest version of Django to address a few open CVEs (https://github.com/ansible/awx/pull/6080)
+- Fixed a critical bug which can cause AWX to hang and stop launching playbooks after a periodic of time (https://github.com/ansible/awx/issues/5617)
+- Fixed a bug which caused delays in project update stdout for certain large SCM clones (as of Ansible 2.9+) (https://github.com/ansible/awx/pull/6254)
+- Fixed a bug which caused certain smart inventory filters to mistakenly return duplicate hosts (https://github.com/ansible/awx/pull/5972)
+- Fixed an unclear server error when creating smart inventories with the AWX collection (https://github.com/ansible/awx/issues/6250)
+- Fixed a bug that broke Grafana notification support (https://github.com/ansible/awx/issues/6137)
+- Fixed a UI bug which prevent users with read access to an organization from editing credentials for that organization (https://github.com/ansible/awx/pull/6241)
+- Fixed a bug which prevent workflow approval records from recording a `started` and `elapsed` date (https://github.com/ansible/awx/issues/6202)
+- Fixed a bug which caused workflow nodes to have a confusing option for `verbosity` (https://github.com/ansible/awx/issues/6196)
+- Fixed an RBAC bug which prevented projects and inventory schedules from being created by certain users in certain contexts (https://github.com/ansible/awx/issues/5717)
+- Fixed a bug that caused `role_path` in a project's config to not be respected due to an error processing `/etc/ansible/ansible.cfg` (https://github.com/ansible/awx/pull/6038)
+- Fixed a bug that broke inventory updates for installs with custom home directories for the awx user (https://github.com/ansible/awx/pull/6152)
+- Fixed a bug that broke fact data collection when AWX encounters invalid/unexpected fact data (https://github.com/ansible/awx/issues/5935)
+
+
 ## 9.2.0 (Feb 12, 2020)
 - Added the ability to configure the convergence behavior of workflow nodes https://github.com/ansible/awx/issues/3054
 - AWX now allows for a configurable global limit for fork count (per-job run).  The default maximum is 200. https://github.com/ansible/awx/pull/5604
@@ -19,7 +132,7 @@ This is a list of high-level changes for each release of AWX. A full list of com
 - Updated the bundled version of openstacksdk to address a known issue https://github.com/ansible/awx/issues/5821
 - Updated the bundled vmware_inventory plugin to the latest version to address a bug https://github.com/ansible/awx/pull/5668
 - Fixed a bug that can cause inventory updates to fail to properly save their output when run within a workflow https://github.com/ansible/awx/pull/5666
- Removed a number of pre-computed fields from the Host and Group models to improve AWX performance.  As part of this change, inventory group UIs throughout the interface no longer display status icons https://github.com/ansible/awx/pull/5448 
+- Removed a number of pre-computed fields from the Host and Group models to improve AWX performance.  As part of this change, inventory group UIs throughout the interface no longer display status icons https://github.com/ansible/awx/pull/5448

 ## 9.1.1 (Jan 14, 2020)

@@ -58,7 +171,7 @@ This is a list of high-level changes for each release of AWX. A full list of com
 - Fixed a bug in the CLI which incorrectly parsed launch time arguments for `awx job_templates launch` and `awx workflow_job_templates launch` (https://github.com/ansible/awx/issues/5093).
 - Fixed a bug that caused inventory updates using "sourced from a project" to stop working (https://github.com/ansible/awx/issues/4750).
 - Fixed a bug that caused Slack notifications to sometimes show the wrong bot avatar (https://github.com/ansible/awx/pull/5125).
- Fixed a bug that prevented the use of digits in Tower's URL settings (https://github.com/ansible/awx/issues/5081).
+- Fixed a bug that prevented the use of digits in AWX's URL settings (https://github.com/ansible/awx/issues/5081).

 ## 8.0.0 (Oct 21, 2019)

--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -155,12 +155,10 @@ If you start a second terminal session, you can take a look at the running conta
 (host)$ docker ps

 $ docker ps
-CONTAINER ID        IMAGE                                              COMMAND                  CREATED             STATUS              PORTS                                                                                                                                      NAMES
-aa4a75d6d77b        gcr.io/ansible-tower-engineering/awx_devel:devel   "/tini -- /bin/sh ..."   23 seconds ago      Up 15 seconds       0.0.0.0:5555->5555/tcp, 0.0.0.0:7899-7999->7899-7999/tcp, 0.0.0.0:8013->8013/tcp, 0.0.0.0:8043->8043/tcp, 22/tcp, 0.0.0.0:8080->8080/tcp   tools_awx_1
-e4c0afeb548c        postgres:10                                       "docker-entrypoint..."   26 seconds ago      Up 23 seconds       5432/tcp                                                                                                                                   tools_postgres_1
-0089699d5afd        tools_logstash                                     "/docker-entrypoin..."   26 seconds ago      Up 25 seconds                                                                                                                                                  tools_logstash_1
-4d4ff0ced266        memcached:alpine                                   "docker-entrypoint..."   26 seconds ago      Up 25 seconds       0.0.0.0:11211->11211/tcp                                                                                                                   tools_memcached_1
-92842acd64cd        rabbitmq:3-management                              "docker-entrypoint..."   26 seconds ago      Up 24 seconds       4369/tcp, 5671-5672/tcp, 15671/tcp, 25672/tcp, 0.0.0.0:15672->15672/tcp                                                                    tools_rabbitmq_1
+CONTAINER ID        IMAGE                                              COMMAND                  CREATED             STATUS              PORTS                                                                                                                                                              NAMES
+44251b476f98        gcr.io/ansible-tower-engineering/awx_devel:devel   "/entrypoint.sh /bin…"   27 seconds ago      Up 23 seconds       0.0.0.0:6899->6899/tcp, 0.0.0.0:7899-7999->7899-7999/tcp, 0.0.0.0:8013->8013/tcp, 0.0.0.0:8043->8043/tcp, 0.0.0.0:8080->8080/tcp, 22/tcp, 0.0.0.0:8888->8888/tcp   tools_awx_run_9e820694d57e
+40de380e3c2e        redis:latest                                       "docker-entrypoint.s…"   28 seconds ago      Up 26 seconds
+b66a506d3007        postgres:10                                        "docker-entrypoint.s…"   28 seconds ago      Up 26 seconds       0.0.0.0:5432->5432/tcp                                                                                                                                             tools_postgres_1
 ```
 **NOTE**

@@ -216,18 +214,23 @@ Using `docker exec`, this will create a session in the running *awx* container,
 If you want to start and use the development environment, you'll first need to bootstrap it by running the following command:

 ```bash
-(container)# /bootstrap_development.sh
+(container)# /usr/bin/bootstrap_development.sh
 ```

-The above will do all the setup tasks, including running database migrations, so it may take a couple minutes.
+The above will do all the setup tasks, including running database migrations, so it may take a couple minutes. Once it's done it
+will drop you back to the shell.

-Now you can start each service individually, or start all services in a pre-configured tmux session like so:
+In order to launch all developer services:

 ```bash
-(container)# cd /awx_devel
-(container)# make server
+(container)# /usr/bin/launch_awx.sh
 ```

+`launch_awx.sh` also calls `bootstrap_development.sh` so if all you are doing is launching the supervisor to start all services, you don't
+need to call `bootstrap_development.sh` first.
+
+
+
 ### Post Build Steps

 Before you can log in and use the system, you will need to create an admin user. Optionally, you may also want to load some demo data.
--- a/INSTALL.md
+++ b/INSTALL.md
@@ -10,7 +10,6 @@ This document provides a guide for installing AWX.
    + [AWX branding](#awx-branding)
    + [Prerequisites](#prerequisites)
    + [System Requirements](#system-requirements)
-    + [AWX Tunables](#awx-tunables)
    + [Choose a deployment platform](#choose-a-deployment-platform)
    + [Official vs Building Images](#official-vs-building-images)
  * [Upgrading from previous versions](#upgrading-from-previous-versions)
@@ -41,13 +40,25 @@ This document provides a guide for installing AWX.
    + [Run the installer](#run-the-installer-2)
    + [Post-install](#post-install-2)
    + [Accessing AWX](#accessing-awx-2)
+- [Installing the AWX CLI](#installing-the-awx-cli)
+  * [Building the CLI Documentation](#building-the-cli-documentation)

    
 ## Getting started

 ### Clone the repo

-If you have not already done so, you will need to clone, or create a local copy, of the [AWX repo](https://github.com/ansible/awx). For more on how to clone the repo, view [git clone help](https://git-scm.com/docs/git-clone).
+If you have not already done so, you will need to clone, or create a local copy, of the [AWX repo](https://github.com/ansible/awx). We generally recommend that you view the releases page:
+
+https://github.com/ansible/awx/releases
+
+...and clone the latest stable release, e.g.,
+
+`git clone -b x.y.z https://github.com/ansible/awx.git`
+
+Please note that deploying from `HEAD` (or the latest commit) is **not** stable, and that if you want to do this, you should proceed at your own risk (also, see the section #official-vs-building-images for building your own image).
+
+For more on how to clone the repo, view [git clone help](https://git-scm.com/docs/git-clone).

 Once you have a local copy, run commands within the root of the project tree.

@@ -69,8 +80,11 @@ Before you can run a deployment, you'll need the following installed in your loc
    + We use this module instead of `docker-py` because it is what the `docker-compose` Python module requires.
 - [GNU Make](https://www.gnu.org/software/make/)
 - [Git](https://git-scm.com/) Requires Version 1.8.4+
+- Python 3.6+
 - [Node 10.x LTS version](https://nodejs.org/en/download/)
+    + This is only required if you're [building your own container images](#official-vs-building-images) with `use_container_for_build=false`
 - [NPM 6.x LTS](https://docs.npmjs.com/)
+    + This is only required if you're [building your own container images](#official-vs-building-images) with `use_container_for_build=false`

 ### System Requirements

@@ -80,11 +94,7 @@ The system that runs the AWX service will need to satisfy the following requirem
 - At least 2 cpu cores
 - At least 20GB of space
 - Running Docker, Openshift, or Kubernetes
- If you choose to use an external PostgreSQL database, please note that the minimum version is 9.6+.
-
-### AWX Tunables
-
-**TODO** add tunable bits
+- If you choose to use an external PostgreSQL database, please note that the minimum version is 10+.

 ### Choose a deployment platform

@@ -99,7 +109,7 @@ In the sections below, you'll find deployment details and instructions for each

 ### Official vs Building Images

-When installing AWX you have the option of building your own images or using the images provided on DockerHub (see [awx_web](https://hub.docker.com/r/ansible/awx_web/) and [awx_task](https://hub.docker.com/r/ansible/awx_task/))
+When installing AWX you have the option of building your own image or using the image provided on DockerHub (see [awx](https://hub.docker.com/r/ansible/awx/))

 This is controlled by the following variables in the `inventory` file

@@ -112,12 +122,16 @@ If these variables are present then all deployments will use these hosted images

 *dockerhub_base*

-> The base location on DockerHub where the images are hosted (by default this pulls container images named `ansible/awx_web:tag` and `ansible/awx_task:tag`)
+> The base location on DockerHub where the images are hosted (by default this pulls a container image named `ansible/awx:tag`)

 *dockerhub_version*

 > Multiple versions are provided. `latest` always pulls the most recent. You may also select version numbers at different granularities: 1, 1.0, 1.0.1, 1.0.0.123

+*use_container_for_build*
+
+> Use a local distribution build container image for building the AWX package. This is helpful if you don't want to bother installing the build-time dependencies as it is taken care of already.
+

 ## Upgrading from previous versions

@@ -128,7 +142,6 @@ For convenience, you can create a file called `vars.yml`:
 ```
 admin_password: 'adminpass'
 pg_password: 'pgpass'
-rabbitmq_password: 'rabbitpass'
 secret_key: 'mysupersecret'
 ```

@@ -142,7 +155,7 @@ $ ansible-playbook -i inventory install.yml -e @vars.yml

 ### Prerequisites

-To complete a deployment to OpenShift, you will obviously need access to an OpenShift cluster. For demo and testing purposes, you can use [Minishift](https://github.com/minishift/minishift) to create a single node cluster running inside a virtual machine.
+To complete a deployment to OpenShift, you will need access to an OpenShift cluster. For demo and testing purposes, you can use [Minishift](https://github.com/minishift/minishift) to create a single node cluster running inside a virtual machine.

 When using OpenShift for deploying AWX make sure you have correct privileges to add the security context 'privileged', otherwise the installation will fail. The privileged context is needed because of the use of [the bubblewrap tool](https://github.com/containers/bubblewrap) to add an additional layer of security when using containers.

@@ -468,15 +481,15 @@ Before starting the install process, review the [inventory](./installer/inventor

 *host_port*

-> Provide a port number that can be mapped from the Docker daemon host to the web server running inside the AWX container. Defaults to *80*.
+> Provide a port number that can be mapped from the Docker daemon host to the web server running inside the AWX container. If undefined no port will be exposed. Defaults to *80*.

 *host_port_ssl*

-> Provide a port number that can be mapped from the Docker daemon host to the web server running inside the AWX container for SSL support. Defaults to *443*, only works if you also set `ssl_certificate` (see below).
+> Provide a port number that can be mapped from the Docker daemon host to the web server running inside the AWX container for SSL support. If undefined no port will be exposed. Defaults to *443*, only works if you also set `ssl_certificate` (see below).

 *ssl_certificate*

-> Optionally, provide the path to a file that contains a certificate and its private key.
+> Optionally, provide the path to a file that contains a certificate and its private key. This needs to be a .pem-file

 *docker_compose_dir*

@@ -506,10 +519,6 @@ If you wish to tag and push built images to a Docker registry, set the following

 > Username of the user that will push images to the registry. Defaults to *developer*.

-*docker_remove_local_images*
-
-> Due to the way that the docker_image module behaves, images will not be pushed to a remote repository if they are present locally.  Set this to delete local versions of the images that will be pushed to the remote.  This will fail if containers are currently running from those images.
-
 **Note**

 > These settings are ignored if using official images
@@ -559,16 +568,7 @@ $ ansible-playbook -i inventory -e docker_registry_password=password install.yml

 ### Post-install

-After the playbook run completes, Docker will report up to 5 running containers. If you chose to use an existing PostgresSQL database, then it will report 4. You can view the running containers using the `docker ps` command, as follows:
-
-```bash
-CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                NAMES
-e240ed8209cd        awx_task:1.0.0.8    "/tini -- /bin/sh ..."   2 minutes ago       Up About a minute   8052/tcp                             awx_task
-1cfd02601690        awx_web:1.0.0.8     "/tini -- /bin/sh ..."   2 minutes ago       Up About a minute   0.0.0.0:443->8052/tcp                 awx_web
-55a552142bcd        memcached:alpine    "docker-entrypoint..."   2 minutes ago       Up 2 minutes        11211/tcp                            memcached
-84011c072aad        rabbitmq:3          "docker-entrypoint..."   2 minutes ago       Up 2 minutes        4369/tcp, 5671-5672/tcp, 25672/tcp   rabbitmq
-97e196120ab3        postgres:9.6        "docker-entrypoint..."   2 minutes ago       Up 2 minutes        5432/tcp                             postgres
-```
+After the playbook run completes, Docker starts a series of containers that provide the services that make up AWX.  You can view the running containers using the `docker ps` command.

 If you're deploying using Docker Compose, container names will be prefixed by the name of the folder where the docker-compose.yml file is created (by default, `awx`).

@@ -634,3 +634,34 @@ Added instance awx to tower
 The AWX web server is accessible on the deployment host, using the *host_port* value set in the *inventory* file. The default URL is [http://localhost](http://localhost).

 You will prompted with a login dialog. The default administrator username is `admin`, and the password is `password`.
+
+
+# Installing the AWX CLI
+
+`awx` is the official command-line client for AWX.  It:
+
+* Uses naming and structure consistent with the AWX HTTP API
+* Provides consistent output formats with optional machine-parsable formats
+* To the extent possible, auto-detects API versions, available endpoints, and
+  feature support across multiple versions of AWX.
+
+Potential uses include:
+
+* Configuring and launching jobs/playbooks
+* Checking on the status and output of job runs
+* Managing objects like organizations, users, teams, etc...
+
+The preferred way to install the AWX CLI is through pip directly from GitHub:
+
+    pip install "https://github.com/ansible/awx/archive/$VERSION.tar.gz#egg=awxkit&subdirectory=awxkit"
+    awx --help
+
+...where ``$VERSION`` is the version of AWX you're running.  To see a list of all available releases, visit: https://github.com/ansible/awx/releases
+
+## Building the CLI Documentation
+
+To build the docs, spin up a real AWX server, `pip install sphinx sphinxcontrib-autoprogram`, and run:
+
+    ~ TOWER_HOST=https://awx.example.org TOWER_USERNAME=example TOWER_PASSWORD=secret make clean html
+    ~ cd build/html/ && python -m http.server
+    Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ..
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -10,6 +10,7 @@ recursive-include awx/playbooks *.yml
 recursive-include awx/lib/site-packages *
 recursive-include awx/plugins *.ps1
 recursive-include requirements *.txt
+recursive-include requirements *.yml
 recursive-include config *
 recursive-include docs/licenses *
 recursive-exclude awx devonly.py*
--- a/137
+++ b/137
@@ -6,26 +6,27 @@ PACKER ?= packer
 PACKER_BUILD_OPTS ?= -var 'official=$(OFFICIAL)' -var 'aw_repo_url=$(AW_REPO_URL)'
 NODE ?= node
 NPM_BIN ?= npm
+CHROMIUM_BIN=/tmp/chrome-linux/chrome
 DEPS_SCRIPT ?= packaging/bundle/deps.py
 GIT_BRANCH ?= $(shell git rev-parse --abbrev-ref HEAD)
 MANAGEMENT_COMMAND ?= awx-manage
 IMAGE_REPOSITORY_AUTH ?=
 IMAGE_REPOSITORY_BASE ?= https://gcr.io
 VERSION := $(shell cat VERSION)
+PYCURL_SSL_LIBRARY ?= openssl

 # NOTE: This defaults the container image version to the branch that's active
 COMPOSE_TAG ?= $(GIT_BRANCH)
 COMPOSE_HOST ?= $(shell hostname)

 VENV_BASE ?= /venv
-COLLECTION_VENV ?= /awx_devel/awx_collection_test_venv
 SCL_PREFIX ?=
 CELERY_SCHEDULE_FILE ?= /var/lib/awx/beat.db

 DEV_DOCKER_TAG_BASE ?= gcr.io/ansible-tower-engineering
 # Python packages to install only from source (not from binary wheels)
 # Comma separated list
-SRC_ONLY_PKGS ?= cffi,pycparser,psycopg2,twilio
+SRC_ONLY_PKGS ?= cffi,pycparser,psycopg2,twilio,pycurl
 # These should be upgraded in the AWX and Ansible venv before attempting
 # to install the actual requirements
 VENV_BOOTSTRAP ?= pip==19.3.1 setuptools==41.6.0
@@ -167,17 +168,16 @@ virtualenv_awx:
 		fi; \
 		if [ ! -d "$(VENV_BASE)/awx" ]; then \
 			virtualenv -p $(PYTHON) $(VENV_BASE)/awx; \
-			$(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) $(VENV_BOOTSTRAP) && \
-			$(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) flit; \
+			$(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) $(VENV_BOOTSTRAP); \
 		fi; \
 	fi

 # --ignore-install flag is not used because *.txt files should specify exact versions
 requirements_ansible: virtualenv_ansible
 	if [[ "$(PIP_OPTIONS)" == *"--no-index"* ]]; then \
-	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_local.txt | $(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) -r /dev/stdin ; \
+	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_local.txt | PYCURL_SSL_LIBRARY=$(PYCURL_SSL_LIBRARY) $(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) -r /dev/stdin ; \
 	else \
-	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_git.txt | $(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) -r /dev/stdin ; \
+	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_git.txt | PYCURL_SSL_LIBRARY=$(PYCURL_SSL_LIBRARY) $(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) -r /dev/stdin ; \
 	fi
 	$(VENV_BASE)/ansible/bin/pip uninstall --yes -r requirements/requirements_ansible_uninstall.txt
 	# Same effect as using --system-site-packages flag on venv creation
@@ -185,9 +185,9 @@ requirements_ansible: virtualenv_ansible

 requirements_ansible_py3: virtualenv_ansible_py3
 	if [[ "$(PIP_OPTIONS)" == *"--no-index"* ]]; then \
-	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_local.txt | $(VENV_BASE)/ansible/bin/pip3 install $(PIP_OPTIONS) -r /dev/stdin ; \
+	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_local.txt | PYCURL_SSL_LIBRARY=$(PYCURL_SSL_LIBRARY) $(VENV_BASE)/ansible/bin/pip3 install $(PIP_OPTIONS) -r /dev/stdin ; \
 	else \
-	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_git.txt | $(VENV_BASE)/ansible/bin/pip3 install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) -r /dev/stdin ; \
+	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_git.txt | PYCURL_SSL_LIBRARY=$(PYCURL_SSL_LIBRARY) $(VENV_BASE)/ansible/bin/pip3 install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) -r /dev/stdin ; \
 	fi
 	$(VENV_BASE)/ansible/bin/pip3 uninstall --yes -r requirements/requirements_ansible_uninstall.txt
 	# Same effect as using --system-site-packages flag on venv creation
@@ -211,7 +211,11 @@ requirements_awx: virtualenv_awx
 requirements_awx_dev:
 	$(VENV_BASE)/awx/bin/pip install -r requirements/requirements_dev.txt

-requirements: requirements_ansible requirements_awx
+requirements_collections:
+	mkdir -p $(COLLECTION_BASE)
+	ansible-galaxy collection install -r requirements/collections_requirements.yml -p $(COLLECTION_BASE)
+
+requirements: requirements_ansible requirements_awx requirements_collections

 requirements_dev: requirements_awx requirements_ansible_py3 requirements_awx_dev requirements_ansible_dev

@@ -266,28 +270,6 @@ migrate:
 dbchange:
 	$(MANAGEMENT_COMMAND) makemigrations

-server_noattach:
-	tmux new-session -d -s awx 'exec make uwsgi'
-	tmux rename-window 'AWX'
-	tmux select-window -t awx:0
-	tmux split-window -v 'exec make dispatcher'
-	tmux new-window 'exec make daphne'
-	tmux select-window -t awx:1
-	tmux rename-window 'WebSockets'
-	tmux split-window -h 'exec make runworker'
-	tmux split-window -v 'exec make nginx'
-	tmux new-window 'exec make receiver'
-	tmux select-window -t awx:2
-	tmux rename-window 'Extra Services'
-	tmux select-window -t awx:0
-
-server: server_noattach
-	tmux -2 attach-session -t awx
-
-# Use with iterm2's native tmux protocol support
-servercc: server_noattach
-	tmux -2 -CC attach-session -t awx
-
 supervisor:
 	@if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
@@ -312,18 +294,11 @@ daphne:
 	fi; \
 	daphne -b 127.0.0.1 -p 8051 awx.asgi:channel_layer

-runworker:
+wsbroadcast:
 	@if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
 	fi; \
-	$(PYTHON) manage.py runworker --only-channels websocket.*
-
-# Run the built-in development webserver (by default on http://localhost:8013).
-runserver:
-	@if [ "$(VENV_BASE)" ]; then \
-		. $(VENV_BASE)/awx/bin/activate; \
-	fi; \
-	$(PYTHON) manage.py runserver
+	$(PYTHON) manage.py run_wsbroadcast

 # Run to start the background task dispatcher for development.
 dispatcher:
@@ -380,55 +355,63 @@ swagger: reports
 check: flake8 pep8 # pyflakes pylint

 awx-link:
-	cp -R /tmp/awx.egg-info /awx_devel/ || true
-	sed -i "s/placeholder/$(shell cat VERSION)/" /awx_devel/awx.egg-info/PKG-INFO
+	[ -d "/awx_devel/awx.egg-info" ] || python3 /awx_devel/setup.py egg_info_dev
 	cp -f /tmp/awx.egg-link /venv/awx/lib/python$(PYTHON_VERSION)/site-packages/awx.egg-link

 TEST_DIRS ?= awx/main/tests/unit awx/main/tests/functional awx/conf/tests awx/sso/tests

 # Run all API unit tests.
 test:
-	@if [ "$(VENV_BASE)" ]; then \
+	if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
 	fi; \
 	PYTHONDONTWRITEBYTECODE=1 py.test -p no:cacheprovider -n auto $(TEST_DIRS)
-	cd awxkit && $(VENV_BASE)/awx/bin/tox -re py2,py3
+	cmp VERSION awxkit/VERSION || "VERSION and awxkit/VERSION *must* match"
+	cd awxkit && $(VENV_BASE)/awx/bin/tox -re py3
 	awx-manage check_migrations --dry-run --check  -n 'vNNN_missing_migration_file'

-prepare_collection_venv:
-	rm -rf $(COLLECTION_VENV)
-	mkdir $(COLLECTION_VENV)
-	$(VENV_BASE)/awx/bin/pip install --target=$(COLLECTION_VENV) git+https://github.com/ansible/tower-cli.git
-
 COLLECTION_TEST_DIRS ?= awx_collection/test/awx
+COLLECTION_TEST_TARGET ?=
 COLLECTION_PACKAGE ?= awx
 COLLECTION_NAMESPACE ?= awx
+COLLECTION_INSTALL = ~/.ansible/collections/ansible_collections/$(COLLECTION_NAMESPACE)/$(COLLECTION_PACKAGE)

 test_collection:
-	@if [ "$(VENV_BASE)" ]; then \
+	rm -f $(shell ls -d $(VENV_BASE)/awx/lib/python* | head -n 1)/no-global-site-packages.txt
+	if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
 	fi; \
-	PYTHONPATH=$(COLLECTION_VENV):/awx_devel/awx_collection:$PYTHONPATH:/usr/lib/python3.6/site-packages py.test $(COLLECTION_TEST_DIRS)
+	py.test $(COLLECTION_TEST_DIRS) -v
+	# The python path needs to be modified so that the tests can find Ansible within the container
+	# First we will use anything expility set as PYTHONPATH
+	# Second we will load any libraries out of the virtualenv (if it's unspecified that should be ok because python should not load out of an empty directory)
+	# Finally we will add the system path so that the tests can find the ansible libraries

 flake8_collection:
 	flake8 awx_collection/  # Different settings, in main exclude list

-test_collection_all: prepare_collection_venv test_collection flake8_collection
+test_collection_all: test_collection flake8_collection

-test_collection_sanity:
-	rm -rf sanity
-	mkdir -p sanity/ansible_collections/awx
-	cp -Ra awx_collection sanity/ansible_collections/awx/awx  # symlinks do not work
-	cd sanity/ansible_collections/awx/awx && git init && git add . # requires both this file structure and a git repo, so there you go
-	cd sanity/ansible_collections/awx/awx && ansible-test sanity
+# WARNING: symlinking a collection is fundamentally unstable
+# this is for rapid development iteration with playbooks, do not use with other test targets
+symlink_collection:
+	rm -rf $(COLLECTION_INSTALL)
+	mkdir -p ~/.ansible/collections/ansible_collections/$(COLLECTION_NAMESPACE)  # in case it does not exist
+	ln -s $(shell pwd)/awx_collection $(COLLECTION_INSTALL)

 build_collection:
-	ansible-playbook -i localhost, awx_collection/template_galaxy.yml -e collection_package=$(COLLECTION_PACKAGE) -e collection_namespace=$(COLLECTION_NAMESPACE) -e collection_version=$(VERSION)
+	ansible-playbook -i localhost, awx_collection/tools/template_galaxy.yml -e collection_package=$(COLLECTION_PACKAGE) -e collection_namespace=$(COLLECTION_NAMESPACE) -e collection_version=$(VERSION) -e '{"awx_template_version":false}'
 	ansible-galaxy collection build awx_collection --force --output-path=awx_collection

 install_collection: build_collection
-	rm -rf ~/.ansible/collections/ansible_collections/awx/awx
-	ansible-galaxy collection install awx_collection/awx-awx-$(VERSION).tar.gz
+	rm -rf $(COLLECTION_INSTALL)
+	ansible-galaxy collection install awx_collection/$(COLLECTION_NAMESPACE)-$(COLLECTION_PACKAGE)-$(VERSION).tar.gz
+
+test_collection_sanity: install_collection
+	cd $(COLLECTION_INSTALL) && ansible-test sanity
+
+test_collection_integration: install_collection
+	cd $(COLLECTION_INSTALL) && ansible-test integration $(COLLECTION_TEST_TARGET)

 test_unit:
 	@if [ "$(VENV_BASE)" ]; then \
@@ -571,11 +554,13 @@ jshint: $(UI_DEPS_FLAG_FILE)
 	$(NPM_BIN) run --prefix awx/ui jshint
 	$(NPM_BIN) run --prefix awx/ui lint

-ui-zuul-lint-and-test: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) run --prefix awx/ui jshint
-	$(NPM_BIN) run --prefix awx/ui lint
-	$(NPM_BIN) --prefix awx/ui run test:ci
-	$(NPM_BIN) --prefix awx/ui run unit
+ui-zuul-lint-and-test:
+	CHROMIUM_BIN=$(CHROMIUM_BIN) ./awx/ui/build/zuul_download_chromium.sh
+	CHROMIUM_BIN=$(CHROMIUM_BIN) PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=1 $(NPM_BIN) --unsafe-perm --prefix awx/ui ci --no-save awx/ui
+	CHROMIUM_BIN=$(CHROMIUM_BIN) $(NPM_BIN) run --prefix awx/ui jshint
+	CHROMIUM_BIN=$(CHROMIUM_BIN) $(NPM_BIN) run --prefix awx/ui lint
+	CHROME_BIN=$(CHROMIUM_BIN) $(NPM_BIN) --prefix awx/ui run test:ci
+	CHROME_BIN=$(CHROMIUM_BIN) $(NPM_BIN) --prefix awx/ui run unit

 # END UI TASKS
 # --------------------------------------
@@ -664,7 +649,7 @@ docker-compose-runtest: awx/projects
 	cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm --service-ports awx /start_tests.sh

 docker-compose-build-swagger: awx/projects
-	cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm --service-ports awx /start_tests.sh swagger
+	cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm --service-ports --no-deps awx /start_tests.sh swagger

 detect-schema-change: genschema
 	curl https://s3.amazonaws.com/awx-public-ci-files/schema.json -o reference-schema.json
@@ -672,29 +657,28 @@ detect-schema-change: genschema
 	diff -u -b reference-schema.json schema.json

 docker-compose-clean: awx/projects
-	cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm -w /awx_devel --service-ports awx make clean
 	cd tools && TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose rm -sf

-docker-compose-build: awx-devel-build
-
 # Base development image build
-awx-devel-build:
+docker-compose-build:
+	ansible localhost -m template -a "src=installer/roles/image_build/templates/Dockerfile.j2 dest=tools/docker-compose/Dockerfile" -e build_dev=True
 	docker build -t ansible/awx_devel -f tools/docker-compose/Dockerfile \
 		--cache-from=$(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG) .
 	docker tag ansible/awx_devel $(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG)
 	#docker push $(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG)

 # For use when developing on "isolated" AWX deployments
-docker-compose-isolated-build: awx-devel-build
+docker-compose-isolated-build: docker-compose-build
 	docker build -t ansible/awx_isolated -f tools/docker-isolated/Dockerfile .
 	docker tag ansible/awx_isolated $(DEV_DOCKER_TAG_BASE)/awx_isolated:$(COMPOSE_TAG)
 	#docker push $(DEV_DOCKER_TAG_BASE)/awx_isolated:$(COMPOSE_TAG)

-MACHINE?=default
 docker-clean:
-	eval $$(docker-machine env $(MACHINE))
 	$(foreach container_id,$(shell docker ps -f name=tools_awx -aq),docker stop $(container_id); docker rm -f $(container_id);)
-	-docker images | grep "awx_devel" | awk '{print $$1 ":" $$2}' | xargs docker rmi
+	docker images | grep "awx_devel" | awk '{print $$1 ":" $$2}' | xargs docker rmi
+
+docker-clean-volumes: docker-compose-clean
+	docker volume rm tools_awx_db

 docker-refresh: docker-clean docker-compose

@@ -708,9 +692,6 @@ docker-compose-cluster-elk: docker-auth awx/projects
 prometheus:
 	docker run -u0 --net=tools_default --link=`docker ps | egrep -o "tools_awx(_run)?_([^ ]+)?"`:awxweb --volume `pwd`/tools/prometheus:/prometheus --name prometheus -d -p 0.0.0.0:9090:9090 prom/prometheus --web.enable-lifecycle --config.file=/prometheus/prometheus.yml

-minishift-dev:
-	ansible-playbook -i localhost, -e devtree_directory=$(CURDIR) tools/clusterdevel/start_minishift_dev.yml
-
 clean-elk:
 	docker stop tools_kibana_1
 	docker stop tools_logstash_1
--- a/2
+++ b/2
@@ -1 +1 @@
-9.2.0
+13.0.0
--- a/awx/init.py
+++ b/awx/init.py
@@ -30,6 +30,7 @@ except ImportError:
    HAS_DJANGO = False
 else:
    from django.db.backends.base import schema
+    from django.db.models import indexes
    from django.db.backends.utils import names_digest


@@ -50,6 +51,7 @@ if HAS_DJANGO is True:
            return h.hexdigest()[:length]

        schema.names_digest = names_digest
+        indexes.names_digest = names_digest


 def find_commands(management_dir):
--- a/awx/api/generics.py
+++ b/awx/api/generics.py
@@ -5,10 +5,12 @@
 import inspect
 import logging
 import time
+import uuid
 import urllib.parse

 # Django
 from django.conf import settings
+from django.core.cache import cache
 from django.db import connection
 from django.db.models.fields import FieldDoesNotExist
 from django.db.models.fields.related import OneToOneRel
@@ -43,7 +45,10 @@ from awx.main.utils import (
    get_search_fields,
    getattrd,
    get_object_or_400,
-    decrypt_field
+    decrypt_field,
+    get_awx_version,
+    get_licenser,
+    StubLicense
 )
 from awx.main.utils.db import get_all_field_names
 from awx.api.serializers import ResourceAccessListElementSerializer, CopySerializer, UserSerializer
@@ -192,9 +197,11 @@ class APIView(views.APIView):
                response.data['detail'] += ' To establish a login session, visit /api/login/.'
                logger.info(status_msg)
            else:
-                logger.warn(status_msg)
+                logger.warning(status_msg)
        response = super(APIView, self).finalize_response(request, response, *args, **kwargs)
        time_started = getattr(self, 'time_started', None)
+        response['X-API-Product-Version'] = get_awx_version()
+        response['X-API-Product-Name'] = 'AWX' if isinstance(get_licenser(), StubLicense) else 'Red Hat Ansible Tower'
        response['X-API-Node'] = settings.CLUSTER_HOST_ID
        if time_started:
            time_elapsed = time.time() - self.time_started
@@ -548,6 +555,15 @@ class SubListCreateAPIView(SubListAPIView, ListCreateAPIView):
        })
        return d

+    def get_queryset(self):
+        if hasattr(self, 'parent_key'):
+            # Prefer this filtering because ForeignKey allows us more assumptions
+            parent = self.get_parent_object()
+            self.check_parent_access(parent)
+            qs = self.request.user.get_queryset(self.model)
+            return qs.filter(**{self.parent_key: parent})
+        return super(SubListCreateAPIView, self).get_queryset()
+
    def create(self, request, *args, **kwargs):
        # If the object ID was not specified, it probably doesn't exist in the
        # DB yet. We want to see if we can create it.  The URL may choose to
@@ -964,6 +980,11 @@ class CopyAPIView(GenericAPIView):
        if hasattr(new_obj, 'admin_role') and request.user not in new_obj.admin_role.members.all():
            new_obj.admin_role.members.add(request.user)
        if sub_objs:
+            # store the copied object dict into cache, because it's
+            # often too large for postgres' notification bus
+            # (which has a default maximum message size of 8k)
+            key = 'deep-copy-{}'.format(str(uuid.uuid4()))
+            cache.set(key, sub_objs, timeout=3600)
            permission_check_func = None
            if hasattr(type(self), 'deep_copy_permission_check_func'):
                permission_check_func = (
@@ -971,7 +992,7 @@ class CopyAPIView(GenericAPIView):
                )
            trigger_delayed_deep_copy(
                self.model.__module__, self.model.__name__,
-                obj.pk, new_obj.pk, request.user.pk, sub_objs,
+                obj.pk, new_obj.pk, request.user.pk, key,
                permission_check_func=permission_check_func
            )
        serializer = self._get_copy_return_serializer(new_obj)
--- a/awx/api/metadata.py
+++ b/awx/api/metadata.py
@@ -2,6 +2,7 @@
 # All Rights Reserved.

 from collections import OrderedDict
+from uuid import UUID

 # Django
 from django.core.exceptions import PermissionDenied
@@ -20,6 +21,7 @@ from rest_framework.fields import JSONField as DRFJSONField
 from rest_framework.request import clone_request

 # AWX
+from awx.api.fields import ChoiceNullField
 from awx.main.fields import JSONField, ImplicitRoleField
 from awx.main.models import InventorySource, NotificationTemplate
 from awx.main.scheduler.kubernetes import PodManager
@@ -59,7 +61,8 @@ class Metadata(metadata.SimpleMetadata):
                'type': _('Data type for this {}.'),
                'url': _('URL for this {}.'),
                'related': _('Data structure with URLs of related resources.'),
-                'summary_fields': _('Data structure with name/description for related resources.'),
+                'summary_fields': _('Data structure with name/description for related resources.  '
+                                    'The output for some objects may be limited for performance reasons.'),
                'created': _('Timestamp when this {} was created.'),
                'modified': _('Timestamp when this {} was last modified.'),
            }
@@ -84,6 +87,8 @@ class Metadata(metadata.SimpleMetadata):
        # FIXME: Still isn't showing all default values?
        try:
            default = field.get_default()
+            if type(default) is UUID:
+                default = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
            if field.field_name == 'TOWER_URL_BASE' and default == 'https://towerhost':
                default = '{}://{}'.format(self.request.scheme, self.request.get_host())
            field_info['default'] = default
@@ -96,7 +101,15 @@ class Metadata(metadata.SimpleMetadata):
            field_info['children'] = self.get_serializer_info(field)

        if not isinstance(field, (RelatedField, ManyRelatedField)) and hasattr(field, 'choices'):
-            field_info['choices'] = [(choice_value, choice_name) for choice_value, choice_name in field.choices.items()]
+            choices = [
+                (choice_value, choice_name) for choice_value, choice_name in field.choices.items()
+            ]
+            if not any(choice in ('', None) for choice, _ in choices):
+                if field.allow_blank:
+                    choices = [("", "---------")] + choices
+                if field.allow_null and not isinstance(field, ChoiceNullField):
+                    choices = [(None, "---------")] + choices
+            field_info['choices'] = choices

        # Indicate if a field is write-only.
        if getattr(field, 'write_only', False):
--- a/awx/api/serializers.py
+++ b/awx/api/serializers.py
@@ -72,6 +72,7 @@ from awx.main.utils import (
    prefetch_page_capabilities, get_external_account, truncate_stdout,
 )
 from awx.main.utils.filters import SmartFilter
+from awx.main.utils.named_url_graph import reset_counters
 from awx.main.redact import UriCleaner, REPLACE_STR

 from awx.main.validators import vars_validate_or_raise
@@ -125,7 +126,7 @@ SUMMARIZABLE_FK_FIELDS = {
    'current_job': DEFAULT_SUMMARY_FIELDS + ('status', 'failed', 'license_error'),
    'inventory_source': ('source', 'last_updated', 'status'),
    'custom_inventory_script': DEFAULT_SUMMARY_FIELDS,
-    'source_script': ('name', 'description'),
+    'source_script': DEFAULT_SUMMARY_FIELDS,
    'role': ('id', 'role_field'),
    'notification_template': DEFAULT_SUMMARY_FIELDS,
    'instance_group': ('id', 'name', 'controller_id', 'is_containerized'),
@@ -347,6 +348,7 @@ class BaseSerializer(serializers.ModelSerializer, metaclass=BaseSerializerMetacl

    def _generate_named_url(self, url_path, obj, node):
        url_units = url_path.split('/')
+        reset_counters()
        named_url = node.generate_named_url(obj)
        url_units[4] = named_url
        return '/'.join(url_units)
@@ -642,7 +644,7 @@ class UnifiedJobTemplateSerializer(BaseSerializer):
    _capabilities_prefetch = [
        'admin', 'execute',
        {'copy': ['jobtemplate.project.use', 'jobtemplate.inventory.use',
-                  'workflowjobtemplate.organization.workflow_admin']}
+                  'organization.workflow_admin']}
    ]

    class Meta:
@@ -804,7 +806,9 @@ class UnifiedJobSerializer(BaseSerializer):
                td = now() - obj.started
                ret['elapsed'] = (td.microseconds + (td.seconds + td.days * 24 * 3600) * 10 ** 6) / (10 ** 6 * 1.0)
            ret['elapsed'] = float(ret['elapsed'])
-
+        # Because this string is saved in the db in the source language,
+        # it must be marked for translation after it is pulled from the db, not when set
+        ret['job_explanation'] = _(obj.job_explanation)
        return ret


@@ -884,6 +888,9 @@ class UserSerializer(BaseSerializer):
        fields = ('*', '-name', '-description', '-modified',
                  'username', 'first_name', 'last_name',
                  'email', 'is_superuser', 'is_system_auditor', 'password', 'ldap_dn', 'last_login', 'external_account')
+        extra_kwargs = {
+            'last_login': {'read_only': True}
+        }

    def to_representation(self, obj):
        ret = super(UserSerializer, self).to_representation(obj)
@@ -1246,6 +1253,7 @@ class OrganizationSerializer(BaseSerializer):
        res.update(dict(
            projects    = self.reverse('api:organization_projects_list',       kwargs={'pk': obj.pk}),
            inventories = self.reverse('api:organization_inventories_list',    kwargs={'pk': obj.pk}),
+            job_templates = self.reverse('api:organization_job_templates_list', kwargs={'pk': obj.pk}),
            workflow_job_templates = self.reverse('api:organization_workflow_job_templates_list', kwargs={'pk': obj.pk}),
            users       = self.reverse('api:organization_users_list',          kwargs={'pk': obj.pk}),
            admins      = self.reverse('api:organization_admins_list',         kwargs={'pk': obj.pk}),
@@ -1274,6 +1282,14 @@ class OrganizationSerializer(BaseSerializer):
                    'job_templates': 0, 'admins': 0, 'projects': 0}
            else:
                summary_dict['related_field_counts'] = counts_dict[obj.id]
+
+        # Organization participation roles (admin, member) can't be assigned
+        # to a team. This provides a hint to the ui so it can know to not
+        # display these roles for team role selection.
+        for key in ('admin_role', 'member_role',):
+            if key in summary_dict.get('object_roles', {}):
+                summary_dict['object_roles'][key]['user_only'] = True
+
        return summary_dict

    def validate(self, attrs):
@@ -1387,12 +1403,6 @@ class ProjectSerializer(UnifiedJobTemplateSerializer, ProjectOptionsSerializer):
        def get_field_from_model_or_attrs(fd):
            return attrs.get(fd, self.instance and getattr(self.instance, fd) or None)

-        organization = None
-        if 'organization' in attrs:
-            organization = attrs['organization']
-        elif self.instance:
-            organization = self.instance.organization
-
        if 'allow_override' in attrs and self.instance:
            # case where user is turning off this project setting
            if self.instance.allow_override and not attrs['allow_override']:
@@ -1408,11 +1418,7 @@ class ProjectSerializer(UnifiedJobTemplateSerializer, ProjectOptionsSerializer):
                            ' '.join([str(pk) for pk in used_by])
                        )})

-        view = self.context.get('view', None)
-        if not organization and not view.request.user.is_superuser:
-            # Only allow super users to create orgless projects
-            raise serializers.ValidationError(_('Organization is missing'))
-        elif get_field_from_model_or_attrs('scm_type') == '':
+        if get_field_from_model_or_attrs('scm_type') == '':
            for fd in ('scm_update_on_launch', 'scm_delete_on_update', 'scm_clean'):
                if get_field_from_model_or_attrs(fd):
                    raise serializers.ValidationError({fd: _('Update options must be set to false for manual projects.')})
@@ -1600,7 +1606,7 @@ class InventorySerializer(BaseSerializerWithVariables):
                        })
                SmartFilter().query_from_string(host_filter)
            except RuntimeError as e:
-                raise models.base.ValidationError(e)
+                raise models.base.ValidationError(str(e))
        return host_filter

    def validate(self, attrs):
@@ -2030,11 +2036,6 @@ class InventorySourceSerializer(UnifiedJobTemplateSerializer, InventorySourceOpt
            res['credentials'] = self.reverse('api:inventory_source_credentials_list', kwargs={'pk': obj.pk})
        return res

-    def get_group(self, obj):  # TODO: remove in 3.3
-        if obj.deprecated_group:
-            return obj.deprecated_group.id
-        return None
-
    def build_relational_field(self, field_name, relation_info):
        field_class, field_kwargs = super(InventorySourceSerializer, self).build_relational_field(field_name, relation_info)
        # SCM Project and inventory are read-only unless creating a new inventory.
@@ -2115,7 +2116,13 @@ class InventorySourceSerializer(UnifiedJobTemplateSerializer, InventorySourceOpt
        def get_field_from_model_or_attrs(fd):
            return attrs.get(fd, self.instance and getattr(self.instance, fd) or None)

-        if get_field_from_model_or_attrs('source') != 'scm':
+        if get_field_from_model_or_attrs('source') == 'scm':
+            if (('source' in attrs or 'source_project' in attrs) and
+                    get_field_from_model_or_attrs('source_project') is None):
+                raise serializers.ValidationError(
+                    {"source_project": _("Project required for scm type sources.")}
+                )
+        else:
            redundant_scm_fields = list(filter(
                lambda x: attrs.get(x, None),
                ['source_project', 'source_path', 'update_on_project_update']
@@ -2301,6 +2308,7 @@ class RoleSerializer(BaseSerializer):
            content_model = obj.content_type.model_class()
            ret['summary_fields']['resource_type'] = get_type_for_model(content_model)
            ret['summary_fields']['resource_type_display_name'] = content_model._meta.verbose_name.title()
+            ret['summary_fields']['resource_id'] = obj.object_id

        return ret

@@ -2636,9 +2644,17 @@ class CredentialSerializerCreate(CredentialSerializer):
                    owner_fields.add(field)
                else:
                    attrs.pop(field)
+
        if not owner_fields:
            raise serializers.ValidationError({"detail": _("Missing 'user', 'team', or 'organization'.")})

+        if len(owner_fields) > 1:
+            received = ", ".join(sorted(owner_fields))
+            raise serializers.ValidationError({"detail": _(
+                "Only one of 'user', 'team', or 'organization' should be provided, "
+                "received {} fields.".format(received)
+            )})
+
        if attrs.get('team'):
            attrs['organization'] = attrs['team'].organization

@@ -2732,7 +2748,8 @@ class JobOptionsSerializer(LabelsListMixin, BaseSerializer):
        fields = ('*', 'job_type', 'inventory', 'project', 'playbook', 'scm_branch',
                  'forks', 'limit', 'verbosity', 'extra_vars', 'job_tags',
                  'force_handlers', 'skip_tags', 'start_at_task', 'timeout',
-                  'use_fact_cache',)
+                  'use_fact_cache', 'organization',)
+        read_only_fields = ('organization',)

    def get_related(self, obj):
        res = super(JobOptionsSerializer, self).get_related(obj)
@@ -2747,17 +2764,14 @@ class JobOptionsSerializer(LabelsListMixin, BaseSerializer):
                res['project'] = self.reverse('api:project_detail', kwargs={'pk': obj.project.pk})
        except ObjectDoesNotExist:
            setattr(obj, 'project', None)
+        if obj.organization_id:
+            res['organization'] = self.reverse('api:organization_detail', kwargs={'pk': obj.organization_id})
        if isinstance(obj, UnifiedJobTemplate):
-            res['extra_credentials'] = self.reverse(
-                'api:job_template_extra_credentials_list',
-                kwargs={'pk': obj.pk}
-            )
            res['credentials'] = self.reverse(
                'api:job_template_credentials_list',
                kwargs={'pk': obj.pk}
            )
        elif isinstance(obj, UnifiedJob):
-            res['extra_credentials'] = self.reverse('api:job_extra_credentials_list', kwargs={'pk': obj.pk})
            res['credentials'] = self.reverse('api:job_credentials_list', kwargs={'pk': obj.pk})

        return res
@@ -2893,6 +2907,10 @@ class JobTemplateSerializer(JobTemplateMixin, UnifiedJobTemplateSerializer, JobO
        )
        if obj.host_config_key:
            res['callback'] = self.reverse('api:job_template_callback', kwargs={'pk': obj.pk})
+        if obj.organization_id:
+            res['organization'] = self.reverse('api:organization_detail',   kwargs={'pk': obj.organization_id})
+        if obj.webhook_credential_id:
+            res['webhook_credential'] = self.reverse('api:credential_detail', kwargs={'pk': obj.webhook_credential_id})
        return res

    def validate(self, attrs):
@@ -2922,7 +2940,6 @@ class JobTemplateSerializer(JobTemplateMixin, UnifiedJobTemplateSerializer, JobO
        summary_fields = super(JobTemplateSerializer, self).get_summary_fields(obj)
        all_creds = []
        # Organize credential data into multitude of deprecated fields
-        extra_creds = []
        if obj.pk:
            for cred in obj.credentials.all():
                summarized_cred = {
@@ -2933,10 +2950,6 @@ class JobTemplateSerializer(JobTemplateMixin, UnifiedJobTemplateSerializer, JobO
                    'cloud': cred.credential_type.kind == 'cloud'
                }
                all_creds.append(summarized_cred)
-                if cred.credential_type.kind in ('cloud', 'net'):
-                    extra_creds.append(summarized_cred)
-        if self.is_detail_view:
-            summary_fields['extra_credentials'] = extra_creds
        summary_fields['credentials'] = all_creds
        return summary_fields

@@ -3011,7 +3024,6 @@ class JobSerializer(UnifiedJobSerializer, JobOptionsSerializer):
        summary_fields = super(JobSerializer, self).get_summary_fields(obj)
        all_creds = []
        # Organize credential data into multitude of deprecated fields
-        extra_creds = []
        if obj.pk:
            for cred in obj.credentials.all():
                summarized_cred = {
@@ -3022,10 +3034,6 @@ class JobSerializer(UnifiedJobSerializer, JobOptionsSerializer):
                    'cloud': cred.credential_type.kind == 'cloud'
                }
                all_creds.append(summarized_cred)
-                if cred.credential_type.kind in ('cloud', 'net'):
-                    extra_creds.append(summarized_cred)
-        if self.is_detail_view:
-            summary_fields['extra_credentials'] = extra_creds
        summary_fields['credentials'] = all_creds
        return summary_fields

@@ -3198,7 +3206,7 @@ class AdHocCommandSerializer(UnifiedJobSerializer):
            field_kwargs['choices'] = module_name_choices
            field_kwargs['required'] = bool(not module_name_default)
            field_kwargs['default'] = module_name_default or serializers.empty
-            field_kwargs['allow_blank'] = bool(module_name_default)
+            field_kwargs['allow_blank'] = False
            field_kwargs.pop('max_length', None)
        return field_class, field_kwargs

@@ -3383,6 +3391,8 @@ class WorkflowJobTemplateSerializer(JobTemplateMixin, LabelsListMixin, UnifiedJo
        )
        if obj.organization:
            res['organization'] = self.reverse('api:organization_detail',   kwargs={'pk': obj.organization.pk})
+        if obj.webhook_credential_id:
+            res['webhook_credential'] = self.reverse('api:credential_detail', kwargs={'pk': obj.webhook_credential_id})
        return res

    def validate_extra_vars(self, value):
@@ -3597,9 +3607,11 @@ class LaunchConfigurationBaseSerializer(BaseSerializer):
        elif self.instance:
            ujt = self.instance.unified_job_template
        if ujt is None:
-            if 'workflow_job_template' in attrs:
-                return {'workflow_job_template': attrs['workflow_job_template']}
-            return {}
+            ret = {}
+            for fd in ('workflow_job_template', 'identifier', 'all_parents_must_converge'):
+                if fd in attrs:
+                    ret[fd] = attrs[fd]
+            return ret

        # build additional field survey_passwords to track redacted variables
        password_dict = {}
@@ -3652,7 +3664,7 @@ class LaunchConfigurationBaseSerializer(BaseSerializer):
                        attrs.get('survey_passwords', {}).pop(key, None)
                    else:
                        errors.setdefault('extra_vars', []).append(
-                            _('"$encrypted$ is a reserved keyword, may not be used for {var_name}."'.format(key))
+                            _('"$encrypted$ is a reserved keyword, may not be used for {}."'.format(key))
                        )

        # Launch configs call extra_vars extra_data for historical reasons
@@ -3677,7 +3689,8 @@ class WorkflowJobTemplateNodeSerializer(LaunchConfigurationBaseSerializer):
    class Meta:
        model = WorkflowJobTemplateNode
        fields = ('*', 'workflow_job_template', '-name', '-description', 'id', 'url', 'related',
-                  'unified_job_template', 'success_nodes', 'failure_nodes', 'always_nodes', 'all_parents_must_converge',)
+                  'unified_job_template', 'success_nodes', 'failure_nodes', 'always_nodes', 'all_parents_must_converge',
+                  'identifier',)

    def get_related(self, obj):
        res = super(WorkflowJobTemplateNodeSerializer, self).get_related(obj)
@@ -3716,8 +3729,8 @@ class WorkflowJobNodeSerializer(LaunchConfigurationBaseSerializer):
    class Meta:
        model = WorkflowJobNode
        fields = ('*', 'job', 'workflow_job', '-name', '-description', 'id', 'url', 'related',
-                  'unified_job_template', 'success_nodes', 'failure_nodes', 'always_nodes', 
-                  'all_parents_must_converge', 'do_not_run',)
+                  'unified_job_template', 'success_nodes', 'failure_nodes', 'always_nodes',
+                  'all_parents_must_converge', 'do_not_run', 'identifier')

    def get_related(self, obj):
        res = super(WorkflowJobNodeSerializer, self).get_related(obj)
@@ -3882,15 +3895,23 @@ class ProjectUpdateEventSerializer(JobEventSerializer):
        return UriCleaner.remove_sensitive(obj.stdout)

    def get_event_data(self, obj):
-        try:
-            return json.loads(
-                UriCleaner.remove_sensitive(
-                    json.dumps(obj.event_data)
+        # the project update playbook uses the git, hg, or svn modules
+        # to clone repositories, and those modules are prone to printing
+        # raw SCM URLs in their stdout (which *could* contain passwords)
+        # attempt to detect and filter HTTP basic auth passwords in the stdout
+        # of these types of events
+        if obj.event_data.get('task_action') in ('git', 'hg', 'svn'):
+            try:
+                return json.loads(
+                    UriCleaner.remove_sensitive(
+                        json.dumps(obj.event_data)
+                    )
                )
-            )
-        except Exception:
-            logger.exception("Failed to sanitize event_data")
-            return {}
+            except Exception:
+                logger.exception("Failed to sanitize event_data")
+                return {}
+        else:
+            return obj.event_data


 class AdHocCommandEventSerializer(BaseSerializer):
@@ -4048,6 +4069,13 @@ class JobLaunchSerializer(BaseSerializer):
            **attrs)
        self._ignored_fields = rejected

+        # Basic validation - cannot run a playbook without a playbook
+        if not template.project:
+            errors['project'] = _("A project is required to run a job.")
+        elif template.project.status in ('error', 'failed'):
+            errors['playbook'] = _("Missing a revision to run due to failed project update.")
+
+        # cannot run a playbook without an inventory
        if template.inventory and template.inventory.pending_deletion is True:
            errors['inventory'] = _("The inventory associated with this Job Template is being deleted.")
        elif 'inventory' in accepted and accepted['inventory'].pending_deletion:
@@ -4512,6 +4540,8 @@ class SchedulePreviewSerializer(BaseSerializer):
        try:
            Schedule.rrulestr(rrule_value)
        except Exception as e:
+            import traceback
+            logger.error(traceback.format_exc())
            raise serializers.ValidationError(_("rrule parsing failed validation: {}").format(e))
        return value

--- a/awx/api/templates/api/setting_logging_test.md
+++ b/awx/api/templates/api/setting_logging_test.md
@@ -1 +1,2 @@
 # Test Logging Configuration
+
--- a/awx/api/urls/oauth2_root.py
+++ b/awx/api/urls/oauth2_root.py
@@ -1,11 +1,15 @@
 # Copyright (c) 2017 Ansible, Inc.
 # All Rights Reserved.
+from datetime import timedelta

+from django.utils.timezone import now
+from django.conf import settings
 from django.conf.urls import url

 from oauthlib import oauth2
 from oauth2_provider import views

+from awx.main.models import RefreshToken
 from awx.api.views import (
    ApiOAuthAuthorizationRootView,
 )
@@ -14,6 +18,21 @@ from awx.api.views import (
 class TokenView(views.TokenView):

    def create_token_response(self, request):
+        # Django OAuth2 Toolkit has a bug whereby refresh tokens are *never*
+        # properly expired (ugh):
+        #
+        # https://github.com/jazzband/django-oauth-toolkit/issues/746
+        #
+        # This code detects and auto-expires them on refresh grant
+        # requests.
+        if request.POST.get('grant_type') == 'refresh_token' and 'refresh_token' in request.POST:
+            refresh_token = RefreshToken.objects.filter(
+                token=request.POST['refresh_token']
+            ).first()
+            if refresh_token:
+                expire_seconds = settings.OAUTH2_PROVIDER.get('REFRESH_TOKEN_EXPIRE_SECONDS', 0)
+                if refresh_token.created + timedelta(seconds=expire_seconds) < now():
+                    return request.build_absolute_uri(), {}, 'The refresh token has expired.', '403'
        try:
            return super(TokenView, self).create_token_response(request)
        except oauth2.AccessDeniedError as e:
--- a/awx/api/urls/organization.py
+++ b/awx/api/urls/organization.py
@@ -10,6 +10,7 @@ from awx.api.views import (
    OrganizationAdminsList,
    OrganizationInventoriesList,
    OrganizationProjectsList,
+    OrganizationJobTemplatesList,
    OrganizationWorkflowJobTemplatesList,
    OrganizationTeamsList,
    OrganizationCredentialList,
@@ -33,6 +34,7 @@ urls = [
    url(r'^(?P<pk>[0-9]+)/admins/$', OrganizationAdminsList.as_view(), name='organization_admins_list'),
    url(r'^(?P<pk>[0-9]+)/inventories/$', OrganizationInventoriesList.as_view(), name='organization_inventories_list'),
    url(r'^(?P<pk>[0-9]+)/projects/$', OrganizationProjectsList.as_view(), name='organization_projects_list'),
+    url(r'^(?P<pk>[0-9]+)/job_templates/$', OrganizationJobTemplatesList.as_view(), name='organization_job_templates_list'),
    url(r'^(?P<pk>[0-9]+)/workflow_job_templates/$', OrganizationWorkflowJobTemplatesList.as_view(), name='organization_workflow_job_templates_list'),
    url(r'^(?P<pk>[0-9]+)/teams/$', OrganizationTeamsList.as_view(), name='organization_teams_list'),
    url(r'^(?P<pk>[0-9]+)/credentials/$', OrganizationCredentialList.as_view(), name='organization_credential_list'),
--- a/awx/api/urls/urls.py
+++ b/awx/api/urls/urls.py
@@ -23,9 +23,7 @@ from awx.api.views import (
    UnifiedJobList,
    HostAnsibleFactsDetail,
    JobCredentialsList,
-    JobExtraCredentialsList,
    JobTemplateCredentialsList,
-    JobTemplateExtraCredentialsList,
    SchedulePreview,
    ScheduleZoneInfo,
    OAuth2ApplicationList,
@@ -34,7 +32,9 @@ from awx.api.views import (
    OAuth2ApplicationDetail,
 )

-from awx.api.views.metrics import MetricsView
+from awx.api.views.metrics import (
+    MetricsView,
+)

 from .organization import urls as organization_urls
 from .user import urls as user_urls
@@ -81,9 +81,7 @@ v2_urls = [
    url(r'^credential_types/', include(credential_type_urls)),
    url(r'^credential_input_sources/', include(credential_input_source_urls)),
    url(r'^hosts/(?P<pk>[0-9]+)/ansible_facts/$', HostAnsibleFactsDetail.as_view(), name='host_ansible_facts_detail'),
-    url(r'^jobs/(?P<pk>[0-9]+)/extra_credentials/$', JobExtraCredentialsList.as_view(), name='job_extra_credentials_list'),
    url(r'^jobs/(?P<pk>[0-9]+)/credentials/$', JobCredentialsList.as_view(), name='job_credentials_list'),
-    url(r'^job_templates/(?P<pk>[0-9]+)/extra_credentials/$', JobTemplateExtraCredentialsList.as_view(), name='job_template_extra_credentials_list'),
    url(r'^job_templates/(?P<pk>[0-9]+)/credentials/$', JobTemplateCredentialsList.as_view(), name='job_template_credentials_list'),
    url(r'^schedules/preview/$', SchedulePreview.as_view(), name='schedule_rrule'),
    url(r'^schedules/zoneinfo/$', ScheduleZoneInfo.as_view(), name='schedule_zoneinfo'),
--- a/awx/api/views/init.py
+++ b/awx/api/views/init.py
@@ -12,7 +12,7 @@ import socket
 import sys
 import time
 from base64 import b64encode
-from collections import OrderedDict, Iterable
+from collections import OrderedDict


 # Django
@@ -111,6 +111,7 @@ from awx.api.views.organization import ( # noqa
    OrganizationUsersList,
    OrganizationAdminsList,
    OrganizationProjectsList,
+    OrganizationJobTemplatesList,
    OrganizationWorkflowJobTemplatesList,
    OrganizationTeamsList,
    OrganizationActivityStreamList,
@@ -1091,7 +1092,7 @@ class UserRolesList(SubListAttachDetachAPIView):

        credential_content_type = ContentType.objects.get_for_model(models.Credential)
        if role.content_type == credential_content_type:
-            if role.content_object.organization and user not in role.content_object.organization.member_role:
+            if 'disassociate' not in request.data and role.content_object.organization and user not in role.content_object.organization.member_role:
                data = dict(msg=_("You cannot grant credential access to a user not in the credentials' organization"))
                return Response(data, status=status.HTTP_400_BAD_REQUEST)

@@ -2336,70 +2337,24 @@ class JobTemplateLaunch(RetrieveAPIView):
        old field structure to launch endpoint
        TODO: delete this method with future API version changes
        '''
-        ignored_fields = {}
        modern_data = data.copy()

        id_fd = '{}_id'.format('inventory')
        if 'inventory' not in modern_data and id_fd in modern_data:
            modern_data['inventory'] = modern_data[id_fd]

-        # Automatically convert legacy launch credential arguments into a list of `.credentials`
-        if 'credentials' in modern_data and 'extra_credentials' in modern_data:
-            raise ParseError({"error": _(
-                "'credentials' cannot be used in combination with 'extra_credentials'."
-            )})
-
-        if 'extra_credentials' in modern_data:
-            # make a list of the current credentials
-            existing_credentials = obj.credentials.all()
-            template_credentials = list(existing_credentials)  # save copy of existing
-            new_credentials = []
-            if 'extra_credentials' in modern_data:
-                existing_credentials = [
-                    cred for cred in existing_credentials
-                    if cred.credential_type.kind not in ('cloud', 'net')
-                ]
-                prompted_value = modern_data.pop('extra_credentials')
-
-                # validate type, since these are not covered by a serializer
-                if not isinstance(prompted_value, Iterable):
-                    msg = _(
-                        "Incorrect type. Expected a list received {}."
-                    ).format(prompted_value.__class__.__name__)
-                    raise ParseError({'extra_credentials': [msg], 'credentials': [msg]})
-
-                # add the deprecated credential specified in the request
-                if not isinstance(prompted_value, Iterable) or isinstance(prompted_value, str):
-                    prompted_value = [prompted_value]
-
-                # If user gave extra_credentials, special case to use exactly
-                # the given list without merging with JT credentials
-                if prompted_value:
-                    obj._deprecated_credential_launch = True  # signal to not merge credentials
-                new_credentials.extend(prompted_value)
-
-            # combine the list of "new" and the filtered list of "old"
-            new_credentials.extend([cred.pk for cred in existing_credentials])
-            if new_credentials:
-                # If provided list doesn't contain the pre-existing credentials
-                # defined on the template, add them back here
-                for cred_obj in template_credentials:
-                    if cred_obj.pk not in new_credentials:
-                        new_credentials.append(cred_obj.pk)
-                modern_data['credentials'] = new_credentials
-
        # credential passwords were historically provided as top-level attributes
        if 'credential_passwords' not in modern_data:
            modern_data['credential_passwords'] = data.copy()

-        return (modern_data, ignored_fields)
+        return modern_data


    def post(self, request, *args, **kwargs):
        obj = self.get_object()

        try:
-            modern_data, ignored_fields = self.modernize_launch_payload(
+            modern_data = self.modernize_launch_payload(
                data=request.data, obj=obj
            )
        except ParseError as exc:
@@ -2409,8 +2364,6 @@ class JobTemplateLaunch(RetrieveAPIView):
        if not serializer.is_valid():
            return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)

-        ignored_fields.update(serializer._ignored_fields)
-
        if not request.user.can_access(models.JobLaunchConfig, 'add', serializer.validated_data, template=obj):
            raise PermissionDenied()

@@ -2426,11 +2379,11 @@ class JobTemplateLaunch(RetrieveAPIView):
            data = OrderedDict()
            if isinstance(new_job, models.WorkflowJob):
                data['workflow_job'] = new_job.id
-                data['ignored_fields'] = self.sanitize_for_response(ignored_fields)
+                data['ignored_fields'] = self.sanitize_for_response(serializer._ignored_fields)
                data.update(serializers.WorkflowJobSerializer(new_job, context=self.get_serializer_context()).to_representation(new_job))
            else:
                data['job'] = new_job.id
-                data['ignored_fields'] = self.sanitize_for_response(ignored_fields)
+                data['ignored_fields'] = self.sanitize_for_response(serializer._ignored_fields)
                data.update(serializers.JobSerializer(new_job, context=self.get_serializer_context()).to_representation(new_job))
            headers = {'Location': new_job.get_absolute_url(request)}
            return Response(data, status=status.HTTP_201_CREATED, headers=headers)
@@ -2710,22 +2663,6 @@ class JobTemplateCredentialsList(SubListCreateAttachDetachAPIView):
        return super(JobTemplateCredentialsList, self).is_valid_relation(parent, sub, created)


-class JobTemplateExtraCredentialsList(JobTemplateCredentialsList):
-
-    deprecated = True
-
-    def get_queryset(self):
-        sublist_qs = super(JobTemplateExtraCredentialsList, self).get_queryset()
-        sublist_qs = sublist_qs.filter(credential_type__kind__in=['cloud', 'net'])
-        return sublist_qs
-
-    def is_valid_relation(self, parent, sub, created=False):
-        valid = super(JobTemplateExtraCredentialsList, self).is_valid_relation(parent, sub, created)
-        if sub.credential_type.kind not in ('cloud', 'net'):
-            return {'error': _('Extra credentials must be network or cloud.')}
-        return valid
-
-
 class JobTemplateLabelList(DeleteLastUnattachLabelMixin, SubListCreateAttachDetachAPIView):

    model = models.Label
@@ -3542,16 +3479,6 @@ class JobCredentialsList(SubListAPIView):
    relationship = 'credentials'


-class JobExtraCredentialsList(JobCredentialsList):
-
-    deprecated = True
-
-    def get_queryset(self):
-        sublist_qs = super(JobExtraCredentialsList, self).get_queryset()
-        sublist_qs = sublist_qs.filter(credential_type__kind__in=['cloud', 'net'])
-        return sublist_qs
-
-
 class JobLabelList(SubListAPIView):

    model = models.Label
@@ -4303,7 +4230,7 @@ class NotificationTemplateTest(GenericAPIView):
        msg = "Tower Notification Test {} {}".format(obj.id, settings.TOWER_URL_BASE)
        if obj.notification_type in ('email', 'pagerduty'):
            body = "Ansible Tower Test Notification {} {}".format(obj.id, settings.TOWER_URL_BASE)
-        elif obj.notification_type == 'webhook':
+        elif obj.notification_type in ('webhook', 'grafana'):
            body = '{{"body": "Ansible Tower Test Notification {} {}"}}'.format(obj.id, settings.TOWER_URL_BASE)
        else:
            body = {"body": "Ansible Tower Test Notification {} {}".format(obj.id, settings.TOWER_URL_BASE)}
@@ -4414,7 +4341,7 @@ class RoleUsersList(SubListAttachDetachAPIView):

        credential_content_type = ContentType.objects.get_for_model(models.Credential)
        if role.content_type == credential_content_type:
-            if role.content_object.organization and user not in role.content_object.organization.member_role:
+            if 'disassociate' not in request.data and role.content_object.organization and user not in role.content_object.organization.member_role:
                data = dict(msg=_("You cannot grant credential access to a user not in the credentials' organization"))
                return Response(data, status=status.HTTP_400_BAD_REQUEST)

--- a/awx/api/views/metrics.py
+++ b/awx/api/views/metrics.py
@@ -39,3 +39,4 @@ class MetricsView(APIView):
        if (request.user.is_superuser or request.user.is_system_auditor):
            return Response(metrics().decode('UTF-8'))
        raise PermissionDenied()
+
--- a/awx/api/views/mixin.py
+++ b/awx/api/views/mixin.py
@@ -4,10 +4,7 @@
 import dateutil
 import logging

-from django.db.models import (
-    Count,
-    F,
-)
+from django.db.models import Count
 from django.db import transaction
 from django.shortcuts import get_object_or_404
 from django.utils.timezone import now
@@ -175,28 +172,18 @@ class OrganizationCountsMixin(object):

        inv_qs = Inventory.accessible_objects(self.request.user, 'read_role')
        project_qs = Project.accessible_objects(self.request.user, 'read_role')
+        jt_qs = JobTemplate.accessible_objects(self.request.user, 'read_role')

        # Produce counts of Foreign Key relationships
-        db_results['inventories'] = inv_qs\
-            .values('organization').annotate(Count('organization')).order_by('organization')
+        db_results['inventories'] = inv_qs.values('organization').annotate(Count('organization')).order_by('organization')

        db_results['teams'] = Team.accessible_objects(
            self.request.user, 'read_role').values('organization').annotate(
            Count('organization')).order_by('organization')

-        JT_project_reference = 'project__organization'
-        JT_inventory_reference = 'inventory__organization'
-        db_results['job_templates_project'] = JobTemplate.accessible_objects(
-            self.request.user, 'read_role').exclude(
-            project__organization=F(JT_inventory_reference)).values(JT_project_reference).annotate(
-            Count(JT_project_reference)).order_by(JT_project_reference)
+        db_results['job_templates'] = jt_qs.values('organization').annotate(Count('organization')).order_by('organization')

-        db_results['job_templates_inventory'] = JobTemplate.accessible_objects(
-            self.request.user, 'read_role').values(JT_inventory_reference).annotate(
-            Count(JT_inventory_reference)).order_by(JT_inventory_reference)
-
-        db_results['projects'] = project_qs\
-            .values('organization').annotate(Count('organization')).order_by('organization')
+        db_results['projects'] = project_qs.values('organization').annotate(Count('organization')).order_by('organization')

        # Other members and admins of organization are always viewable
        db_results['users'] = org_qs.annotate(
@@ -212,11 +199,7 @@ class OrganizationCountsMixin(object):
                'admins': 0, 'projects': 0}

        for res, count_qs in db_results.items():
-            if res == 'job_templates_project':
-                org_reference = JT_project_reference
-            elif res == 'job_templates_inventory':
-                org_reference = JT_inventory_reference
-            elif res == 'users':
+            if res == 'users':
                org_reference = 'id'
            else:
                org_reference = 'organization'
@@ -229,14 +212,6 @@ class OrganizationCountsMixin(object):
                        continue
                    count_context[org_id][res] = entry['%s__count' % org_reference]

-        # Combine the counts for job templates by project and inventory
-        for org in org_id_list:
-            org_id = org['id']
-            count_context[org_id]['job_templates'] = 0
-            for related_path in ['job_templates_project', 'job_templates_inventory']:
-                if related_path in count_context[org_id]:
-                    count_context[org_id]['job_templates'] += count_context[org_id].pop(related_path)
-
        full_context['related_field_counts'] = count_context

        return full_context
--- a/awx/api/views/organization.py
+++ b/awx/api/views/organization.py
@@ -20,7 +20,7 @@ from awx.main.models import (
    Role,
    User,
    Team,
-    InstanceGroup,
+    InstanceGroup
 )
 from awx.api.generics import (
    ListCreateAPIView,
@@ -28,6 +28,7 @@ from awx.api.generics import (
    SubListAPIView,
    SubListCreateAttachDetachAPIView,
    SubListAttachDetachAPIView,
+    SubListCreateAPIView,
    ResourceAccessList,
    BaseUsersList,
 )
@@ -35,14 +36,13 @@ from awx.api.generics import (
 from awx.api.serializers import (
    OrganizationSerializer,
    InventorySerializer,
-    ProjectSerializer,
    UserSerializer,
    TeamSerializer,
    ActivityStreamSerializer,
    RoleSerializer,
    NotificationTemplateSerializer,
-    WorkflowJobTemplateSerializer,
    InstanceGroupSerializer,
+    ProjectSerializer, JobTemplateSerializer, WorkflowJobTemplateSerializer
 )
 from awx.api.views.mixin import (
    RelatedJobsPreventDeleteMixin,
@@ -94,7 +94,7 @@ class OrganizationDetail(RelatedJobsPreventDeleteMixin, RetrieveUpdateDestroyAPI
        org_counts['projects'] = Project.accessible_objects(**access_kwargs).filter(
            organization__id=org_id).count()
        org_counts['job_templates'] = JobTemplate.accessible_objects(**access_kwargs).filter(
-            project__organization__id=org_id).count()
+            organization__id=org_id).count()

        full_context['related_field_counts'] = {}
        full_context['related_field_counts'][org_id] = org_counts
@@ -128,21 +128,27 @@ class OrganizationAdminsList(BaseUsersList):
    ordering = ('username',)


-class OrganizationProjectsList(SubListCreateAttachDetachAPIView):
+class OrganizationProjectsList(SubListCreateAPIView):

    model = Project
    serializer_class = ProjectSerializer
    parent_model = Organization
-    relationship = 'projects'
    parent_key = 'organization'


-class OrganizationWorkflowJobTemplatesList(SubListCreateAttachDetachAPIView):
+class OrganizationJobTemplatesList(SubListCreateAPIView):
+
+    model = JobTemplate
+    serializer_class = JobTemplateSerializer
+    parent_model = Organization
+    parent_key = 'organization'
+
+
+class OrganizationWorkflowJobTemplatesList(SubListCreateAPIView):

    model = WorkflowJobTemplate
    serializer_class = WorkflowJobTemplateSerializer
    parent_model = Organization
-    relationship = 'workflows'
    parent_key = 'organization'


--- a/awx/asgi.py
+++ b/awx/asgi.py
@@ -2,14 +2,14 @@
 # All Rights Reserved.
 import os
 import logging
+import django
 from awx import __version__ as tower_version
-
 # Prepare the AWX environment.
 from awx import prepare_env, MODE
+from channels.routing import get_default_application  # noqa
 prepare_env() # NOQA

-from django.core.wsgi import get_wsgi_application  # NOQA
-from channels.asgi import get_channel_layer
+

 """
 ASGI config for AWX project.
@@ -32,6 +32,5 @@ if MODE == 'production':


 os.environ.setdefault("DJANGO_SETTINGS_MODULE", "awx.settings")
-
-
-channel_layer = get_channel_layer()
+django.setup()
+channel_layer = get_default_application()
--- a/awx/conf/fields.py
+++ b/awx/conf/fields.py
@@ -172,9 +172,9 @@ class URLField(CharField):
                        netloc = '{}:{}'.format(netloc, url_parts.port)
                    if url_parts.username:
                        if url_parts.password:
-                            netloc = '{}:{}@{}' % (url_parts.username, url_parts.password, netloc)
+                            netloc = '{}:{}@{}'.format(url_parts.username, url_parts.password, netloc)
                        else:
-                            netloc = '{}@{}' % (url_parts.username, netloc)
+                            netloc = '{}@{}'.format(url_parts.username, netloc)
                    value = urlparse.urlunsplit([url_parts.scheme, netloc, url_parts.path, url_parts.query, url_parts.fragment])
            except Exception:
                raise  # If something fails here, just fall through and let the validators check it.
--- a/awx/conf/settings.py
+++ b/awx/conf/settings.py
@@ -1,11 +1,9 @@
 # Python
 import contextlib
 import logging
-import re
 import sys
 import threading
 import time
-import urllib.parse

 # Django
 from django.conf import LazySettings
@@ -13,7 +11,7 @@ from django.conf import settings, UserSettingsHolder
 from django.core.cache import cache as django_cache
 from django.core.exceptions import ImproperlyConfigured
 from django.db import transaction, connection
-from django.db.utils import Error as DBError
+from django.db.utils import Error as DBError, ProgrammingError
 from django.utils.functional import cached_property

 # Django REST Framework
@@ -33,18 +31,18 @@ logger = logging.getLogger('awx.conf.settings')
 # Store a special value to indicate when a setting is not set in the database.
 SETTING_CACHE_NOTSET = '___notset___'

-# Cannot store None in memcached; use a special value instead to indicate None.
+# Cannot store None in cache; use a special value instead to indicate None.
 # If the special value for None is the same as the "not set" value, then a value
 # of None will be equivalent to the setting not being set (and will raise an
 # AttributeError if there is no other default defined).
 # SETTING_CACHE_NONE = '___none___'
 SETTING_CACHE_NONE = SETTING_CACHE_NOTSET

-# Cannot store empty list/tuple in memcached; use a special value instead to
+# Cannot store empty list/tuple in cache; use a special value instead to
 # indicate an empty list.
 SETTING_CACHE_EMPTY_LIST = '___[]___'

-# Cannot store empty dict in memcached; use a special value instead to indicate
+# Cannot store empty dict in cache; use a special value instead to indicate
 # an empty dict.
 SETTING_CACHE_EMPTY_DICT = '___{}___'

@@ -57,15 +55,6 @@ SETTING_CACHE_DEFAULTS = True
 __all__ = ['SettingsWrapper', 'get_settings_to_cache', 'SETTING_CACHE_NOTSET']


-def normalize_broker_url(value):
-    parts = value.rsplit('@', 1)
-    match = re.search('(amqp://[^:]+:)(.*)', parts[0])
-    if match:
-        prefix, password = match.group(1), match.group(2)
-        parts[0] = prefix + urllib.parse.quote(password)
-    return '@'.join(parts)
-
-
@contextlib.contextmanager
 def _ctit_db_wrapper(trans_safe=False):
    '''
@@ -85,10 +74,19 @@ def _ctit_db_wrapper(trans_safe=False):
                    logger.debug('Obtaining database settings in spite of broken transaction.')
                    transaction.set_rollback(False)
        yield
-    except DBError:
+    except DBError as exc:
        if trans_safe:
            if 'migrate' not in sys.argv and 'check_migrations' not in sys.argv:
-                logger.exception('Database settings are not available, using defaults.')
+                level = logger.exception
+                if isinstance(exc, ProgrammingError):
+                    if 'relation' in str(exc) and 'does not exist' in str(exc):
+                        # this generally means we can't fetch Tower configuration
+                        # because the database hasn't actually finished migrating yet;
+                        # this is usually a sign that a service in a container (such as ws_broadcast)
+                        # has come up *before* the database has finished migrating, and
+                        # especially that the conf.settings table doesn't exist yet
+                        level = logger.debug
+                level('Database settings are not available, using defaults.')
        else:
            logger.exception('Error modifying something related to database settings.')
    finally:
@@ -415,22 +413,13 @@ class SettingsWrapper(UserSettingsHolder):
                value = self._get_local(name)
        if value is not empty:
            return value
-        value = self._get_default(name)
-        # sometimes users specify RabbitMQ passwords that contain
-        # unescaped : and @ characters that confused urlparse, e.g.,
-        # amqp://guest:a@ns:ibl3#@localhost:5672//
-        #
-        # detect these scenarios, and automatically escape the user's
-        # password so it just works
-        if name == 'BROKER_URL':
-            value = normalize_broker_url(value)
-        return value
+        return self._get_default(name)

    def _set_local(self, name, value):
        field = self.registry.get_setting_field(name)
        if field.read_only:
            logger.warning('Attempt to set read only setting "%s".', name)
-            raise ImproperlyConfigured('Setting "%s" is read only.'.format(name))
+            raise ImproperlyConfigured('Setting "{}" is read only.'.format(name))

        try:
            data = field.to_representation(value)
@@ -461,7 +450,7 @@ class SettingsWrapper(UserSettingsHolder):
        field = self.registry.get_setting_field(name)
        if field.read_only:
            logger.warning('Attempt to delete read only setting "%s".', name)
-            raise ImproperlyConfigured('Setting "%s" is read only.'.format(name))
+            raise ImproperlyConfigured('Setting "{}" is read only.'.format(name))
        for setting in Setting.objects.filter(key=name, user__isnull=True):
            setting.delete()
            # pre_delete handler will delete from cache.
--- a/awx/conf/tests/functional/test_api.py
+++ b/awx/conf/tests/functional/test_api.py
@@ -325,17 +325,3 @@ def test_setting_singleton_delete_no_read_only_fields(api_request, dummy_setting
        )
        assert response.data['FOO_BAR'] == 23

-
-@pytest.mark.django_db
-def test_setting_logging_test(api_request):
-    with mock.patch('awx.conf.views.AWXProxyHandler.perform_test') as mock_func:
-        api_request(
-            'post',
-            reverse('api:setting_logging_test'),
-            data={'LOG_AGGREGATOR_HOST': 'http://foobar', 'LOG_AGGREGATOR_TYPE': 'logstash'}
-        )
-        call = mock_func.call_args_list[0]
-        args, kwargs = call
-        given_settings = kwargs['custom_settings']
-        assert given_settings.LOG_AGGREGATOR_HOST == 'http://foobar'
-        assert given_settings.LOG_AGGREGATOR_TYPE == 'logstash'
--- a/awx/conf/tests/unit/test_registry.py
+++ b/awx/conf/tests/unit/test_registry.py
@@ -29,9 +29,10 @@ def reg(request):
    # as "defined in a settings file".  This is analogous to manually
    # specifying a setting on the filesystem (e.g., in a local_settings.py in
    # development, or in /etc/tower/conf.d/<something>.py)
-    defaults = request.node.get_marker('defined_in_file')
-    if defaults:
-        settings.configure(**defaults.kwargs)
+    for marker in request.node.own_markers:
+        if marker.name == 'defined_in_file':
+            settings.configure(**marker.kwargs)
+
    settings._wrapped = SettingsWrapper(settings._wrapped,
                                        cache,
                                        registry)
--- a/awx/conf/tests/unit/test_settings.py
+++ b/awx/conf/tests/unit/test_settings.py
@@ -41,13 +41,16 @@ def settings(request):
    cache = LocMemCache(str(uuid4()), {})  # make a new random cache each time
    settings = LazySettings()
    registry = SettingsRegistry(settings)
+    defaults = {}

    # @pytest.mark.defined_in_file can be used to mark specific setting values
    # as "defined in a settings file".  This is analogous to manually
    # specifying a setting on the filesystem (e.g., in a local_settings.py in
    # development, or in /etc/tower/conf.d/<something>.py)
-    in_file_marker = request.node.get_marker('defined_in_file')
-    defaults = in_file_marker.kwargs if in_file_marker else {}
+    for marker in request.node.own_markers:
+        if marker.name == 'defined_in_file':
+            defaults = marker.kwargs
+
    defaults['DEFAULTS_SNAPSHOT'] = {}
    settings.configure(**defaults)
    settings._wrapped = SettingsWrapper(settings._wrapped,
@@ -63,15 +66,6 @@ def test_unregistered_setting(settings):
    assert settings.cache.get('DEBUG') is None


-def test_cached_settings_unicode_is_auto_decoded(settings):
-    # https://github.com/linsomniac/python-memcached/issues/79
-    # https://github.com/linsomniac/python-memcached/blob/288c159720eebcdf667727a859ef341f1e908308/memcache.py#L961
-
-    value = 'Iñtërnâtiônàlizætiøn'  # this simulates what python-memcached does on cache.set()
-    settings.cache.set('DEBUG', value)
-    assert settings.cache.get('DEBUG') == 'Iñtërnâtiônàlizætiøn'
-
-
 def test_read_only_setting(settings):
    settings.registry.register(
        'AWX_READ_ONLY',
@@ -251,31 +245,6 @@ def test_setting_from_db(settings, mocker):
        assert settings.cache.get('AWX_SOME_SETTING') == 'FROM_DB'


-@pytest.mark.parametrize('encrypted', (True, False))
-def test_setting_from_db_with_unicode(settings, mocker, encrypted):
-    settings.registry.register(
-        'AWX_SOME_SETTING',
-        field_class=fields.CharField,
-        category=_('System'),
-        category_slug='system',
-        default='DEFAULT',
-        encrypted=encrypted
-    )
-    # this simulates a bug in python-memcached; see https://github.com/linsomniac/python-memcached/issues/79
-    value = 'Iñtërnâtiônàlizætiøn'
-
-    setting_from_db = mocker.Mock(id=1, key='AWX_SOME_SETTING', value=value)
-    mocks = mocker.Mock(**{
-        'order_by.return_value': mocker.Mock(**{
-            '__iter__': lambda self: iter([setting_from_db]),
-            'first.return_value': setting_from_db
-        }),
-    })
-    with mocker.patch('awx.conf.models.Setting.objects.filter', return_value=mocks):
-        assert settings.AWX_SOME_SETTING == 'Iñtërnâtiônàlizætiøn'
-        assert settings.cache.get('AWX_SOME_SETTING') == 'Iñtërnâtiônàlizætiøn'
-
-
@pytest.mark.defined_in_file(AWX_SOME_SETTING='DEFAULT')
 def test_read_only_setting_assignment(settings):
    "read-only settings cannot be overwritten"
--- a/awx/conf/views.py
+++ b/awx/conf/views.py
@@ -3,15 +3,20 @@

 # Python
 import collections
+import logging
+import subprocess
 import sys
+import socket
+from socket import SHUT_RDWR

 # Django
+from django.db import connection
 from django.conf import settings
 from django.http import Http404
 from django.utils.translation import ugettext_lazy as _

 # Django REST Framework
-from rest_framework.exceptions import PermissionDenied, ValidationError
+from rest_framework.exceptions import PermissionDenied
 from rest_framework.response import Response
 from rest_framework import serializers
 from rest_framework import status
@@ -26,7 +31,6 @@ from awx.api.generics import (
 from awx.api.permissions import IsSuperUser
 from awx.api.versioning import reverse
 from awx.main.utils import camelcase_to_underscore
-from awx.main.utils.handlers import AWXProxyHandler, LoggingConnectivityException
 from awx.main.tasks import handle_setting_changes
 from awx.conf.models import Setting
 from awx.conf.serializers import SettingCategorySerializer, SettingSingletonSerializer
@@ -127,7 +131,8 @@ class SettingSingletonDetail(RetrieveUpdateDestroyAPIView):
                setting.save(update_fields=['value'])
                settings_change_list.append(key)
        if settings_change_list:
-            handle_setting_changes.delay(settings_change_list)
+            connection.on_commit(lambda: handle_setting_changes.delay(settings_change_list))
+

    def destroy(self, request, *args, **kwargs):
        instance = self.get_object()
@@ -142,7 +147,7 @@ class SettingSingletonDetail(RetrieveUpdateDestroyAPIView):
            setting.delete()
            settings_change_list.append(setting.key)
        if settings_change_list:
-            handle_setting_changes.delay(settings_change_list)
+            connection.on_commit(lambda: handle_setting_changes.delay(settings_change_list))

        # When TOWER_URL_BASE is deleted from the API, reset it to the hostname
        # used to make the request as a default.
@@ -161,40 +166,53 @@ class SettingLoggingTest(GenericAPIView):
    filter_backends = []

    def post(self, request, *args, **kwargs):
-        defaults = dict()
-        for key in settings_registry.get_registered_settings(category_slug='logging'):
-            try:
-                defaults[key] = settings_registry.get_setting_field(key).get_default()
-            except serializers.SkipField:
-                defaults[key] = None
-        obj = type('Settings', (object,), defaults)()
-        serializer = self.get_serializer(obj, data=request.data)
-        serializer.is_valid(raise_exception=True)
-        # Special validation specific to logging test.
-        errors = {}
-        for key in ['LOG_AGGREGATOR_TYPE', 'LOG_AGGREGATOR_HOST']:
-            if not request.data.get(key, ''):
-                errors[key] = 'This field is required.'
-        if errors:
-            raise ValidationError(errors)
-
-        if request.data.get('LOG_AGGREGATOR_PASSWORD', '').startswith('$encrypted$'):
-            serializer.validated_data['LOG_AGGREGATOR_PASSWORD'] = getattr(
-                settings, 'LOG_AGGREGATOR_PASSWORD', ''
-            )
+        # Error if logging is not enabled
+        enabled = getattr(settings, 'LOG_AGGREGATOR_ENABLED', False)
+        if not enabled:
+            return Response({'error': 'Logging not enabled'}, status=status.HTTP_409_CONFLICT)
+        
+        # Send test message to configured logger based on db settings
+        try:
+            default_logger = settings.LOG_AGGREGATOR_LOGGERS[0]
+            if default_logger != 'awx':
+                default_logger = f'awx.analytics.{default_logger}'
+        except IndexError:
+            default_logger = 'awx'
+        logging.getLogger(default_logger).error('AWX Connection Test Message')
+        
+        hostname = getattr(settings, 'LOG_AGGREGATOR_HOST', None)
+        protocol = getattr(settings, 'LOG_AGGREGATOR_PROTOCOL', None)

        try:
-            class MockSettings:
-                pass
-            mock_settings = MockSettings()
-            for k, v in serializer.validated_data.items():
-                setattr(mock_settings, k, v)
-            AWXProxyHandler().perform_test(custom_settings=mock_settings)
-            if mock_settings.LOG_AGGREGATOR_PROTOCOL.upper() == 'UDP':
-                return Response(status=status.HTTP_201_CREATED)
-        except LoggingConnectivityException as e:
-            return Response({'error': str(e)}, status=status.HTTP_500_INTERNAL_SERVER_ERROR)
-        return Response(status=status.HTTP_200_OK)
+            subprocess.check_output(
+                ['rsyslogd', '-N1', '-f', '/var/lib/awx/rsyslog/rsyslog.conf'],
+                stderr=subprocess.STDOUT
+            )
+        except subprocess.CalledProcessError as exc:
+            return Response({'error': exc.output}, status=status.HTTP_400_BAD_REQUEST)
+        
+        # Check to ensure port is open at host
+        if protocol in ['udp', 'tcp']:
+            port = getattr(settings, 'LOG_AGGREGATOR_PORT', None)
+            # Error if port is not set when using UDP/TCP
+            if not port:
+                return Response({'error': 'Port required for ' + protocol}, status=status.HTTP_400_BAD_REQUEST)
+        else:
+            # if http/https by this point, domain is reacheable
+            return Response(status=status.HTTP_202_ACCEPTED)
+
+        if protocol == 'udp':
+            s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+        else:
+            s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        try:
+            s.settimeout(.5)
+            s.connect((hostname, int(port)))
+            s.shutdown(SHUT_RDWR)
+            s.close()
+            return Response(status=status.HTTP_202_ACCEPTED)
+        except Exception as e:
+            return Response({'error': str(e)}, status=status.HTTP_400_BAD_REQUEST)


 # Create view functions for all of the class-based views to simplify inclusion
--- a/awx/locale/django.pot
+++ b/awx/locale/django.pot
--- a/awx/locale/en-us/LC_MESSAGES/django.po
+++ b/awx/locale/en-us/LC_MESSAGES/django.po
--- a/awx/locale/es/LC_MESSAGES/django.po
+++ b/awx/locale/es/LC_MESSAGES/django.po
--- a/awx/locale/fr/LC_MESSAGES/django.po
+++ b/awx/locale/fr/LC_MESSAGES/django.po
--- a/awx/locale/ja/LC_MESSAGES/django.po
+++ b/awx/locale/ja/LC_MESSAGES/django.po
--- a/awx/locale/nl/LC_MESSAGES/django.po
+++ b/awx/locale/nl/LC_MESSAGES/django.po
--- a/awx/locale/zh/LC_MESSAGES/django.po
+++ b/awx/locale/zh/LC_MESSAGES/django.po
--- a/awx/main/access.py
+++ b/awx/main/access.py
@@ -11,7 +11,6 @@ from functools import reduce
 from django.conf import settings
 from django.db.models import Q, Prefetch
 from django.contrib.auth.models import User
-from django.contrib.contenttypes.models import ContentType
 from django.utils.translation import ugettext_lazy as _
 from django.core.exceptions import ObjectDoesNotExist

@@ -405,14 +404,6 @@ class BaseAccess(object):
                # Cannot copy manual project without errors
                user_capabilities[display_method] = False
                continue
-            elif display_method in ['start', 'schedule'] and isinstance(obj, Group):  # TODO: remove in 3.3
-                try:
-                    if obj.deprecated_inventory_source and not obj.deprecated_inventory_source._can_update():
-                        user_capabilities[display_method] = False
-                        continue
-                except Group.deprecated_inventory_source.RelatedObjectDoesNotExist:
-                    user_capabilities[display_method] = False
-                    continue
            elif display_method in ['start', 'schedule'] and isinstance(obj, (Project)):
                if obj.scm_type == '':
                    user_capabilities[display_method] = False
@@ -504,7 +495,7 @@ class NotificationAttachMixin(BaseAccess):
            # due to this special case, we use symmetrical logic with attach permission
            return self._can_attach(notification_template=sub_obj, resource_obj=obj)
        return super(NotificationAttachMixin, self).can_unattach(
-            obj, sub_obj, relationship, relationship, data=data
+            obj, sub_obj, relationship, data=data
        )


@@ -650,8 +641,8 @@ class UserAccess(BaseAccess):
                # in these cases only superusers can modify orphan users
                return False
            return not obj.roles.all().exclude(
-                content_type=ContentType.objects.get_for_model(User)
-            ).filter(ancestors__in=self.user.roles.all()).exists()
+                ancestors__in=self.user.roles.all()
+            ).exists()
        else:
            return self.is_all_org_admin(obj)

@@ -789,7 +780,6 @@ class OrganizationAccess(NotificationAttachMixin, BaseAccess):
        return self.user in obj.admin_role

    def can_delete(self, obj):
-        self.check_license(check_expiration=False)
        is_change_possible = self.can_change(obj, None)
        if not is_change_possible:
            return False
@@ -1411,7 +1401,7 @@ class JobTemplateAccess(NotificationAttachMixin, BaseAccess):
    '''

    model = JobTemplate
-    select_related = ('created_by', 'modified_by', 'inventory', 'project',
+    select_related = ('created_by', 'modified_by', 'inventory', 'project', 'organization',
                      'next_schedule',)
    prefetch_related = (
        'instance_groups',
@@ -1435,16 +1425,11 @@ class JobTemplateAccess(NotificationAttachMixin, BaseAccess):
        Users who are able to create deploy jobs can also run normal and check (dry run) jobs.
        '''
        if not data:  # So the browseable API will work
-            return (
-                Project.accessible_objects(self.user, 'use_role').exists() or
-                Inventory.accessible_objects(self.user, 'use_role').exists())
+            return Project.accessible_objects(self.user, 'use_role').exists()

        # if reference_obj is provided, determine if it can be copied
        reference_obj = data.get('reference_obj', None)

-        if 'survey_enabled' in data and data['survey_enabled']:
-            self.check_license(feature='surveys')
-
        if self.user.is_superuser:
            return True

@@ -1504,22 +1489,23 @@ class JobTemplateAccess(NotificationAttachMixin, BaseAccess):
        return self.user in obj.execute_role

    def can_change(self, obj, data):
-        data_for_change = data
        if self.user not in obj.admin_role and not self.user.is_superuser:
            return False
-        if data is not None:
-            data = dict(data)
+        if data is None:
+            return True

-            if self.changes_are_non_sensitive(obj, data):
-                if 'survey_enabled' in data and obj.survey_enabled != data['survey_enabled'] and data['survey_enabled']:
-                    self.check_license(feature='surveys')
-                return True
+        data = dict(data)

-            for required_field in ('inventory', 'project'):
-                required_obj = getattr(obj, required_field, None)
-                if required_field not in data_for_change and required_obj is not None:
-                    data_for_change[required_field] = required_obj.pk
-        return self.can_read(obj) and (self.can_add(data_for_change) if data is not None else True)
+        if self.changes_are_non_sensitive(obj, data):
+            return True
+
+        for required_field, cls in (('inventory', Inventory), ('project', Project)):
+            is_mandatory = True
+            if not getattr(obj, '{}_id'.format(required_field)):
+                is_mandatory = False
+            if not self.check_related(required_field, cls, data, obj=obj, role_field='use_role', mandatory=is_mandatory):
+                return False
+        return True

    def changes_are_non_sensitive(self, obj, data):
        '''
@@ -1554,9 +1540,9 @@ class JobTemplateAccess(NotificationAttachMixin, BaseAccess):
    @check_superuser
    def can_attach(self, obj, sub_obj, relationship, data, skip_sub_obj_read_check=False):
        if relationship == "instance_groups":
-            if not obj.project.organization:
+            if not obj.organization:
                return False
-            return self.user.can_access(type(sub_obj), "read", sub_obj) and self.user in obj.project.organization.admin_role
+            return self.user.can_access(type(sub_obj), "read", sub_obj) and self.user in obj.organization.admin_role
        if relationship == 'credentials' and isinstance(sub_obj, Credential):
            return self.user in obj.admin_role and self.user in sub_obj.use_role
        return super(JobTemplateAccess, self).can_attach(
@@ -1587,6 +1573,7 @@ class JobAccess(BaseAccess):
    select_related = ('created_by', 'modified_by', 'job_template', 'inventory',
                      'project', 'project_update',)
    prefetch_related = (
+        'organization',
        'unified_job_template',
        'instance_group',
        'credentials__credential_type',
@@ -1607,42 +1594,19 @@ class JobAccess(BaseAccess):

        return qs.filter(
            Q(job_template__in=JobTemplate.accessible_objects(self.user, 'read_role')) |
-            Q(inventory__organization__in=org_access_qs) |
-            Q(project__organization__in=org_access_qs)).distinct()
-
-    def related_orgs(self, obj):
-        orgs = []
-        if obj.inventory and obj.inventory.organization:
-            orgs.append(obj.inventory.organization)
-        if obj.project and obj.project.organization and obj.project.organization not in orgs:
-            orgs.append(obj.project.organization)
-        return orgs
-
-    def org_access(self, obj, role_types=['admin_role']):
-        orgs = self.related_orgs(obj)
-        for org in orgs:
-            for role_type in role_types:
-                role = getattr(org, role_type)
-                if self.user in role:
-                    return True
-        return False
+            Q(organization__in=org_access_qs)).distinct()

    def can_add(self, data, validate_license=True):
-        if validate_license:
-            self.check_license()
-
-        if not data:  # So the browseable API will work
-            return True
-        return self.user.is_superuser
+        raise NotImplementedError('Direct job creation not possible in v2 API')

    def can_change(self, obj, data):
-        return (obj.status == 'new' and
-                self.can_read(obj) and
-                self.can_add(data, validate_license=False))
+        raise NotImplementedError('Direct job editing not supported in v2 API')

    @check_superuser
    def can_delete(self, obj):
-        return self.org_access(obj)
+        if not obj.organization:
+            return False
+        return self.user in obj.organization.admin_role

    def can_start(self, obj, validate_license=True):
        if validate_license:
@@ -1662,6 +1626,7 @@ class JobAccess(BaseAccess):
        except JobLaunchConfig.DoesNotExist:
            config = None

+        # Standard permissions model
        if obj.job_template and (self.user not in obj.job_template.execute_role):
            return False

@@ -1676,24 +1641,17 @@ class JobAccess(BaseAccess):
                if JobLaunchConfigAccess(self.user).can_add({'reference_obj': config}):
                    return True

-        org_access = bool(obj.inventory) and self.user in obj.inventory.organization.inventory_admin_role
-        project_access = obj.project is None or self.user in obj.project.admin_role
-        credential_access = all([self.user in cred.use_role for cred in obj.credentials.all()])
+        # Standard permissions model without job template involved
+        if obj.organization and self.user in obj.organization.execute_role:
+            return True
+        elif not (obj.job_template or obj.organization):
+            raise PermissionDenied(_('Job has been orphaned from its job template and organization.'))
+        elif obj.job_template and config is not None:
+            raise PermissionDenied(_('Job was launched with prompted fields you do not have access to.'))
+        elif obj.job_template and config is None:
+            raise PermissionDenied(_('Job was launched with unknown prompted fields. Organization admin permissions required.'))

-        # job can be relaunched if user could make an equivalent JT
-        ret = org_access and credential_access and project_access
-        if not ret and self.save_messages and not self.messages:
-            if not obj.job_template:
-                pretext = _('Job has been orphaned from its job template.')
-            elif config is None:
-                pretext = _('Job was launched with unknown prompted fields.')
-            else:
-                pretext = _('Job was launched with prompted fields.')
-            if credential_access:
-                self.messages['detail'] = '{} {}'.format(pretext, _(' Organization level permissions required.'))
-            else:
-                self.messages['detail'] = '{} {}'.format(pretext, _(' You do not have permission to related resources.'))
-        return ret
+        return False

    def get_method_capability(self, method, obj, parent_obj):
        if method == 'start':
@@ -1706,10 +1664,16 @@ class JobAccess(BaseAccess):
    def can_cancel(self, obj):
        if not obj.can_cancel:
            return False
-        # Delete access allows org admins to stop running jobs
-        if self.user == obj.created_by or self.can_delete(obj):
+        # Users may always cancel their own jobs
+        if self.user == obj.created_by:
            return True
-        return obj.job_template is not None and self.user in obj.job_template.admin_role
+        # Users with direct admin to JT may cancel jobs started by anyone
+        if obj.job_template and self.user in obj.job_template.admin_role:
+            return True
+        # If orphaned, allow org JT admins to stop running jobs
+        if not obj.job_template and obj.organization and self.user in obj.organization.job_template_admin_role:
+            return True
+        return False


 class SystemJobTemplateAccess(BaseAccess):
@@ -1944,11 +1908,11 @@ class WorkflowJobNodeAccess(BaseAccess):
 # TODO: notification attachments?
 class WorkflowJobTemplateAccess(NotificationAttachMixin, BaseAccess):
    '''
-    I can only see/manage Workflow Job Templates if I'm a super user
+    I can see/manage Workflow Job Templates based on object roles
    '''

    model = WorkflowJobTemplate
-    select_related = ('created_by', 'modified_by', 'next_schedule',
+    select_related = ('created_by', 'modified_by', 'organization', 'next_schedule',
                      'admin_role', 'execute_role', 'read_role',)

    def filtered_queryset(self):
@@ -1966,10 +1930,6 @@ class WorkflowJobTemplateAccess(NotificationAttachMixin, BaseAccess):
        if not data:  # So the browseable API will work
            return Organization.accessible_objects(self.user, 'workflow_admin_role').exists()

-        # will check this if surveys are added to WFJT
-        if 'survey_enabled' in data and data['survey_enabled']:
-            self.check_license(feature='surveys')
-
        return (
            self.check_related('organization', Organization, data, role_field='workflow_admin_role', mandatory=True) and
            self.check_related('inventory', Inventory, data, role_field='use_role')
@@ -2038,7 +1998,7 @@ class WorkflowJobAccess(BaseAccess):
       I can also cancel it if I started it
    '''
    model = WorkflowJob
-    select_related = ('created_by', 'modified_by',)
+    select_related = ('created_by', 'modified_by', 'organization',)

    def filtered_queryset(self):
        return WorkflowJob.objects.filter(
@@ -2332,6 +2292,7 @@ class UnifiedJobTemplateAccess(BaseAccess):
    prefetch_related = (
        'last_job',
        'current_job',
+        'organization',
        'credentials__credential_type',
        Prefetch('labels', queryset=Label.objects.all().order_by('name')),
    )
@@ -2371,6 +2332,7 @@ class UnifiedJobAccess(BaseAccess):
    prefetch_related = (
        'created_by',
        'modified_by',
+        'organization',
        'unified_job_node__workflow_job',
        'unified_job_template',
        'instance_group',
@@ -2401,8 +2363,7 @@ class UnifiedJobAccess(BaseAccess):
            Q(unified_job_template_id__in=UnifiedJobTemplate.accessible_pk_qs(self.user, 'read_role')) |
            Q(inventoryupdate__inventory_source__inventory__id__in=inv_pk_qs) |
            Q(adhoccommand__inventory__id__in=inv_pk_qs) |
-            Q(job__inventory__organization__in=org_auditor_qs) |
-            Q(job__project__organization__in=org_auditor_qs)
+            Q(organization__in=org_auditor_qs)
        )
        return qs

@@ -2429,6 +2390,9 @@ class ScheduleAccess(BaseAccess):
    def can_add(self, data):
        if not JobLaunchConfigAccess(self.user).can_add(data):
            return False
+        if not data:
+            return Role.objects.filter(role_field__in=['update_role', 'execute_role'], ancestors__in=self.user.roles.all()).exists()
+
        return self.check_related('unified_job_template', UnifiedJobTemplate, data, role_field='execute_role', mandatory=True)

    @check_superuser
--- a/awx/main/analytics/broadcast_websocket.py
+++ b/awx/main/analytics/broadcast_websocket.py
@@ -0,0 +1,169 @@
+import datetime
+import asyncio
+import logging
+import aioredis
+import redis
+import re
+
+from prometheus_client import (
+    generate_latest,
+    Gauge,
+    Counter,
+    Enum,
+    CollectorRegistry,
+    parser,
+)
+
+from django.conf import settings
+
+
+BROADCAST_WEBSOCKET_REDIS_KEY_NAME = 'broadcast_websocket_stats'
+
+
+logger = logging.getLogger('awx.main.analytics.broadcast_websocket')
+
+
+def dt_to_seconds(dt):
+    return int((dt - datetime.datetime(1970,1,1)).total_seconds())
+
+
+def now_seconds():
+    return dt_to_seconds(datetime.datetime.now())
+
+
+def safe_name(s):
+    # Replace all non alpha-numeric characters with _
+    return re.sub('[^0-9a-zA-Z]+', '_', s)
+
+
+# Second granularity; Per-minute
+class FixedSlidingWindow():
+    def __init__(self, start_time=None):
+        self.buckets = dict()
+        self.start_time = start_time or now_seconds()
+
+    def cleanup(self, now_bucket=None):
+        now_bucket = now_bucket or now_seconds()
+        if self.start_time + 60 < now_bucket:
+            self.start_time = now_bucket - 60
+
+            # Delete old entries
+            for k in list(self.buckets.keys()):
+                if k < self.start_time:
+                    del self.buckets[k]
+
+    def record(self, ts=None):
+        now_bucket = ts or dt_to_seconds(datetime.datetime.now())
+
+        val = self.buckets.get(now_bucket, 0)
+        self.buckets[now_bucket] = val + 1
+
+        self.cleanup(now_bucket)
+
+    def render(self, ts=None):
+        self.cleanup(now_bucket=ts)
+        return sum(self.buckets.values()) or 0
+
+
+class BroadcastWebsocketStatsManager():
+    def __init__(self, event_loop, local_hostname):
+        self._local_hostname = local_hostname
+
+        self._event_loop = event_loop
+        self._stats = dict()
+        self._redis_key = BROADCAST_WEBSOCKET_REDIS_KEY_NAME
+
+    def new_remote_host_stats(self, remote_hostname):
+        self._stats[remote_hostname] = BroadcastWebsocketStats(self._local_hostname,
+                                                               remote_hostname)
+        return self._stats[remote_hostname]
+
+    def delete_remote_host_stats(self, remote_hostname):
+        del self._stats[remote_hostname]
+
+    async def run_loop(self):
+        try:
+            redis_conn = await aioredis.create_redis_pool(settings.BROKER_URL)
+            while True:
+                stats_data_str = ''.join(stat.serialize() for stat in self._stats.values())
+                await redis_conn.set(self._redis_key, stats_data_str)
+
+                await asyncio.sleep(settings.BROADCAST_WEBSOCKET_STATS_POLL_RATE_SECONDS)
+        except Exception as e:
+            logger.warn(e)
+            await asyncio.sleep(settings.BROADCAST_WEBSOCKET_STATS_POLL_RATE_SECONDS)
+            self.start()
+
+    def start(self):
+        self.async_task = self._event_loop.create_task(self.run_loop())
+        return self.async_task
+
+    @classmethod
+    def get_stats_sync(cls):
+        '''
+        Stringified verion of all the stats
+        '''
+        redis_conn = redis.Redis.from_url(settings.BROKER_URL)
+        stats_str = redis_conn.get(BROADCAST_WEBSOCKET_REDIS_KEY_NAME) or b''
+        return parser.text_string_to_metric_families(stats_str.decode('UTF-8'))
+
+
+class BroadcastWebsocketStats():
+    def __init__(self, local_hostname, remote_hostname):
+        self._local_hostname = local_hostname
+        self._remote_hostname = remote_hostname
+        self._registry = CollectorRegistry()
+
+        # TODO: More robust replacement
+        self.name = safe_name(self._local_hostname)
+        self.remote_name = safe_name(self._remote_hostname)
+
+        self._messages_received_total = Counter(f'awx_{self.remote_name}_messages_received_total',
+                                                'Number of messages received, to be forwarded, by the broadcast websocket system',
+                                                registry=self._registry)
+        self._messages_received = Gauge(f'awx_{self.remote_name}_messages_received',
+                                        'Number forwarded messages received by the broadcast websocket system, for the duration of the current connection',
+                                        registry=self._registry)
+        self._connection = Enum(f'awx_{self.remote_name}_connection',
+                                'Websocket broadcast connection',
+                                states=['disconnected', 'connected'],
+                                registry=self._registry)
+        self._connection.state('disconnected')
+        self._connection_start = Gauge(f'awx_{self.remote_name}_connection_start',
+                                       'Time the connection was established',
+                                       registry=self._registry)
+
+        self._messages_received_per_minute = Gauge(f'awx_{self.remote_name}_messages_received_per_minute',
+                                                   'Messages received per minute',
+                                                   registry=self._registry)
+        self._internal_messages_received_per_minute = FixedSlidingWindow()
+
+    def unregister(self):
+        self._registry.unregister(f'awx_{self.remote_name}_messages_received')
+        self._registry.unregister(f'awx_{self.remote_name}_connection')
+
+    def record_message_received(self):
+        self._internal_messages_received_per_minute.record()
+        self._messages_received.inc()
+        self._messages_received_total.inc()
+
+    def record_connection_established(self):
+        self._connection.state('connected')
+        self._connection_start.set_to_current_time()
+        self._messages_received.set(0)
+
+    def record_connection_lost(self):
+        self._connection.state('disconnected')
+
+    def get_connection_duration(self):
+        return (datetime.datetime.now() - self._connection_established_ts).total_seconds()
+
+    def render(self):
+        msgs_per_min = self._internal_messages_received_per_minute.render()
+        self._messages_received_per_minute.set(msgs_per_min)
+
+    def serialize(self):
+        self.render()
+
+        registry_data = generate_latest(self._registry).decode('UTF-8')
+        return registry_data
--- a/awx/main/analytics/collectors.py
+++ b/awx/main/analytics/collectors.py
@@ -31,7 +31,7 @@ data _since_ the last report date - i.e., new data in the last 24 hours)
 '''


-@register('config', '1.0')
+@register('config', '1.1')
 def config(since):
    license_info = get_license(show_key=False)
    install_type = 'traditional'
@@ -53,6 +53,7 @@ def config(since):
        'ansible_version': get_ansible_version(),
        'license_type': license_info.get('license_type', 'UNLICENSED'),
        'free_instances': license_info.get('free_instances', 0),
+        'total_licensed_instances': license_info.get('instance_count', 0),
        'license_expiry': license_info.get('time_remaining', 0),
        'pendo_tracking': settings.PENDO_TRACKING_STATE,
        'authentication_backends': settings.AUTHENTICATION_BACKENDS,
@@ -121,22 +122,27 @@ def cred_type_counts(since):
    return counts
    
    
-@register('inventory_counts', '1.0')
+@register('inventory_counts', '1.2')
 def inventory_counts(since):
    counts = {}
    for inv in models.Inventory.objects.filter(kind='').annotate(num_sources=Count('inventory_sources', distinct=True), 
                                                                 num_hosts=Count('hosts', distinct=True)).only('id', 'name', 'kind'):
+        source_list = []
+        for source in inv.inventory_sources.filter().annotate(num_hosts=Count('hosts', distinct=True)).values('name','source', 'num_hosts'):
+            source_list.append(source)
        counts[inv.id] = {'name': inv.name,
                          'kind': inv.kind,
                          'hosts': inv.num_hosts,
-                          'sources': inv.num_sources
+                          'sources': inv.num_sources,
+                          'source_list': source_list
                          }

    for smart_inv in models.Inventory.objects.filter(kind='smart'):
        counts[smart_inv.id] = {'name': smart_inv.name,
                                'kind': smart_inv.kind,
-                                'num_hosts': smart_inv.hosts.count(),
-                                'num_sources': smart_inv.inventory_sources.count()
+                                'hosts': smart_inv.hosts.count(),
+                                'sources': 0,
+                                'source_list': []
                                }
    return counts

@@ -221,10 +227,12 @@ def query_info(since, collection_type):


 # Copies Job Events from db to a .csv to be shipped
-@table_version('events_table.csv', '1.0')
+@table_version('events_table.csv', '1.1')
@table_version('unified_jobs_table.csv', '1.0')
@table_version('unified_job_template_table.csv', '1.0')
-def copy_tables(since, full_path):
+@table_version('workflow_job_node_table.csv', '1.0')
+@table_version('workflow_job_template_node_table.csv', '1.0')
+def copy_tables(since, full_path, subset=None):
    def _copy_table(table, query, path):
        file_path = os.path.join(path, table + '_table.csv')
        file = open(file_path, 'w', encoding='utf-8')
@@ -248,15 +256,21 @@ def copy_tables(since, full_path):
                              main_jobevent.job_id, 
                              main_jobevent.host_id, 
                              main_jobevent.host_name
+                              , CAST(main_jobevent.event_data::json->>'start' AS TIMESTAMP WITH TIME ZONE) AS start,
+                              CAST(main_jobevent.event_data::json->>'end' AS TIMESTAMP WITH TIME ZONE) AS end,
+                              main_jobevent.event_data::json->'duration' AS duration,
+                              main_jobevent.event_data::json->'res'->'warnings' AS warnings,
+                              main_jobevent.event_data::json->'res'->'deprecations' AS deprecations
                              FROM main_jobevent 
                              WHERE main_jobevent.created > {}
                              ORDER BY main_jobevent.id ASC) TO STDOUT WITH CSV HEADER'''.format(since.strftime("'%Y-%m-%d %H:%M:%S'"))
-    _copy_table(table='events', query=events_query, path=full_path)
+    if not subset or 'events' in subset:
+        _copy_table(table='events', query=events_query, path=full_path)

    unified_job_query = '''COPY (SELECT main_unifiedjob.id,
                                 main_unifiedjob.polymorphic_ctype_id,
                                 django_content_type.model,
-                                 main_project.organization_id,
+                                 main_unifiedjob.organization_id,
                                 main_organization.name as organization_name,
                                 main_unifiedjob.created,  
                                 main_unifiedjob.name,  
@@ -274,14 +288,13 @@ def copy_tables(since, full_path):
                                 main_unifiedjob.job_explanation, 
                                 main_unifiedjob.instance_group_id
                                 FROM main_unifiedjob
-                                 JOIN main_job ON main_unifiedjob.id = main_job.unifiedjob_ptr_id
                                 JOIN django_content_type ON main_unifiedjob.polymorphic_ctype_id = django_content_type.id
-                                 JOIN main_project ON main_project.unifiedjobtemplate_ptr_id = main_job.project_id
-                                 JOIN main_organization ON main_organization.id = main_project.organization_id
-                                 WHERE main_unifiedjob.created > {} 
-                                 AND main_unifiedjob.launch_type != 'sync'
+                                 LEFT JOIN main_organization ON main_organization.id = main_unifiedjob.organization_id
+                                 WHERE (main_unifiedjob.created > {0} OR main_unifiedjob.finished > {0})
+                                       AND main_unifiedjob.launch_type != 'sync'
                                 ORDER BY main_unifiedjob.id ASC) TO STDOUT WITH CSV HEADER'''.format(since.strftime("'%Y-%m-%d %H:%M:%S'"))    
-    _copy_table(table='unified_jobs', query=unified_job_query, path=full_path)
+    if not subset or 'unified_jobs' in subset:
+        _copy_table(table='unified_jobs', query=unified_job_query, path=full_path)

    unified_job_template_query = '''COPY (SELECT main_unifiedjobtemplate.id, 
                                 main_unifiedjobtemplate.polymorphic_ctype_id,
@@ -300,6 +313,71 @@ def copy_tables(since, full_path):
                                 main_unifiedjobtemplate.status 
                                 FROM main_unifiedjobtemplate, django_content_type
                                 WHERE main_unifiedjobtemplate.polymorphic_ctype_id = django_content_type.id
-                                 ORDER BY main_unifiedjobtemplate.id ASC) TO STDOUT WITH CSV HEADER'''.format(since.strftime("'%Y-%m-%d %H:%M:%S'"))    
-    _copy_table(table='unified_job_template', query=unified_job_template_query, path=full_path)
+                                 ORDER BY main_unifiedjobtemplate.id ASC) TO STDOUT WITH CSV HEADER'''  
+    if not subset or 'unified_job_template' in subset:
+        _copy_table(table='unified_job_template', query=unified_job_template_query, path=full_path)
+
+    workflow_job_node_query = '''COPY (SELECT main_workflowjobnode.id,
+                                 main_workflowjobnode.created,
+                                 main_workflowjobnode.modified, 
+                                 main_workflowjobnode.job_id, 
+                                 main_workflowjobnode.unified_job_template_id, 
+                                 main_workflowjobnode.workflow_job_id, 
+                                 main_workflowjobnode.inventory_id, 
+                                 success_nodes.nodes AS success_nodes,
+                                 failure_nodes.nodes AS failure_nodes,
+                                 always_nodes.nodes AS always_nodes,
+                                 main_workflowjobnode.do_not_run, 
+                                 main_workflowjobnode.all_parents_must_converge
+                                 FROM main_workflowjobnode
+                                 LEFT JOIN (
+                                     SELECT from_workflowjobnode_id, ARRAY_AGG(to_workflowjobnode_id) AS nodes
+                                     FROM main_workflowjobnode_success_nodes
+                                     GROUP BY from_workflowjobnode_id
+                                 ) success_nodes ON main_workflowjobnode.id = success_nodes.from_workflowjobnode_id
+                                 LEFT JOIN (
+                                     SELECT from_workflowjobnode_id, ARRAY_AGG(to_workflowjobnode_id) AS nodes
+                                     FROM main_workflowjobnode_failure_nodes
+                                     GROUP BY from_workflowjobnode_id
+                                 ) failure_nodes ON main_workflowjobnode.id = failure_nodes.from_workflowjobnode_id
+                                 LEFT JOIN (
+                                     SELECT from_workflowjobnode_id, ARRAY_AGG(to_workflowjobnode_id) AS nodes
+                                     FROM main_workflowjobnode_always_nodes
+                                     GROUP BY from_workflowjobnode_id
+                                 ) always_nodes ON main_workflowjobnode.id = always_nodes.from_workflowjobnode_id
+                                 WHERE main_workflowjobnode.modified > {}
+                                 ORDER BY main_workflowjobnode.id ASC) TO STDOUT WITH CSV HEADER'''.format(since.strftime("'%Y-%m-%d %H:%M:%S'"))    
+    if not subset or 'workflow_job_node' in subset:
+        _copy_table(table='workflow_job_node', query=workflow_job_node_query, path=full_path)
+
+    workflow_job_template_node_query = '''COPY (SELECT main_workflowjobtemplatenode.id, 
+                                 main_workflowjobtemplatenode.created,
+                                 main_workflowjobtemplatenode.modified, 
+                                 main_workflowjobtemplatenode.unified_job_template_id, 
+                                 main_workflowjobtemplatenode.workflow_job_template_id, 
+                                 main_workflowjobtemplatenode.inventory_id, 
+                                 success_nodes.nodes AS success_nodes,
+                                 failure_nodes.nodes AS failure_nodes,
+                                 always_nodes.nodes AS always_nodes,
+                                 main_workflowjobtemplatenode.all_parents_must_converge
+                                 FROM main_workflowjobtemplatenode
+                                 LEFT JOIN (
+                                     SELECT from_workflowjobtemplatenode_id, ARRAY_AGG(to_workflowjobtemplatenode_id) AS nodes
+                                     FROM main_workflowjobtemplatenode_success_nodes
+                                     GROUP BY from_workflowjobtemplatenode_id
+                                 ) success_nodes ON main_workflowjobtemplatenode.id = success_nodes.from_workflowjobtemplatenode_id
+                                 LEFT JOIN (
+                                     SELECT from_workflowjobtemplatenode_id, ARRAY_AGG(to_workflowjobtemplatenode_id) AS nodes
+                                     FROM main_workflowjobtemplatenode_failure_nodes
+                                     GROUP BY from_workflowjobtemplatenode_id
+                                 ) failure_nodes ON main_workflowjobtemplatenode.id = failure_nodes.from_workflowjobtemplatenode_id
+                                 LEFT JOIN (
+                                     SELECT from_workflowjobtemplatenode_id, ARRAY_AGG(to_workflowjobtemplatenode_id) AS nodes
+                                     FROM main_workflowjobtemplatenode_always_nodes
+                                     GROUP BY from_workflowjobtemplatenode_id
+                                 ) always_nodes ON main_workflowjobtemplatenode.id = always_nodes.from_workflowjobtemplatenode_id
+                                 ORDER BY main_workflowjobtemplatenode.id ASC) TO STDOUT WITH CSV HEADER'''   
+    if not subset or 'workflow_job_template_node' in subset:
+        _copy_table(table='workflow_job_template_node', query=workflow_job_template_node_query, path=full_path)
+
    return
--- a/awx/main/analytics/core.py
+++ b/awx/main/analytics/core.py
@@ -134,13 +134,17 @@ def gather(dest=None, module=None, collection_type='scheduled'):
        settings.SYSTEM_UUID,
        run_now.strftime('%Y-%m-%d-%H%M%S%z')
    ])
-    tgz = shutil.make_archive(
-        os.path.join(os.path.dirname(dest), tarname),
-        'gztar',
-        dest
-    )
-    shutil.rmtree(dest)
-    return tgz
+    try:
+        tgz = shutil.make_archive(
+            os.path.join(os.path.dirname(dest), tarname),
+            'gztar',
+            dest
+        )
+        return tgz
+    except Exception:
+        logger.exception("Failed to write analytics archive file")
+    finally: 
+        shutil.rmtree(dest)


 def ship(path):
--- a/awx/main/conf.py
+++ b/awx/main/conf.py
@@ -246,18 +246,6 @@ register(
    category_slug='jobs',
 )

-register(
-    'AWX_ISOLATED_VERBOSITY',
-    field_class=fields.IntegerField,
-    min_value=0,
-    max_value=5,
-    label=_('Verbosity level for isolated node management tasks'),
-    help_text=_('This can be raised to aid in debugging connection issues for isolated task execution'),
-    category=_('Jobs'),
-    category_slug='jobs',
-    default=0
-)
-
 register(
    'AWX_ISOLATED_CHECK_INTERVAL',
    field_class=fields.IntegerField,
@@ -435,6 +423,19 @@ register(
    category_slug='jobs',
 )

+register(
+    'AWX_SHOW_PLAYBOOK_LINKS',
+    field_class=fields.BooleanField,
+    default=False,
+    label=_('Follow symlinks'),
+    help_text=_(
+        'Follow symbolic links when scanning for playbooks. Be aware that setting this to True can lead '
+        'to infinite recursion if a link points to a parent directory of itself.'
+    ),
+    category=_('Jobs'),
+    category_slug='jobs',
+)
+
 register(
    'PRIMARY_GALAXY_URL',
    field_class=fields.URLField,
@@ -667,7 +668,7 @@ register(
    allow_blank=True,
    default='',
    label=_('Logging Aggregator Username'),
-    help_text=_('Username for external log aggregator (if required).'),
+    help_text=_('Username for external log aggregator (if required; HTTP/s only).'),
    category=_('Logging'),
    category_slug='logging',
    required=False,
@@ -679,7 +680,7 @@ register(
    default='',
    encrypted=True,
    label=_('Logging Aggregator Password/Token'),
-    help_text=_('Password or authentication token for external log aggregator (if required).'),
+    help_text=_('Password or authentication token for external log aggregator (if required; HTTP/s only).'),
    category=_('Logging'),
    category_slug='logging',
    required=False,
@@ -778,26 +779,40 @@ register(
    category_slug='logging',
 )
 register(
-    'LOG_AGGREGATOR_AUDIT',
+    'LOG_AGGREGATOR_MAX_DISK_USAGE_GB',
+    field_class=fields.IntegerField,
+    default=1,
+    min_value=1,
+    label=_('Maximum disk persistance for external log aggregation (in GB)'),
+    help_text=_('Amount of data to store (in gigabytes) during an outage of '
+                'the external log aggregator (defaults to 1). '
+                'Equivalent to the rsyslogd queue.maxdiskspace setting.'),
+    category=_('Logging'),
+    category_slug='logging',
+)
+register(
+    'LOG_AGGREGATOR_MAX_DISK_USAGE_PATH',
+    field_class=fields.CharField,
+    default='/var/lib/awx',
+    label=_('File system location for rsyslogd disk persistence'),
+    help_text=_('Location to persist logs that should be retried after an outage '
+                'of the external log aggregator (defaults to /var/lib/awx). '
+                'Equivalent to the rsyslogd queue.spoolDirectory setting.'),
+    category=_('Logging'),
+    category_slug='logging',
+)
+register(
+    'LOG_AGGREGATOR_RSYSLOGD_DEBUG',
    field_class=fields.BooleanField,
-    allow_null=True,
    default=False,
-    label=_('Enabled external log aggregation auditing'),
-    help_text=_('When enabled, all external logs emitted by Tower will also be written to /var/log/tower/external.log.  This is an experimental setting intended to be used for debugging external log aggregation issues (and may be subject to change in the future).'),  # noqa
+    label=_('Enable rsyslogd debugging'),
+    help_text=_('Enabled high verbosity debugging for rsyslogd.  '
+                'Useful for debugging connection issues for external log aggregation.'),
    category=_('Logging'),
    category_slug='logging',
 )


-register(
-    'BROKER_DURABILITY',
-    field_class=fields.BooleanField,
-    label=_('Message Durability'),
-    help_text=_('When set (the default), underlying queues will be persisted to disk.  Disable this to enable higher message bus throughput.'),
-    category=_('System'),
-    category_slug='system',
-)
-

 register(
    'AUTOMATION_ANALYTICS_LAST_GATHER',
--- a/awx/main/constants.py
+++ b/awx/main/constants.py
@@ -10,8 +10,7 @@ __all__ = [
    'ANSI_SGR_PATTERN', 'CAN_CANCEL', 'ACTIVE_STATES', 'STANDARD_INVENTORY_UPDATE_ENV'
 ]

-
-CLOUD_PROVIDERS = ('azure_rm', 'ec2', 'gce', 'vmware', 'openstack', 'rhv', 'satellite6', 'cloudforms', 'tower')
+CLOUD_PROVIDERS = ('azure_rm', 'ec2', 'gce', 'vmware', 'openstack', 'rhv', 'satellite6', 'tower')
 SCHEDULEABLE_PROVIDERS = CLOUD_PROVIDERS + ('custom', 'scm',)
 PRIVILEGE_ESCALATION_METHODS = [
    ('sudo', _('Sudo')), ('su', _('Su')), ('pbrun', _('Pbrun')), ('pfexec', _('Pfexec')),
@@ -38,7 +37,7 @@ ENV_BLACKLIST = frozenset((
    'AD_HOC_COMMAND_ID', 'REST_API_URL', 'REST_API_TOKEN', 'MAX_EVENT_RES',
    'CALLBACK_QUEUE', 'CALLBACK_CONNECTION', 'CACHE',
    'JOB_CALLBACK_DEBUG', 'INVENTORY_HOSTVARS',
-    'AWX_HOST', 'PROJECT_REVISION'
+    'AWX_HOST', 'PROJECT_REVISION', 'SUPERVISOR_WEB_CONFIG_PATH'
 ))

 # loggers that may be called in process of emitting a log
--- a/awx/main/consumers.py
+++ b/awx/main/consumers.py
@@ -1,97 +1,246 @@
 import json
 import logging
+import time
+import hmac
+import asyncio

-from channels import Group
-from channels.auth import channel_session_user_from_http, channel_session_user
-
-from django.utils.encoding import smart_str
-from django.http.cookie import parse_cookie
 from django.core.serializers.json import DjangoJSONEncoder
+from django.conf import settings
+from django.utils.encoding import force_bytes
+from django.contrib.auth.models import User
+
+from channels.generic.websocket import AsyncJsonWebsocketConsumer
+from channels.layers import get_channel_layer
+from channels.db import database_sync_to_async


 logger = logging.getLogger('awx.main.consumers')
 XRF_KEY = '_auth_user_xrf'


-def discard_groups(message):
-    if 'groups' in message.channel_session:
-        for group in message.channel_session['groups']:
-            Group(group).discard(message.reply_channel)
+class WebsocketSecretAuthHelper:
+    """
+    Middlewareish for websockets to verify node websocket broadcast interconnect.
+
+    Note: The "ish" is due to the channels routing interface. Routing occurs 
+    _after_ authentication; making it hard to apply this auth to _only_ a subset of
+    websocket endpoints.
+    """
+
+    @classmethod
+    def construct_secret(cls):
+        nonce_serialized = f"{int(time.time())}"
+        payload_dict = {
+            'secret': settings.BROADCAST_WEBSOCKET_SECRET,
+            'nonce': nonce_serialized
+        }
+        payload_serialized = json.dumps(payload_dict)
+
+        secret_serialized = hmac.new(force_bytes(settings.BROADCAST_WEBSOCKET_SECRET),
+                                     msg=force_bytes(payload_serialized),
+                                     digestmod='sha256').hexdigest()
+
+        return 'HMAC-SHA256 {}:{}'.format(nonce_serialized, secret_serialized)


-@channel_session_user_from_http
-def ws_connect(message):
-    headers = dict(message.content.get('headers', ''))
-    message.reply_channel.send({"accept": True})
-    message.content['method'] = 'FAKE'
-    if message.user.is_authenticated:
-        message.reply_channel.send(
-            {"text": json.dumps({"accept": True, "user": message.user.id})}
-        )
-        # store the valid CSRF token from the cookie so we can compare it later
-        # on ws_receive
-        cookie_token = parse_cookie(
-            smart_str(headers.get(b'cookie'))
-        ).get('csrftoken')
-        if cookie_token:
-            message.channel_session[XRF_KEY] = cookie_token
-    else:
-        logger.error("Request user is not authenticated to use websocket.")
-        message.reply_channel.send({"close": True})
-    return None
+    @classmethod
+    def verify_secret(cls, s, nonce_tolerance=300):
+        try:
+            (prefix, payload) = s.split(' ')
+            if prefix != 'HMAC-SHA256':
+                raise ValueError('Unsupported encryption algorithm')
+            (nonce_parsed, secret_parsed) = payload.split(':')
+        except Exception:
+            raise ValueError("Failed to parse secret")
+
+        try:
+            payload_expected = {
+                'secret': settings.BROADCAST_WEBSOCKET_SECRET,
+                'nonce': nonce_parsed,
+            }
+            payload_serialized = json.dumps(payload_expected)
+        except Exception:
+            raise ValueError("Failed to create hash to compare to secret.")
+
+        secret_serialized = hmac.new(force_bytes(settings.BROADCAST_WEBSOCKET_SECRET),
+                                     msg=force_bytes(payload_serialized),
+                                     digestmod='sha256').hexdigest()
+
+        if secret_serialized != secret_parsed:
+            raise ValueError("Invalid secret")
+
+        # Avoid timing attack and check the nonce after all the heavy lifting
+        now = int(time.time())
+        nonce_parsed = int(nonce_parsed)
+        nonce_diff = now - nonce_parsed
+        if abs(nonce_diff) > nonce_tolerance:
+            logger.warn(f"Potential replay attack or machine(s) time out of sync by {nonce_diff} seconds.")
+            raise ValueError("Potential replay attack or machine(s) time out of sync by {nonce_diff} seconds.")
+
+        return True
+
+    @classmethod
+    def is_authorized(cls, scope):
+        secret = ''
+        for k, v in scope['headers']:
+            if k.decode("utf-8") == 'secret':
+                secret = v.decode("utf-8")
+                break
+        WebsocketSecretAuthHelper.verify_secret(secret)


-@channel_session_user
-def ws_disconnect(message):
-    discard_groups(message)
+class BroadcastConsumer(AsyncJsonWebsocketConsumer):
+
+    async def connect(self):
+        try:
+            WebsocketSecretAuthHelper.is_authorized(self.scope)
+        except Exception:
+            logger.warn(f"client '{self.channel_name}' failed to authorize against the broadcast endpoint.")
+            await self.close()
+            return
+
+        await self.accept()
+        await self.channel_layer.group_add(settings.BROADCAST_WEBSOCKET_GROUP_NAME, self.channel_name)
+        logger.info(f"client '{self.channel_name}' joined the broadcast group.")
+
+    async def disconnect(self, code):
+        logger.info(f"client '{self.channel_name}' disconnected from the broadcast group.")
+        await self.channel_layer.group_discard(settings.BROADCAST_WEBSOCKET_GROUP_NAME, self.channel_name)
+
+    async def internal_message(self, event):
+        await self.send(event['text'])


-@channel_session_user
-def ws_receive(message):
-    from awx.main.access import consumer_access
-    user = message.user
-    raw_data = message.content['text']
-    data = json.loads(raw_data)
+class EventConsumer(AsyncJsonWebsocketConsumer):
+    async def connect(self):
+        user = self.scope['user']
+        if user and not user.is_anonymous:
+            await self.accept()
+            await self.send_json({"accept": True, "user": user.id})
+            # store the valid CSRF token from the cookie so we can compare it later
+            # on ws_receive
+            cookie_token = self.scope['cookies'].get('csrftoken')
+            if cookie_token:
+                self.scope['session'][XRF_KEY] = cookie_token
+        else:
+            logger.error("Request user is not authenticated to use websocket.")
+            # TODO: Carry over from channels 1 implementation
+            # We should never .accept() the client and close without sending a close message
+            await self.accept()
+            await self.send_json({"close": True})
+            await self.close()

-    xrftoken = data.get('xrftoken')
-    if (
-        not xrftoken or
-        XRF_KEY not in message.channel_session or
-        xrftoken != message.channel_session[XRF_KEY]
-    ):
-        logger.error(
-            "access denied to channel, XRF mismatch for {}".format(user.username)
-        )
-        message.reply_channel.send({
-            "text": json.dumps({"error": "access denied to channel"})
-        })
-        return
+    async def disconnect(self, code):
+        current_groups = set(self.scope['session'].pop('groups') if 'groups' in self.scope['session'] else [])
+        for group_name in current_groups:
+            await self.channel_layer.group_discard(
+                group_name,
+                self.channel_name,
+            )

-    if 'groups' in data:
-        discard_groups(message)
-        groups = data['groups']
-        current_groups = set(message.channel_session.pop('groups') if 'groups' in message.channel_session else [])
-        for group_name,v in groups.items():
-            if type(v) is list:
-                for oid in v:
-                    name = '{}-{}'.format(group_name, oid)
-                    access_cls = consumer_access(group_name)
-                    if access_cls is not None:
-                        user_access = access_cls(user)
-                        if not user_access.get_queryset().filter(pk=oid).exists():
-                            message.reply_channel.send({"text": json.dumps(
-                                {"error": "access denied to channel {0} for resource id {1}".format(group_name, oid)})})
-                            continue
-                    current_groups.add(name)
-                    Group(name).add(message.reply_channel)
-            else:
-                current_groups.add(group_name)
-                Group(group_name).add(message.reply_channel)
-        message.channel_session['groups'] = list(current_groups)
+    @database_sync_to_async
+    def user_can_see_object_id(self, user_access, oid):
+        # At this point user is a channels.auth.UserLazyObject object
+        # This causes problems with our generic role permissions checking.
+        # Specifically, type(user) != User
+        # Therefore, get the "real" User objects from the database before
+        # calling the access permission methods
+        user_access.user = User.objects.get(id=user_access.user.id)
+        res = user_access.get_queryset().filter(pk=oid).exists()
+        return res
+
+    async def receive_json(self, data):
+        from awx.main.access import consumer_access
+        user = self.scope['user']
+        xrftoken = data.get('xrftoken')
+        if (
+            not xrftoken or
+            XRF_KEY not in self.scope["session"] or
+            xrftoken != self.scope["session"][XRF_KEY]
+        ):
+            logger.error(f"access denied to channel, XRF mismatch for {user.username}")
+            await self.send_json({"error": "access denied to channel"})
+            return
+
+        if 'groups' in data:
+            groups = data['groups']
+            new_groups = set()
+            current_groups = set(self.scope['session'].pop('groups') if 'groups' in self.scope['session'] else [])
+            for group_name,v in groups.items():
+                if type(v) is list:
+                    for oid in v:
+                        name = '{}-{}'.format(group_name, oid)
+                        access_cls = consumer_access(group_name)
+                        if access_cls is not None:
+                            user_access = access_cls(user)
+                            if not await self.user_can_see_object_id(user_access, oid):
+                                await self.send_json({"error": "access denied to channel {0} for resource id {1}".format(group_name, oid)})
+                                continue
+                        new_groups.add(name)
+                else:
+                    await self.send_json({"error": "access denied to channel"})
+                    logger.error(f"groups must be a list, not {groups}")
+                    return
+
+            old_groups = current_groups - new_groups
+            for group_name in old_groups:
+                await self.channel_layer.group_discard(
+                    group_name,
+                    self.channel_name,
+                )
+
+            new_groups_exclusive = new_groups - current_groups
+            for group_name in new_groups_exclusive:
+                await self.channel_layer.group_add(
+                    group_name,
+                    self.channel_name
+                )
+            self.scope['session']['groups'] = new_groups
+            await self.send_json({
+                "groups_current": list(new_groups),
+                "groups_left": list(old_groups),
+                "groups_joined": list(new_groups_exclusive)
+            })
+
+    async def internal_message(self, event):
+        await self.send(event['text'])
+
+
+def run_sync(func):
+    event_loop = asyncio.new_event_loop()
+    event_loop.run_until_complete(func)
+    event_loop.close()
+
+
+def _dump_payload(payload):
+    try:
+        return json.dumps(payload, cls=DjangoJSONEncoder)
+    except ValueError:
+        logger.error("Invalid payload to emit")
+        return None


 def emit_channel_notification(group, payload):
-    try:
-        Group(group).send({"text": json.dumps(payload, cls=DjangoJSONEncoder)})
-    except ValueError:
-        logger.error("Invalid payload emitting channel {} on topic: {}".format(group, payload))
+    from awx.main.wsbroadcast import wrap_broadcast_msg # noqa
+
+    payload_dumped = _dump_payload(payload)
+    if payload_dumped is None:
+        return
+
+    channel_layer = get_channel_layer()
+
+    run_sync(channel_layer.group_send(
+        group,
+        {
+            "type": "internal.message",
+            "text": payload_dumped
+        },
+    ))
+
+    run_sync(channel_layer.group_send(
+        settings.BROADCAST_WEBSOCKET_GROUP_NAME,
+        {
+            "type": "internal.message",
+            "text": wrap_broadcast_msg(group, payload_dumped),
+        },
+    ))
--- a/awx/main/credential_plugins/aim.py
+++ b/awx/main/credential_plugins/aim.py
@@ -1,15 +1,10 @@
-from .plugin import CredentialPlugin
+from .plugin import CredentialPlugin, CertFiles

 from urllib.parse import quote, urlencode, urljoin

 from django.utils.translation import ugettext_lazy as _
 import requests

-# AWX
-from awx.main.utils import (
-    create_temporary_fifo,
-)
-
 aim_inputs = {
    'fields': [{
        'id': 'url',
@@ -43,7 +38,7 @@ aim_inputs = {
        'id': 'object_query',
        'label': _('Object Query'),
        'type': 'string',
-        'help_text': _('Lookup query for the object. Ex: "Safe=TestSafe;Object=testAccountName123"'),
+        'help_text': _('Lookup query for the object. Ex: Safe=TestSafe;Object=testAccountName123'),
    }, {
        'id': 'object_query_format',
        'label': _('Object Query Format'),
@@ -81,21 +76,13 @@ def aim_backend(**kwargs):
    request_qs = '?' + urlencode(query_params, quote_via=quote)
    request_url = urljoin(url, '/'.join(['AIMWebService', 'api', 'Accounts']))

-    cert = None
-    if client_cert and client_key:
-        cert = (
-            create_temporary_fifo(client_cert.encode()),
-            create_temporary_fifo(client_key.encode())
+    with CertFiles(client_cert, client_key) as cert:
+        res = requests.get(
+            request_url + request_qs,
+            timeout=30,
+            cert=cert,
+            verify=verify,
        )
-    elif client_cert:
-        cert = create_temporary_fifo(client_cert.encode())
-
-    res = requests.get(
-        request_url + request_qs,
-        timeout=30,
-        cert=cert,
-        verify=verify,
-    )
    res.raise_for_status()
    return res.json()['Content']

--- a/awx/main/credential_plugins/conjur.py
+++ b/awx/main/credential_plugins/conjur.py
@@ -1,16 +1,11 @@
-from .plugin import CredentialPlugin
+from .plugin import CredentialPlugin, CertFiles

 import base64
-from urllib.parse import urljoin, quote_plus
+from urllib.parse import urljoin, quote

 from django.utils.translation import ugettext_lazy as _
 import requests

-# AWX
-from awx.main.utils import (
-    create_temporary_fifo,
-)
-

 conjur_inputs = {
    'fields': [{
@@ -55,9 +50,9 @@ conjur_inputs = {
 def conjur_backend(**kwargs):
    url = kwargs['url']
    api_key = kwargs['api_key']
-    account = quote_plus(kwargs['account'])
-    username = quote_plus(kwargs['username'])
-    secret_path = quote_plus(kwargs['secret_path'])
+    account = quote(kwargs['account'], safe='') 
+    username = quote(kwargs['username'], safe='')
+    secret_path = quote(kwargs['secret_path'], safe='')
    version = kwargs.get('secret_version')
    cacert = kwargs.get('cacert', None)

@@ -65,22 +60,20 @@ def conjur_backend(**kwargs):
        'headers': {'Content-Type': 'text/plain'},
        'data': api_key
    }
-    if cacert:
-        auth_kwargs['verify'] = create_temporary_fifo(cacert.encode())

-    # https://www.conjur.org/api.html#authentication-authenticate-post
-    resp = requests.post(
-        urljoin(url, '/'.join(['authn', account, username, 'authenticate'])),
-        **auth_kwargs
-    )
+    with CertFiles(cacert) as cert:
+        # https://www.conjur.org/api.html#authentication-authenticate-post
+        auth_kwargs['verify'] = cert
+        resp = requests.post(
+            urljoin(url, '/'.join(['authn', account, username, 'authenticate'])),
+            **auth_kwargs
+        )
    resp.raise_for_status()
    token = base64.b64encode(resp.content).decode('utf-8')

    lookup_kwargs = {
        'headers': {'Authorization': 'Token token="{}"'.format(token)},
    }
-    if cacert:
-        lookup_kwargs['verify'] = create_temporary_fifo(cacert.encode())

    # https://www.conjur.org/api.html#secrets-retrieve-a-secret-get
    path = urljoin(url, '/'.join([
@@ -92,7 +85,9 @@ def conjur_backend(**kwargs):
    if version:
        path = '?'.join([path, version])

-    resp = requests.get(path, timeout=30, **lookup_kwargs)
+    with CertFiles(cacert) as cert:
+        lookup_kwargs['verify'] = cert
+        resp = requests.get(path, timeout=30, **lookup_kwargs)
    resp.raise_for_status()
    return resp.text

--- a/awx/main/credential_plugins/hashivault.py
+++ b/awx/main/credential_plugins/hashivault.py
@@ -3,16 +3,11 @@ import os
 import pathlib
 from urllib.parse import urljoin

-from .plugin import CredentialPlugin
+from .plugin import CredentialPlugin, CertFiles

 import requests
 from django.utils.translation import ugettext_lazy as _

-# AWX
-from awx.main.utils import (
-    create_temporary_fifo,
-)
-
 base_inputs = {
    'fields': [{
        'id': 'url',
@@ -32,14 +27,33 @@ base_inputs = {
        'type': 'string',
        'multiline': True,
        'help_text': _('The CA certificate used to verify the SSL certificate of the Vault server')
-    }],
+    }, {
+        'id': 'role_id',
+        'label': _('AppRole role_id'),
+        'type': 'string',
+        'multiline': False,
+        'help_text': _('The Role ID for AppRole Authentication')
+    }, {
+        'id': 'secret_id',
+        'label': _('AppRole secret_id'),
+        'type': 'string',
+        'multiline': False,
+        'secret': True,
+        'help_text': _('The Secret ID for AppRole Authentication')
+    }
+    ],
    'metadata': [{
        'id': 'secret_path',
        'label': _('Path to Secret'),
        'type': 'string',
        'help_text': _('The path to the secret stored in the secret backend e.g, /some/secret/')
+    },{
+        'id': 'auth_path',
+        'label': _('Path to Auth'),
+        'type': 'string',
+        'help_text': _('The path where the Authentication method is mounted e.g, approle')
    }],
-    'required': ['url', 'token', 'secret_path'],
+    'required': ['url', 'secret_path'],
 }

 hashi_kv_inputs = copy.deepcopy(base_inputs)
@@ -88,8 +102,42 @@ hashi_ssh_inputs['metadata'] = [{
 hashi_ssh_inputs['required'].extend(['public_key', 'role'])


+def handle_auth(**kwargs):
+    token = None
+
+    if kwargs.get('token'):
+        token = kwargs['token']
+    elif kwargs.get('role_id') and kwargs.get('secret_id'):
+        token = approle_auth(**kwargs)
+    else:
+        raise Exception('Either token or AppRole parameters must be set')
+
+    return token
+
+
+def approle_auth(**kwargs):
+    role_id = kwargs['role_id']
+    secret_id = kwargs['secret_id']
+    auth_path = kwargs.get('auth_path') or 'approle'
+
+    url = urljoin(kwargs['url'], 'v1')
+    cacert = kwargs.get('cacert', None)
+
+    request_kwargs = {'timeout': 30}
+    # AppRole Login
+    request_kwargs['json'] = {'role_id': role_id, 'secret_id': secret_id}
+    sess = requests.Session()
+    request_url = '/'.join([url, 'auth', auth_path, 'login']).rstrip('/')
+    with CertFiles(cacert) as cert:
+        request_kwargs['verify'] = cert
+        resp = sess.post(request_url, **request_kwargs)
+    resp.raise_for_status()
+    token = resp.json()['auth']['client_token']
+    return token
+
+
 def kv_backend(**kwargs):
-    token = kwargs['token']
+    token = handle_auth(**kwargs)
    url = kwargs['url']
    secret_path = kwargs['secret_path']
    secret_backend = kwargs.get('secret_backend', None)
@@ -98,8 +146,6 @@ def kv_backend(**kwargs):
    api_version = kwargs['api_version']

    request_kwargs = {'timeout': 30}
-    if cacert:
-        request_kwargs['verify'] = create_temporary_fifo(cacert.encode())

    sess = requests.Session()
    sess.headers['Authorization'] = 'Bearer {}'.format(token)
@@ -126,7 +172,9 @@ def kv_backend(**kwargs):
            path_segments = [secret_path]

    request_url = urljoin(url, '/'.join(['v1'] + path_segments)).rstrip('/')
-    response = sess.get(request_url, **request_kwargs)
+    with CertFiles(cacert) as cert:
+        request_kwargs['verify'] = cert
+        response = sess.get(request_url, **request_kwargs)
    response.raise_for_status()

    json = response.json()
@@ -144,15 +192,13 @@ def kv_backend(**kwargs):


 def ssh_backend(**kwargs):
-    token = kwargs['token']
+    token = handle_auth(**kwargs)
    url = urljoin(kwargs['url'], 'v1')
    secret_path = kwargs['secret_path']
    role = kwargs['role']
    cacert = kwargs.get('cacert', None)

    request_kwargs = {'timeout': 30}
-    if cacert:
-        request_kwargs['verify'] = create_temporary_fifo(cacert.encode())

    request_kwargs['json'] = {'public_key': kwargs['public_key']}
    if kwargs.get('valid_principals'):
@@ -164,7 +210,10 @@ def ssh_backend(**kwargs):
    sess.headers['X-Vault-Token'] = token
    # https://www.vaultproject.io/api/secret/ssh/index.html#sign-ssh-key
    request_url = '/'.join([url, secret_path, 'sign', role]).rstrip('/')
-    resp = sess.post(request_url, **request_kwargs)
+
+    with CertFiles(cacert) as cert:
+        request_kwargs['verify'] = cert
+        resp = sess.post(request_url, **request_kwargs)

    resp.raise_for_status()
    return resp.json()['data']['signed_key']
--- a/awx/main/credential_plugins/plugin.py
+++ b/awx/main/credential_plugins/plugin.py
@@ -1,3 +1,45 @@
+import os
+import tempfile
+
 from collections import namedtuple

 CredentialPlugin = namedtuple('CredentialPlugin', ['name', 'inputs', 'backend'])
+
+
+class CertFiles():
+    """
+    A context manager used for writing a certificate and (optional) key
+    to $TMPDIR, and cleaning up afterwards.
+
+    This is particularly useful as a shared resource for credential plugins
+    that want to pull cert/key data out of the database and persist it
+    temporarily to the file system so that it can loaded into the openssl
+    certificate chain (generally, for HTTPS requests plugins make via the
+    Python requests library)
+
+    with CertFiles(cert_data, key_data) as cert:
+        # cert is string representing a path to the cert or pemfile
+        # temporarily written to disk
+        requests.post(..., cert=cert)
+    """
+
+    certfile = None
+
+    def __init__(self, cert, key=None):
+        self.cert = cert
+        self.key = key
+
+    def __enter__(self):
+        if not self.cert:
+            return None
+        self.certfile = tempfile.NamedTemporaryFile('wb', delete=False)
+        self.certfile.write(self.cert.encode())
+        if self.key:
+            self.certfile.write(b'\n')
+            self.certfile.write(self.key.encode())
+        self.certfile.flush()
+        return str(self.certfile.name)
+
+    def __exit__(self, *args):
+        if self.certfile and os.path.exists(self.certfile.name):
+            os.remove(self.certfile.name)
--- a/awx/main/db/profiled_pg/base.py
+++ b/awx/main/db/profiled_pg/base.py
@@ -24,7 +24,7 @@ class RecordedQueryLog(object):
        try:
            self.threshold = cache.get('awx-profile-sql-threshold')
        except Exception:
-            # if we can't reach memcached, just assume profiling's off
+            # if we can't reach the cache, just assume profiling's off
            self.threshold = None

    def append(self, query):
@@ -64,7 +64,7 @@ class RecordedQueryLog(object):
            if not os.path.isdir(self.dest):
                os.makedirs(self.dest)
            progname = ' '.join(sys.argv)
-            for match in ('uwsgi', 'dispatcher', 'callback_receiver', 'runworker'):
+            for match in ('uwsgi', 'dispatcher', 'callback_receiver', 'wsbroadcast'):
                if match in progname:
                    progname = match
                    break
@@ -110,7 +110,7 @@ class RecordedQueryLog(object):
 class DatabaseWrapper(BaseDatabaseWrapper):
    """
    This is a special subclass of Django's postgres DB backend which - based on
-    the value of a special flag in memcached - captures slow queries and
+    the value of a special flag in cache - captures slow queries and
    writes profile and Python stack metadata to the disk.
    """

@@ -133,19 +133,19 @@ class DatabaseWrapper(BaseDatabaseWrapper):
        # is the same mechanism used by libraries like the django-debug-toolbar)
        #
        # in _this_ implementation, we represent it as a property which will
-        # check memcache for a special flag to be set (when the flag is set, it
+        # check the cache for a special flag to be set (when the flag is set, it
        # means we should start recording queries because somebody called
        # `awx-manage profile_sql`)
        #
        # it's worth noting that this property is wrapped w/ @memoize because
        # Django references this attribute _constantly_ (in particular, once
-        # per executed query); doing a memcached.get()  _at most_ once per
+        # per executed query); doing a cache.get()  _at most_ once per
        # second is a good enough window to detect when profiling is turned
        # on/off by a system administrator
        try:
            threshold = cache.get('awx-profile-sql-threshold')
        except Exception:
-            # if we can't reach memcached, just assume profiling's off
+            # if we can't reach the cache, just assume profiling's off
            threshold = None
        self.queries_log.threshold = threshold
        return threshold is not None
--- a/awx/main/dispatch/init.py
+++ b/awx/main/dispatch/init.py
@@ -1,5 +1,62 @@
+import psycopg2
+import select
+
+from contextlib import contextmanager
+
 from django.conf import settings


+NOT_READY = ([], [], [])
+
+
 def get_local_queuename():
    return settings.CLUSTER_HOST_ID
+
+
+class PubSub(object):
+    def __init__(self, conn):
+        assert conn.autocommit, "Connection must be in autocommit mode."
+        self.conn = conn
+
+    def listen(self, channel):
+        with self.conn.cursor() as cur:
+            cur.execute('LISTEN "%s";' % channel)
+
+    def unlisten(self, channel):
+        with self.conn.cursor() as cur:
+            cur.execute('UNLISTEN "%s";' % channel)
+
+    def notify(self, channel, payload):
+        with self.conn.cursor() as cur:
+            cur.execute('SELECT pg_notify(%s, %s);', (channel, payload))
+
+    def events(self, select_timeout=5, yield_timeouts=False):
+        while True:
+            if select.select([self.conn], [], [], select_timeout) == NOT_READY:
+                if yield_timeouts:
+                    yield None
+            else:
+                self.conn.poll()
+                while self.conn.notifies:
+                    yield self.conn.notifies.pop(0)
+
+    def close(self):
+        self.conn.close()
+
+
+@contextmanager
+def pg_bus_conn():
+    conf = settings.DATABASES['default']
+    conn = psycopg2.connect(dbname=conf['NAME'],
+                            host=conf['HOST'],
+                            user=conf['USER'],
+                            password=conf['PASSWORD'],
+                            port=conf['PORT'],
+                            **conf.get("OPTIONS", {}))
+    # Django connection.cursor().connection doesn't have autocommit=True on
+    conn.set_session(autocommit=True)
+    pubsub = PubSub(conn)
+    yield pubsub
+    conn.close()
+
+
--- a/awx/main/dispatch/control.py
+++ b/awx/main/dispatch/control.py
@@ -1,11 +1,10 @@
 import logging
-import socket
-
-from django.conf import settings
+import uuid
+import json

 from awx.main.dispatch import get_local_queuename
-from awx.main.dispatch.kombu import Connection
-from kombu import Queue, Exchange, Producer, Consumer
+
+from . import pg_bus_conn

 logger = logging.getLogger('awx.main.dispatch')

@@ -20,15 +19,6 @@ class Control(object):
            raise RuntimeError('{} must be in {}'.format(service, self.services))
        self.service = service
        self.queuename = host or get_local_queuename()
-        self.queue = Queue(self.queuename, Exchange(self.queuename), routing_key=self.queuename)
-
-    def publish(self, msg, conn, **kwargs):
-        producer = Producer(
-            exchange=self.queue.exchange,
-            channel=conn,
-            routing_key=self.queuename
-        )
-        producer.publish(msg, expiration=5, **kwargs)

    def status(self, *args, **kwargs):
        return self.control_with_reply('status', *args, **kwargs)
@@ -36,24 +26,28 @@ class Control(object):
    def running(self, *args, **kwargs):
        return self.control_with_reply('running', *args, **kwargs)

+    @classmethod
+    def generate_reply_queue_name(cls):
+        return f"reply_to_{str(uuid.uuid4()).replace('-','_')}"
+
    def control_with_reply(self, command, timeout=5):
        logger.warn('checking {} {} for {}'.format(self.service, command, self.queuename))
-        reply_queue = Queue(name="amq.rabbitmq.reply-to")
+        reply_queue = Control.generate_reply_queue_name()
        self.result = None
-        with Connection(settings.BROKER_URL) as conn:
-            with Consumer(conn, reply_queue, callbacks=[self.process_message], no_ack=True):
-                self.publish({'control': command}, conn, reply_to='amq.rabbitmq.reply-to')
-                try:
-                    conn.drain_events(timeout=timeout)
-                except socket.timeout:
-                    logger.error('{} did not reply within {}s'.format(self.service, timeout))
-                    raise
-        return self.result
+
+        with pg_bus_conn() as conn:
+            conn.listen(reply_queue)
+            conn.notify(self.queuename,
+                        json.dumps({'control': command, 'reply_to': reply_queue}))
+
+            for reply in conn.events(select_timeout=timeout, yield_timeouts=True):
+                if reply is None:
+                    logger.error(f'{self.service} did not reply within {timeout}s')
+                    raise RuntimeError("{self.service} did not reply within {timeout}s")
+                break
+
+        return json.loads(reply.payload)

    def control(self, msg, **kwargs):
-        with Connection(settings.BROKER_URL) as conn:
-            self.publish(msg, conn)
-
-    def process_message(self, body, message):
-        self.result = body
-        message.ack()
+        with pg_bus_conn() as conn:
+            conn.notify(self.queuename, json.dumps(msg))
--- a/awx/main/dispatch/kombu.py
+++ b/awx/main/dispatch/kombu.py
@@ -1,42 +0,0 @@
-from amqp.exceptions import PreconditionFailed
-from django.conf import settings
-from kombu.connection import Connection as KombuConnection
-from kombu.transport import pyamqp
-
-import logging
-
-logger = logging.getLogger('awx.main.dispatch')
-
-
-__all__ = ['Connection']
-
-
-class Connection(KombuConnection):
-
-    def __init__(self, *args, **kwargs):
-        super(Connection, self).__init__(*args, **kwargs)
-        class _Channel(pyamqp.Channel):
-
-            def queue_declare(self, queue, *args, **kwargs):
-                kwargs['durable'] = settings.BROKER_DURABILITY
-                try:
-                    return super(_Channel, self).queue_declare(queue, *args, **kwargs)
-                except PreconditionFailed as e:
-                    if "inequivalent arg 'durable'" in getattr(e, 'reply_text', None):
-                        logger.error(
-                            'queue {} durability is not {}, deleting and recreating'.format(
-
-                                queue,
-                                kwargs['durable']
-                            )
-                        )
-                        self.queue_delete(queue)
-                    return super(_Channel, self).queue_declare(queue, *args, **kwargs)
-
-        class _Connection(pyamqp.Connection):
-            Channel = _Channel
-
-        class _Transport(pyamqp.Transport):
-            Connection = _Connection
-
-        self.transport_cls = _Transport
--- a/awx/main/dispatch/periodic.py
+++ b/awx/main/dispatch/periodic.py
@@ -1,6 +1,7 @@
 import logging
-import threading
+import os
 import time
+from multiprocessing import Process

 from django.conf import settings
 from django.db import connections
@@ -14,33 +15,36 @@ logger = logging.getLogger('awx.main.dispatch.periodic')
 class Scheduler(Scheduler):

    def run_continuously(self):
-        cease_continuous_run = threading.Event()
        idle_seconds = max(
            1,
            min(self.jobs).period.total_seconds() / 2
        )

-        class ScheduleThread(threading.Thread):
-            @classmethod
-            def run(cls):
-                while not cease_continuous_run.is_set():
-                    try:
-                        for conn in connections.all():
-                            # If the database connection has a hiccup, re-establish a new
-                            # connection
-                            conn.close_if_unusable_or_obsolete()
-                        self.run_pending()
-                    except Exception:
-                        logger.exception(
-                            'encountered an error while scheduling periodic tasks'
-                        )
-                    time.sleep(idle_seconds)
-                logger.debug('periodic thread exiting...')
+        def run():
+            ppid = os.getppid()
+            logger.warn('periodic beat started')
+            while True:
+                if os.getppid() != ppid:
+                    # if the parent PID changes, this process has been orphaned
+                    # via e.g., segfault or sigkill, we should exit too
+                    pid = os.getpid()
+                    logger.warn(f'periodic beat exiting gracefully pid:{pid}')
+                    raise SystemExit()
+                try:
+                    for conn in connections.all():
+                        # If the database connection has a hiccup, re-establish a new
+                        # connection
+                        conn.close_if_unusable_or_obsolete()
+                    self.run_pending()
+                except Exception:
+                    logger.exception(
+                        'encountered an error while scheduling periodic tasks'
+                    )
+                time.sleep(idle_seconds)

-        thread = ScheduleThread()
-        thread.daemon = True
-        thread.start()
-        return cease_continuous_run
+        process = Process(target=run)
+        process.daemon = True
+        process.start()


 def run_continuously():
@@ -49,4 +53,4 @@ def run_continuously():
        apply_async = TaskWorker.resolve_callable(task['task']).apply_async
        total_seconds = task['schedule'].total_seconds()
        scheduler.every(total_seconds).seconds.do(apply_async)
-    return scheduler.run_continuously()
+    scheduler.run_continuously()
--- a/awx/main/dispatch/pool.py
+++ b/awx/main/dispatch/pool.py
@@ -1,7 +1,9 @@
 import logging
 import os
 import random
+import signal
 import sys
+import time
 import traceback
 from uuid import uuid4

@@ -220,7 +222,7 @@ class WorkerPool(object):
        idx = len(self.workers)
        # It's important to close these because we're _about_ to fork, and we
        # don't want the forked processes to inherit the open sockets
-        # for the DB and memcached connections (that way lies race conditions)
+        # for the DB and cache connections (that way lies race conditions)
        django_connection.close()
        django_cache.close()
        worker = PoolWorker(self.queue_size, self.target, (idx,) + self.target_args)
@@ -244,7 +246,7 @@ class WorkerPool(object):
            ' qsize={{ w.managed_tasks|length }}'
            ' rss={{ w.mb }}MB'
            '{% for task in w.managed_tasks.values() %}'
-            '\n     - {% if loop.index0 == 0 %}running {% else %}queued {% endif %}'
+            '\n     - {% if loop.index0 == 0 %}running {% if "age" in task %}for: {{ "%.1f" % task["age"] }}s {% endif %}{% else %}queued {% endif %}'
            '{{ task["uuid"] }} '
            '{% if "task" in task %}'
            '{{ task["task"].rsplit(".", 1)[-1] }}'
@@ -365,6 +367,26 @@ class AutoscalePool(WorkerPool):
                logger.warn('scaling down worker pid:{}'.format(w.pid))
                w.quit()
                self.workers.remove(w)
+            if w.alive:
+                # if we discover a task manager invocation that's been running
+                # too long, reap it (because otherwise it'll just hold the postgres
+                # advisory lock forever); the goal of this code is to discover
+                # deadlocks or other serious issues in the task manager that cause
+                # the task manager to never do more work
+                current_task = w.current_task
+                if current_task and isinstance(current_task, dict):
+                    if current_task.get('task', '').endswith('tasks.run_task_manager'):
+                        if 'started' not in current_task:
+                            w.managed_tasks[
+                                current_task['uuid']
+                            ]['started'] = time.time()
+                        age = time.time() - current_task['started']
+                        w.managed_tasks[current_task['uuid']]['age'] = age
+                        if age > (60 * 5):
+                            logger.error(
+                                f'run_task_manager has held the advisory lock for >5m, sending SIGTERM to {w.pid}'
+                            )  # noqa
+                            os.kill(w.pid, signal.SIGTERM)

        for m in orphaned:
            # if all the workers are dead, spawn at least one
--- a/awx/main/dispatch/publish.py
+++ b/awx/main/dispatch/publish.py
@@ -1,12 +1,12 @@
 import inspect
 import logging
 import sys
+import json
 from uuid import uuid4

 from django.conf import settings
-from kombu import Exchange, Producer

-from awx.main.dispatch.kombu import Connection
+from . import pg_bus_conn

 logger = logging.getLogger('awx.main.dispatch')

@@ -39,24 +39,22 @@ class task:
    add.apply_async([1, 1])
    Adder.apply_async([1, 1])

-    # Tasks can also define a specific target queue or exchange type:
+    # Tasks can also define a specific target queue or use the special fan-out queue tower_broadcast:

    @task(queue='slow-tasks')
    def snooze():
        time.sleep(10)

-    @task(queue='tower_broadcast', exchange_type='fanout')
+    @task(queue='tower_broadcast')
    def announce():
        print("Run this everywhere!")
    """

-    def __init__(self, queue=None, exchange_type=None):
+    def __init__(self, queue=None):
        self.queue = queue
-        self.exchange_type = exchange_type

    def __call__(self, fn=None):
        queue = self.queue
-        exchange_type = self.exchange_type

        class PublisherMixin(object):

@@ -73,9 +71,12 @@ class task:
                kwargs = kwargs or {}
                queue = (
                    queue or
-                    getattr(cls.queue, 'im_func', cls.queue) or
-                    settings.CELERY_DEFAULT_QUEUE
+                    getattr(cls.queue, 'im_func', cls.queue)
                )
+                if not queue:
+                    msg = f'{cls.name}: Queue value required and may not be None'
+                    logger.error(msg)
+                    raise ValueError(msg)
                obj = {
                    'uuid': task_id,
                    'args': args,
@@ -86,21 +87,8 @@ class task:
                if callable(queue):
                    queue = queue()
                if not settings.IS_TESTING(sys.argv):
-                    with Connection(settings.BROKER_URL) as conn:
-                        exchange = Exchange(queue, type=exchange_type or 'direct')
-                        producer = Producer(conn)
-                        logger.debug('publish {}({}, queue={})'.format(
-                            cls.name,
-                            task_id,
-                            queue
-                        ))
-                        producer.publish(obj,
-                                         serializer='json',
-                                         compression='bzip2',
-                                         exchange=exchange,
-                                         declare=[exchange],
-                                         delivery_mode="persistent",
-                                         routing_key=queue)
+                    with pg_bus_conn() as conn:
+                        conn.notify(queue, json.dumps(obj))
                return (obj, queue)

        # If the object we're wrapping *is* a class (e.g., RunJob), return
--- a/awx/main/dispatch/worker/init.py
+++ b/awx/main/dispatch/worker/init.py
@@ -1,3 +1,3 @@
-from .base import AWXConsumer, BaseWorker  # noqa
+from .base import AWXConsumerRedis, AWXConsumerPG, BaseWorker  # noqa
 from .callback import CallbackBrokerWorker  # noqa
 from .task import TaskWorker  # noqa
--- a/awx/main/dispatch/worker/base.py
+++ b/awx/main/dispatch/worker/base.py
@@ -5,14 +5,18 @@ import os
 import logging
 import signal
 import sys
+import redis
+import json
+import psycopg2
+import time
 from uuid import UUID
 from queue import Empty as QueueEmpty

 from django import db
-from kombu import Producer
-from kombu.mixins import ConsumerMixin
+from django.conf import settings

 from awx.main.dispatch.pool import WorkerPool
+from awx.main.dispatch import pg_bus_conn

 if 'run_callback_receiver' in sys.argv:
    logger = logging.getLogger('awx.main.commands.run_callback_receiver')
@@ -31,16 +35,18 @@ class WorkerSignalHandler:

    def __init__(self):
        self.kill_now = False
+        signal.signal(signal.SIGTERM, signal.SIG_DFL)
        signal.signal(signal.SIGINT, self.exit_gracefully)

    def exit_gracefully(self, *args, **kwargs):
        self.kill_now = True


-class AWXConsumer(ConsumerMixin):
+class AWXConsumerBase(object):
+    def __init__(self, name, worker, queues=[], pool=None):
+        self.should_stop = False

-    def __init__(self, name, connection, worker, queues=[], pool=None):
-        self.connection = connection
+        self.name = name
        self.total_messages = 0
        self.queues = queues
        self.worker = worker
@@ -49,25 +55,15 @@ class AWXConsumer(ConsumerMixin):
            self.pool = WorkerPool()
        self.pool.init_workers(self.worker.work_loop)

-    def get_consumers(self, Consumer, channel):
-        logger.debug(self.listening_on)
-        return [Consumer(queues=self.queues, accept=['json'],
-                         callbacks=[self.process_task])]
-
    @property
    def listening_on(self):
-        return 'listening on {}'.format([
-            '{} [{}]'.format(q.name, q.exchange.type) for q in self.queues
-        ])
+        return f'listening on {self.queues}'

-    def control(self, body, message):
-        logger.warn('Consumer received control message {}'.format(body))
+    def control(self, body):
+        logger.warn(body)
        control = body.get('control')
        if control in ('status', 'running'):
-            producer = Producer(
-                channel=self.connection,
-                routing_key=message.properties['reply_to']
-            )
+            reply_queue = body['reply_to']
            if control == 'status':
                msg = '\n'.join([self.listening_on, self.pool.debug()])
            elif control == 'running':
@@ -75,20 +71,21 @@ class AWXConsumer(ConsumerMixin):
                for worker in self.pool.workers:
                    worker.calculate_managed_tasks()
                    msg.extend(worker.managed_tasks.keys())
-            producer.publish(msg)
+
+            with pg_bus_conn() as conn:
+                conn.notify(reply_queue, json.dumps(msg))
        elif control == 'reload':
            for worker in self.pool.workers:
                worker.quit()
        else:
            logger.error('unrecognized control message: {}'.format(control))
-        message.ack()

-    def process_task(self, body, message):
+    def process_task(self, body):
        if 'control' in body:
            try:
-                return self.control(body, message)
+                return self.control(body)
            except Exception:
-                logger.exception("Exception handling control message:")
+                logger.exception(f"Exception handling control message: {body}")
                return
        if len(self.pool):
            if "uuid" in body and body['uuid']:
@@ -102,21 +99,68 @@ class AWXConsumer(ConsumerMixin):
            queue = 0
        self.pool.write(queue, body)
        self.total_messages += 1
-        message.ack()

    def run(self, *args, **kwargs):
        signal.signal(signal.SIGINT, self.stop)
        signal.signal(signal.SIGTERM, self.stop)
-        self.worker.on_start()
-        super(AWXConsumer, self).run(*args, **kwargs)
+
+        # Child should implement other things here

    def stop(self, signum, frame):
-        self.should_stop = True  # this makes the kombu mixin stop consuming
+        self.should_stop = True
        logger.warn('received {}, stopping'.format(signame(signum)))
        self.worker.on_stop()
        raise SystemExit()


+class AWXConsumerRedis(AWXConsumerBase):
+    def run(self, *args, **kwargs):
+        super(AWXConsumerRedis, self).run(*args, **kwargs)
+        self.worker.on_start()
+
+        time_to_sleep = 1
+        while True:
+            queue = redis.Redis.from_url(settings.BROKER_URL)
+            while True:
+                try:
+                    res = queue.blpop(self.queues)
+                    time_to_sleep = 1
+                    res = json.loads(res[1])
+                    self.process_task(res)
+                except redis.exceptions.RedisError:
+                    time_to_sleep = min(time_to_sleep * 2, 30)
+                    logger.exception(f"encountered an error communicating with redis. Reconnect attempt in {time_to_sleep} seconds")
+                    time.sleep(time_to_sleep)
+                except (json.JSONDecodeError, KeyError):
+                    logger.exception("failed to decode JSON message from redis")
+                if self.should_stop:
+                    return
+
+
+class AWXConsumerPG(AWXConsumerBase):
+    def run(self, *args, **kwargs):
+        super(AWXConsumerPG, self).run(*args, **kwargs)
+
+        logger.warn(f"Running worker {self.name} listening to queues {self.queues}")
+        init = False
+
+        while True:
+            try:
+                with pg_bus_conn() as conn:
+                    for queue in self.queues:
+                        conn.listen(queue)
+                    if init is False:
+                        self.worker.on_start()
+                        init = True
+                    for e in conn.events():
+                        self.process_task(json.loads(e.payload))
+                    if self.should_stop:
+                        return
+            except psycopg2.InterfaceError:
+                logger.warn("Stale Postgres message bus connection, reconnecting")
+                continue
+
+
 class BaseWorker(object):

    def read(self, queue):
--- a/awx/main/dispatch/worker/callback.py
+++ b/awx/main/dispatch/worker/callback.py
@@ -15,7 +15,9 @@ from django.db.utils import InterfaceError, InternalError, IntegrityError

 from awx.main.consumers import emit_channel_notification
 from awx.main.models import (JobEvent, AdHocCommandEvent, ProjectUpdateEvent,
-                             InventoryUpdateEvent, SystemJobEvent, UnifiedJob)
+                             InventoryUpdateEvent, SystemJobEvent, UnifiedJob,
+                             Job)
+from awx.main.tasks import handle_success_and_failure_notifications
 from awx.main.models.events import emit_event_detail

 from .base import BaseWorker
@@ -89,7 +91,7 @@ class CallbackBrokerWorker(BaseWorker):
                    for e in events:
                        try:
                            if (
-                                isinstance(exc, IntegrityError),
+                                isinstance(exc, IntegrityError) and
                                getattr(e, 'host_id', '')
                            ):
                                # this is one potential IntegrityError we can
@@ -137,19 +139,14 @@ class CallbackBrokerWorker(BaseWorker):
                        # have all the data we need to send out success/failure
                        # notification templates
                        uj = UnifiedJob.objects.get(pk=job_identifier)
-                        if hasattr(uj, 'send_notification_templates'):
-                            retries = 0
-                            while retries < 5:
-                                if uj.finished:
-                                    uj.send_notification_templates('succeeded' if uj.status == 'successful' else 'failed')
-                                    break
-                                else:
-                                    # wait a few seconds to avoid a race where the
-                                    # events are persisted _before_ the UJ.status
-                                    # changes from running -> successful
-                                    retries += 1
-                                    time.sleep(1)
-                                    uj = UnifiedJob.objects.get(pk=job_identifier)
+
+                        if isinstance(uj, Job):
+                            # *actual playbooks* send their success/failure
+                            # notifications in response to the playbook_on_stats
+                            # event handling code in main.models.events
+                            pass
+                        elif hasattr(uj, 'send_notification_templates'):
+                            handle_success_and_failure_notifications.apply_async([uj.id])
                    except Exception:
                        logger.exception('Worker failed to emit notifications: Job {}'.format(job_identifier))
                    return
--- a/awx/main/fields.py
+++ b/awx/main/fields.py
@@ -56,7 +56,8 @@ from awx.main import utils

 __all__ = ['AutoOneToOneField', 'ImplicitRoleField', 'JSONField',
           'SmartFilterField', 'OrderedManyToManyField',
-           'update_role_parentage_for_instance', 'is_implicit_parent']
+           'update_role_parentage_for_instance',
+           'is_implicit_parent']


 # Provide a (better) custom error message for enum jsonschema validation
@@ -140,8 +141,9 @@ def resolve_role_field(obj, field):
        return []

    if len(field_components) == 1:
-        role_cls = str(utils.get_current_apps().get_model('main', 'Role'))
-        if not str(type(obj)) == role_cls:
+        # use extremely generous duck typing to accomidate all possible forms
+        # of the model that may be used during various migrations
+        if obj._meta.model_name != 'role' or obj._meta.app_label != 'main':
            raise Exception(smart_text('{} refers to a {}, not a Role'.format(field, type(obj))))
        ret.append(obj.id)
    else:
@@ -197,18 +199,27 @@ def update_role_parentage_for_instance(instance):
    updates the parents listing for all the roles
    of a given instance if they have changed
    '''
+    parents_removed = set()
+    parents_added = set()
    for implicit_role_field in getattr(instance.__class__, '__implicit_role_fields'):
        cur_role = getattr(instance, implicit_role_field.name)
        original_parents = set(json.loads(cur_role.implicit_parents))
        new_parents = implicit_role_field._resolve_parent_roles(instance)
-        cur_role.parents.remove(*list(original_parents - new_parents))
-        cur_role.parents.add(*list(new_parents - original_parents))
+        removals = original_parents - new_parents
+        if removals:
+            cur_role.parents.remove(*list(removals))
+            parents_removed.add(cur_role.pk)
+        additions = new_parents - original_parents
+        if additions:
+            cur_role.parents.add(*list(additions))
+            parents_added.add(cur_role.pk)
        new_parents_list = list(new_parents)
        new_parents_list.sort()
        new_parents_json = json.dumps(new_parents_list)
        if cur_role.implicit_parents != new_parents_json:
            cur_role.implicit_parents = new_parents_json
-            cur_role.save()
+            cur_role.save(update_fields=['implicit_parents'])
+    return (parents_added, parents_removed)


 class ImplicitRoleDescriptor(ForwardManyToOneDescriptor):
@@ -256,20 +267,18 @@ class ImplicitRoleField(models.ForeignKey):
            field_names = [field_names]

        for field_name in field_names:
-            # Handle the OR syntax for role parents
-            if type(field_name) == tuple:
-                continue
-
-            if type(field_name) == bytes:
-                field_name = field_name.decode('utf-8')

            if field_name.startswith('singleton:'):
                continue

            field_name, sep, field_attr = field_name.partition('.')
-            field = getattr(cls, field_name)
+            # Non existent fields will occur if ever a parent model is
+            # moved inside a migration, needed for job_template_organization_field
+            # migration in particular
+            # consistency is assured by unit test awx.main.tests.functional
+            field = getattr(cls, field_name, None)

-            if type(field) is ReverseManyToOneDescriptor or \
+            if field and type(field) is ReverseManyToOneDescriptor or \
               type(field) is ManyToManyDescriptor:

                if '.' in field_attr:
--- a/awx/main/isolated/manager.py
+++ b/awx/main/isolated/manager.py
@@ -15,7 +15,6 @@ import awx
 from awx.main.utils import (
    get_system_task_capacity
 )
-from awx.main.queue import CallbackQueueDispatcher

 logger = logging.getLogger('awx.isolated.manager')
 playbook_logger = logging.getLogger('awx.isolated.manager.playbooks')
@@ -32,12 +31,14 @@ def set_pythonpath(venv_libdir, env):

 class IsolatedManager(object):

-    def __init__(self, canceled_callback=None, check_callback=None, pod_manager=None):
+    def __init__(self, event_handler, canceled_callback=None, check_callback=None, pod_manager=None):
        """
+        :param event_handler: a callable used to persist event data from isolated nodes
        :param canceled_callback:  a callable - which returns `True` or `False`
                                    - signifying if the job has been prematurely
                                      canceled
        """
+        self.event_handler = event_handler
        self.canceled_callback = canceled_callback
        self.check_callback = check_callback
        self.started_at = None
@@ -57,7 +58,7 @@ class IsolatedManager(object):
                os.chmod(temp.name, stat.S_IRUSR | stat.S_IWUSR | stat.S_IXUSR)
            for host in hosts:
                inventory['all']['hosts'][host] = {
-                    "ansible_connection": "kubectl",
+                    "ansible_connection": "community.kubernetes.kubectl",
                    "ansible_kubectl_config": path,
                }
        else:
@@ -73,6 +74,7 @@ class IsolatedManager(object):
        env['ANSIBLE_RETRY_FILES_ENABLED'] = 'False'
        env['ANSIBLE_HOST_KEY_CHECKING'] = str(settings.AWX_ISOLATED_HOST_KEY_CHECKING)
        env['ANSIBLE_LIBRARY'] = os.path.join(os.path.dirname(awx.__file__), 'plugins', 'isolated')
+        env['ANSIBLE_COLLECTIONS_PATHS'] = settings.AWX_ANSIBLE_COLLECTIONS_PATHS
        set_pythonpath(os.path.join(settings.ANSIBLE_VENV_PATH, 'lib'), env)

        def finished_callback(runner_obj):
@@ -108,7 +110,6 @@ class IsolatedManager(object):
            'cancel_callback': self.canceled_callback,
            'settings': {
                'job_timeout': settings.AWX_ISOLATED_LAUNCH_TIMEOUT,
-                'pexpect_timeout': getattr(settings, 'PEXPECT_TIMEOUT', 5),
                'suppress_ansible_output': True,
            },
        }
@@ -208,7 +209,6 @@ class IsolatedManager(object):
        status = 'failed'
        rc = None
        last_check = time.time()
-        dispatcher = CallbackQueueDispatcher()

        while status == 'failed':
            canceled = self.canceled_callback() if self.canceled_callback else False
@@ -238,7 +238,7 @@ class IsolatedManager(object):
                    except json.decoder.JSONDecodeError:  # Just in case it's not fully here yet.
                        pass

-            self.consume_events(dispatcher)
+            self.consume_events()

            last_check = time.time()

@@ -266,19 +266,11 @@ class IsolatedManager(object):

        # consume events one last time just to be sure we didn't miss anything
        # in the final sync
-        self.consume_events(dispatcher)
-
-        # emit an EOF event
-        event_data = {
-            'event': 'EOF',
-            'final_counter': len(self.handled_events)
-        }
-        event_data.setdefault(self.event_data_key, self.instance.id)
-        dispatcher.dispatch(event_data)
+        self.consume_events()

        return status, rc

-    def consume_events(self, dispatcher):
+    def consume_events(self):
        # discover new events and ingest them
        events_path = self.path_to('artifacts', self.ident, 'job_events')

@@ -288,7 +280,7 @@ class IsolatedManager(object):
        if os.path.exists(events_path):
            for event in set(os.listdir(events_path)) - self.handled_events:
                path = os.path.join(events_path, event)
-                if os.path.exists(path):
+                if os.path.exists(path) and os.path.isfile(path):
                    try:
                        event_data = json.load(
                            open(os.path.join(events_path, event), 'r')
@@ -302,16 +294,10 @@ class IsolatedManager(object):
                        # practice
                        # in this scenario, just ignore this event and try it
                        # again on the next sync
-                        pass
-                    event_data.setdefault(self.event_data_key, self.instance.id)
-                    dispatcher.dispatch(event_data)
+                        continue
+                    self.event_handler(event_data)
                    self.handled_events.add(event)

-                    # handle artifacts
-                    if event_data.get('event_data', {}).get('artifact_data', {}):
-                        self.instance.artifacts = event_data['event_data']['artifact_data']
-                        self.instance.save(update_fields=['artifacts'])
-

    def cleanup(self):
        extravars = {
@@ -400,8 +386,7 @@ class IsolatedManager(object):
            if os.path.exists(private_data_dir):
                shutil.rmtree(private_data_dir)

-    def run(self, instance, private_data_dir, playbook, module, module_args,
-            event_data_key, ident=None):
+    def run(self, instance, private_data_dir, playbook, module, module_args, ident=None):
        """
        Run a job on an isolated host.

@@ -412,14 +397,12 @@ class IsolatedManager(object):
        :param playbook:         the playbook to run
        :param module:           the module to run
        :param module_args:      the module args to use
-        :param event_data_key:   e.g., job_id, inventory_id, ...

        For a completed job run, this function returns (status, rc),
        representing the status and return code of the isolated
        `ansible-playbook` run.
        """
        self.ident = ident
-        self.event_data_key = event_data_key
        self.instance = instance
        self.private_data_dir = private_data_dir
        self.runner_params = self.build_runner_params(
@@ -430,9 +413,4 @@ class IsolatedManager(object):
        status, rc = self.dispatch(playbook, module, module_args)
        if status == 'successful':
            status, rc = self.check()
-        else:
-            # emit an EOF event
-            event_data = {'event': 'EOF', 'final_counter': 0}
-            event_data.setdefault(self.event_data_key, self.instance.id)
-            CallbackQueueDispatcher().dispatch(event_data)
        return status, rc
--- a/awx/main/management/commands/callback_stats.py
+++ b/awx/main/management/commands/callback_stats.py
@@ -34,7 +34,7 @@ class Command(BaseCommand):
                if clear:
                    for i in range(12):
                        sys.stdout.write('\x1b[1A\x1b[2K')
-                for l in lines:
-                    print(l)
+                for line in lines:
+                    print(line)
                clear = True
                time.sleep(.25)
--- a/awx/main/management/commands/cleanup_jobs.py
+++ b/awx/main/management/commands/cleanup_jobs.py
@@ -21,6 +21,8 @@ from awx.main.signals import (
    disable_computed_fields
 )

+from awx.main.management.commands.deletion import AWXCollector, pre_delete
+

 class Command(BaseCommand):
    '''
@@ -57,27 +59,37 @@ class Command(BaseCommand):
                            action='store_true', dest='only_workflow_jobs',
                            help='Remove workflow jobs')

-    def cleanup_jobs(self):
-        #jobs_qs = Job.objects.exclude(status__in=('pending', 'running'))
-        #jobs_qs = jobs_qs.filter(created__lte=self.cutoff)
-        skipped, deleted = 0, 0
-        jobs = Job.objects.filter(created__lt=self.cutoff)
-        for job in jobs.iterator():
-            job_display = '"%s" (%d host summaries, %d events)' % \
-                          (str(job),
-                           job.job_host_summaries.count(), job.job_events.count())
-            if job.status in ('pending', 'waiting', 'running'):
-                action_text = 'would skip' if self.dry_run else 'skipping'
-                self.logger.debug('%s %s job %s', action_text, job.status, job_display)
-                skipped += 1
-            else:
-                action_text = 'would delete' if self.dry_run else 'deleting'
-                self.logger.info('%s %s', action_text, job_display)
-                if not self.dry_run:
-                    job.delete()
-                deleted += 1

-        skipped += Job.objects.filter(created__gte=self.cutoff).count()
+    def cleanup_jobs(self):
+        skipped, deleted = 0, 0
+
+        batch_size = 1000000
+
+        while True:
+            # get queryset for available jobs to remove
+            qs = Job.objects.filter(created__lt=self.cutoff).exclude(status__in=['pending', 'waiting', 'running'])
+            # get pk list for the first N (batch_size) objects
+            pk_list = qs[0:batch_size].values_list('pk')
+            # You cannot delete queries with sql LIMIT set, so we must
+            # create a new query from this pk_list
+            qs_batch = Job.objects.filter(pk__in=pk_list)
+            just_deleted = 0
+            if not self.dry_run:
+                del_query = pre_delete(qs_batch)
+                collector = AWXCollector(del_query.db)
+                collector.collect(del_query)
+                _, models_deleted = collector.delete()
+                if models_deleted:
+                    just_deleted = models_deleted['main.Job']
+                deleted += just_deleted
+            else:
+                just_deleted = 0 # break from loop, this is dry run
+                deleted = qs.count()
+
+            if just_deleted == 0:
+                break
+
+        skipped += (Job.objects.filter(created__gte=self.cutoff) | Job.objects.filter(status__in=['pending', 'waiting', 'running'])).count()
        return skipped, deleted

    def cleanup_ad_hoc_commands(self):
--- a/awx/main/management/commands/deletion.py
+++ b/awx/main/management/commands/deletion.py
@@ -0,0 +1,177 @@
+from django.contrib.contenttypes.models import ContentType
+from django.db.models.deletion import (
+    DO_NOTHING, Collector, get_candidate_relations_to_delete,
+)
+from collections import Counter, OrderedDict
+from django.db import transaction
+from django.db.models import sql
+
+
+def bulk_related_objects(field, objs, using):
+    # This overrides the method in django.contrib.contenttypes.fields.py
+    """
+    Return all objects related to ``objs`` via this ``GenericRelation``.
+    """
+    return field.remote_field.model._base_manager.db_manager(using).filter(**{
+        "%s__pk" % field.content_type_field_name: ContentType.objects.db_manager(using).get_for_model(
+            field.model, for_concrete_model=field.for_concrete_model).pk,
+        "%s__in" % field.object_id_field_name: list(objs.values_list('pk', flat=True))
+    })
+
+
+def pre_delete(qs):
+    # taken from .delete method in django.db.models.query.py
+    assert qs.query.can_filter(), \
+        "Cannot use 'limit' or 'offset' with delete."
+
+    if qs._fields is not None:
+        raise TypeError("Cannot call delete() after .values() or .values_list()")
+
+    del_query = qs._chain()
+
+    # The delete is actually 2 queries - one to find related objects,
+    # and one to delete. Make sure that the discovery of related
+    # objects is performed on the same database as the deletion.
+    del_query._for_write = True
+
+    # Disable non-supported fields.
+    del_query.query.select_for_update = False
+    del_query.query.select_related = False
+    del_query.query.clear_ordering(force_empty=True)
+    return del_query
+
+
+class AWXCollector(Collector):
+
+    def add(self, objs, source=None, nullable=False, reverse_dependency=False):
+        """
+        Add 'objs' to the collection of objects to be deleted.  If the call is
+        the result of a cascade, 'source' should be the model that caused it,
+        and 'nullable' should be set to True if the relation can be null.
+
+        Return a list of all objects that were not already collected.
+        """
+        if not objs.exists():
+            return objs
+        model = objs.model
+        self.data.setdefault(model, [])
+        self.data[model].append(objs)
+        # Nullable relationships can be ignored -- they are nulled out before
+        # deleting, and therefore do not affect the order in which objects have
+        # to be deleted.
+        if source is not None and not nullable:
+            if reverse_dependency:
+                source, model = model, source
+            self.dependencies.setdefault(
+                source._meta.concrete_model, set()).add(model._meta.concrete_model)
+        return objs
+
+    def add_field_update(self, field, value, objs):
+        """
+        Schedule a field update. 'objs' must be a homogeneous iterable
+        collection of model instances (e.g. a QuerySet).
+        """
+        if not objs.exists():
+            return
+        model = objs.model
+        self.field_updates.setdefault(model, {})
+        self.field_updates[model].setdefault((field, value), [])
+        self.field_updates[model][(field, value)].append(objs)
+
+    def collect(self, objs, source=None, nullable=False, collect_related=True,
+                source_attr=None, reverse_dependency=False, keep_parents=False):
+        """
+        Add 'objs' to the collection of objects to be deleted as well as all
+        parent instances.  'objs' must be a homogeneous iterable collection of
+        model instances (e.g. a QuerySet).  If 'collect_related' is True,
+        related objects will be handled by their respective on_delete handler.
+
+        If the call is the result of a cascade, 'source' should be the model
+        that caused it and 'nullable' should be set to True, if the relation
+        can be null.
+
+        If 'reverse_dependency' is True, 'source' will be deleted before the
+        current model, rather than after. (Needed for cascading to parent
+        models, the one case in which the cascade follows the forwards
+        direction of an FK rather than the reverse direction.)
+
+        If 'keep_parents' is True, data of parent model's will be not deleted.
+        """
+
+        if hasattr(objs, 'polymorphic_disabled'):
+            objs.polymorphic_disabled = True
+
+        if self.can_fast_delete(objs):
+            self.fast_deletes.append(objs)
+            return
+        new_objs = self.add(objs, source, nullable,
+                            reverse_dependency=reverse_dependency)
+        if not new_objs.exists():
+            return
+
+        model = new_objs.model
+
+        if not keep_parents:
+            # Recursively collect concrete model's parent models, but not their
+            # related objects. These will be found by meta.get_fields()
+            concrete_model = model._meta.concrete_model
+            for ptr in concrete_model._meta.parents.keys():
+                if ptr:
+                    parent_objs = ptr.objects.filter(pk__in = new_objs.values_list('pk', flat=True))
+                    self.collect(parent_objs, source=model,
+                                 collect_related=False,
+                                 reverse_dependency=True)
+        if collect_related:
+            parents = model._meta.parents
+            for related in get_candidate_relations_to_delete(model._meta):
+                # Preserve parent reverse relationships if keep_parents=True.
+                if keep_parents and related.model in parents:
+                    continue
+                field = related.field
+                if field.remote_field.on_delete == DO_NOTHING:
+                    continue
+                related_qs = self.related_objects(related, new_objs)
+                if self.can_fast_delete(related_qs, from_field=field):
+                    self.fast_deletes.append(related_qs)
+                elif related_qs:
+                    field.remote_field.on_delete(self, field, related_qs, self.using)
+            for field in model._meta.private_fields:
+                if hasattr(field, 'bulk_related_objects'):
+                    # It's something like generic foreign key.
+                    sub_objs = bulk_related_objects(field, new_objs, self.using)
+                    self.collect(sub_objs, source=model, nullable=True)
+
+    def delete(self):
+        self.sort()
+
+        # collect pk_list before deletion (once things start to delete
+        # queries might not be able to retreive pk list)
+        del_dict = OrderedDict()
+        for model, instances in self.data.items():
+            del_dict.setdefault(model, [])
+            for inst in instances:
+                del_dict[model] += list(inst.values_list('pk', flat=True))
+
+        deleted_counter = Counter()
+
+        with transaction.atomic(using=self.using, savepoint=False):
+
+            # update fields
+            for model, instances_for_fieldvalues in self.field_updates.items():
+                for (field, value), instances in instances_for_fieldvalues.items():
+                    for inst in instances:
+                        query = sql.UpdateQuery(model)
+                        query.update_batch(inst.values_list('pk', flat=True),
+                                           {field.name: value}, self.using)
+            # fast deletes
+            for qs in self.fast_deletes:
+                count = qs._raw_delete(using=self.using)
+                deleted_counter[qs.model._meta.label] += count
+
+            # delete instances
+            for model, pk_list in del_dict.items():
+                query = sql.DeleteQuery(model)
+                count = query.delete_batch(pk_list, self.using)
+                deleted_counter[model._meta.label] += count
+
+            return sum(deleted_counter.values()), dict(deleted_counter)
--- a/awx/main/management/commands/deprovision_instance.py
+++ b/awx/main/management/commands/deprovision_instance.py
@@ -1,8 +1,6 @@
 # Copyright (c) 2016 Ansible, Inc.
 # All Rights Reserved

-import subprocess
-
 from django.db import transaction
 from django.core.management.base import BaseCommand, CommandError

@@ -33,18 +31,9 @@ class Command(BaseCommand):
        with advisory_lock('instance_registration_%s' % hostname):
            instance = Instance.objects.filter(hostname=hostname)
            if instance.exists():
-                isolated = instance.first().is_isolated()
                instance.delete()
                print("Instance Removed")
-                if isolated:
-                    print('Successfully deprovisioned {}'.format(hostname))
-                else:
-                    result = subprocess.Popen("rabbitmqctl forget_cluster_node rabbitmq@{}".format(hostname), shell=True).wait()
-                    if result != 0:
-                        print("Node deprovisioning may have failed when attempting to "
-                              "remove the RabbitMQ instance {} from the cluster".format(hostname))
-                    else:
-                        print('Successfully deprovisioned {}'.format(hostname))
+                print('Successfully deprovisioned {}'.format(hostname))
                print('(changed: True)')
            else:
                print('No instance found matching name {}'.format(hostname))
--- a/awx/main/management/commands/inventory_import.py
+++ b/awx/main/management/commands/inventory_import.py
@@ -169,7 +169,7 @@ class AnsibleInventoryLoader(object):
            self.tmp_private_dir = build_proot_temp_dir()
            logger.debug("Using fresh temporary directory '{}' for isolation.".format(self.tmp_private_dir))
            kwargs['proot_temp_dir'] = self.tmp_private_dir
-            kwargs['proot_show_paths'] = [functioning_dir(self.source)]
+            kwargs['proot_show_paths'] = [functioning_dir(self.source), settings.AWX_ANSIBLE_COLLECTIONS_PATHS]
        logger.debug("Running from `{}` working directory.".format(cwd))

        if self.venv_path != settings.ANSIBLE_VENV_PATH:
@@ -271,7 +271,7 @@ class Command(BaseCommand):
                                     logging.DEBUG, 0]))
        logger.setLevel(log_levels.get(self.verbosity, 0))

-    def _get_instance_id(self, from_dict, default=''):
+    def _get_instance_id(self, variables, default=''):
        '''
        Retrieve the instance ID from the given dict of host variables.

@@ -279,15 +279,23 @@ class Command(BaseCommand):
        the lookup will traverse into nested dicts, equivalent to:

        from_dict.get('foo', {}).get('bar', default)
+
+        Multiple ID variables may be specified as 'foo.bar,foobar', so that
+        it will first try to find 'bar' inside of 'foo', and if unable,
+        will try to find 'foobar' as a fallback
        '''
        instance_id = default
        if getattr(self, 'instance_id_var', None):
-            for key in self.instance_id_var.split('.'):
-                if not hasattr(from_dict, 'get'):
-                    instance_id = default
+            for single_instance_id in self.instance_id_var.split(','):
+                from_dict = variables
+                for key in single_instance_id.split('.'):
+                    if not hasattr(from_dict, 'get'):
+                        instance_id = default
+                        break
+                    instance_id = from_dict.get(key, default)
+                    from_dict = instance_id
+                if instance_id:
                    break
-                instance_id = from_dict.get(key, default)
-                from_dict = instance_id
        return smart_text(instance_id)

    def _get_enabled(self, from_dict, default=None):
@@ -422,7 +430,7 @@ class Command(BaseCommand):
            for mem_host in self.all_group.all_hosts.values():
                instance_id = self._get_instance_id(mem_host.variables)
                if not instance_id:
-                    logger.warning('Host "%s" has no "%s" variable',
+                    logger.warning('Host "%s" has no "%s" variable(s)',
                                   mem_host.name, self.instance_id_var)
                    continue
                mem_host.instance_id = instance_id
@@ -496,12 +504,6 @@ class Command(BaseCommand):
            group_names = all_group_names[offset:(offset + self._batch_size)]
            for group_pk in groups_qs.filter(name__in=group_names).values_list('pk', flat=True):
                del_group_pks.discard(group_pk)
-        if self.inventory_source.deprecated_group_id in del_group_pks:  # TODO: remove in 3.3
-            logger.warning(
-                'Group "%s" from v1 API is not deleted by overwrite',
-                self.inventory_source.deprecated_group.name
-            )
-            del_group_pks.discard(self.inventory_source.deprecated_group_id)
        # Now delete all remaining groups in batches.
        all_del_pks = sorted(list(del_group_pks))
        for offset in range(0, len(all_del_pks), self._batch_size):
@@ -534,12 +536,6 @@ class Command(BaseCommand):
        # Set of all host pks managed by this inventory source
        all_source_host_pks = self._existing_host_pks()
        for db_group in db_groups.all():
-            if self.inventory_source.deprecated_group_id == db_group.id:  # TODO: remove in 3.3
-                logger.debug(
-                    'Group "%s" from v1 API child group/host connections preserved',
-                    db_group.name
-                )
-                continue
            # Delete child group relationships not present in imported data.
            db_children = db_group.children
            db_children_name_pk_map = dict(db_children.values_list('name', 'pk'))
@@ -661,11 +657,12 @@ class Command(BaseCommand):
            if group_name in existing_group_names:
                continue
            mem_group = self.all_group.all_groups[group_name]
+            group_desc = mem_group.variables.pop('_awx_description', 'imported')
            group = self.inventory.groups.update_or_create(
                name=group_name,
                defaults={
                    'variables':json.dumps(mem_group.variables),
-                    'description':'imported'
+                    'description':group_desc
                }
            )[0]
            logger.debug('Group "%s" added', group.name)
@@ -788,8 +785,9 @@ class Command(BaseCommand):
        # Create any new hosts.
        for mem_host_name in sorted(mem_host_names_to_update):
            mem_host = self.all_group.all_hosts[mem_host_name]
-            host_attrs = dict(variables=json.dumps(mem_host.variables),
-                              description='imported')
+            import_vars = mem_host.variables
+            host_desc = import_vars.pop('_awx_description', 'imported')
+            host_attrs = dict(variables=json.dumps(import_vars), description=host_desc)
            enabled = self._get_enabled(mem_host.variables)
            if enabled is not None:
                host_attrs['enabled'] = enabled
@@ -1006,12 +1004,6 @@ class Command(BaseCommand):
        except re.error:
            raise CommandError('invalid regular expression for --host-filter')

-        '''
-        TODO: Remove this deprecation when we remove support for rax.py
-        '''
-        if self.source == "rax.py":
-            logger.info("Rackspace inventory sync is Deprecated in Tower 3.1.0 and support for Rackspace will be removed in a future release.")
-
        begin = time.time()
        self.load_inventory_from_database()

@@ -1096,7 +1088,7 @@ class Command(BaseCommand):
                            if settings.SQL_DEBUG:
                                logger.warning('update computed fields took %d queries',
                                               len(connection.queries) - queries_before2)
-                        # Check if the license is valid. 
+                        # Check if the license is valid.
                        # If the license is not valid, a CommandError will be thrown,
                        # and inventory update will be marked as invalid.
                        # with transaction.atomic() will roll back the changes.
--- a/awx/main/management/commands/regenerate_secret_key.py
+++ b/awx/main/management/commands/regenerate_secret_key.py
@@ -82,7 +82,7 @@ class Command(BaseCommand):
            OAuth2Application.objects.filter(pk=app.pk).update(client_secret=encrypted)

    def _settings(self):
-        # don't update memcached, the *actual* value isn't changing
+        # don't update the cache, the *actual* value isn't changing
        post_save.disconnect(on_post_save_setting, sender=Setting)
        for setting in Setting.objects.filter().order_by('pk'):
            if settings_registry.is_setting_encrypted(setting.key):
--- a/awx/main/management/commands/register_queue.py
+++ b/awx/main/management/commands/register_queue.py
@@ -6,6 +6,7 @@ from awx.main.utils.pglock import advisory_lock
 from awx.main.models import Instance, InstanceGroup

 from django.core.management.base import BaseCommand, CommandError
+from django.db import transaction


 class InstanceNotFound(Exception):
@@ -15,32 +16,24 @@ class InstanceNotFound(Exception):
        super(InstanceNotFound, self).__init__(*args, **kwargs)


-class Command(BaseCommand):
+class RegisterQueue:
+    def __init__(self, queuename, controller, instance_percent, inst_min, hostname_list):
+        self.instance_not_found_err = None
+        self.queuename = queuename
+        self.controller = controller
+        self.instance_percent = instance_percent
+        self.instance_min = inst_min
+        self.hostname_list = hostname_list

-    def add_arguments(self, parser):
-        parser.add_argument('--queuename', dest='queuename', type=str,
-                            help='Queue to create/update')
-        parser.add_argument('--hostnames', dest='hostnames', type=str,
-                            help='Comma-Delimited Hosts to add to the Queue (will not remove already assigned instances)')
-        parser.add_argument('--controller', dest='controller', type=str,
-                            default='', help='The controlling group (makes this an isolated group)')
-        parser.add_argument('--instance_percent', dest='instance_percent', type=int, default=0,
-                            help='The percentage of active instances that will be assigned to this group'),
-        parser.add_argument('--instance_minimum', dest='instance_minimum', type=int, default=0,
-                            help='The minimum number of instance that will be retained for this group from available instances')
-
-
-    def get_create_update_instance_group(self, queuename, instance_percent, instance_min):
-        ig = InstanceGroup.objects.filter(name=queuename)
+    def get_create_update_instance_group(self):
        created = False
        changed = False
-
-        (ig, created) = InstanceGroup.objects.get_or_create(name=queuename)
-        if ig.policy_instance_percentage != instance_percent:
-            ig.policy_instance_percentage = instance_percent
+        (ig, created) = InstanceGroup.objects.get_or_create(name=self.queuename)
+        if ig.policy_instance_percentage != self.instance_percent:
+            ig.policy_instance_percentage = self.instance_percent
            changed = True
-        if ig.policy_instance_minimum != instance_min:
-            ig.policy_instance_minimum = instance_min
+        if ig.policy_instance_minimum != self.instance_min:
+            ig.policy_instance_minimum = self.instance_min
            changed = True

        if changed:
@@ -48,12 +41,12 @@ class Command(BaseCommand):

        return (ig, created, changed)

-    def update_instance_group_controller(self, ig, controller):
+    def update_instance_group_controller(self, ig):
        changed = False
        control_ig = None

-        if controller:
-            control_ig = InstanceGroup.objects.filter(name=controller).first()
+        if self.controller:
+            control_ig = InstanceGroup.objects.filter(name=self.controller).first()

        if control_ig and ig.controller_id != control_ig.pk:
            ig.controller = control_ig
@@ -62,10 +55,10 @@ class Command(BaseCommand):

        return (control_ig, changed)

-    def add_instances_to_group(self, ig, hostname_list):
+    def add_instances_to_group(self, ig):
        changed = False

-        instance_list_unique = set([x.strip() for x in hostname_list if x])
+        instance_list_unique = set([x.strip() for x in self.hostname_list if x])
        instances = []
        for inst_name in instance_list_unique:
            instance = Instance.objects.filter(hostname=inst_name)
@@ -86,42 +79,61 @@ class Command(BaseCommand):

        return (instances, changed)

+    def register(self):
+        with advisory_lock('cluster_policy_lock'):
+            with transaction.atomic():
+                changed2 = False
+                changed3 = False
+                (ig, created, changed1) = self.get_create_update_instance_group()
+                if created:
+                    print("Creating instance group {}".format(ig.name))
+                elif not created:
+                    print("Instance Group already registered {}".format(ig.name))
+
+                if self.controller:
+                    (ig_ctrl, changed2) = self.update_instance_group_controller(ig)
+                    if changed2:
+                        print("Set controller group {} on {}.".format(self.controller, self.queuename))
+
+                try:
+                    (instances, changed3) = self.add_instances_to_group(ig)
+                    for i in instances:
+                        print("Added instance {} to {}".format(i.hostname, ig.name))
+                except InstanceNotFound as e:
+                    self.instance_not_found_err = e
+
+        if any([changed1, changed2, changed3]):
+            print('(changed: True)')
+
+
+class Command(BaseCommand):
+
+    def add_arguments(self, parser):
+        parser.add_argument('--queuename', dest='queuename', type=str,
+                            help='Queue to create/update')
+        parser.add_argument('--hostnames', dest='hostnames', type=str,
+                            help='Comma-Delimited Hosts to add to the Queue (will not remove already assigned instances)')
+        parser.add_argument('--controller', dest='controller', type=str,
+                            default='', help='The controlling group (makes this an isolated group)')
+        parser.add_argument('--instance_percent', dest='instance_percent', type=int, default=0,
+                            help='The percentage of active instances that will be assigned to this group'),
+        parser.add_argument('--instance_minimum', dest='instance_minimum', type=int, default=0,
+                            help='The minimum number of instance that will be retained for this group from available instances')
+
+
    def handle(self, **options):
-        instance_not_found_err = None
        queuename = options.get('queuename')
        if not queuename:
            raise CommandError("Specify `--queuename` to use this command.")
        ctrl = options.get('controller')
        inst_per = options.get('instance_percent')
-        inst_min = options.get('instance_minimum')
+        instance_min = options.get('instance_minimum')
        hostname_list = []
        if options.get('hostnames'):
            hostname_list = options.get('hostnames').split(",")

-        with advisory_lock('instance_group_registration_{}'.format(queuename)):
-            changed2 = False
-            changed3 = False
-            (ig, created, changed1) = self.get_create_update_instance_group(queuename, inst_per, inst_min)
-            if created:
-                print("Creating instance group {}".format(ig.name))
-            elif not created:
-                print("Instance Group already registered {}".format(ig.name))
-
-            if ctrl:
-                (ig_ctrl, changed2) = self.update_instance_group_controller(ig, ctrl)
-                if changed2:
-                    print("Set controller group {} on {}.".format(ctrl, queuename))
-
-            try:
-                (instances, changed3) = self.add_instances_to_group(ig, hostname_list)
-                for i in instances:
-                    print("Added instance {} to {}".format(i.hostname, ig.name))
-            except InstanceNotFound as e:
-                instance_not_found_err = e
-
-        if any([changed1, changed2, changed3]):
-            print('(changed: True)')
-
-        if instance_not_found_err:
-            print(instance_not_found_err.message)
+        rq = RegisterQueue(queuename, ctrl, inst_per, instance_min, hostname_list)
+        rq.register()
+        if rq.instance_not_found_err:
+            print(rq.instance_not_found_err.message)
            sys.exit(1)
--- a/awx/main/management/commands/run_callback_receiver.py
+++ b/awx/main/management/commands/run_callback_receiver.py
@@ -3,10 +3,8 @@

 from django.conf import settings
 from django.core.management.base import BaseCommand
-from kombu import Exchange, Queue

-from awx.main.dispatch.kombu import Connection
-from awx.main.dispatch.worker import AWXConsumer, CallbackBrokerWorker
+from awx.main.dispatch.worker import AWXConsumerRedis, CallbackBrokerWorker


 class Command(BaseCommand):
@@ -18,23 +16,15 @@ class Command(BaseCommand):
    help = 'Launch the job callback receiver'

    def handle(self, *arg, **options):
-        with Connection(settings.BROKER_URL) as conn:
-            consumer = None
-            try:
-                consumer = AWXConsumer(
-                    'callback_receiver',
-                    conn,
-                    CallbackBrokerWorker(),
-                    [
-                        Queue(
-                            settings.CALLBACK_QUEUE,
-                            Exchange(settings.CALLBACK_QUEUE, type='direct'),
-                            routing_key=settings.CALLBACK_QUEUE
-                        )
-                    ]
-                )
-                consumer.run()
-            except KeyboardInterrupt:
-                print('Terminating Callback Receiver')
-                if consumer:
-                    consumer.stop()
+        consumer = None
+        try:
+            consumer = AWXConsumerRedis(
+                'callback_receiver',
+                CallbackBrokerWorker(),
+                queues=[getattr(settings, 'CALLBACK_QUEUE', '')],
+            )
+            consumer.run()
+        except KeyboardInterrupt:
+            print('Terminating Callback Receiver')
+            if consumer:
+                consumer.stop()
--- a/awx/main/management/commands/run_dispatcher.py
+++ b/awx/main/management/commands/run_dispatcher.py
@@ -1,21 +1,16 @@
 # Copyright (c) 2015 Ansible, Inc.
 # All Rights Reserved.
-import os
 import logging
-from multiprocessing import Process

 from django.conf import settings
 from django.core.cache import cache as django_cache
 from django.core.management.base import BaseCommand
-from django.db import connection as django_connection, connections
-from kombu import Exchange, Queue
+from django.db import connection as django_connection

-from awx.main.utils.handlers import AWXProxyHandler
 from awx.main.dispatch import get_local_queuename, reaper
 from awx.main.dispatch.control import Control
-from awx.main.dispatch.kombu import Connection
 from awx.main.dispatch.pool import AutoscalePool
-from awx.main.dispatch.worker import AWXConsumer, TaskWorker
+from awx.main.dispatch.worker import AWXConsumerPG, TaskWorker
 from awx.main.dispatch import periodic

 logger = logging.getLogger('awx.main.dispatch')
@@ -49,47 +44,27 @@ class Command(BaseCommand):

        # It's important to close these because we're _about_ to fork, and we
        # don't want the forked processes to inherit the open sockets
-        # for the DB and memcached connections (that way lies race conditions)
+        # for the DB and cache connections (that way lies race conditions)
        django_connection.close()
        django_cache.close()

        # spawn a daemon thread to periodically enqueues scheduled tasks
        # (like the node heartbeat)
-        cease_continuous_run = periodic.run_continuously()
+        periodic.run_continuously()

        reaper.reap()
        consumer = None

-        # don't ship external logs inside the dispatcher's parent process
-        # this exists to work around a race condition + deadlock bug on fork
-        # in cpython itself:
-        # https://bugs.python.org/issue37429
-        AWXProxyHandler.disable()
-        with Connection(settings.BROKER_URL) as conn:
-            try:
-                bcast = 'tower_broadcast_all'
-                queues = [
-                    Queue(q, Exchange(q), routing_key=q)
-                    for q in (settings.AWX_CELERY_QUEUES_STATIC + [get_local_queuename()])
-                ]
-                queues.append(
-                    Queue(
-                        construct_bcast_queue_name(bcast),
-                        exchange=Exchange(bcast, type='fanout'),
-                        routing_key=bcast,
-                        reply=True
-                    )
-                )
-                consumer = AWXConsumer(
-                    'dispatcher',
-                    conn,
-                    TaskWorker(),
-                    queues,
-                    AutoscalePool(min_workers=4)
-                )
-                consumer.run()
-            except KeyboardInterrupt:
-                cease_continuous_run.set()
-                logger.debug('Terminating Task Dispatcher')
-                if consumer:
-                    consumer.stop()
+        try:
+            queues = ['tower_broadcast_all', get_local_queuename()]
+            consumer = AWXConsumerPG(
+                'dispatcher',
+                TaskWorker(),
+                queues,
+                AutoscalePool(min_workers=4)
+            )
+            consumer.run()
+        except KeyboardInterrupt:
+            logger.debug('Terminating Task Dispatcher')
+            if consumer:
+                consumer.stop()
--- a/awx/main/management/commands/run_wsbroadcast.py
+++ b/awx/main/management/commands/run_wsbroadcast.py
@@ -0,0 +1,167 @@
+# Copyright (c) 2015 Ansible, Inc.
+# All Rights Reserved.
+import logging
+import asyncio
+import datetime
+import re
+import redis
+import time
+from datetime import datetime as dt
+
+from django.core.management.base import BaseCommand
+from django.db import connection
+from django.db.models import Q
+from django.db.migrations.executor import MigrationExecutor
+
+from awx.main.analytics.broadcast_websocket import (
+    BroadcastWebsocketStatsManager,
+    safe_name,
+)
+from awx.main.wsbroadcast import BroadcastWebsocketManager
+
+
+logger = logging.getLogger('awx.main.wsbroadcast')
+
+
+class Command(BaseCommand):
+    help = 'Launch the websocket broadcaster'
+
+    def add_arguments(self, parser):
+        parser.add_argument('--status', dest='status', action='store_true',
+                            help='print the internal state of any running broadcast websocket')
+
+    @classmethod
+    def display_len(cls, s):
+        return len(re.sub('\x1b.*?m', '', s))
+
+    @classmethod
+    def _format_lines(cls, host_stats, padding=5):
+        widths = [0 for i in host_stats[0]]
+        for entry in host_stats:
+            for i, e in enumerate(entry):
+                if Command.display_len(e) > widths[i]:
+                    widths[i] = Command.display_len(e)
+        paddings = [padding for i in widths]
+
+        lines = []
+        for entry in host_stats:
+            line = ""
+            for pad, width, value in zip(paddings, widths, entry):
+                if len(value) > Command.display_len(value):
+                    width += len(value) - Command.display_len(value)
+                total_width = width + pad
+                line += f'{value:{total_width}}'
+            lines.append(line)
+        return lines
+
+    @classmethod
+    def get_connection_status(cls, me, hostnames, data):
+        host_stats = [('hostname', 'state', 'start time', 'duration (sec)')]
+        for h in hostnames:
+            connection_color = '91'    # red
+            h_safe = safe_name(h)
+            prefix = f'awx_{h_safe}'
+            connection_state = data.get(f'{prefix}_connection', 'N/A')
+            connection_started = 'N/A'
+            connection_duration = 'N/A'
+            if connection_state is None:
+                connection_state = 'unknown'
+            if connection_state == 'connected':
+                connection_color = '92' # green
+                connection_started = data.get(f'{prefix}_connection_start', 'Error')
+                if connection_started != 'Error':
+                    connection_started = datetime.datetime.fromtimestamp(connection_started)
+                    connection_duration = int((dt.now() - connection_started).total_seconds())
+
+            connection_state = f'\033[{connection_color}m{connection_state}\033[0m'
+
+            host_stats.append((h, connection_state, str(connection_started), str(connection_duration)))
+
+        return host_stats
+
+    @classmethod
+    def get_connection_stats(cls, me, hostnames, data):
+        host_stats = [('hostname', 'total', 'per minute')]
+        for h in hostnames:
+            h_safe = safe_name(h)
+            prefix = f'awx_{h_safe}'
+            messages_total = data.get(f'{prefix}_messages_received', '0')
+            messages_per_minute = data.get(f'{prefix}_messages_received_per_minute', '0')
+
+            host_stats.append((h, str(int(messages_total)), str(int(messages_per_minute))))
+
+        return host_stats
+
+    def handle(self, *arg, **options):
+        # it's necessary to delay this import in case
+        # database migrations are still running
+        from awx.main.models.ha import Instance
+
+        executor = MigrationExecutor(connection)
+        migrating = bool(executor.migration_plan(executor.loader.graph.leaf_nodes()))
+        registered = False
+
+        if not migrating:
+            try:
+                Instance.objects.me()
+                registered = True
+            except RuntimeError:
+                pass
+
+        if migrating or not registered:
+            # In containerized deployments, migrations happen in the task container,
+            # and the services running there don't start until migrations are
+            # finished.
+            # *This* service runs in the web container, and it's possible that it can
+            # start _before_ migrations are finished, thus causing issues with the ORM
+            # queries it makes (specifically, conf.settings queries).
+            # This block is meant to serve as a sort of bail-out for the situation
+            # where migrations aren't yet finished (similar to the migration
+            # detection middleware that the uwsgi processes have) or when instance
+            # registration isn't done yet
+            logger.error('AWX is currently installing/upgrading.  Trying again in 5s...')
+            time.sleep(5)
+            return
+
+        if options.get('status'):
+            try:
+                stats_all = BroadcastWebsocketStatsManager.get_stats_sync()
+            except redis.exceptions.ConnectionError as e:
+                print(f"Unable to get Broadcast Websocket Status. Failed to connect to redis {e}")
+                return
+
+            data = {}
+            for family in stats_all:
+                if family.type == 'gauge' and len(family.samples) > 1:
+                    for sample in family.samples:
+                        if sample.value >= 1:
+                            data[family.name] = sample.labels[family.name]
+                            break
+                else:
+                    data[family.name] = family.samples[0].value
+
+            me = Instance.objects.me()
+            hostnames = [i.hostname for i in Instance.objects.exclude(Q(hostname=me.hostname) | Q(rampart_groups__controller__isnull=False))]
+
+            host_stats = Command.get_connection_status(me, hostnames, data)
+            lines = Command._format_lines(host_stats)
+
+            print(f'Broadcast websocket connection status from "{me.hostname}" to:')
+            print('\n'.join(lines))
+
+            host_stats = Command.get_connection_stats(me, hostnames, data)
+            lines = Command._format_lines(host_stats)
+
+            print(f'\nBroadcast websocket connection stats from "{me.hostname}" to:')
+            print('\n'.join(lines))
+
+            return
+
+        try:
+            broadcast_websocket_mgr = BroadcastWebsocketManager()
+            task = broadcast_websocket_mgr.start()
+
+            loop = asyncio.get_event_loop()
+            loop.run_until_complete(task)
+        except KeyboardInterrupt:
+            logger.debug('Terminating Websocket Broadcaster')
--- a/awx/main/managers.py
+++ b/awx/main/managers.py
@@ -3,6 +3,7 @@

 import sys
 import logging
+import os

 from django.db import models
 from django.conf import settings
@@ -43,20 +44,6 @@ class HostManager(models.Manager):
            inventory_sources__source='tower'
        ).filter(inventory__organization=org_id).values('name').distinct().count()

-    def active_counts_by_org(self):
-        """Return the counts of active, unique hosts for each organization.
-        Construction of query involves:
-         - remove any ordering specified in model's Meta
-         - Exclude hosts sourced from another Tower
-         - Consider only hosts where the canonical inventory is owned by each organization
-         - Restrict the query to only count distinct names
-         - Return the counts
-        """
-        return self.order_by().exclude(
-            inventory_sources__source='tower'
-        ).values('inventory__organization').annotate(
-            inventory__organization__count=models.Count('name', distinct=True))
-
    def get_queryset(self):
        """When the parent instance of the host query set has a `kind=smart` and a `host_filter`
        set. Use the `host_filter` to generate the queryset for the hosts.
@@ -78,8 +65,7 @@ class HostManager(models.Manager):
                self.core_filters = {}

                qs = qs & q
-                unique_by_name = qs.order_by('name', 'pk').distinct('name')
-                return qs.filter(pk__in=unique_by_name)
+                return qs.order_by('name', 'pk').distinct('name')
        return qs


@@ -115,21 +101,45 @@ class InstanceManager(models.Manager):
            return node[0]
        raise RuntimeError("No instance found with the current cluster host id")

-    def register(self, uuid=None, hostname=None):
+    def register(self, uuid=None, hostname=None, ip_address=None):
        if not uuid:
            uuid = settings.SYSTEM_UUID
        if not hostname:
            hostname = settings.CLUSTER_HOST_ID
        with advisory_lock('instance_registration_%s' % hostname):
+            if settings.AWX_AUTO_DEPROVISION_INSTANCES:
+                # detect any instances with the same IP address.
+                # if one exists, set it to None
+                inst_conflicting_ip = self.filter(ip_address=ip_address).exclude(hostname=hostname)
+                if inst_conflicting_ip.exists():
+                    for other_inst in inst_conflicting_ip:
+                        other_hostname = other_inst.hostname
+                        other_inst.ip_address = None
+                        other_inst.save(update_fields=['ip_address'])
+                        logger.warning("IP address {0} conflict detected, ip address unset for host {1}.".format(ip_address, other_hostname))
+
            instance = self.filter(hostname=hostname)
            if instance.exists():
-                return (False, instance[0])
-            instance = self.create(uuid=uuid, hostname=hostname, capacity=0)
+                instance = instance.get()
+                if instance.ip_address != ip_address:
+                    instance.ip_address = ip_address
+                    instance.save(update_fields=['ip_address'])
+                    return (True, instance)
+                else:
+                    return (False, instance)
+            instance = self.create(uuid=uuid,
+                                   hostname=hostname,
+                                   ip_address=ip_address,
+                                   capacity=0)
        return (True, instance)

    def get_or_register(self):
        if settings.AWX_AUTO_DEPROVISION_INSTANCES:
-            return self.register()
+            from awx.main.management.commands.register_queue import RegisterQueue
+            pod_ip = os.environ.get('MY_POD_IP')
+            registered = self.register(ip_address=pod_ip)
+            RegisterQueue('tower', None, 100, 0, []).register()
+            return registered
        else:
            return (False, self.me())

--- a/awx/main/middleware.py
+++ b/awx/main/middleware.py
@@ -12,23 +12,19 @@ import urllib.parse

 from django.conf import settings
 from django.contrib.auth.models import User
-from django.db.models.signals import post_save
 from django.db.migrations.executor import MigrationExecutor
-from django.db import IntegrityError, connection
-from django.utils.functional import curry
+from django.db import connection
 from django.shortcuts import get_object_or_404, redirect
 from django.apps import apps
 from django.utils.deprecation import MiddlewareMixin
 from django.utils.translation import ugettext_lazy as _
 from django.urls import reverse, resolve

-from awx.main.models import ActivityStream
 from awx.main.utils.named_url_graph import generate_graph, GraphNode
 from awx.conf import fields, register


 logger = logging.getLogger('awx.main.middleware')
-analytics_logger = logging.getLogger('awx.analytics.activity_stream')
 perf_logger = logging.getLogger('awx.analytics.performance')


@@ -76,61 +72,6 @@ class TimingMiddleware(threading.local, MiddlewareMixin):
        return filepath


-class ActivityStreamMiddleware(threading.local, MiddlewareMixin):
-
-    def __init__(self, get_response=None):
-        self.disp_uid = None
-        self.instance_ids = []
-        super().__init__(get_response)
-
-    def process_request(self, request):
-        if hasattr(request, 'user') and request.user.is_authenticated:
-            user = request.user
-        else:
-            user = None
-
-        set_actor = curry(self.set_actor, user)
-        self.disp_uid = str(uuid.uuid1())
-        self.instance_ids = []
-        post_save.connect(set_actor, sender=ActivityStream, dispatch_uid=self.disp_uid, weak=False)
-
-    def process_response(self, request, response):
-        drf_request = getattr(request, 'drf_request', None)
-        drf_user = getattr(drf_request, 'user', None)
-        if self.disp_uid is not None:
-            post_save.disconnect(dispatch_uid=self.disp_uid)
-
-        for instance in ActivityStream.objects.filter(id__in=self.instance_ids):
-            if drf_user and drf_user.id:
-                instance.actor = drf_user
-                try:
-                    instance.save(update_fields=['actor'])
-                    analytics_logger.info('Activity Stream update entry for %s' % str(instance.object1),
-                                          extra=dict(changes=instance.changes, relationship=instance.object_relationship_type,
-                                          actor=drf_user.username, operation=instance.operation,
-                                          object1=instance.object1, object2=instance.object2))
-                except IntegrityError:
-                    logger.debug("Integrity Error saving Activity Stream instance for id : " + str(instance.id))
-            # else:
-            #     obj1_type_actual = instance.object1_type.split(".")[-1]
-            #     if obj1_type_actual in ("InventoryUpdate", "ProjectUpdate", "Job") and instance.id is not None:
-            #         instance.delete()
-
-        self.instance_ids = []
-        return response
-
-    def set_actor(self, user, sender, instance, **kwargs):
-        if sender == ActivityStream:
-            if isinstance(user, User) and instance.actor is None:
-                user = User.objects.filter(id=user.id)
-                if user.exists():
-                    user = user[0]
-                    instance.actor = user
-            else:
-                if instance.id not in self.instance_ids:
-                    self.instance_ids.append(instance.id)
-
-
 class SessionTimeoutMiddleware(MiddlewareMixin):
    """
    Resets the session timeout for both the UI and the actual session for the API
@@ -192,21 +133,41 @@ class URLModificationMiddleware(MiddlewareMixin):
        )
        super().__init__(get_response)

-    def _named_url_to_pk(self, node, named_url):
-        kwargs = {}
-        if not node.populate_named_url_query_kwargs(kwargs, named_url):
-            return named_url
-        return str(get_object_or_404(node.model, **kwargs).pk)
+    @staticmethod
+    def _hijack_for_old_jt_name(node, kwargs, named_url):
+        try:
+            int(named_url)
+            return False
+        except ValueError:
+            pass
+        JobTemplate = node.model
+        name = urllib.parse.unquote(named_url)
+        return JobTemplate.objects.filter(name=name).order_by('organization__created').first()

-    def _convert_named_url(self, url_path):
+    @classmethod
+    def _named_url_to_pk(cls, node, resource, named_url):
+        kwargs = {}
+        if node.populate_named_url_query_kwargs(kwargs, named_url):
+            return str(get_object_or_404(node.model, **kwargs).pk)
+        if resource == 'job_templates' and '++' not in named_url:
+            # special case for deprecated job template case
+            # will not raise a 404 on its own
+            jt = cls._hijack_for_old_jt_name(node, kwargs, named_url)
+            if jt:
+                return str(jt.pk)
+        return named_url
+
+    @classmethod
+    def _convert_named_url(cls, url_path):
        url_units = url_path.split('/')
        # If the identifier is an empty string, it is always invalid.
        if len(url_units) < 6 or url_units[1] != 'api' or url_units[2] not in ['v2'] or not url_units[4]:
            return url_path
        resource = url_units[3]
        if resource in settings.NAMED_URL_MAPPINGS:
-            url_units[4] = self._named_url_to_pk(settings.NAMED_URL_GRAPH[settings.NAMED_URL_MAPPINGS[resource]],
-                                                 url_units[4])
+            url_units[4] = cls._named_url_to_pk(
+                settings.NAMED_URL_GRAPH[settings.NAMED_URL_MAPPINGS[resource]],
+                resource, url_units[4])
        return '/'.join(url_units)

    def process_request(self, request):
--- a/awx/main/migrations/0006_v320_release.py
+++ b/awx/main/migrations/0006_v320_release.py
@@ -464,7 +464,7 @@ class Migration(migrations.Migration):
        migrations.AddField(
            model_name='unifiedjob',
            name='instance_group',
-            field=models.ForeignKey(on_delete=models.SET_NULL, default=None, blank=True, to='main.InstanceGroup', help_text='The Rampart/Instance group the job was run under', null=True),
+            field=models.ForeignKey(on_delete=models.SET_NULL, default=None, blank=True, to='main.InstanceGroup', help_text='The Instance group the job was run under', null=True),
        ),
        migrations.AddField(
            model_name='unifiedjobtemplate',
--- a/awx/main/migrations/0032_v330_polymorphic_delete.py
+++ b/awx/main/migrations/0032_v330_polymorphic_delete.py
@@ -16,6 +16,6 @@ class Migration(migrations.Migration):
        migrations.AlterField(
            model_name='unifiedjob',
            name='instance_group',
-            field=models.ForeignKey(blank=True, default=None, help_text='The Rampart/Instance group the job was run under', null=True, on_delete=awx.main.utils.polymorphic.SET_NULL, to='main.InstanceGroup'),
+            field=models.ForeignKey(blank=True, default=None, help_text='The Instance group the job was run under', null=True, on_delete=awx.main.utils.polymorphic.SET_NULL, to='main.InstanceGroup'),
        ),
    ]
--- a/awx/main/migrations/0109_v370_job_template_organization_field.py
+++ b/awx/main/migrations/0109_v370_job_template_organization_field.py
@@ -0,0 +1,81 @@
+# Generated by Django 2.2.4 on 2019-08-07 19:56
+
+import awx.main.utils.polymorphic
+import awx.main.fields
+from django.db import migrations, models
+import django.db.models.deletion
+
+from awx.main.migrations._rbac import (
+    rebuild_role_parentage, rebuild_role_hierarchy,
+    migrate_ujt_organization, migrate_ujt_organization_backward,
+    restore_inventory_admins, restore_inventory_admins_backward
+)
+
+
+def rebuild_jt_parents(apps, schema_editor):
+    rebuild_role_parentage(apps, schema_editor, models=('jobtemplate',))
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('main', '0108_v370_unifiedjob_dependencies_processed'),
+    ]
+
+    operations = [
+        # backwards parents and ancestors caching
+        migrations.RunPython(migrations.RunPython.noop, rebuild_jt_parents),
+        # add new organization field for JT and all other unified jobs
+        migrations.AddField(
+            model_name='unifiedjob',
+            name='tmp_organization',
+            field=models.ForeignKey(blank=True, help_text='The organization used to determine access to this unified job.', null=True, on_delete=awx.main.utils.polymorphic.SET_NULL, related_name='unifiedjobs', to='main.Organization'),
+        ),
+        migrations.AddField(
+            model_name='unifiedjobtemplate',
+            name='tmp_organization',
+            field=models.ForeignKey(blank=True, help_text='The organization used to determine access to this template.', null=True, on_delete=awx.main.utils.polymorphic.SET_NULL, related_name='unifiedjobtemplates', to='main.Organization'),
+        ),
+        # while new and old fields exist, copy the organization fields
+        migrations.RunPython(migrate_ujt_organization, migrate_ujt_organization_backward),
+        # with data saved, remove old fields
+        migrations.RemoveField(
+            model_name='project',
+            name='organization',
+        ),
+        migrations.RemoveField(
+            model_name='workflowjobtemplate',
+            name='organization',
+        ),
+        # now, without safely rename the new field without conflicts from old field
+        migrations.RenameField(
+            model_name='unifiedjobtemplate',
+            old_name='tmp_organization',
+            new_name='organization',
+        ),
+        migrations.RenameField(
+            model_name='unifiedjob',
+            old_name='tmp_organization',
+            new_name='organization',
+        ),
+        # parentage of job template roles has genuinely changed at this point
+        migrations.AlterField(
+            model_name='jobtemplate',
+            name='admin_role',
+            field=awx.main.fields.ImplicitRoleField(editable=False, null='True', on_delete=django.db.models.deletion.CASCADE, parent_role=['organization.job_template_admin_role'], related_name='+', to='main.Role'),
+        ),
+        migrations.AlterField(
+            model_name='jobtemplate',
+            name='execute_role',
+            field=awx.main.fields.ImplicitRoleField(editable=False, null='True', on_delete=django.db.models.deletion.CASCADE, parent_role=['admin_role', 'organization.execute_role'], related_name='+', to='main.Role'),
+        ),
+        migrations.AlterField(
+            model_name='jobtemplate',
+            name='read_role',
+            field=awx.main.fields.ImplicitRoleField(editable=False, null='True', on_delete=django.db.models.deletion.CASCADE, parent_role=['organization.auditor_role', 'inventory.organization.auditor_role', 'execute_role', 'admin_role'], related_name='+', to='main.Role'),
+        ),
+        # Re-compute the role parents and ancestors caching
+        migrations.RunPython(rebuild_jt_parents, migrations.RunPython.noop),
+        # for all permissions that will be removed, make them explicit
+        migrations.RunPython(restore_inventory_admins, restore_inventory_admins_backward),
+    ]
--- a/awx/main/migrations/0110_v370_instance_ip_address.py
+++ b/awx/main/migrations/0110_v370_instance_ip_address.py
@@ -0,0 +1,18 @@
+# Generated by Django 2.2.8 on 2020-02-12 17:55
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('main', '0109_v370_job_template_organization_field'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='instance',
+            name='ip_address',
+            field=models.CharField(blank=True, default=None, max_length=50, null=True, unique=True),
+        ),
+    ]
--- a/awx/main/migrations/0111_v370_delete_channelgroup.py
+++ b/awx/main/migrations/0111_v370_delete_channelgroup.py
@@ -0,0 +1,16 @@
+# Generated by Django 2.2.8 on 2020-02-17 14:50
+
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('main', '0110_v370_instance_ip_address'),
+    ]
+
+    operations = [
+        migrations.DeleteModel(
+            name='ChannelGroup',
+        ),
+    ]
--- a/awx/main/migrations/0112_v370_workflow_node_identifier.py
+++ b/awx/main/migrations/0112_v370_workflow_node_identifier.py
@@ -0,0 +1,61 @@
+# Generated by Django 2.2.8 on 2020-03-14 02:29
+
+from django.db import migrations, models
+import uuid
+import logging
+
+
+logger = logging.getLogger('awx.main.migrations')
+
+
+def create_uuid(apps, schema_editor):
+    WorkflowJobTemplateNode = apps.get_model('main', 'WorkflowJobTemplateNode')
+    ct = 0
+    for node in WorkflowJobTemplateNode.objects.iterator():
+        node.identifier = uuid.uuid4()
+        node.save(update_fields=['identifier'])
+        ct += 1
+    if ct:
+        logger.info(f'Automatically created uuid4 identifier for {ct} workflow nodes')
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('main', '0111_v370_delete_channelgroup'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='workflowjobnode',
+            name='identifier',
+            field=models.CharField(blank=True, help_text='An identifier coresponding to the workflow job template node that this node was created from.', max_length=512),
+        ),
+        migrations.AddField(
+            model_name='workflowjobtemplatenode',
+            name='identifier',
+            field=models.CharField(blank=True, null=True, help_text='An identifier for this node that is unique within its workflow. It is copied to workflow job nodes corresponding to this node.', max_length=512),
+        ),
+        migrations.RunPython(create_uuid, migrations.RunPython.noop),  # this fixes the uuid4 issue
+        migrations.AlterField(
+            model_name='workflowjobtemplatenode',
+            name='identifier',
+            field=models.CharField(default=uuid.uuid4, help_text='An identifier for this node that is unique within its workflow. It is copied to workflow job nodes corresponding to this node.', max_length=512),
+        ),
+        migrations.AlterUniqueTogether(
+            name='workflowjobtemplatenode',
+            unique_together={('identifier', 'workflow_job_template')},
+        ),
+        migrations.AddIndex(
+            model_name='workflowjobnode',
+            index=models.Index(fields=['identifier', 'workflow_job'], name='main_workfl_identif_87b752_idx'),
+        ),
+        migrations.AddIndex(
+            model_name='workflowjobnode',
+            index=models.Index(fields=['identifier'], name='main_workfl_identif_efdfe8_idx'),
+        ),
+        migrations.AddIndex(
+            model_name='workflowjobtemplatenode',
+            index=models.Index(fields=['identifier'], name='main_workfl_identif_0cc025_idx'),
+        ),
+    ]
--- a/awx/main/migrations/0113_v370_event_bigint.py
+++ b/awx/main/migrations/0113_v370_event_bigint.py
@@ -0,0 +1,118 @@
+# Generated by Django 2.2.8 on 2020-02-21 16:31
+
+from django.db import migrations, models, connection
+
+
+def migrate_event_data(apps, schema_editor):
+    # see: https://github.com/ansible/awx/issues/6010
+    #
+    # the goal of this function is to end with event tables (e.g., main_jobevent)
+    # that have a bigint primary key (because the old usage of an integer
+    # numeric isn't enough, as its range is about 2.1B, see:
+    # https://www.postgresql.org/docs/9.1/datatype-numeric.html)
+
+    # unfortunately, we can't do this with a simple ALTER TABLE, because
+    # for tables with hundreds of millions or billions of rows, the ALTER TABLE
+    # can take *hours* on modest hardware.
+    #
+    # the approach in this migration means that post-migration, event data will
+    # *not* immediately show up, but will be repopulated over time progressively
+    # the trade-off here is not having to wait hours for the full data migration
+    # before you can start and run AWX again (including new playbook runs)
+    for tblname in (
+        'main_jobevent', 'main_inventoryupdateevent',
+        'main_projectupdateevent', 'main_adhoccommandevent',
+        'main_systemjobevent'
+    ):
+        with connection.cursor() as cursor:
+            # rename the current event table
+            cursor.execute(
+                f'ALTER TABLE {tblname} RENAME TO _old_{tblname};'
+            )
+            # create a *new* table with the same schema
+            cursor.execute(
+                f'CREATE TABLE {tblname} (LIKE _old_{tblname} INCLUDING ALL);'
+            )
+            # alter the *new* table so that the primary key is a big int
+            cursor.execute(
+                f'ALTER TABLE {tblname} ALTER COLUMN id TYPE bigint USING id::bigint;'
+            )
+
+            # recreate counter for the new table's primary key to
+            # start where the *old* table left off (we have to do this because the
+            # counter changed from an int to a bigint)
+            cursor.execute(f'DROP SEQUENCE IF EXISTS "{tblname}_id_seq" CASCADE;')
+            cursor.execute(f'CREATE SEQUENCE "{tblname}_id_seq";')
+            cursor.execute(
+                f'ALTER TABLE "{tblname}" ALTER COLUMN "id" '
+                f"SET DEFAULT nextval('{tblname}_id_seq');"
+            )
+            cursor.execute(
+                f"SELECT setval('{tblname}_id_seq', (SELECT MAX(id) FROM _old_{tblname}), true);"
+            )
+
+            # replace the BTREE index on main_jobevent.job_id with
+            # a BRIN index to drastically improve per-UJ lookup performance
+            # see: https://info.crunchydata.com/blog/postgresql-brin-indexes-big-data-performance-with-minimal-storage
+            if tblname == 'main_jobevent':
+                cursor.execute("SELECT indexname FROM pg_indexes WHERE tablename='main_jobevent' AND indexdef LIKE '%USING btree (job_id)';")
+                old_index = cursor.fetchone()[0]
+                cursor.execute(f'DROP INDEX {old_index}')
+                cursor.execute('CREATE INDEX main_jobevent_job_id_brin_idx ON main_jobevent USING brin (job_id);')
+
+            # remove all of the indexes and constraints from the old table
+            # (they just slow down the data migration)
+            cursor.execute(f"SELECT indexname, indexdef FROM pg_indexes WHERE tablename='_old_{tblname}' AND indexname != '{tblname}_pkey';")
+            indexes = cursor.fetchall()
+
+            cursor.execute(f"SELECT conname, contype, pg_catalog.pg_get_constraintdef(r.oid, true) as condef FROM pg_catalog.pg_constraint r WHERE r.conrelid = '_old_{tblname}'::regclass AND conname != '{tblname}_pkey';")
+            constraints = cursor.fetchall()
+
+            for indexname, indexdef in indexes:
+                cursor.execute(f'DROP INDEX IF EXISTS {indexname}')
+            for conname, contype, condef in constraints:
+                cursor.execute(f'ALTER TABLE _old_{tblname} DROP CONSTRAINT IF EXISTS {conname}')
+
+
+class FakeAlterField(migrations.AlterField):
+
+    def database_forwards(self, *args):
+        # this is intentionally left blank, because we're
+        # going to accomplish the migration with some custom raw SQL
+        pass
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('main', '0112_v370_workflow_node_identifier'),
+    ]
+
+    operations = [
+        migrations.RunPython(migrate_event_data),
+        FakeAlterField(
+            model_name='adhoccommandevent',
+            name='id',
+            field=models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID'),
+        ),
+        FakeAlterField(
+            model_name='inventoryupdateevent',
+            name='id',
+            field=models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID'),
+        ),
+        FakeAlterField(
+            model_name='jobevent',
+            name='id',
+            field=models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID'),
+        ),
+        FakeAlterField(
+            model_name='projectupdateevent',
+            name='id',
+            field=models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID'),
+        ),
+        FakeAlterField(
+            model_name='systemjobevent',
+            name='id',
+            field=models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID'),
+        ),
+    ]
--- a/awx/main/migrations/0114_v370_remove_deprecated_manual_inventory_sources.py
+++ b/awx/main/migrations/0114_v370_remove_deprecated_manual_inventory_sources.py
@@ -0,0 +1,39 @@
+# Generated by Django 2.2.11 on 2020-04-03 00:11
+
+from django.db import migrations, models
+
+
+def remove_manual_inventory_sources(apps, schema_editor):
+    '''Previously we would automatically create inventory sources after
+    Group creation and we would use the parent Group as our interface for the user.
+    During that process we would create InventorySource that had a source of "manual".
+    '''
+    InventoryUpdate = apps.get_model('main', 'InventoryUpdate')
+    InventoryUpdate.objects.filter(source='').delete()
+    InventorySource = apps.get_model('main', 'InventorySource')
+    InventorySource.objects.filter(source='').delete()
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('main', '0113_v370_event_bigint'),
+    ]
+
+    operations = [
+        migrations.RemoveField(
+            model_name='inventorysource',
+            name='deprecated_group',
+        ),
+        migrations.RunPython(remove_manual_inventory_sources),
+        migrations.AlterField(
+            model_name='inventorysource',
+            name='source',
+            field=models.CharField(choices=[('file', 'File, Directory or Script'), ('scm', 'Sourced from a Project'), ('ec2', 'Amazon EC2'), ('gce', 'Google Compute Engine'), ('azure_rm', 'Microsoft Azure Resource Manager'), ('vmware', 'VMware vCenter'), ('satellite6', 'Red Hat Satellite 6'), ('cloudforms', 'Red Hat CloudForms'), ('openstack', 'OpenStack'), ('rhv', 'Red Hat Virtualization'), ('tower', 'Ansible Tower'), ('custom', 'Custom Script')], default=None, max_length=32),
+        ),
+        migrations.AlterField(
+            model_name='inventoryupdate',
+            name='source',
+            field=models.CharField(choices=[('file', 'File, Directory or Script'), ('scm', 'Sourced from a Project'), ('ec2', 'Amazon EC2'), ('gce', 'Google Compute Engine'), ('azure_rm', 'Microsoft Azure Resource Manager'), ('vmware', 'VMware vCenter'), ('satellite6', 'Red Hat Satellite 6'), ('cloudforms', 'Red Hat CloudForms'), ('openstack', 'OpenStack'), ('rhv', 'Red Hat Virtualization'), ('tower', 'Ansible Tower'), ('custom', 'Custom Script')], default=None, max_length=32),
+        ),
+    ]
--- a/awx/main/migrations/0115_v370_schedule_set_null.py
+++ b/awx/main/migrations/0115_v370_schedule_set_null.py
@@ -0,0 +1,24 @@
+# Generated by Django 2.2.11 on 2020-05-04 02:26
+
+import awx.main.utils.polymorphic
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('main', '0114_v370_remove_deprecated_manual_inventory_sources'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='unifiedjob',
+            name='schedule',
+            field=models.ForeignKey(default=None, editable=False, null=True, on_delete=awx.main.utils.polymorphic.SET_NULL, to='main.Schedule'),
+        ),
+        migrations.AlterField(
+            model_name='unifiedjobtemplate',
+            name='next_schedule',
+            field=models.ForeignKey(default=None, editable=False, null=True, on_delete=awx.main.utils.polymorphic.SET_NULL, related_name='unifiedjobtemplate_as_next_schedule+', to='main.Schedule'),
+        ),
+    ]
--- a/awx/main/migrations/0116_v400_remove_hipchat_notifications.py
+++ b/awx/main/migrations/0116_v400_remove_hipchat_notifications.py
@@ -0,0 +1,34 @@
+# Generated by Django 2.2.11 on 2020-05-19 02:27
+
+from django.db import migrations, models
+
+
+def remove_hipchat_notifications(apps, schema_editor):
+    '''
+    HipChat notifications are no longer in service, remove any that are found.
+    '''
+    Notification = apps.get_model('main', 'Notification')
+    Notification.objects.filter(notification_type='hipchat').delete()
+    NotificationTemplate = apps.get_model('main', 'NotificationTemplate')
+    NotificationTemplate.objects.filter(notification_type='hipchat').delete()
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('main', '0115_v370_schedule_set_null'),
+    ]
+
+    operations = [
+        migrations.RunPython(remove_hipchat_notifications),
+        migrations.AlterField(
+            model_name='notification',
+            name='notification_type',
+            field=models.CharField(choices=[('email', 'Email'), ('grafana', 'Grafana'), ('irc', 'IRC'), ('mattermost', 'Mattermost'), ('pagerduty', 'Pagerduty'), ('rocketchat', 'Rocket.Chat'), ('slack', 'Slack'), ('twilio', 'Twilio'), ('webhook', 'Webhook')], max_length=32),
+        ),
+        migrations.AlterField(
+            model_name='notificationtemplate',
+            name='notification_type',
+            field=models.CharField(choices=[('email', 'Email'), ('grafana', 'Grafana'), ('irc', 'IRC'), ('mattermost', 'Mattermost'), ('pagerduty', 'Pagerduty'), ('rocketchat', 'Rocket.Chat'), ('slack', 'Slack'), ('twilio', 'Twilio'), ('webhook', 'Webhook')], max_length=32),
+        ),
+    ]
--- a/awx/main/migrations/0117_v400_remove_cloudforms_inventory.py
+++ b/awx/main/migrations/0117_v400_remove_cloudforms_inventory.py
@@ -0,0 +1,29 @@
+# Generated by Django 2.2.11 on 2020-05-01 13:25
+
+from django.db import migrations, models
+from awx.main.migrations._inventory_source import create_scm_script_substitute
+
+
+def convert_cloudforms_to_scm(apps, schema_editor):
+    create_scm_script_substitute(apps, 'cloudforms')
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('main', '0116_v400_remove_hipchat_notifications'),
+    ]
+
+    operations = [
+        migrations.RunPython(convert_cloudforms_to_scm),
+        migrations.AlterField(
+            model_name='inventorysource',
+            name='source',
+            field=models.CharField(choices=[('file', 'File, Directory or Script'), ('scm', 'Sourced from a Project'), ('ec2', 'Amazon EC2'), ('gce', 'Google Compute Engine'), ('azure_rm', 'Microsoft Azure Resource Manager'), ('vmware', 'VMware vCenter'), ('satellite6', 'Red Hat Satellite 6'), ('openstack', 'OpenStack'), ('rhv', 'Red Hat Virtualization'), ('tower', 'Ansible Tower'), ('custom', 'Custom Script')], default=None, max_length=32),
+        ),
+        migrations.AlterField(
+            model_name='inventoryupdate',
+            name='source',
+            field=models.CharField(choices=[('file', 'File, Directory or Script'), ('scm', 'Sourced from a Project'), ('ec2', 'Amazon EC2'), ('gce', 'Google Compute Engine'), ('azure_rm', 'Microsoft Azure Resource Manager'), ('vmware', 'VMware vCenter'), ('satellite6', 'Red Hat Satellite 6'), ('openstack', 'OpenStack'), ('rhv', 'Red Hat Virtualization'), ('tower', 'Ansible Tower'), ('custom', 'Custom Script')], default=None, max_length=32),
+        ),
+    ]
--- a/awx/main/migrations/_inventory_source.py
+++ b/awx/main/migrations/_inventory_source.py
@@ -1,6 +1,9 @@
 import logging

+from uuid import uuid4
+
 from django.utils.encoding import smart_text
+from django.utils.timezone import now

 from awx.main.utils.common import parse_yaml_or_json

@@ -87,3 +90,44 @@ def back_out_new_instance_id(apps, source, new_id):
            modified_ct, source
        ))

+
+def create_scm_script_substitute(apps, source):
+    """Only applies for cloudforms in practice, but written generally.
+    Given a source type, this will replace all inventory sources of that type
+    with SCM inventory sources that source the script from Ansible core
+    """
+    # the revision in the Ansible 2.9 stable branch this project will start out as
+    # it can still be updated manually later (but staying within 2.9 branch), if desired
+    ansible_rev = '6f83b9aff42331e15c55a171de0a8b001208c18c'
+    InventorySource = apps.get_model('main', 'InventorySource')
+    ContentType = apps.get_model('contenttypes', 'ContentType')
+    Project = apps.get_model('main', 'Project')
+    if not InventorySource.objects.filter(source=source).exists():
+        logger.debug('No sources of type {} to migrate'.format(source))
+        return
+    proj_name = 'Replacement project for {} type sources - {}'.format(source, uuid4())
+    right_now = now()
+    project = Project.objects.create(
+        name=proj_name,
+        created=right_now,
+        modified=right_now,
+        description='Created by migration',
+        polymorphic_ctype=ContentType.objects.get(model='project'),
+        # project-specific fields
+        scm_type='git',
+        scm_url='https://github.com/ansible/ansible.git',
+        scm_branch='stable-2.9',
+        scm_revision=ansible_rev
+    )
+    ct = 0
+    for inv_src in InventorySource.objects.filter(source=source).iterator():
+        inv_src.source = 'scm'
+        inv_src.source_project = project
+        inv_src.source_path = 'contrib/inventory/{}.py'.format(source)
+        inv_src.scm_last_revision = ansible_rev
+        inv_src.save(update_fields=['source', 'source_project', 'source_path', 'scm_last_revision'])
+        logger.debug('Changed inventory source {} to scm type'.format(inv_src.pk))
+        ct += 1
+    if ct:
+        logger.info('Changed total of {} inventory sources from {} type to scm'.format(ct, source))
+
--- a/awx/main/migrations/_rbac.py
+++ b/awx/main/migrations/_rbac.py
@@ -1,6 +1,9 @@
 import logging
 from time import time

+from django.db.models import Subquery, OuterRef, F
+
+from awx.main.fields import update_role_parentage_for_instance
 from awx.main.models.rbac import Role, batch_role_ancestor_rebuilding

 logger = logging.getLogger('rbac_migrations')
@@ -10,11 +13,11 @@ def create_roles(apps, schema_editor):
    '''
    Implicit role creation happens in our post_save hook for all of our
    resources. Here we iterate through all of our resource types and call
-    .save() to ensure all that happens for every object in the system before we
-    get busy with the actual migration work.
+    .save() to ensure all that happens for every object in the system.

-    This gets run after migrate_users, which does role creation for users a
-    little differently.
+    This can be used whenever new roles are introduced in a migration to
+    create those roles for pre-existing objects that did not previously
+    have them created via signals.
    '''

    models = [
@@ -35,7 +38,189 @@ def create_roles(apps, schema_editor):
                obj.save()


+def delete_all_user_roles(apps, schema_editor):
+    ContentType = apps.get_model('contenttypes', "ContentType")
+    Role = apps.get_model('main', "Role")
+    User = apps.get_model('auth', "User")
+    user_content_type = ContentType.objects.get_for_model(User)
+    for role in Role.objects.filter(content_type=user_content_type).iterator():
+        role.delete()
+
+
+UNIFIED_ORG_LOOKUPS = {
+    # Job Templates had an implicit organization via their project
+    'jobtemplate': 'project',
+    # Inventory Sources had an implicit organization via their inventory
+    'inventorysource': 'inventory',
+    # Projects had an explicit organization in their subclass table
+    'project': None,
+    # Workflow JTs also had an explicit organization in their subclass table
+    'workflowjobtemplate': None,
+    # Jobs inherited project from job templates as a convenience field
+    'job': 'project',
+    # Inventory Sources had an convenience field of inventory
+    'inventoryupdate': 'inventory',
+    # Project Updates did not have a direct organization field, obtained it from project
+    'projectupdate': 'project',
+    # Workflow Jobs are handled same as project updates
+    # Sliced jobs are a special case, but old data is not given special treatment for simplicity
+    'workflowjob': 'workflow_job_template',
+    # AdHocCommands do not have a template, but still migrate them
+    'adhoccommand': 'inventory'
+}
+
+
+def implicit_org_subquery(UnifiedClass, cls, backward=False):
+    """Returns a subquery that returns the so-called organization for objects
+    in the class in question, before migration to the explicit unified org field.
+    In some cases, this can still be applied post-migration.
+    """
+    if cls._meta.model_name not in UNIFIED_ORG_LOOKUPS:
+        return None
+    cls_name = cls._meta.model_name
+    source_field = UNIFIED_ORG_LOOKUPS[cls_name]
+
+    unified_field = UnifiedClass._meta.get_field(cls_name)
+    unified_ptr = unified_field.remote_field.name
+    if backward:
+        qs = UnifiedClass.objects.filter(**{cls_name: OuterRef('id')}).order_by().values_list('tmp_organization')[:1]
+    elif source_field is None:
+        qs = cls.objects.filter(**{unified_ptr: OuterRef('id')}).order_by().values_list('organization')[:1]
+    else:
+        intermediary_field = cls._meta.get_field(source_field)
+        intermediary_model = intermediary_field.related_model
+        intermediary_reverse_rel = intermediary_field.remote_field.name
+        qs = intermediary_model.objects.filter(**{
+            # this filter leverages the fact that the Unified models have same pk as subclasses.
+            # For instance... filters projects used in job template, where that job template
+            # has same id same as UJT from the outer reference (which it does)
+            intermediary_reverse_rel: OuterRef('id')}
+        ).order_by().values_list('organization')[:1]
+    return Subquery(qs)
+
+
+def _migrate_unified_organization(apps, unified_cls_name, backward=False):
+    """Given a unified base model (either UJT or UJ)
+    and a dict org_field_mapping which gives related model to get org from
+    saves organization for those objects to the temporary migration
+    variable tmp_organization on the unified model
+    (optimized method)
+    """
+    start = time()
+    UnifiedClass = apps.get_model('main', unified_cls_name)
+    ContentType = apps.get_model('contenttypes', 'ContentType')
+
+    for cls in UnifiedClass.__subclasses__():
+        cls_name = cls._meta.model_name
+        if backward and UNIFIED_ORG_LOOKUPS.get(cls_name, 'not-found') is not None:
+            logger.debug('Not reverse migrating {}, existing data should remain valid'.format(cls_name))
+            continue
+        logger.debug('{}Migrating {} to new organization field'.format('Reverse ' if backward else '', cls_name))
+
+        sub_qs = implicit_org_subquery(UnifiedClass, cls, backward=backward)
+        if sub_qs is None:
+            logger.debug('Class {} has no organization migration'.format(cls_name))
+            continue
+
+        this_ct = ContentType.objects.get_for_model(cls)
+        if backward:
+            r = cls.objects.order_by().update(organization=sub_qs)
+        else:
+            r = UnifiedClass.objects.order_by().filter(polymorphic_ctype=this_ct).update(tmp_organization=sub_qs)
+        if r:
+            logger.info('Organization migration on {} affected {} rows.'.format(cls_name, r))
+    logger.info('Unified organization migration completed in {:.4f} seconds'.format(time() - start))
+
+
+def migrate_ujt_organization(apps, schema_editor):
+    '''Move organization field to UJT and UJ models'''
+    _migrate_unified_organization(apps, 'UnifiedJobTemplate')
+    _migrate_unified_organization(apps, 'UnifiedJob')
+
+
+def migrate_ujt_organization_backward(apps, schema_editor):
+    '''Move organization field from UJT and UJ models back to their original places'''
+    _migrate_unified_organization(apps, 'UnifiedJobTemplate', backward=True)
+    _migrate_unified_organization(apps, 'UnifiedJob', backward=True)
+
+
+def _restore_inventory_admins(apps, schema_editor, backward=False):
+    """With the JT.organization changes, admins of organizations connected to
+    job templates via inventory will have their permissions demoted.
+    This maintains current permissions over the migration by granting the
+    permissions they used to have explicitly on the JT itself.
+    """
+    start = time()
+    JobTemplate = apps.get_model('main', 'JobTemplate')
+    User = apps.get_model('auth', 'User')
+    changed_ct = 0
+    jt_qs = JobTemplate.objects.filter(inventory__isnull=False)
+    jt_qs = jt_qs.exclude(inventory__organization=F('project__organization'))
+    jt_qs = jt_qs.only('id', 'admin_role_id', 'execute_role_id', 'inventory_id')
+    for jt in jt_qs.iterator():
+        org = jt.inventory.organization
+        for jt_role, org_roles in (
+                ('admin_role', ('admin_role', 'job_template_admin_role',)),
+                ('execute_role', ('execute_role',))
+            ):
+            role_id = getattr(jt, '{}_id'.format(jt_role))
+
+            user_qs = User.objects
+            if not backward:
+                # In this specific case, the name for the org role and JT roles were the same
+                org_role_ids = [getattr(org, '{}_id'.format(role_name)) for role_name in org_roles]
+                user_qs = user_qs.filter(roles__in=org_role_ids)
+                # bizarre migration behavior - ancestors / descendents of
+                # migration version of Role model is reversed, using current model briefly
+                ancestor_ids = list(
+                    Role.objects.filter(descendents=role_id).values_list('id', flat=True)
+                )
+                # same as Role.__contains__, filter for "user in jt.admin_role"
+                user_qs = user_qs.exclude(roles__in=ancestor_ids)
+            else:
+                # use the database to filter intersection of users without access
+                # to the JT role and either organization role
+                user_qs = user_qs.filter(roles__in=[org.admin_role_id, org.execute_role_id])
+                # in reverse, intersection of users who have both
+                user_qs = user_qs.filter(roles=role_id)
+
+            user_ids = list(user_qs.values_list('id', flat=True))
+            if not user_ids:
+                continue
+
+            role = getattr(jt, jt_role)
+            logger.debug('{} {} on jt {} for users {} via inventory.organization {}'.format(
+                'Removing' if backward else 'Setting',
+                jt_role, jt.pk, user_ids, org.pk
+            ))
+            if not backward:
+                # in reverse, explit role becomes redundant
+                role.members.add(*user_ids)
+            else:
+                role.members.remove(*user_ids)
+            changed_ct += len(user_ids)
+
+    if changed_ct:
+        logger.info('{} explicit JT permission for {} users in {:.4f} seconds'.format(
+            'Removed' if backward else 'Added',
+            changed_ct, time() - start
+        ))
+
+
+def restore_inventory_admins(apps, schema_editor):
+    _restore_inventory_admins(apps, schema_editor)
+
+
+def restore_inventory_admins_backward(apps, schema_editor):
+    _restore_inventory_admins(apps, schema_editor, backward=True)
+
+
 def rebuild_role_hierarchy(apps, schema_editor):
+    '''
+    This should be called in any migration when ownerships are changed.
+    Ex. I remove a user from the admin_role of a credential.
+    Ancestors are cached from parents for performance, this re-computes ancestors.
+    '''
    logger.info('Computing role roots..')
    start = time()
    roots = Role.objects \
@@ -46,14 +231,74 @@ def rebuild_role_hierarchy(apps, schema_editor):
    start = time()
    Role.rebuild_role_ancestor_list(roots, [])
    stop = time()
-    logger.info('Rebuild completed in %f seconds' % (stop - start))
+    logger.info('Rebuild ancestors completed in %f seconds' % (stop - start))
    logger.info('Done.')


-def delete_all_user_roles(apps, schema_editor):
+def rebuild_role_parentage(apps, schema_editor, models=None):
+    '''
+    This should be called in any migration when any parent_role entry
+    is modified so that the cached parent fields will be updated. Ex:
+        foo_role = ImplicitRoleField(
+            parent_role=['bar_role']  # change to parent_role=['admin_role']
+        )
+
+    This is like rebuild_role_hierarchy, but that method updates ancestors,
+    whereas this method updates parents.
+    '''
+    start = time()
+    seen_models = set()
+    model_ct = 0
+    noop_ct = 0
    ContentType = apps.get_model('contenttypes', "ContentType")
-    Role = apps.get_model('main', "Role")
-    User = apps.get_model('auth', "User")
-    user_content_type = ContentType.objects.get_for_model(User)
-    for role in Role.objects.filter(content_type=user_content_type).iterator():
-        role.delete()
+    additions = set()
+    removals = set()
+
+    role_qs = Role.objects
+    if models:
+        # update_role_parentage_for_instance is expensive
+        # if the models have been downselected, ignore those which are not in the list
+        ct_ids = list(ContentType.objects.filter(
+            model__in=[name.lower() for name in models]
+        ).values_list('id', flat=True))
+        role_qs = role_qs.filter(content_type__in=ct_ids)
+
+    for role in role_qs.iterator():
+        if not role.object_id:
+            continue
+        model_tuple = (role.content_type_id, role.object_id)
+        if model_tuple in seen_models:
+            continue
+        seen_models.add(model_tuple)
+
+        # The GenericForeignKey does not work right in migrations
+        # with the usage as role.content_object
+        # so we do the lookup ourselves with current migration models
+        ct = role.content_type
+        app = ct.app_label
+        ct_model = apps.get_model(app, ct.model)
+        content_object = ct_model.objects.get(pk=role.object_id)
+
+        parents_added, parents_removed = update_role_parentage_for_instance(content_object)
+        additions.update(parents_added)
+        removals.update(parents_removed)
+        if parents_added:
+            model_ct += 1
+            logger.debug('Added to parents of roles {} of {}'.format(parents_added, content_object))
+        if parents_removed:
+            model_ct += 1
+            logger.debug('Removed from parents of roles {} of {}'.format(parents_removed, content_object))
+        else:
+            noop_ct += 1
+
+    logger.debug('No changes to role parents for {} resources'.format(noop_ct))
+    logger.debug('Added parents to {} roles'.format(len(additions)))
+    logger.debug('Removed parents from {} roles'.format(len(removals)))
+    if model_ct:
+        logger.info('Updated implicit parents of {} resources'.format(model_ct))
+
+    logger.info('Rebuild parentage completed in %f seconds' % (time() - start))
+
+    # this is ran because the ordinary signals for
+    # Role.parents.add and Role.parents.remove not called in migration
+    Role.rebuild_role_ancestor_list(list(additions), list(removals))
--- a/awx/main/models/init.py
+++ b/awx/main/models/init.py
@@ -3,6 +3,7 @@

 # Django
 from django.conf import settings # noqa
+from django.db import connection
 from django.db.models.signals import pre_delete  # noqa

 # AWX
@@ -58,7 +59,6 @@ from awx.main.models.workflow import (  # noqa
    WorkflowJob, WorkflowJobNode, WorkflowJobOptions, WorkflowJobTemplate,
    WorkflowJobTemplateNode, WorkflowApproval, WorkflowApprovalTemplate,
 )
-from awx.main.models.channels import ChannelGroup # noqa
 from awx.api.versioning import reverse
 from awx.main.models.oauth import ( # noqa
    OAuth2AccessToken, OAuth2Application
@@ -80,6 +80,26 @@ User.add_to_class('can_access_with_errors', check_user_access_with_errors)
 User.add_to_class('accessible_objects', user_accessible_objects)


+def enforce_bigint_pk_migration():
+    # see: https://github.com/ansible/awx/issues/6010
+    # look at all the event tables and verify that they have been fully migrated
+    # from the *old* int primary key table to the replacement bigint table
+    # if not, attempt to migrate them in the background
+    for tblname in (
+        'main_jobevent', 'main_inventoryupdateevent',
+        'main_projectupdateevent', 'main_adhoccommandevent',
+        'main_systemjobevent'
+    ):
+        with connection.cursor() as cursor:
+            cursor.execute(
+                'SELECT 1 FROM information_schema.tables WHERE table_name=%s',
+                (f'_old_{tblname}',)
+            )
+            if bool(cursor.rowcount):
+                from awx.main.tasks import migrate_legacy_event_data
+                migrate_legacy_event_data.apply_async([tblname])
+
+
 def cleanup_created_modified_by(sender, **kwargs):
    # work around a bug in django-polymorphic that doesn't properly
    # handle cascades for reverse foreign keys on the polymorphic base model
--- a/awx/main/models/base.py
+++ b/awx/main/models/base.py
@@ -15,6 +15,7 @@ from crum import get_current_user

 # AWX
 from awx.main.utils import encrypt_field, parse_yaml_or_json
+from awx.main.constants import CLOUD_PROVIDERS

 __all__ = ['prevent_search', 'VarsDictProperty', 'BaseModel', 'CreatedModifiedModel',
           'PasswordFieldsModel', 'PrimordialModel', 'CommonModel',
@@ -50,7 +51,7 @@ PROJECT_UPDATE_JOB_TYPE_CHOICES = [
    (PERM_INVENTORY_CHECK, _('Check')),
 ]

-CLOUD_INVENTORY_SOURCES = ['ec2', 'vmware', 'gce', 'azure_rm', 'openstack', 'rhv', 'custom', 'satellite6', 'cloudforms', 'scm', 'tower',]
+CLOUD_INVENTORY_SOURCES = list(CLOUD_PROVIDERS) + ['scm', 'custom']

 VERBOSITY_CHOICES = [
    (0, '0 (Normal)'),
--- a/awx/main/models/channels.py
+++ b/awx/main/models/channels.py
@@ -1,6 +0,0 @@
-from django.db import models
-
-
-class ChannelGroup(models.Model):
-    group = models.CharField(max_length=200, unique=True)
-    channels = models.TextField()
--- a/awx/main/models/credential/init.py
+++ b/awx/main/models/credential/init.py
@@ -799,6 +799,10 @@ ManagedCredentialType(
            'id': 'project',
            'label': ugettext_noop('Project (Tenant Name)'),
            'type': 'string',
+        }, {
+            'id': 'project_domain_name',
+            'label': ugettext_noop('Project (Domain Name)'),
+            'type': 'string',
        }, {
            'id': 'domain',
            'label': ugettext_noop('Domain Name'),
--- a/awx/main/models/credential/injectors.py
+++ b/awx/main/models/credential/injectors.py
@@ -77,6 +77,8 @@ def _openstack_data(cred):
                          username=cred.get_input('username', default=''),
                          password=cred.get_input('password', default=''),
                          project_name=cred.get_input('project', default=''))
+    if cred.has_input('project_domain_name'):
+        openstack_auth['project_domain_name'] = cred.get_input('project_domain_name', default='')
    if cred.has_input('domain'):
        openstack_auth['domain_name'] = cred.get_input('domain', default='')
    verify_state = cred.get_input('verify_ssl', default=True)
--- a/awx/main/models/events.py
+++ b/awx/main/models/events.py
@@ -4,10 +4,10 @@ import datetime
 import logging
 from collections import defaultdict

-from django.db import models, DatabaseError
+from django.db import models, DatabaseError, connection
 from django.utils.dateparse import parse_datetime
 from django.utils.text import Truncator
-from django.utils.timezone import utc
+from django.utils.timezone import utc, now
 from django.utils.translation import ugettext_lazy as _
 from django.utils.encoding import force_text

@@ -338,7 +338,7 @@ class BasePlaybookEvent(CreatedModifiedModel):

            if isinstance(self, JobEvent):
                hostnames = self._hostnames()
-                self._update_host_summary_from_stats(hostnames)
+                self._update_host_summary_from_stats(set(hostnames))
                if self.job.inventory:
                    try:
                        self.job.inventory.update_computed_fields()
@@ -356,6 +356,14 @@ class BasePlaybookEvent(CreatedModifiedModel):
                    job_id=self.job_id, uuid__in=failed
                ).update(failed=True)

+                # send success/failure notifications when we've finished handling the playbook_on_stats event
+                from awx.main.tasks import handle_success_and_failure_notifications  # circular import
+
+                def _send_notifications():
+                    handle_success_and_failure_notifications.apply_async([self.job.id])
+                connection.on_commit(_send_notifications)
+
+
        for field in ('playbook', 'play', 'task', 'role'):
            value = force_text(event_data.get(field, '')).strip()
            if value != getattr(self, field):
@@ -399,11 +407,14 @@ class BasePlaybookEvent(CreatedModifiedModel):
        except (KeyError, ValueError):
            kwargs.pop('created', None)

+        host_map = kwargs.pop('host_map', {})
+
        sanitize_event_keys(kwargs, cls.VALID_KEYS)
        workflow_job_id = kwargs.pop('workflow_job_id', None)
        event = cls(**kwargs)
        if workflow_job_id:
            setattr(event, 'workflow_job_id', workflow_job_id)
+        setattr(event, 'host_map', host_map)
        event._update_from_event_data()
        return event

@@ -430,6 +441,7 @@ class JobEvent(BasePlaybookEvent):
            ('job', 'parent_uuid'),
        ]

+    id = models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')
    job = models.ForeignKey(
        'Job',
        related_name='job_events',
@@ -475,29 +487,47 @@ class JobEvent(BasePlaybookEvent):
            if not self.job or not self.job.inventory:
                logger.info('Event {} missing job or inventory, host summaries not updated'.format(self.pk))
                return
-            qs = self.job.inventory.hosts.filter(name__in=hostnames)
            job = self.job
+
+            from awx.main.models import Host, JobHostSummary  # circular import
+            all_hosts = Host.objects.filter(
+                pk__in=self.host_map.values()
+            ).only('id')
+            existing_host_ids = set(h.id for h in all_hosts)
+
+            summaries = dict()
            for host in hostnames:
+                host_id = self.host_map.get(host, None)
+                if host_id not in existing_host_ids:
+                    host_id = None
                host_stats = {}
                for stat in ('changed', 'dark', 'failures', 'ignored', 'ok', 'processed', 'rescued', 'skipped'):
                    try:
                        host_stats[stat] = self.event_data.get(stat, {}).get(host, 0)
                    except AttributeError:  # in case event_data[stat] isn't a dict.
                        pass
-                if qs.filter(name=host).exists():
-                    host_actual = qs.get(name=host)
-                    host_summary, created = job.job_host_summaries.get_or_create(host=host_actual, host_name=host_actual.name, defaults=host_stats)
-                else:
-                    host_summary, created = job.job_host_summaries.get_or_create(host_name=host, defaults=host_stats)
+                summary = JobHostSummary(
+                    created=now(), modified=now(), job_id=job.id, host_id=host_id, host_name=host, **host_stats
+                )
+                summary.failed = bool(summary.dark or summary.failures)
+                summaries[(host_id, host)] = summary
+
+            JobHostSummary.objects.bulk_create(summaries.values())
+
+            # update the last_job_id and last_job_host_summary_id
+            # in single queries
+            host_mapping = dict(
+                (summary['host_id'], summary['id'])
+                for summary in JobHostSummary.objects.filter(job_id=job.id).values('id', 'host_id')
+            )
+            for h in all_hosts:
+                # if the hostname *shows up* in the playbook_on_stats event
+                if h.name in hostnames:
+                    h.last_job_id = job.id
+                if h.id in host_mapping:
+                    h.last_job_host_summary_id = host_mapping[h.id]
+            Host.objects.bulk_update(all_hosts, ['last_job_id', 'last_job_host_summary_id'])

-                if not created:
-                    update_fields = []
-                    for stat, value in host_stats.items():
-                        if getattr(host_summary, stat) != value:
-                            setattr(host_summary, stat, value)
-                            update_fields.append(stat)
-                    if update_fields:
-                        host_summary.save(update_fields=update_fields)

    @property
    def job_verbosity(self):
@@ -518,6 +548,7 @@ class ProjectUpdateEvent(BasePlaybookEvent):
            ('project_update', 'end_line'),
        ]

+    id = models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')
    project_update = models.ForeignKey(
        'ProjectUpdate',
        related_name='project_update_events',
@@ -669,6 +700,7 @@ class AdHocCommandEvent(BaseCommandEvent):
    FAILED_EVENTS = [x[0] for x in EVENT_TYPES if x[2]]
    EVENT_CHOICES = [(x[0], x[1]) for x in EVENT_TYPES]

+    id = models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')
    event = models.CharField(
        max_length=100,
        choices=EVENT_CHOICES,
@@ -731,6 +763,7 @@ class InventoryUpdateEvent(BaseCommandEvent):
            ('inventory_update', 'end_line'),
        ]

+    id = models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')
    inventory_update = models.ForeignKey(
        'InventoryUpdate',
        related_name='inventory_update_events',
@@ -764,6 +797,7 @@ class SystemJobEvent(BaseCommandEvent):
            ('system_job', 'end_line'),
        ]

+    id = models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')
    system_job = models.ForeignKey(
        'SystemJob',
        related_name='system_job_events',
--- a/awx/main/models/ha.py
+++ b/awx/main/models/ha.py
@@ -53,6 +53,13 @@ class Instance(HasPolicyEditsMixin, BaseModel):

    uuid = models.CharField(max_length=40)
    hostname = models.CharField(max_length=250, unique=True)
+    ip_address = models.CharField(
+        blank=True,
+        null=True,
+        default=None,
+        max_length=50,
+        unique=True,
+    )
    created = models.DateTimeField(auto_now_add=True)
    modified = models.DateTimeField(auto_now=True)
    last_isolated_check = models.DateTimeField(
--- a/awx/main/models/inventory.py
+++ b/awx/main/models/inventory.py
@@ -4,16 +4,13 @@
 # Python
 import datetime
 import time
+import json
 import logging
 import re
 import copy
 import os.path
 from urllib.parse import urljoin
 import yaml
-import configparser
-import tempfile
-from io import StringIO
-from distutils.version import LooseVersion as Version

 # Django
 from django.conf import settings
@@ -59,7 +56,7 @@ from awx.main.models.notifications import (
    JobNotificationMixin,
 )
 from awx.main.models.credential.injectors import _openstack_data
-from awx.main.utils import _inventory_updates, region_sorting, get_licenser
+from awx.main.utils import _inventory_updates, region_sorting
 from awx.main.utils.safe_yaml import sanitize_jinja


@@ -426,9 +423,9 @@ class Inventory(CommonModelNameNotUnique, ResourceMixin, RelatedJobsMixin):
    '''
    def _get_related_jobs(self):
        return UnifiedJob.objects.non_polymorphic().filter(
-            Q(Job___inventory=self) |
-            Q(InventoryUpdate___inventory_source__inventory=self) |
-            Q(AdHocCommand___inventory=self)
+            Q(job__inventory=self) |
+            Q(inventoryupdate__inventory=self) |
+            Q(adhoccommand__inventory=self)
        )


@@ -808,8 +805,8 @@ class Group(CommonModelNameNotUnique, RelatedJobsMixin):
    '''
    def _get_related_jobs(self):
        return UnifiedJob.objects.non_polymorphic().filter(
-            Q(Job___inventory=self.inventory) |
-            Q(InventoryUpdate___inventory_source__groups=self)
+            Q(job__inventory=self.inventory) |
+            Q(inventoryupdate__inventory_source__groups=self)
        )


@@ -821,7 +818,6 @@ class InventorySourceOptions(BaseModel):
    injectors = dict()

    SOURCE_CHOICES = [
-        ('', _('Manual')),
        ('file', _('File, Directory or Script')),
        ('scm', _('Sourced from a Project')),
        ('ec2', _('Amazon EC2')),
@@ -829,7 +825,6 @@ class InventorySourceOptions(BaseModel):
        ('azure_rm', _('Microsoft Azure Resource Manager')),
        ('vmware', _('VMware vCenter')),
        ('satellite6', _('Red Hat Satellite 6')),
-        ('cloudforms', _('Red Hat CloudForms')),
        ('openstack', _('OpenStack')),
        ('rhv', _('Red Hat Virtualization')),
        ('tower', _('Ansible Tower')),
@@ -932,8 +927,8 @@ class InventorySourceOptions(BaseModel):
    source = models.CharField(
        max_length=32,
        choices=SOURCE_CHOICES,
-        blank=True,
-        default='',
+        blank=False,
+        default=None,
    )
    source_path = models.CharField(
        max_length=1024,
@@ -1069,11 +1064,6 @@ class InventorySourceOptions(BaseModel):
        """Red Hat Satellite 6 region choices (not implemented)"""
        return [('all', 'All')]

-    @classmethod
-    def get_cloudforms_region_choices(self):
-        """Red Hat CloudForms region choices (not implemented)"""
-        return [('all', 'All')]
-
    @classmethod
    def get_rhv_region_choices(self):
        """No region supprt"""
@@ -1237,14 +1227,6 @@ class InventorySource(UnifiedJobTemplate, InventorySourceOptions, CustomVirtualE
        on_delete=models.CASCADE,
    )

-    deprecated_group = models.OneToOneField(
-        'Group',
-        related_name='deprecated_inventory_source',
-        null=True,
-        default=None,
-        on_delete=models.CASCADE,
-    )
-
    source_project = models.ForeignKey(
        'Project',
        related_name='scm_inventory_sources',
@@ -1277,10 +1259,14 @@ class InventorySource(UnifiedJobTemplate, InventorySourceOptions, CustomVirtualE
    @classmethod
    def _get_unified_job_field_names(cls):
        return set(f.name for f in InventorySourceOptions._meta.fields) | set(
-            ['name', 'description', 'credentials', 'inventory']
+            ['name', 'description', 'organization', 'credentials', 'inventory']
        )

    def save(self, *args, **kwargs):
+        # if this is a new object, inherit organization from its inventory
+        if not self.pk and self.inventory and self.inventory.organization_id and not self.organization_id:
+            self.organization_id = self.inventory.organization_id
+
        # If update_fields has been specified, add our field names to it,
        # if it hasn't been specified, then we're just doing a normal save.
        update_fields = kwargs.get('update_fields', [])
@@ -1410,16 +1396,6 @@ class InventorySource(UnifiedJobTemplate, InventorySourceOptions, CustomVirtualE
                    started=list(started_notification_templates),
                    success=list(success_notification_templates))

-    def clean_source(self):  # TODO: remove in 3.3
-        source = self.source
-        if source and self.deprecated_group:
-            qs = self.deprecated_group.inventory_sources.filter(source__in=CLOUD_INVENTORY_SOURCES)
-            existing_sources = qs.exclude(pk=self.pk)
-            if existing_sources.count():
-                s = u', '.join([x.deprecated_group.name for x in existing_sources])
-                raise ValidationError(_('Unable to configure this item for cloud sync. It is already managed by %s.') % s)
-        return source
-
    def clean_update_on_project_update(self):
        if self.update_on_project_update is True and \
                self.source == 'scm' and \
@@ -1508,8 +1484,6 @@ class InventoryUpdate(UnifiedJob, InventorySourceOptions, JobNotificationMixin,
        if self.inventory_source.inventory is not None:
            websocket_data.update(dict(inventory_id=self.inventory_source.inventory.pk))

-        if self.inventory_source.deprecated_group is not None:  # TODO: remove in 3.3
-            websocket_data.update(dict(group_id=self.inventory_source.deprecated_group.id))
        return websocket_data

    def get_absolute_url(self, request=None):
@@ -1618,17 +1592,15 @@ class CustomInventoryScript(CommonModelNameNotUnique, ResourceMixin):
        return reverse('api:inventory_script_detail', kwargs={'pk': self.pk}, request=request)


-# TODO: move to awx/main/models/inventory/injectors.py
 class PluginFileInjector(object):
-    # if plugin_name is not given, no inventory plugin functionality exists
    plugin_name = None  # Ansible core name used to reference plugin
-    # if initial_version is None, but we have plugin name, injection logic exists,
-    # but it is vaporware, meaning we do not use it for some reason in Ansible core
-    initial_version = None  # at what version do we switch to the plugin
-    ini_env_reference = None  # env var name that points to old ini config file
    # base injector should be one of None, "managed", or "template"
    # this dictates which logic to borrow from playbook injectors
    base_injector = None
+    # every source should have collection, these are for the collection name
+    namespace = None
+    collection = None
+    collection_migration = '2.9'  # Starting with this version, we use collections

    def __init__(self, ansible_version):
        # This is InventoryOptions instance, could be source or inventory update
@@ -1641,12 +1613,6 @@ class PluginFileInjector(object):
        """
        return '{0}.yml'.format(self.plugin_name)

-    @property
-    def script_name(self):
-        """Name of the script located in awx/plugins/inventory
-        """
-        return '{0}.py'.format(self.__class__.__name__)
-
    def inventory_as_dict(self, inventory_update, private_data_dir):
        """Default implementation of inventory plugin file contents.
        There are some valid cases when all parameters can be obtained from
@@ -1655,7 +1621,8 @@ class PluginFileInjector(object):
        """
        if self.plugin_name is None:
            raise NotImplementedError('At minimum the plugin name is needed for inventory plugin use.')
-        return {'plugin': self.plugin_name}
+        proper_name = f'{self.namespace}.{self.collection}.{self.plugin_name}'
+        return {'plugin': proper_name}

    def inventory_contents(self, inventory_update, private_data_dir):
        """Returns a string that is the content for the inventory file for the inventory plugin
@@ -1666,17 +1633,8 @@ class PluginFileInjector(object):
            width=1000
        )

-    def should_use_plugin(self):
-        return bool(
-            self.plugin_name and self.initial_version and
-            Version(self.ansible_version) >= Version(self.initial_version)
-        )
-
    def build_env(self, inventory_update, env, private_data_dir, private_data_files):
-        if self.should_use_plugin():
-            injector_env = self.get_plugin_env(inventory_update, private_data_dir, private_data_files)
-        else:
-            injector_env = self.get_script_env(inventory_update, private_data_dir, private_data_files)
+        injector_env = self.get_plugin_env(inventory_update, private_data_dir, private_data_files)
        env.update(injector_env)
        # Preserves current behavior for Ansible change in default planned for 2.10
        env['ANSIBLE_TRANSFORM_INVALID_GROUP_CHARS'] = 'never'
@@ -1684,7 +1642,6 @@ class PluginFileInjector(object):

    def _get_shared_env(self, inventory_update, private_data_dir, private_data_files):
        """By default, we will apply the standard managed_by_tower injectors
-        for the script injection
        """
        injected_env = {}
        credential = inventory_update.get_cloud_credential()
@@ -1710,51 +1667,22 @@ class PluginFileInjector(object):
        return injected_env

    def get_plugin_env(self, inventory_update, private_data_dir, private_data_files):
-        return self._get_shared_env(inventory_update, private_data_dir, private_data_files)
-
-    def get_script_env(self, inventory_update, private_data_dir, private_data_files):
-        injected_env = self._get_shared_env(inventory_update, private_data_dir, private_data_files)
-
-        # Put in env var reference to private ini data files, if relevant
-        if self.ini_env_reference:
-            credential = inventory_update.get_cloud_credential()
-            cred_data = private_data_files['credentials']
-            injected_env[self.ini_env_reference] = cred_data[credential]
-
-        return injected_env
+        env = self._get_shared_env(inventory_update, private_data_dir, private_data_files)
+        env['ANSIBLE_COLLECTIONS_PATHS'] = settings.AWX_ANSIBLE_COLLECTIONS_PATHS
+        return env

    def build_private_data(self, inventory_update, private_data_dir):
-        if self.should_use_plugin():
-            return self.build_plugin_private_data(inventory_update, private_data_dir)
-        else:
-            return self.build_script_private_data(inventory_update, private_data_dir)
-
-    def build_script_private_data(self, inventory_update, private_data_dir):
-        return None
+        return self.build_plugin_private_data(inventory_update, private_data_dir)

    def build_plugin_private_data(self, inventory_update, private_data_dir):
        return None

-    @staticmethod
-    def dump_cp(cp, credential):
-        """Dump config parser data and return it as a string.
-        Helper method intended for use by build_script_private_data
-        """
-        if cp.sections():
-            f = StringIO()
-            cp.write(f)
-            private_data = {'credentials': {}}
-            private_data['credentials'][credential] = f.getvalue()
-            return private_data
-        else:
-            return None
-

 class azure_rm(PluginFileInjector):
    plugin_name = 'azure_rm'
-    initial_version = '2.8'  # Driven by unsafe group names issue, hostvars, host names
-    ini_env_reference = 'AZURE_INI_PATH'
    base_injector = 'managed'
+    namespace = 'azure'
+    collection = 'azcollection'

    def get_plugin_env(self, *args, **kwargs):
        ret = super(azure_rm, self).get_plugin_env(*args, **kwargs)
@@ -1862,33 +1790,12 @@ class azure_rm(PluginFileInjector):
            ret['exclude_host_filters'].append("location not in {}".format(repr(python_regions)))
        return ret

-    def build_script_private_data(self, inventory_update, private_data_dir):
-        cp = configparser.RawConfigParser()
-        section = 'azure'
-        cp.add_section(section)
-        cp.set(section, 'include_powerstate', 'yes')
-        cp.set(section, 'group_by_resource_group', 'yes')
-        cp.set(section, 'group_by_location', 'yes')
-        cp.set(section, 'group_by_tag', 'yes')
-
-        if inventory_update.source_regions and 'all' not in inventory_update.source_regions:
-            cp.set(
-                section, 'locations',
-                ','.join([x.strip() for x in inventory_update.source_regions.split(',')])
-            )
-
-        azure_rm_opts = dict(inventory_update.source_vars_dict.items())
-        for k, v in azure_rm_opts.items():
-            cp.set(section, k, str(v))
-        return self.dump_cp(cp, inventory_update.get_cloud_credential())
-

 class ec2(PluginFileInjector):
    plugin_name = 'aws_ec2'
-    # blocked by https://github.com/ansible/ansible/issues/54059
-    # initial_version = '2.8'  # Driven by unsafe group names issue, parent_group templating, hostvars
-    ini_env_reference = 'EC2_INI_PATH'
    base_injector = 'managed'
+    namespace = 'amazon'
+    collection = 'aws'

    def get_plugin_env(self, *args, **kwargs):
        ret = super(ec2, self).get_plugin_env(*args, **kwargs)
@@ -2028,6 +1935,9 @@ class ec2(PluginFileInjector):
                grouping_data['key'] += ' | regex_replace("{rx}", "_")'.format(rx=legacy_regex)
        # end compatibility content

+        if source_vars.get('iam_role_arn', None):
+            ret['iam_role_arn'] = source_vars['iam_role_arn']
+
        # This was an allowed ec2.ini option, also plugin option, so pass through
        if source_vars.get('boto_profile', None):
            ret['boto_profile'] = source_vars['boto_profile']
@@ -2036,6 +1946,10 @@ class ec2(PluginFileInjector):
            # Using the plugin, but still want dashes whitelisted
            ret['use_contrib_script_compatible_sanitization'] = True

+        if source_vars.get('nested_groups') is False:
+            for this_keyed_group in keyed_groups:
+                this_keyed_group.pop('parent_group', None)
+
        if keyed_groups:
            ret['keyed_groups'] = keyed_groups

@@ -2047,18 +1961,35 @@ class ec2(PluginFileInjector):
        compose_dict.update(self._compat_compose_vars())
        # plugin provides "aws_ec2", but not this which the script gave
        ret['groups'] = {'ec2': True}
-        # public_ip as hostname is non-default plugin behavior, script behavior
-        ret['hostnames'] = [
-            'network-interface.addresses.association.public-ip',
-            'dns-name',
-            'private-dns-name'
-        ]
+        if source_vars.get('hostname_variable') is not None:
+            hnames = []
+            for expr in source_vars.get('hostname_variable').split(','):
+                if expr == 'public_dns_name':
+                    hnames.append('dns-name')
+                elif not expr.startswith('tag:') and '_' in expr:
+                    hnames.append(expr.replace('_', '-'))
+                else:
+                    hnames.append(expr)
+            ret['hostnames'] = hnames
+        else:
+            # public_ip as hostname is non-default plugin behavior, script behavior
+            ret['hostnames'] = [
+                'network-interface.addresses.association.public-ip',
+                'dns-name',
+                'private-dns-name'
+            ]
        # The script returned only running state by default, the plugin does not
        # https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html#options
        # options: pending | running | shutting-down | terminated | stopping | stopped
        inst_filters['instance-state-name'] = ['running']
        # end compatibility content

+        if source_vars.get('destination_variable') or source_vars.get('vpc_destination_variable'):
+            for fd in ('destination_variable', 'vpc_destination_variable'):
+                if source_vars.get(fd):
+                    compose_dict['ansible_host'] = source_vars.get(fd)
+                    break
+
        if compose_dict:
            ret['compose'] = compose_dict

@@ -2084,47 +2015,12 @@ class ec2(PluginFileInjector):

        return ret

-    def build_script_private_data(self, inventory_update, private_data_dir):
-        cp = configparser.RawConfigParser()
-        # Build custom ec2.ini for ec2 inventory script to use.
-        section = 'ec2'
-        cp.add_section(section)
-        ec2_opts = dict(inventory_update.source_vars_dict.items())
-        regions = inventory_update.source_regions or 'all'
-        regions = ','.join([x.strip() for x in regions.split(',')])
-        regions_blacklist = ','.join(settings.EC2_REGIONS_BLACKLIST)
-        ec2_opts['regions'] = regions
-        ec2_opts.setdefault('regions_exclude', regions_blacklist)
-        ec2_opts.setdefault('destination_variable', 'public_dns_name')
-        ec2_opts.setdefault('vpc_destination_variable', 'ip_address')
-        ec2_opts.setdefault('route53', 'False')
-        ec2_opts.setdefault('all_instances', 'True')
-        ec2_opts.setdefault('all_rds_instances', 'False')
-        ec2_opts.setdefault('include_rds_clusters', 'False')
-        ec2_opts.setdefault('rds', 'False')
-        ec2_opts.setdefault('nested_groups', 'True')
-        ec2_opts.setdefault('elasticache', 'False')
-        ec2_opts.setdefault('stack_filters', 'False')
-        if inventory_update.instance_filters:
-            ec2_opts.setdefault('instance_filters', inventory_update.instance_filters)
-        group_by = [x.strip().lower() for x in inventory_update.group_by.split(',') if x.strip()]
-        for choice in inventory_update.get_ec2_group_by_choices():
-            value = bool((group_by and choice[0] in group_by) or (not group_by and choice[0] != 'instance_id'))
-            ec2_opts.setdefault('group_by_%s' % choice[0], str(value))
-        if 'cache_path' not in ec2_opts:
-            cache_path = tempfile.mkdtemp(prefix='ec2_cache', dir=private_data_dir)
-            ec2_opts['cache_path'] = cache_path
-        ec2_opts.setdefault('cache_max_age', '300')
-        for k, v in ec2_opts.items():
-            cp.set(section, k, str(v))
-        return self.dump_cp(cp, inventory_update.get_cloud_credential())
-

 class gce(PluginFileInjector):
    plugin_name = 'gcp_compute'
-    initial_version = '2.8'  # Driven by unsafe group names issue, hostvars
-    ini_env_reference = 'GCE_INI_PATH'
    base_injector = 'managed'
+    namespace = 'google'
+    collection = 'cloud'

    def get_plugin_env(self, *args, **kwargs):
        ret = super(gce, self).get_plugin_env(*args, **kwargs)
@@ -2132,17 +2028,6 @@ class gce(PluginFileInjector):
        ret['ANSIBLE_JINJA2_NATIVE'] = str(True)
        return ret

-    def get_script_env(self, inventory_update, private_data_dir, private_data_files):
-        env = super(gce, self).get_script_env(inventory_update, private_data_dir, private_data_files)
-        cred = inventory_update.get_cloud_credential()
-        # these environment keys are unique to the script operation, and are not
-        # concepts in the modern inventory plugin or gce Ansible module
-        # email and project are redundant with the creds file
-        env['GCE_EMAIL'] = cred.get_input('username', default='')
-        env['GCE_PROJECT'] = cred.get_input('project', default='')
-        env['GCE_ZONE'] = inventory_update.source_regions if inventory_update.source_regions != 'all' else ''  # noqa
-        return env
-
    def _compat_compose_vars(self):
        # missing: gce_image, gce_uuid
        # https://github.com/ansible/ansible/issues/51884
@@ -2215,73 +2100,123 @@ class gce(PluginFileInjector):
            ret['zones'] = inventory_update.source_regions.split(',')
        return ret

-    def build_script_private_data(self, inventory_update, private_data_dir):
-        cp = configparser.RawConfigParser()
-        # by default, the GCE inventory source caches results on disk for
-        # 5 minutes; disable this behavior
-        cp.add_section('cache')
-        cp.set('cache', 'cache_max_age', '0')
-        return self.dump_cp(cp, inventory_update.get_cloud_credential())
-

 class vmware(PluginFileInjector):
-    # plugin_name = 'vmware_vm_inventory'  # FIXME: implement me
-    ini_env_reference = 'VMWARE_INI_PATH'
+    plugin_name = 'vmware_vm_inventory'
    base_injector = 'managed'
+    namespace = 'community'
+    collection = 'vmware'

-    @property
-    def script_name(self):
-        return 'vmware_inventory.py'  # exception
-
-    def build_script_private_data(self, inventory_update, private_data_dir):
-        cp = configparser.RawConfigParser()
-        credential = inventory_update.get_cloud_credential()
-
-        # Allow custom options to vmware inventory script.
-        section = 'vmware'
-        cp.add_section(section)
-        cp.set('vmware', 'cache_max_age', '0')
-        cp.set('vmware', 'validate_certs', str(settings.VMWARE_VALIDATE_CERTS))
-        cp.set('vmware', 'username', credential.get_input('username', default=''))
-        cp.set('vmware', 'password', credential.get_input('password', default=''))
-        cp.set('vmware', 'server', credential.get_input('host', default=''))
+    def inventory_as_dict(self, inventory_update, private_data_dir):
+        ret = super(vmware, self).inventory_as_dict(inventory_update, private_data_dir)
+        ret['strict'] = False
+        # Documentation of props, see
+        # https://github.com/ansible/ansible/blob/devel/docs/docsite/rst/scenario_guides/vmware_scenarios/vmware_inventory_vm_attributes.rst
+        UPPERCASE_PROPS = [
+            "availableField",
+            "configIssue",
+            "configStatus",
+            "customValue",  # optional
+            "datastore",
+            "effectiveRole",
+            "guestHeartbeatStatus",  # optional
+            "layout",  # optional
+            "layoutEx",  # optional
+            "name",
+            "network",
+            "overallStatus",
+            "parentVApp",  # optional
+            "permission",
+            "recentTask",
+            "resourcePool",
+            "rootSnapshot",
+            "snapshot",  # optional
+            "triggeredAlarmState",
+            "value"
+        ]
+        NESTED_PROPS = [
+            "capability",
+            "config",
+            "guest",
+            "runtime",
+            "storage",
+            "summary",  # repeat of other properties
+        ]
+        ret['properties'] = UPPERCASE_PROPS + NESTED_PROPS
+        ret['compose'] = {'ansible_host': 'guest.ipAddress'}  # default value
+        ret['compose']['ansible_ssh_host'] = ret['compose']['ansible_host']
+        # the ansible_uuid was unique every host, every import, from the script
+        ret['compose']['ansible_uuid'] = '99999999 | random | to_uuid'
+        for prop in UPPERCASE_PROPS:
+            if prop == prop.lower():
+                continue
+            ret['compose'][prop.lower()] = prop
+        ret['with_nested_properties'] = True
+        # ret['property_name_format'] = 'lower_case'  # only dacrystal/topic/vmware-inventory-plugin-property-format

+        # process custom options
        vmware_opts = dict(inventory_update.source_vars_dict.items())
        if inventory_update.instance_filters:
            vmware_opts.setdefault('host_filters', inventory_update.instance_filters)
        if inventory_update.group_by:
            vmware_opts.setdefault('groupby_patterns', inventory_update.group_by)

-        for k, v in vmware_opts.items():
-            cp.set(section, k, str(v))
+        alias_pattern = vmware_opts.get('alias_pattern')
+        if alias_pattern:
+            ret.setdefault('hostnames', [])
+            for alias in alias_pattern.split(','):  # make best effort
+                striped_alias = alias.replace('{', '').replace('}', '').strip()  # make best effort
+                if not striped_alias:
+                    continue
+                ret['hostnames'].append(striped_alias)

-        return self.dump_cp(cp, credential)
+        host_pattern = vmware_opts.get('host_pattern')  # not working in script
+        if host_pattern:
+            stripped_hp = host_pattern.replace('{', '').replace('}', '').strip()  # make best effort
+            ret['compose']['ansible_host'] = stripped_hp
+            ret['compose']['ansible_ssh_host'] = stripped_hp
+
+        host_filters = vmware_opts.get('host_filters')
+        if host_filters:
+            ret.setdefault('filters', [])
+            for hf in host_filters.split(','):
+                striped_hf = hf.replace('{', '').replace('}', '').strip()  # make best effort
+                if not striped_hf:
+                    continue
+                ret['filters'].append(striped_hf)
+        else:
+            # default behavior filters by power state
+            ret['filters'] = ['runtime.powerState == "poweredOn"']
+
+        groupby_patterns = vmware_opts.get('groupby_patterns')
+        ret.setdefault('keyed_groups', [])
+        if groupby_patterns:
+            for pattern in groupby_patterns.split(','):
+                stripped_pattern = pattern.replace('{', '').replace('}', '').strip()  # make best effort
+                ret['keyed_groups'].append({
+                    'prefix': '', 'separator': '',
+                    'key': stripped_pattern
+                })
+        else:
+            # default groups from script
+            for entry in ('config.guestId', '"templates" if config.template else "guests"'):
+                ret['keyed_groups'].append({
+                    'prefix': '', 'separator': '',
+                    'key': entry
+                })
+
+        return ret


 class openstack(PluginFileInjector):
-    ini_env_reference = 'OS_CLIENT_CONFIG_FILE'
    plugin_name = 'openstack'
-    # minimum version of 2.7.8 may be theoretically possible
-    initial_version = '2.8'  # Driven by consistency with other sources
+    namespace = 'openstack'
+    collection = 'cloud'

-    @property
-    def script_name(self):
-        return 'openstack_inventory.py'  # exception
-
-    def _get_clouds_dict(self, inventory_update, cred, private_data_dir, mk_cache=True):
+    def _get_clouds_dict(self, inventory_update, cred, private_data_dir):
        openstack_data = _openstack_data(cred)

        openstack_data['clouds']['devstack']['private'] = inventory_update.source_vars_dict.get('private', True)
-        if mk_cache:
-            # Retrieve cache path from inventory update vars if available,
-            # otherwise create a temporary cache path only for this update.
-            cache = inventory_update.source_vars_dict.get('cache', {})
-            if not isinstance(cache, dict):
-                cache = {}
-            if not cache.get('path', ''):
-                cache_path = tempfile.mkdtemp(prefix='openstack_cache', dir=private_data_dir)
-                cache['path'] = cache_path
-            openstack_data['cache'] = cache
        ansible_variables = {
            'use_hostnames': True,
            'expand_hostvars': False,
@@ -2298,24 +2233,16 @@ class openstack(PluginFileInjector):
            openstack_data['ansible'] = ansible_variables
        return openstack_data

-    def build_script_private_data(self, inventory_update, private_data_dir, mk_cache=True):
+    def build_plugin_private_data(self, inventory_update, private_data_dir):
        credential = inventory_update.get_cloud_credential()
        private_data = {'credentials': {}}

-        openstack_data = self._get_clouds_dict(inventory_update, credential, private_data_dir, mk_cache=mk_cache)
+        openstack_data = self._get_clouds_dict(inventory_update, credential, private_data_dir)
        private_data['credentials'][credential] = yaml.safe_dump(
            openstack_data, default_flow_style=False, allow_unicode=True
        )
        return private_data

-    def build_plugin_private_data(self, inventory_update, private_data_dir):
-        # Credentials can be passed in the same way as the script did
-        # but do not create the tmp cache file
-        return self.build_script_private_data(inventory_update, private_data_dir, mk_cache=False)
-
-    def get_plugin_env(self, inventory_update, private_data_dir, private_data_files):
-        return self.get_script_env(inventory_update, private_data_dir, private_data_files)
-
    def inventory_as_dict(self, inventory_update, private_data_dir):
        def use_host_name_for_name(a_bool_maybe):
            if not isinstance(a_bool_maybe, bool):
@@ -2326,12 +2253,10 @@ class openstack(PluginFileInjector):
            else:
                return 'uuid'

-        ret = dict(
-            plugin=self.plugin_name,
-            fail_on_errors=True,
-            expand_hostvars=True,
-            inventory_hostname=use_host_name_for_name(False),
-        )
+        ret = super(openstack, self).inventory_as_dict(inventory_update, private_data_dir)
+        ret['fail_on_errors'] = True
+        ret['expand_hostvars'] = True
+        ret['inventory_hostname'] = use_host_name_for_name(False)
        # Note: mucking with defaults will break import integrity
        # For the plugin, we need to use the same defaults as the old script
        # or else imports will conflict. To find script defaults you have
@@ -2352,43 +2277,70 @@ class openstack(PluginFileInjector):
            ret['inventory_hostname'] = use_host_name_for_name(source_vars['use_hostnames'])
        return ret

+    def get_plugin_env(self, inventory_update, private_data_dir, private_data_files):
+        env = super(openstack, self).get_plugin_env(inventory_update, private_data_dir, private_data_files)
+        credential = inventory_update.get_cloud_credential()
+        cred_data = private_data_files['credentials']
+        env['OS_CLIENT_CONFIG_FILE'] = cred_data[credential]
+        return env
+

 class rhv(PluginFileInjector):
    """ovirt uses the custom credential templating, and that is all
    """
-    # plugin_name = 'FIXME'  # contribute inventory plugin to Ansible
+    plugin_name = 'ovirt'
    base_injector = 'template'
+    initial_version = '2.9'
+    namespace = 'ovirt'
+    collection = 'ovirt'

-    @property
-    def script_name(self):
-        return 'ovirt4.py'  # exception
+    def inventory_as_dict(self, inventory_update, private_data_dir):
+        ret = super(rhv, self).inventory_as_dict(inventory_update, private_data_dir)
+        ret['ovirt_insecure'] = False  # Default changed from script
+        # TODO: process strict option upstream
+        ret['compose'] = {
+            'ansible_host': '(devices.values() | list)[0][0] if devices else None'
+        }
+        ret['keyed_groups'] = []
+        for key in ('cluster', 'status'):
+            ret['keyed_groups'].append({'prefix': key, 'separator': '_', 'key': key})
+        ret['keyed_groups'].append({'prefix': 'tag', 'separator': '_', 'key': 'tags'})
+        ret['ovirt_hostname_preference'] = ['name', 'fqdn']
+        source_vars = inventory_update.source_vars_dict
+        for key, value in source_vars.items():
+            if key == 'plugin':
+                continue
+            ret[key] = value
+        return ret


 class satellite6(PluginFileInjector):
    plugin_name = 'foreman'
-    ini_env_reference = 'FOREMAN_INI_PATH'
-    # initial_version = '2.8'  # FIXME: turn on after plugin is validated
-    # No base injector, because this does not work in playbooks. Bug??
+    namespace = 'theforeman'
+    collection = 'foreman'

-    @property
-    def script_name(self):
-        return 'foreman.py'  # exception
-
-    def build_script_private_data(self, inventory_update, private_data_dir):
-        cp = configparser.RawConfigParser()
+    def get_plugin_env(self, inventory_update, private_data_dir, private_data_files):
+        # this assumes that this is merged
+        # https://github.com/ansible/ansible/pull/52693
        credential = inventory_update.get_cloud_credential()
+        ret = super(satellite6, self).get_plugin_env(inventory_update, private_data_dir, private_data_files)
+        if credential:
+            ret['FOREMAN_SERVER'] = credential.get_input('host', default='')
+            ret['FOREMAN_USER'] = credential.get_input('username', default='')
+            ret['FOREMAN_PASSWORD'] = credential.get_input('password', default='')
+        return ret

-        section = 'foreman'
-        cp.add_section(section)
+    def inventory_as_dict(self, inventory_update, private_data_dir):
+        ret = super(satellite6, self).inventory_as_dict(inventory_update, private_data_dir)
+        ret['validate_certs'] = False

        group_patterns = '[]'
        group_prefix = 'foreman_'
-        want_hostcollections = 'False'
-        want_ansible_ssh_host = 'False'
-        rich_params = 'False'
-        want_facts = 'True'
-        foreman_opts = dict(inventory_update.source_vars_dict.items())
-        foreman_opts.setdefault('ssl_verify', 'False')
+        want_hostcollections = False
+        want_ansible_ssh_host = False
+        want_facts = True
+
+        foreman_opts = inventory_update.source_vars_dict.copy()
        for k, v in foreman_opts.items():
            if k == 'satellite6_group_patterns' and isinstance(v, str):
                group_patterns = v
@@ -2398,93 +2350,103 @@ class satellite6(PluginFileInjector):
                want_hostcollections = v
            elif k == 'satellite6_want_ansible_ssh_host' and isinstance(v, bool):
                want_ansible_ssh_host = v
-            elif k == 'satellite6_rich_params' and isinstance(v, bool):
-                rich_params = v
            elif k == 'satellite6_want_facts' and isinstance(v, bool):
                want_facts = v
+            # add backwards support for ssl_verify
+            # plugin uses new option, validate_certs, instead
+            elif k == 'ssl_verify' and isinstance(v, bool):
+                ret['validate_certs'] = v
            else:
-                cp.set(section, k, str(v))
+                ret[k] = str(v)

-        if credential:
-            cp.set(section, 'url', credential.get_input('host', default=''))
-            cp.set(section, 'user', credential.get_input('username', default=''))
-            cp.set(section, 'password', credential.get_input('password', default=''))
+        # Compatibility content
+        group_by_hostvar = {
+            "environment":           {"prefix": "{}environment_".format(group_prefix),
+                                      "separator": "",
+                                      "key": "foreman['environment_name'] | lower | regex_replace(' ', '') | "
+                                             "regex_replace('[^A-Za-z0-9_]', '_') | regex_replace('none', '')"},
+            "location":              {"prefix": "{}location_".format(group_prefix),
+                                      "separator": "",
+                                      "key": "foreman['location_name'] | lower | regex_replace(' ', '') | regex_replace('[^A-Za-z0-9_]', '_')"},
+            "organization":          {"prefix": "{}organization_".format(group_prefix),
+                                      "separator": "",
+                                      "key": "foreman['organization_name'] | lower | regex_replace(' ', '') | regex_replace('[^A-Za-z0-9_]', '_')"},
+            "lifecycle_environment": {"prefix": "{}lifecycle_environment_".format(group_prefix),
+                                      "separator": "",
+                                      "key": "foreman['content_facet_attributes']['lifecycle_environment_name'] | "
+                                             "lower | regex_replace(' ', '') | regex_replace('[^A-Za-z0-9_]', '_')"},
+            "content_view":          {"prefix": "{}content_view_".format(group_prefix),
+                                      "separator": "",
+                                      "key": "foreman['content_facet_attributes']['content_view_name'] | "
+                                             "lower | regex_replace(' ', '') | regex_replace('[^A-Za-z0-9_]', '_')"}
+        }

-        section = 'ansible'
-        cp.add_section(section)
-        cp.set(section, 'group_patterns', group_patterns)
-        cp.set(section, 'want_facts', str(want_facts))
-        cp.set(section, 'want_hostcollections', str(want_hostcollections))
-        cp.set(section, 'group_prefix', group_prefix)
-        cp.set(section, 'want_ansible_ssh_host', str(want_ansible_ssh_host))
-        cp.set(section, 'rich_params', str(rich_params))
+        ret['legacy_hostvars'] = True  # convert hostvar structure to the form used by the script
+        ret['want_params'] = True
+        ret['group_prefix'] = group_prefix
+        ret['want_hostcollections'] = want_hostcollections
+        ret['want_facts'] = want_facts

-        section = 'cache'
-        cp.add_section(section)
-        cp.set(section, 'path', '/tmp')
-        cp.set(section, 'max_age', '0')
+        if want_ansible_ssh_host:
+            ret['compose'] = {'ansible_ssh_host': "foreman['ip6'] | default(foreman['ip'], true)"}
+        ret['keyed_groups'] = [group_by_hostvar[grouping_name] for grouping_name in group_by_hostvar]

-        return self.dump_cp(cp, credential)
+        def form_keyed_group(group_pattern):
+            """
+            Converts foreman group_pattern to
+            inventory plugin keyed_group
+
+            e.g. {app_param}-{tier_param}-{dc_param}
+                 becomes
+                 "%s-%s-%s" | format(app_param, tier_param, dc_param)
+            """
+            if type(group_pattern) is not str:
+                return None
+            params = re.findall('{[^}]*}', group_pattern)
+            if len(params) == 0:
+                return None
+
+            param_names = []
+            for p in params:
+                param_names.append(p[1:-1].strip())  # strip braces and space
+
+            # form keyed_group key by
+            # replacing curly braces with '%s'
+            # (for use with jinja's format filter)
+            key = group_pattern
+            for p in params:
+                key = key.replace(p, '%s', 1)
+
+            # apply jinja filter to key
+            key = '"{}" | format({})'.format(key, ', '.join(param_names))
+
+            keyed_group = {'key': key,
+                           'separator': ''}
+            return keyed_group
+
+        try:
+            group_patterns = json.loads(group_patterns)
+
+            if type(group_patterns) is list:
+                for group_pattern in group_patterns:
+                    keyed_group = form_keyed_group(group_pattern)
+                    if keyed_group:
+                        ret['keyed_groups'].append(keyed_group)
+        except json.JSONDecodeError:
+            logger.warning('Could not parse group_patterns. Expected JSON-formatted string, found: {}'
+                           .format(group_patterns))

-    def get_plugin_env(self, inventory_update, private_data_dir, private_data_files):
-        # this assumes that this is merged
-        # https://github.com/ansible/ansible/pull/52693
-        credential = inventory_update.get_cloud_credential()
-        ret = {}
-        if credential:
-            ret['FOREMAN_SERVER'] = credential.get_input('host', default='')
-            ret['FOREMAN_USER'] = credential.get_input('username', default='')
-            ret['FOREMAN_PASSWORD'] = credential.get_input('password', default='')
        return ret


-class cloudforms(PluginFileInjector):
-    # plugin_name = 'FIXME'  # contribute inventory plugin to Ansible
-    ini_env_reference = 'CLOUDFORMS_INI_PATH'
-    # Also no base_injector because this does not work in playbooks
-
-    def build_script_private_data(self, inventory_update, private_data_dir):
-        cp = configparser.RawConfigParser()
-        credential = inventory_update.get_cloud_credential()
-
-        section = 'cloudforms'
-        cp.add_section(section)
-
-        if credential:
-            cp.set(section, 'url', credential.get_input('host', default=''))
-            cp.set(section, 'username', credential.get_input('username', default=''))
-            cp.set(section, 'password', credential.get_input('password', default=''))
-            cp.set(section, 'ssl_verify', "false")
-
-        cloudforms_opts = dict(inventory_update.source_vars_dict.items())
-        for opt in ['version', 'purge_actions', 'clean_group_keys', 'nest_tags', 'suffix', 'prefer_ipv4']:
-            if opt in cloudforms_opts:
-                cp.set(section, opt, str(cloudforms_opts[opt]))
-
-        section = 'cache'
-        cp.add_section(section)
-        cp.set(section, 'max_age', "0")
-        cache_path = tempfile.mkdtemp(
-            prefix='cloudforms_cache',
-            dir=private_data_dir
-        )
-        cp.set(section, 'path', cache_path)
-
-        return self.dump_cp(cp, credential)
-
-
 class tower(PluginFileInjector):
    plugin_name = 'tower'
    base_injector = 'template'
-    initial_version = '2.8'  # Driven by "include_metadata" hostvars
-
-    def get_script_env(self, inventory_update, private_data_dir, private_data_files):
-        env = super(tower, self).get_script_env(inventory_update, private_data_dir, private_data_files)
-        env['TOWER_INVENTORY'] = inventory_update.instance_filters
-        env['TOWER_LICENSE_TYPE'] = get_licenser().validate().get('license_type', 'unlicensed')
-        return env
+    namespace = 'awx'
+    collection = 'awx'

    def inventory_as_dict(self, inventory_update, private_data_dir):
+        ret = super(tower, self).inventory_as_dict(inventory_update, private_data_dir)
        # Credentials injected as env vars, same as script
        try:
            # plugin can take an actual int type
@@ -2492,11 +2454,9 @@ class tower(PluginFileInjector):
        except ValueError:
            # inventory_id could be a named URL
            identifier = iri_to_uri(inventory_update.instance_filters)
-        return {
-            'plugin': self.plugin_name,
-            'inventory_id': identifier,
-            'include_metadata': True  # used for license check
-        }
+        ret['inventory_id'] = identifier
+        ret['include_metadata'] = True  # used for license check
+        return ret


 for cls in PluginFileInjector.__subclasses__():
--- a/awx/main/models/jobs.py
+++ b/awx/main/models/jobs.py
@@ -199,7 +199,7 @@ class JobTemplate(UnifiedJobTemplate, JobOptions, SurveyJobTemplateMixin, Resour
        'labels', 'instance_groups', 'credentials', 'survey_spec'
    ]
    FIELDS_TO_DISCARD_AT_COPY = ['vault_credential', 'credential']
-    SOFT_UNIQUE_TOGETHER = [('polymorphic_ctype', 'name')]
+    SOFT_UNIQUE_TOGETHER = [('polymorphic_ctype', 'name', 'organization')]

    class Meta:
        app_label = 'main'
@@ -262,13 +262,17 @@ class JobTemplate(UnifiedJobTemplate, JobOptions, SurveyJobTemplateMixin, Resour
    )

    admin_role = ImplicitRoleField(
-        parent_role=['project.organization.job_template_admin_role', 'inventory.organization.job_template_admin_role']
+        parent_role=['organization.job_template_admin_role']
    )
    execute_role = ImplicitRoleField(
-        parent_role=['admin_role', 'project.organization.execute_role', 'inventory.organization.execute_role'],
+        parent_role=['admin_role', 'organization.execute_role'],
    )
    read_role = ImplicitRoleField(
-        parent_role=['project.organization.auditor_role', 'inventory.organization.auditor_role', 'execute_role', 'admin_role'],
+        parent_role=[
+            'organization.auditor_role',
+            'inventory.organization.auditor_role',  # partial support for old inheritance via inventory
+            'execute_role', 'admin_role'
+        ],
    )


@@ -279,7 +283,7 @@ class JobTemplate(UnifiedJobTemplate, JobOptions, SurveyJobTemplateMixin, Resour
    @classmethod
    def _get_unified_job_field_names(cls):
        return set(f.name for f in JobOptions._meta.fields) | set(
-            ['name', 'description', 'survey_passwords', 'labels', 'credentials',
+            ['name', 'description', 'organization', 'survey_passwords', 'labels', 'credentials',
             'job_slice_number', 'job_slice_count']
        )

@@ -319,6 +323,41 @@ class JobTemplate(UnifiedJobTemplate, JobOptions, SurveyJobTemplateMixin, Resour
        else:
            return self.job_slice_count

+    def save(self, *args, **kwargs):
+        update_fields = kwargs.get('update_fields', [])
+        # if project is deleted for some reason, then keep the old organization
+        # to retain ownership for organization admins
+        if self.project and self.project.organization_id != self.organization_id:
+            self.organization_id = self.project.organization_id
+            if 'organization' not in update_fields and 'organization_id' not in update_fields:
+                update_fields.append('organization_id')
+        return super(JobTemplate, self).save(*args, **kwargs)
+
+    def validate_unique(self, exclude=None):
+        """Custom over-ride for JT specifically
+        because organization is inferred from project after full_clean is finished
+        thus the organization field is not yet set when validation happens
+        """
+        errors = []
+        for ut in JobTemplate.SOFT_UNIQUE_TOGETHER:
+            kwargs = {'name': self.name}
+            if self.project:
+                kwargs['organization'] = self.project.organization_id
+            else:
+                kwargs['organization'] = None
+            qs = JobTemplate.objects.filter(**kwargs)
+            if self.pk:
+                qs = qs.exclude(pk=self.pk)
+            if qs.exists():
+                errors.append(
+                    '%s with this (%s) combination already exists.' % (
+                        JobTemplate.__name__,
+                        ', '.join(set(ut) - {'polymorphic_ctype'})
+                    )
+                )
+        if errors:
+            raise ValidationError(errors)
+
    def create_unified_job(self, **kwargs):
        prevent_slicing = kwargs.pop('_prevent_slicing', False)
        slice_ct = self.get_effective_slice_ct(kwargs)
@@ -400,13 +439,9 @@ class JobTemplate(UnifiedJobTemplate, JobOptions, SurveyJobTemplateMixin, Resour
            field = self._meta.get_field(field_name)
            if isinstance(field, models.ManyToManyField):
                old_value = set(old_value.all())
-                if getattr(self, '_deprecated_credential_launch', False):
-                    # TODO: remove this code branch when support for `extra_credentials` goes away
-                    new_value = set(kwargs[field_name])
-                else:
-                    new_value = set(kwargs[field_name]) - old_value
-                    if not new_value:
-                        continue
+                new_value = set(kwargs[field_name]) - old_value
+                if not new_value:
+                    continue

            if new_value == old_value:
                # no-op case: Fields the same as template's value
@@ -479,13 +514,13 @@ class JobTemplate(UnifiedJobTemplate, JobOptions, SurveyJobTemplateMixin, Resour
        success_notification_templates = list(base_notification_templates.filter(
            unifiedjobtemplate_notification_templates_for_success__in=[self, self.project]))
        # Get Organization NotificationTemplates
-        if self.project is not None and self.project.organization is not None:
+        if self.organization is not None:
            error_notification_templates = set(error_notification_templates + list(base_notification_templates.filter(
-                organization_notification_templates_for_errors=self.project.organization)))
+                organization_notification_templates_for_errors=self.organization)))
            started_notification_templates = set(started_notification_templates + list(base_notification_templates.filter(
-                organization_notification_templates_for_started=self.project.organization)))
+                organization_notification_templates_for_started=self.organization)))
            success_notification_templates = set(success_notification_templates + list(base_notification_templates.filter(
-                organization_notification_templates_for_success=self.project.organization)))
+                organization_notification_templates_for_success=self.organization)))
        return dict(error=list(error_notification_templates),
                    started=list(started_notification_templates),
                    success=list(success_notification_templates))
@@ -588,7 +623,7 @@ class Job(UnifiedJob, JobOptions, SurveyJobMixin, JobNotificationMixin, TaskMana
        for virtualenv in (
            self.job_template.custom_virtualenv if self.job_template else None,
            self.project.custom_virtualenv,
-            self.project.organization.custom_virtualenv if self.project.organization else None
+            self.organization.custom_virtualenv if self.organization else None
        ):
            if virtualenv:
                return virtualenv
@@ -741,8 +776,8 @@ class Job(UnifiedJob, JobOptions, SurveyJobMixin, JobNotificationMixin, TaskMana

    @property
    def preferred_instance_groups(self):
-        if self.project is not None and self.project.organization is not None:
-            organization_groups = [x for x in self.project.organization.instance_groups.all()]
+        if self.organization is not None:
+            organization_groups = [x for x in self.organization.instance_groups.all()]
        else:
            organization_groups = []
        if self.inventory is not None:
@@ -829,8 +864,10 @@ class Job(UnifiedJob, JobOptions, SurveyJobMixin, JobNotificationMixin, TaskMana
                            continue
                        host.ansible_facts = ansible_facts
                        host.ansible_facts_modified = now()
-                        ansible_local_system_id = ansible_facts.get('ansible_local', {}).get('insights', {}).get('system_id', None)
-                        ansible_facts_system_id = ansible_facts.get('insights', {}).get('system_id', None)
+                        ansible_local = ansible_facts.get('ansible_local', {}).get('insights', {})
+                        ansible_facts = ansible_facts.get('insights', {})
+                        ansible_local_system_id = ansible_local.get('system_id', None) if isinstance(ansible_local, dict) else None
+                        ansible_facts_system_id = ansible_facts.get('system_id', None) if isinstance(ansible_facts, dict) else None
                        if ansible_local_system_id:
                            print("Setting local {}".format(ansible_local_system_id))
                            logger.debug("Insights system_id {} found for host <{}, {}> in"
@@ -1092,20 +1129,6 @@ class JobHostSummary(CreatedModifiedModel):
        self.failed = bool(self.dark or self.failures)
        update_fields.append('failed')
        super(JobHostSummary, self).save(*args, **kwargs)
-        self.update_host_last_job_summary()
-
-    def update_host_last_job_summary(self):
-        update_fields = []
-        if self.host is None:
-            return
-        if self.host.last_job_id != self.job_id:
-            self.host.last_job_id = self.job_id
-            update_fields.append('last_job_id')
-        if self.host.last_job_host_summary_id != self.id:
-            self.host.last_job_host_summary_id = self.id
-            update_fields.append('last_job_host_summary_id')
-        if update_fields:
-            self.host.save(update_fields=update_fields)


 class SystemJobOptions(BaseModel):
@@ -1142,7 +1165,7 @@ class SystemJobTemplate(UnifiedJobTemplate, SystemJobOptions):

    @classmethod
    def _get_unified_job_field_names(cls):
-        return ['name', 'description', 'job_type', 'extra_vars']
+        return ['name', 'description', 'organization', 'job_type', 'extra_vars']

    def get_absolute_url(self, request=None):
        return reverse('api:system_job_template_detail', kwargs={'pk': self.pk}, request=request)
--- a/awx/main/models/mixins.py
+++ b/awx/main/models/mixins.py
@@ -566,7 +566,6 @@ class WebhookMixin(models.Model):

    def update_webhook_status(self, status):
        if not self.webhook_credential:
-            logger.debug("No credential configured to post back webhook status, skipping.")
            return

        status_api = self.extra_vars_dict.get('tower_webhook_status_api')
--- a/awx/main/models/notifications.py
+++ b/awx/main/models/notifications.py
@@ -23,7 +23,6 @@ from awx.main.notifications.email_backend import CustomEmailBackend
 from awx.main.notifications.slack_backend import SlackBackend
 from awx.main.notifications.twilio_backend import TwilioBackend
 from awx.main.notifications.pagerduty_backend import PagerDutyBackend
-from awx.main.notifications.hipchat_backend import HipChatBackend
 from awx.main.notifications.webhook_backend import WebhookBackend
 from awx.main.notifications.mattermost_backend import MattermostBackend
 from awx.main.notifications.grafana_backend import GrafanaBackend
@@ -44,7 +43,6 @@ class NotificationTemplate(CommonModelNameNotUnique):
                          ('twilio', _('Twilio'), TwilioBackend),
                          ('pagerduty', _('Pagerduty'), PagerDutyBackend),
                          ('grafana', _('Grafana'), GrafanaBackend),
-                          ('hipchat', _('HipChat'), HipChatBackend),
                          ('webhook', _('Webhook'), WebhookBackend),
                          ('mattermost', _('Mattermost'), MattermostBackend),
                          ('rocketchat', _('Rocket.Chat'), RocketChatBackend),
@@ -269,22 +267,20 @@ class JobNotificationMixin(object):
                            'timeout', 'use_fact_cache', 'launch_type', 'status', 'failed', 'started', 'finished',
                            'elapsed', 'job_explanation', 'execution_node', 'controller_node', 'allow_simultaneous',
                            'scm_revision', 'diff_mode', 'job_slice_number', 'job_slice_count', 'custom_virtualenv',
-                            'approval_status', 'approval_node_name', 'workflow_url',
-                            {'host_status_counts': ['skipped', 'ok', 'changed', 'failures', 'dark']},
-                            {'playbook_counts': ['play_count', 'task_count']},
+                            'approval_status', 'approval_node_name', 'workflow_url', 'scm_branch',
+                            {'host_status_counts': ['skipped', 'ok', 'changed', 'failed', 'failures', 'dark'
+                                                    'processed', 'rescued', 'ignored']},
                            {'summary_fields': [{'inventory': ['id', 'name', 'description', 'has_active_failures',
                                                               'total_hosts', 'hosts_with_active_failures', 'total_groups',
                                                               'has_inventory_sources',
                                                               'total_inventory_sources', 'inventory_sources_with_failures',
                                                               'organization_id', 'kind']},
                                                {'project': ['id', 'name', 'description', 'status', 'scm_type']},
-                                                {'project_update': ['id', 'name', 'description', 'status', 'failed']},
                                                {'job_template': ['id', 'name', 'description']},
                                                {'unified_job_template': ['id', 'name', 'description', 'unified_job_type']},
                                                {'instance_group': ['name', 'id']},
                                                {'created_by': ['id', 'username', 'first_name', 'last_name']},
-                                                {'labels': ['count', 'results']},
-                                                {'source_workflow_job': ['description', 'elapsed', 'failed', 'id', 'name', 'status']}]}]
+                                                {'labels': ['count', 'results']}]}]

    @classmethod
    def context_stub(cls):
@@ -303,7 +299,7 @@ class JobNotificationMixin(object):
                           'finished': False,
                           'force_handlers': False,
                           'forks': 0,
-                           'host_status_counts': {'skipped': 1, 'ok': 5, 'changed': 3, 'failures': 0, 'dark': 0},
+                           'host_status_counts': {'skipped': 1, 'ok': 5, 'changed': 3, 'failures': 0, 'dark': 0, 'failed': False, 'processed': 0, 'rescued': 0},
                           'id': 42,
                           'job_explanation': 'Sample job explanation',
                           'job_slice_count': 1,
@@ -314,8 +310,8 @@ class JobNotificationMixin(object):
                           'limit': 'bar_limit',
                           'modified': datetime.datetime(2018, 12, 13, 6, 4, 0, 0, tzinfo=datetime.timezone.utc),
                           'name': 'Stub JobTemplate',
-                           'playbook_counts': {'play_count': 5, 'task_count': 10},
                           'playbook': 'ping.yml',
+                           'scm_branch': '',
                           'scm_revision': '',
                           'skip_tags': '',
                           'start_at_task': '',
@@ -347,18 +343,10 @@ class JobNotificationMixin(object):
                                                          'name': 'Stub project',
                                                          'scm_type': 'git',
                                                          'status': 'successful'},
-                                              'project_update': {'id': 5, 'name': 'Stub Project Update', 'description': 'Project Update',
-                                                                 'status': 'running', 'failed': False},
                                              'unified_job_template': {'description': 'Sample unified job template description',
                                                                       'id': 39,
                                                                       'name': 'Stub Job Template',
-                                                                       'unified_job_type': 'job'},
-                                              'source_workflow_job': {'description': 'Sample workflow job description',
-                                                                      'elapsed': 0.000,
-                                                                      'failed': False,
-                                                                      'id': 88,
-                                                                      'name': 'Stub WorkflowJobTemplate',
-                                                                      'status': 'running'}},
+                                                                       'unified_job_type': 'job'}},
                           'timeout': 0,
                           'type': 'job',
                           'url': '/api/v2/jobs/13/',
@@ -392,10 +380,20 @@ class JobNotificationMixin(object):
        The context will contain whitelisted content retrieved from a serialized job object
        (see JobNotificationMixin.JOB_FIELDS_WHITELIST), the job's friendly name,
        and a url to the job run."""
-        context = {'job': {},
-                   'job_friendly_name': self.get_notification_friendly_name(),
-                   'url': self.get_ui_url(),
-                   'job_metadata': json.dumps(self.notification_data(), indent=4)}
+        job_context = {'host_status_counts': {}}
+        summary = None
+        if hasattr(self, 'job_host_summaries'):
+            summary = self.job_host_summaries.first()
+        if summary:
+            from awx.api.serializers import JobHostSummarySerializer
+            summary_data = JobHostSummarySerializer(summary).to_representation(summary)
+            job_context['host_status_counts'] = summary_data
+        context = {
+            'job': job_context,
+            'job_friendly_name': self.get_notification_friendly_name(),
+            'url': self.get_ui_url(),
+            'job_metadata': json.dumps(self.notification_data(), indent=4)
+        }

        def build_context(node, fields, whitelisted_fields):
            for safe_field in whitelisted_fields:
--- a/awx/main/models/organization.py
+++ b/awx/main/models/organization.py
@@ -6,7 +6,6 @@
 # Django
 from django.conf import settings
 from django.db import models
-from django.db.models import Q
 from django.contrib.auth.models import User
 from django.contrib.sessions.models import Session
 from django.utils.timezone import now as tz_now
@@ -106,12 +105,7 @@ class Organization(CommonModel, NotificationFieldsModel, ResourceMixin, CustomVi
    RelatedJobsMixin
    '''
    def _get_related_jobs(self):
-        project_ids = self.projects.all().values_list('id')
-        return UnifiedJob.objects.non_polymorphic().filter(
-            Q(Job___project__in=project_ids) |
-            Q(ProjectUpdate___project__in=project_ids) |
-            Q(InventoryUpdate___inventory_source__inventory__organization=self)
-        )
+        return UnifiedJob.objects.non_polymorphic().filter(organization=self)


 class Team(CommonModelNameNotUnique, ResourceMixin):
--- a/awx/main/models/projects.py
+++ b/awx/main/models/projects.py
@@ -199,7 +199,7 @@ class ProjectOptions(models.Model):
        results = []
        project_path = self.get_project_path()
        if project_path:
-            for dirpath, dirnames, filenames in os.walk(smart_str(project_path)):
+            for dirpath, dirnames, filenames in os.walk(smart_str(project_path), followlinks=settings.AWX_SHOW_PLAYBOOK_LINKS):
                if skip_directory(dirpath):
                    continue
                for filename in filenames:
@@ -254,13 +254,6 @@ class Project(UnifiedJobTemplate, ProjectOptions, ResourceMixin, CustomVirtualEn
        app_label = 'main'
        ordering = ('id',)

-    organization = models.ForeignKey(
-        'Organization',
-        blank=True,
-        null=True,
-        on_delete=models.CASCADE,
-        related_name='projects',
-    )
    scm_update_on_launch = models.BooleanField(
        default=False,
        help_text=_('Update the project when a job is launched that uses the project.'),
@@ -329,9 +322,16 @@ class Project(UnifiedJobTemplate, ProjectOptions, ResourceMixin, CustomVirtualEn
    @classmethod
    def _get_unified_job_field_names(cls):
        return set(f.name for f in ProjectOptions._meta.fields) | set(
-            ['name', 'description']
+            ['name', 'description', 'organization']
        )

+    def clean_organization(self):
+        if self.pk:
+            old_org_id = getattr(self, '_prior_values_store', {}).get('organization_id', None)
+            if self.organization_id != old_org_id and self.jobtemplates.exists():
+                raise ValidationError({'organization': _('Organization cannot be changed when in use by job templates.')})
+        return self.organization
+
    def save(self, *args, **kwargs):
        new_instance = not bool(self.pk)
        pre_save_vals = getattr(self, '_prior_values_store', {})
@@ -450,8 +450,8 @@ class Project(UnifiedJobTemplate, ProjectOptions, ResourceMixin, CustomVirtualEn
    '''
    def _get_related_jobs(self):
        return UnifiedJob.objects.non_polymorphic().filter(
-            models.Q(Job___project=self) |
-            models.Q(ProjectUpdate___project=self)
+            models.Q(job__project=self) |
+            models.Q(projectupdate__project=self)
        )

    def delete(self, *args, **kwargs):
@@ -584,8 +584,8 @@ class ProjectUpdate(UnifiedJob, ProjectOptions, JobNotificationMixin, TaskManage

    @property
    def preferred_instance_groups(self):
-        if self.project is not None and self.project.organization is not None:
-            organization_groups = [x for x in self.project.organization.instance_groups.all()]
+        if self.organization is not None:
+            organization_groups = [x for x in self.organization.instance_groups.all()]
        else:
            organization_groups = []
        template_groups = [x for x in super(ProjectUpdate, self).preferred_instance_groups]
--- a/awx/main/models/schedules.py
+++ b/awx/main/models/schedules.py
@@ -191,7 +191,7 @@ class Schedule(PrimordialModel, LaunchTimeConfig):
        return rrule

    @classmethod
-    def rrulestr(cls, rrule, **kwargs):
+    def rrulestr(cls, rrule, fast_forward=True, **kwargs):
        """
        Apply our own custom rrule parsing requirements
        """
@@ -205,11 +205,17 @@ class Schedule(PrimordialModel, LaunchTimeConfig):
                    'A valid TZID must be provided (e.g., America/New_York)'
                )

-        if 'MINUTELY' in rrule or 'HOURLY' in rrule:
+        if fast_forward and ('MINUTELY' in rrule or 'HOURLY' in rrule):
            try:
                first_event = x[0]
-                if first_event < now() - datetime.timedelta(days=365 * 5):
-                    raise ValueError('RRULE values with more than 1000 events are not allowed.')
+                if first_event < now():
+                    # hourly/minutely rrules with far-past DTSTART values
+                    # are *really* slow to precompute
+                    # start *from* one week ago to speed things up drastically
+                    dtstart = x._rrule[0]._dtstart.strftime(':%Y%m%dT')
+                    new_start = (now() - datetime.timedelta(days=7)).strftime(':%Y%m%dT')
+                    new_rrule = rrule.replace(dtstart, new_start)
+                    return Schedule.rrulestr(new_rrule, fast_forward=False)
            except IndexError:
                pass
        return x
--- a/awx/main/models/unified_jobs.py
+++ b/awx/main/models/unified_jobs.py
@@ -3,6 +3,7 @@

 # Python
 from io import StringIO
+import datetime
 import codecs
 import json
 import logging
@@ -35,6 +36,7 @@ from awx.main.models.base import (
    NotificationFieldsModel,
    prevent_search
 )
+from awx.main.dispatch import get_local_queuename
 from awx.main.dispatch.control import Control as ControlDispatcher
 from awx.main.registrar import activity_stream_registrar
 from awx.main.models.mixins import ResourceMixin, TaskManagerUnifiedJobMixin
@@ -101,7 +103,7 @@ class UnifiedJobTemplate(PolymorphicModel, CommonModelNameNotUnique, Notificatio
        ordering = ('name',)
        # unique_together here is intentionally commented out. Please make sure sub-classes of this model
        # contain at least this uniqueness restriction: SOFT_UNIQUE_TOGETHER = [('polymorphic_ctype', 'name')]
-        #unique_together = [('polymorphic_ctype', 'name')]
+        #unique_together = [('polymorphic_ctype', 'name', 'organization')]

    old_pk = models.PositiveIntegerField(
        null=True,
@@ -148,7 +150,7 @@ class UnifiedJobTemplate(PolymorphicModel, CommonModelNameNotUnique, Notificatio
        default=None,
        editable=False,
        related_name='%(class)s_as_next_schedule+',
-        on_delete=models.SET_NULL,
+        on_delete=polymorphic.SET_NULL,
    )
    status = models.CharField(
        max_length=32,
@@ -156,6 +158,14 @@ class UnifiedJobTemplate(PolymorphicModel, CommonModelNameNotUnique, Notificatio
        default='ok',
        editable=False,
    )
+    organization = models.ForeignKey(
+        'Organization',
+        blank=True,
+        null=True,
+        on_delete=polymorphic.SET_NULL,
+        related_name='%(class)ss',
+        help_text=_('The organization used to determine access to this template.'),
+    )
    credentials = models.ManyToManyField(
        'Credential',
        related_name='%(class)ss',
@@ -403,9 +413,8 @@ class UnifiedJobTemplate(PolymorphicModel, CommonModelNameNotUnique, Notificatio
        if 'extra_vars' in validated_kwargs:
            unified_job.handle_extra_data(validated_kwargs['extra_vars'])

-        if not getattr(self, '_deprecated_credential_launch', False):
-            # Create record of provided prompts for relaunch and rescheduling
-            unified_job.create_config_from_prompts(kwargs, parent=self)
+        # Create record of provided prompts for relaunch and rescheduling
+        unified_job.create_config_from_prompts(kwargs, parent=self)

        # manually issue the create activity stream entry _after_ M2M relations
        # have been associated to the UJ
@@ -577,7 +586,7 @@ class UnifiedJob(PolymorphicModel, PasswordFieldsModel, CommonModelNameNotUnique
        null=True,
        default=None,
        editable=False,
-        on_delete=models.SET_NULL,
+        on_delete=polymorphic.SET_NULL,
    )
    dependent_jobs = models.ManyToManyField(
        'self',
@@ -697,7 +706,15 @@ class UnifiedJob(PolymorphicModel, PasswordFieldsModel, CommonModelNameNotUnique
        null=True,
        default=None,
        on_delete=polymorphic.SET_NULL,
-        help_text=_('The Rampart/Instance group the job was run under'),
+        help_text=_('The Instance group the job was run under'),
+    )
+    organization = models.ForeignKey(
+        'Organization',
+        blank=True,
+        null=True,
+        on_delete=polymorphic.SET_NULL,
+        related_name='%(class)ss',
+        help_text=_('The organization used to determine access to this unified job.'),
    )
    credentials = models.ManyToManyField(
        'Credential',
@@ -1218,12 +1235,17 @@ class UnifiedJob(PolymorphicModel, PasswordFieldsModel, CommonModelNameNotUnique
                    status_data['instance_group_name'] = self.instance_group.name
                else:
                    status_data['instance_group_name'] = None
+            elif status in ['successful', 'failed', 'canceled'] and self.finished:
+                status_data['finished'] = datetime.datetime.strftime(self.finished, "%Y-%m-%dT%H:%M:%S.%fZ")
            status_data.update(self.websocket_emit_data())
            status_data['group_name'] = 'jobs'
+            if getattr(self, 'unified_job_template_id', None):
+                status_data['unified_job_template_id'] = self.unified_job_template_id
            emit_channel_notification('jobs-status_changed', status_data)

            if self.spawned_by_workflow:
                status_data['group_name'] = "workflow_events"
+                status_data['workflow_job_template_id'] = self.unified_job_template.id
                emit_channel_notification('workflow_events-' + str(self.workflow_job_id), status_data)
        except IOError:  # includes socket errors
            logger.exception('%s failed to emit channel msg about status change', self.log_format)
@@ -1338,7 +1360,7 @@ class UnifiedJob(PolymorphicModel, PasswordFieldsModel, CommonModelNameNotUnique
            timeout = 5
            try:
                running = self.celery_task_id in ControlDispatcher(
-                    'dispatcher', self.execution_node
+                    'dispatcher', self.controller_node or self.execution_node
                ).running(timeout=timeout)
            except socket.timeout:
                logger.error('could not reach dispatcher on {} within {}s'.format(
@@ -1444,7 +1466,7 @@ class UnifiedJob(PolymorphicModel, PasswordFieldsModel, CommonModelNameNotUnique
        return r

    def get_queue_name(self):
-        return self.controller_node or self.execution_node or settings.CELERY_DEFAULT_QUEUE
+        return self.controller_node or self.execution_node or get_local_queuename()

    def is_isolated(self):
        return bool(self.controller_node)
--- a/awx/main/models/workflow.py
+++ b/awx/main/models/workflow.py
@@ -4,6 +4,7 @@
 # Python
 import json
 import logging
+from uuid import uuid4
 from copy import copy
 from urllib.parse import urljoin

@@ -121,6 +122,7 @@ class WorkflowNodeBase(CreatedModifiedModel, LaunchTimeConfig):
                create_kwargs[field_name] = kwargs[field_name]
            elif hasattr(self, field_name):
                create_kwargs[field_name] = getattr(self, field_name)
+        create_kwargs['identifier'] = self.identifier
        new_node = WorkflowJobNode.objects.create(**create_kwargs)
        if self.pk:
            allowed_creds = self.credentials.all()
@@ -135,7 +137,7 @@ class WorkflowJobTemplateNode(WorkflowNodeBase):
    FIELDS_TO_PRESERVE_AT_COPY = [
        'unified_job_template', 'workflow_job_template', 'success_nodes', 'failure_nodes',
        'always_nodes', 'credentials', 'inventory', 'extra_data', 'survey_passwords',
-        'char_prompts', 'all_parents_must_converge'
+        'char_prompts', 'all_parents_must_converge', 'identifier'
    ]
    REENCRYPTION_BLACKLIST_AT_COPY = ['extra_data', 'survey_passwords']

@@ -144,6 +146,21 @@ class WorkflowJobTemplateNode(WorkflowNodeBase):
        related_name='workflow_job_template_nodes',
        on_delete=models.CASCADE,
    )
+    identifier = models.CharField(
+        max_length=512,
+        default=uuid4,
+        blank=False,
+        help_text=_(
+            'An identifier for this node that is unique within its workflow. '
+            'It is copied to workflow job nodes corresponding to this node.'),
+    )
+
+    class Meta:
+        app_label = 'main'
+        unique_together = (("identifier", "workflow_job_template"),)
+        indexes = [
+            models.Index(fields=['identifier']),
+        ]

    def get_absolute_url(self, request=None):
        return reverse('api:workflow_job_template_node_detail', kwargs={'pk': self.pk}, request=request)
@@ -213,6 +230,18 @@ class WorkflowJobNode(WorkflowNodeBase):
                    "semantics will mark this True if the node is in a path that will "
                    "decidedly not be ran. A value of False means the node may not run."),
    )
+    identifier = models.CharField(
+        max_length=512,
+        blank=True,  # blank denotes pre-migration job nodes
+        help_text=_('An identifier coresponding to the workflow job template node that this node was created from.'),
+    )
+
+    class Meta:
+        app_label = 'main'
+        indexes = [
+            models.Index(fields=["identifier", "workflow_job"]),
+            models.Index(fields=['identifier']),
+        ]

    def get_absolute_url(self, request=None):
        return reverse('api:workflow_job_node_detail', kwargs={'pk': self.pk}, request=request)
@@ -335,7 +364,7 @@ class WorkflowJobOptions(LaunchTimeConfigBase):
    @classmethod
    def _get_unified_job_field_names(cls):
        r = set(f.name for f in WorkflowJobOptions._meta.fields) | set(
-            ['name', 'description', 'survey_passwords', 'labels', 'limit', 'scm_branch']
+            ['name', 'description', 'organization', 'survey_passwords', 'labels', 'limit', 'scm_branch']
        )
        r.remove('char_prompts')  # needed due to copying launch config to launch config
        return r
@@ -376,19 +405,12 @@ class WorkflowJobTemplate(UnifiedJobTemplate, WorkflowJobOptions, SurveyJobTempl

    SOFT_UNIQUE_TOGETHER = [('polymorphic_ctype', 'name', 'organization')]
    FIELDS_TO_PRESERVE_AT_COPY = [
-        'labels', 'instance_groups', 'workflow_job_template_nodes', 'credentials', 'survey_spec'
+        'labels', 'organization', 'instance_groups', 'workflow_job_template_nodes', 'credentials', 'survey_spec'
    ]

    class Meta:
        app_label = 'main'

-    organization = models.ForeignKey(
-        'Organization',
-        blank=True,
-        null=True,
-        on_delete=models.SET_NULL,
-        related_name='workflows',
-    )
    ask_inventory_on_launch = AskForField(
        blank=True,
        default=False,
@@ -749,6 +771,8 @@ class WorkflowApproval(UnifiedJob, JobNotificationMixin):

    def signal_start(self, **kwargs):
        can_start = super(WorkflowApproval, self).signal_start(**kwargs)
+        self.started = self.created
+        self.save(update_fields=['started'])
        self.send_approval_notification('running')
        return can_start

--- a/awx/main/notifications/grafana_backend.py
+++ b/awx/main/notifications/grafana_backend.py
@@ -2,6 +2,7 @@
 # All Rights Reserved.

 import datetime
+import json
 import logging
 import requests
 import dateutil.parser as dp
@@ -12,6 +13,19 @@ from django.utils.translation import ugettext_lazy as _
 from awx.main.notifications.base import AWXBaseEmailBackend
 from awx.main.notifications.custom_notification_base import CustomNotificationBase

+DEFAULT_MSG = CustomNotificationBase.DEFAULT_MSG
+
+DEFAULT_APPROVAL_RUNNING_MSG = CustomNotificationBase.DEFAULT_APPROVAL_RUNNING_MSG
+DEFAULT_APPROVAL_RUNNING_BODY = CustomNotificationBase.DEFAULT_APPROVAL_RUNNING_BODY
+
+DEFAULT_APPROVAL_APPROVED_MSG = CustomNotificationBase.DEFAULT_APPROVAL_APPROVED_MSG
+DEFAULT_APPROVAL_APPROVED_BODY = CustomNotificationBase.DEFAULT_APPROVAL_APPROVED_BODY
+
+DEFAULT_APPROVAL_TIMEOUT_MSG = CustomNotificationBase.DEFAULT_APPROVAL_TIMEOUT_MSG
+DEFAULT_APPROVAL_TIMEOUT_BODY = CustomNotificationBase.DEFAULT_APPROVAL_TIMEOUT_BODY
+
+DEFAULT_APPROVAL_DENIED_MSG = CustomNotificationBase.DEFAULT_APPROVAL_DENIED_MSG
+DEFAULT_APPROVAL_DENIED_BODY = CustomNotificationBase.DEFAULT_APPROVAL_DENIED_BODY

 logger = logging.getLogger('awx.main.notifications.grafana_backend')

@@ -23,6 +37,15 @@ class GrafanaBackend(AWXBaseEmailBackend, CustomNotificationBase):
    recipient_parameter = "grafana_url"
    sender_parameter = None

+    DEFAULT_BODY = "{{ job_metadata }}"
+    default_messages = {"started": {"body": DEFAULT_BODY, "message": DEFAULT_MSG},
+                        "success": {"body": DEFAULT_BODY, "message": DEFAULT_MSG},
+                        "error": {"body": DEFAULT_BODY, "message": DEFAULT_MSG},
+                        "workflow_approval": {"running": {"message": DEFAULT_APPROVAL_RUNNING_MSG, "body": DEFAULT_APPROVAL_RUNNING_BODY},
+                                              "approved": {"message": DEFAULT_APPROVAL_APPROVED_MSG,"body": DEFAULT_APPROVAL_APPROVED_BODY},
+                                              "timed_out": {"message": DEFAULT_APPROVAL_TIMEOUT_MSG, "body": DEFAULT_APPROVAL_TIMEOUT_BODY},
+                                              "denied": {"message": DEFAULT_APPROVAL_DENIED_MSG, "body": DEFAULT_APPROVAL_DENIED_BODY}}}
+
    def __init__(self, grafana_key,dashboardId=None, panelId=None, annotation_tags=None, grafana_no_verify_ssl=False, isRegion=True,
                 fail_silently=False, **kwargs):
        super(GrafanaBackend, self).__init__(fail_silently=fail_silently)
@@ -34,6 +57,13 @@ class GrafanaBackend(AWXBaseEmailBackend, CustomNotificationBase):
        self.isRegion = isRegion

    def format_body(self, body):
+        # expect body to be a string representing a dict
+        try:
+            potential_body = json.loads(body)
+            if isinstance(potential_body, dict):
+                body = potential_body
+        except json.JSONDecodeError:
+            body = {}
        return body

    def send_messages(self, messages):
@@ -41,18 +71,21 @@ class GrafanaBackend(AWXBaseEmailBackend, CustomNotificationBase):
        for m in messages:
            grafana_data = {}
            grafana_headers = {}
-            try:
-                epoch=datetime.datetime.utcfromtimestamp(0)
-                grafana_data['time'] = int((dp.parse(m.body['started']).replace(tzinfo=None) - epoch).total_seconds() * 1000)
-                grafana_data['timeEnd'] = int((dp.parse(m.body['finished']).replace(tzinfo=None) - epoch).total_seconds() * 1000)
-            except ValueError:
-                logger.error(smart_text(_("Error converting time {} or timeEnd {} to int.").format(m.body['started'],m.body['finished'])))
-                if not self.fail_silently:
-                    raise Exception(smart_text(_("Error converting time {} and/or timeEnd {} to int.").format(m.body['started'],m.body['finished'])))
+            if 'started' in m.body:
+                try:
+                    epoch=datetime.datetime.utcfromtimestamp(0)
+                    grafana_data['time'] = grafana_data['timeEnd'] = int((dp.parse(m.body['started']).replace(tzinfo=None) - epoch).total_seconds() * 1000)
+                    if m.body.get('finished'):
+                        grafana_data['timeEnd'] = int((dp.parse(m.body['finished']).replace(tzinfo=None) - epoch).total_seconds() * 1000)
+                except ValueError:
+                    logger.error(smart_text(_("Error converting time {} or timeEnd {} to int.").format(m.body['started'],m.body['finished'])))
+                    if not self.fail_silently:
+                        raise Exception(smart_text(_("Error converting time {} and/or timeEnd {} to int.").format(m.body['started'],m.body['finished'])))
            grafana_data['isRegion'] = self.isRegion
            grafana_data['dashboardId'] = self.dashboardId
            grafana_data['panelId'] = self.panelId
-            grafana_data['tags'] = self.annotation_tags
+            if self.annotation_tags:
+                grafana_data['tags'] = self.annotation_tags
            grafana_data['text'] = m.subject
            grafana_headers['Authorization'] = "Bearer {}".format(self.grafana_key)
            grafana_headers['Content-Type'] = "application/json"
--- a/awx/main/notifications/hipchat_backend.py
+++ b/awx/main/notifications/hipchat_backend.py
@@ -1,54 +0,0 @@
-# Copyright (c) 2016 Ansible, Inc.
-# All Rights Reserved.
-
-import logging
-
-import requests
-
-from django.utils.encoding import smart_text
-from django.utils.translation import ugettext_lazy as _
-
-from awx.main.notifications.base import AWXBaseEmailBackend
-from awx.main.notifications.custom_notification_base import CustomNotificationBase
-
-logger = logging.getLogger('awx.main.notifications.hipchat_backend')
-
-
-class HipChatBackend(AWXBaseEmailBackend, CustomNotificationBase):
-
-    init_parameters = {"token": {"label": "Token", "type": "password"},
-                       "rooms": {"label": "Destination Rooms", "type": "list"},
-                       "color": {"label": "Notification Color", "type": "string"},
-                       "api_url": {"label": "API Url (e.g: https://mycompany.hipchat.com)", "type": "string"},
-                       "notify": {"label": "Notify room", "type": "bool"},
-                       "message_from": {"label": "Label to be shown with notification", "type": "string"}}
-    recipient_parameter = "rooms"
-    sender_parameter = "message_from"
-
-    def __init__(self, token, color, api_url, notify, fail_silently=False, **kwargs):
-        super(HipChatBackend, self).__init__(fail_silently=fail_silently)
-        self.token = token
-        if color is not None:
-            self.color = color.lower()
-        self.api_url = api_url
-        self.notify = notify
-        
-    def send_messages(self, messages):
-        sent_messages = 0
-
-        for m in messages:
-            for rcp in m.recipients():
-                r = requests.post("{}/v2/room/{}/notification".format(self.api_url, rcp),
-                                  params={"auth_token": self.token},
-                                  verify=False,
-                                  json={"color": self.color,
-                                        "message": m.subject,
-                                        "notify": self.notify,
-                                        "from": m.from_email,
-                                        "message_format": "text"})
-                if r.status_code != 204:
-                    logger.error(smart_text(_("Error sending messages: {}").format(r.text)))
-                    if not self.fail_silently:
-                        raise Exception(smart_text(_("Error sending message to hipchat: {}").format(r.text)))
-                sent_messages += 1
-        return sent_messages
--- a/Show More
+++ b/Show More
@@ -1 +1 @@
 .2.0
 .0.0