Merge pull request #5656 from ryanpetrello/pygments-minus-minus

remove some unnecessary callback receiver debugging code

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul

.github/BOTMETA.yml

@@ -1,3 +1,4 @@
+---
 files:
     awx/ui/:
         labels: component:ui

.yamllint

@@ -0,0 +1,12 @@
+---
+ignore: |
+  .tox
+  awx/main/tests/data/inventory/plugins/**
+  # vault files
+  awx/main/tests/data/ansible_utils/playbooks/valid/vault.yml
+  awx/ui/test/e2e/tests/smoke-vars.yml
+
+extends: default
+
+rules:
+  line-length: disable

CHANGELOG.md

@@ -0,0 +1,128 @@
+# Changelog
+
+This is a list of high-level changes for each release of AWX. A full list of commits can be found at `https://github.com/ansible/awx/releases/tag/<version>`.
+
+## 9.1.1 (Jan 14, 2020)
+
+- Fixed a bug that caused database migrations on Kubernetes installs to hang https://github.com/ansible/awx/pull/5579
+- Upgraded Python-level app dependencies in AWX virtual environment https://github.com/ansible/awx/pull/5407
+- Running jobs no longer block associated inventory updates https://github.com/ansible/awx/pull/5519
+- Fixed invalid_response SAML error https://github.com/ansible/awx/pull/5577
+- Optimized the callback receiver to drastically improve the write speed of stdout for parallel jobs (https://github.com/ansible/awx/pull/5618)
+
+## 9.1.0 (Dec 17, 2019)
+- Added a command to generate a new SECRET_KEY and rekey the secrets in the database
+- Removed project update locking when jobs using it are running
+- Fixed slow queries for /api/v2/instances and /api/v2/instance_groups when smart inventories are used
+- Fixed a partial password disclosure when special characters existed in the RabbitMQ password (CVE-2019-19342)
+- Fixed hang in error handling for source control checkouts
+- Fixed an error on subsequent job runs that override the branch of a project on an instance that did not have a prior project checkout
+- Fixed an issue where jobs launched in isolated or container groups would incorrectly timeout
+- Fixed an incorrect link to instance groups documentation in the user interface
+- Fixed editing of inventory on Workflow templates
+- Fixed multiple issues with OAuth2 token cleanup system jobs
+- Fixed a bug that broke email notifications for workflow approval/deny https://github.com/ansible/awx/issues/5401
+- Updated SAML implementation to automatically login if authorization already exists
+- Updated AngularJS to 1.7.9 for CVE-2019-10768
+
+## 9.0.1 (Nov 4, 2019)
+
+- Fixed a bug in the installer that broke certain types of k8s installs https://github.com/ansible/awx/issues/5205
+
+## 9.0.0 (Oct 31, 2019)
+
+- Updated AWX images to use centos:8 as the parent image.
+- Updated to ansible-runner 1.4.4 to address various bugs.
+- Added oc and kubectl to the AWX images to support new container-based execution introduced in 8.0.0.
+- Added some optimizations to speed up the deletion of large Inventory Groups.
+- Fixed a bug that broke webhook launches for Job Templates that define a survey (https://github.com/ansible/awx/issues/5062).
+- Fixed a bug in the CLI which incorrectly parsed launch time arguments for `awx job_templates launch` and `awx workflow_job_templates launch` (https://github.com/ansible/awx/issues/5093).
+- Fixed a bug that caused inventory updates using "sourced from a project" to stop working (https://github.com/ansible/awx/issues/4750).
+- Fixed a bug that caused Slack notifications to sometimes show the wrong bot avatar (https://github.com/ansible/awx/pull/5125).
+- Fixed a bug that prevented the use of digits in Tower's URL settings (https://github.com/ansible/awx/issues/5081).
+
+## 8.0.0 (Oct 21, 2019)
+
+- The Ansible Tower Ansible modules have been migrated to a new official Ansible AWX collection: https://galaxy.ansible.com/awx/AWX
+  Please note that this functionality is only supported in Ansible 2.9+
+- AWX now supports the ability to launch jobs from external webhooks (GitHub and GitLab integration are supported).
+- AWX now supports Container Groups, a new feature that allows you to schedule and run playbooks on single-use kubernetes pods on-demand.
+- AWX now supports sending notifications when Workflow steps are approved, denied, or time out.
+- AWX now records the user who approved or denied Workflow steps.
+- AWX now supports fetching Ansible Collections from private galaxy servers.
+- AWX now checks the user's ansible.cfg for paths where role/collections may live when running project updates.
+- AWX now uses PostgreSQL 10 by default.
+- AWX now warns more loudly about underlying AMQP connectivity issues (https://github.com/ansible/awx/pull/4857).
+- Added a few optimizations to drastically improve dashboard performance for larger AWX installs (installs with several hundred thousand jobs or more).
+- Updated to the latest version of Ansible's VMWare inventory script (which adds support for vmware_guest_facts).
+- Deprecated /api/v2/inventory_scripts/ (this endpoint - and the Custom Inventory Script feature - will be removed in a future release of AWX).
+- Fixed a bug which prevented Organization Admins from removing users from their own Organization (https://github.com/ansible/awx/issues/2979)
+- Fixed a bug which sometimes caused cluster nodes to fail to re-join with a cryptic error, "No instance found with the current cluster host id" (https://github.com/ansible/awx/issues/4294)
+- Fixed a bug that prevented the use of launch-time passphrases when using credential plugins (https://github.com/ansible/awx/pull/4807)
+- Fixed a bug that caused notifications assigned at the Organization level not to take effect for Workflows in that Organization (https://github.com/ansible/awx/issues/4712)
+- Fixed a bug which caused a notable amount of CPU overhead on RabbitMQ health checks (https://github.com/ansible/awx/pull/5009)
+- Fixed a bug which sometimes caused the <return> key to stop functioning in <textarea> elements (https://github.com/ansible/awx/issues/4192)
+- Fixed a bug which caused request contention when the same OAuth2.0 token was used in multiple simultaneous requests (https://github.com/ansible/awx/issues/4694)
+- Fixed a bug related to parsing multiple choice survey options (https://github.com/ansible/awx/issues/4452).
+- Fixed a bug that caused single-sign-on icons on the login page to fail to render in certain Windows browsers (https://github.com/ansible/awx/issues/3924)
+- Fixed a number of bugs that caused certain OAuth2 settings to not be properly respected, such as REFRESH_TOKEN_EXPIRE_SECONDS.
+- Fixed a number of bugs in the AWX CLI, including a bug which sometimes caused long lines of stdout output to be unexpectedly truncated.
+- Fixed a number of bugs on the job details UI which sometimes caused auto-scrolling stdout to become stuck.
+- Fixed a bug which caused LDAP authentication to fail if the TLD of the server URL contained digits (https://github.com/ansible/awx/issues/3646)
+- Fixed a bug which broke HashiCorp Vault integration on older versions of HashiCorp Vault.
+
+## 7.0.0 (Sept 4, 2019)
+
+- AWX now detects and installs Ansible Collections defined in your project (note - this feature only works in Ansible 2.9+) (https://github.com/ansible/awx/issues/2534)
+- AWX now includes an official command line client.  Keep an eye out for a follow-up email on this mailing list for information on how to install it and try it out.
+- Added the ability to provide a specific SCM branch on jobs (https://github.com/ansible/awx/issues/282)
+- Added support for Workflow Approval Nodes, a new feature which allows you to add "pause and wait for approval" steps into your workflows (https://github.com/ansible/awx/issues/1206)
+- Added the ability to specify a specific HTTP method for webhook notifications (POST vs PUT) (https://github.com/ansible/awx/pull/4124)
+- Added the ability to specify a username and password for HTTP Basic Authorization for webhook notifications (https://github.com/ansible/awx/pull/4124)
+- Added support for customizing the text content of notifications (https://github.com/ansible/awx/issues/79)
+- Added the ability to enable and disable hosts in dynamic inventory (https://github.com/ansible/awx/pull/4420)
+- Added the description (if any) to the Job Template list (https://github.com/ansible/awx/issues/4359)
+- Added new metrics for instance hostnames and pending jobs to the /api/v2/metrics/ endpoint (https://github.com/ansible/awx/pull/4375)
+- Changed AWX's on/off toggle buttons to a non-text based style to simplify internationalization (https://github.com/ansible/awx/pull/4425)
+- Events emitted by ansible for adhoc commands are now sent to the external log aggregrator (https://github.com/ansible/awx/issues/4545)
+- Fixed a bug which allowed a user to make an organization credential in another organization without permissions to that organization (https://github.com/ansible/awx/pull/4483)
+- Fixed a bug that caused `extra_vars` on workflows to break when edited (https://github.com/ansible/awx/issues/4293)
+- Fixed a slow SQL query that caused performance issues when large numbers of groups exist (https://github.com/ansible/awx/issues/4461)
+- Fixed a few minor bugs in survey field validation (https://github.com/ansible/awx/pull/4509) (https://github.com/ansible/awx/pull/4479)
+- Fixed a bug that sometimes resulted in orphaned `ansible_runner_pi` directories in `/tmp` after playbook execution (https://github.com/ansible/awx/pull/4409)
+- Fixed a bug that caused the `is_system_auditor` flag in LDAP configuration to not work (https://github.com/ansible/awx/pull/4396)
+- Fixed a bug which caused schedules to disappear from the UI when toggled off (https://github.com/ansible/awx/pull/4378)
+- Fixed a bug that sometimes caused stdout content to contain extraneous blank lines in newer versions of Ansible (https://github.com/ansible/awx/pull/4391)
+- Updated to the latest Django security release, 2.2.4 (https://github.com/ansible/awx/pull/4410) (https://www.djangoproject.com/weblog/2019/aug/01/security-releases/)
+- Updated the default version of git to a version that includes support for x509 certificates (https://github.com/ansible/awx/issues/4362)
+- Removed the deprecated `credential` field from `/api/v2/workflow_job_templates/N/` (as part of the `/api/v1/` removal in prior AWX versions - https://github.com/ansible/awx/pull/4490).
+
+## 6.1.0 (Jul 18, 2019)
+
+- Updated AWX to use Django 2.2.2.
+- Updated the provided openstacksdk version to support new functionality (such as Nova scheduler_hints)
+- Added the ability to specify a custom cacert for the HashiCorp Vault credential plugin
+- Fixed a number of bugs related to path lookups for the HashiCorp Vault credential plugin
+- Fixed a bug which prevented signed SSH certificates from working, including the HashiCorp Vault Signed SSH backend
+- Fixed a bug which prevented custom logos from displaying on the login page (as a result of a new Content Security Policy in 6.0.0)
+- Fixed a bug which broke websocket connectivity in Apple Safari (as a result of a new Content Security Policy in 6.0.0)
+- Fixed a bug on the job output page that occasionally caused the "up" and "down" buttons to not load additional output
+- Fixed a bug on the job output page that caused quoted task names to display incorrectly
+
+## 6.0.0 (Jul 1, 2019)
+
+- Removed support for "Any" notification templates and their API endpoints e.g., /api/v2/job_templates/N/notification_templates/any/ (https://github.com/ansible/awx/issues/4022)
+- Fixed a bug which prevented credentials from properly being applied to inventory sources (https://github.com/ansible/awx/issues/4059)
+- Fixed a bug which can cause the task dispatcher to hang indefinitely when external logging support (e.g., Splunk, Logstash) is enabled (https://github.com/ansible/awx/issues/4181)
+- Fixed a bug which causes slow stdout display when running jobs against smart inventories. (https://github.com/ansible/awx/issues/3106)
+- Fixed a bug that caused SSL verification flags to fail to be respected for LDAP authentication in certain environments. (https://github.com/ansible/awx/pull/4190)
+- Added a simple Content Security Policy (https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) to restrict access to third-party resources in the browser. (https://github.com/ansible/awx/pull/4167)
+- Updated ovirt4 library dependencies to work with newer versions of oVirt (https://github.com/ansible/awx/issues/4138)
+
+## 5.0.0 (Jun 21, 2019)
+
+- Bump Django Rest Framework from 3.7.7 to 3.9.4
+- Bump setuptools / pip dependencies
+- Fixed bug where Recent Notification list would not appear
+- Added notifications on job start
+- Default to Ansible 2.8

INSTALL.md

@@ -4,41 +4,45 @@ This document provides a guide for installing AWX.
 
 ## Table of contents
 
-- [Getting started](#getting-started)
-  - [Clone the repo](#clone-the-repo)
-  - [AWX branding](#awx-branding)
-  - [Prerequisites](#prerequisites)
-  - [System Requirements](#system-requirements)
-  - [AWX Tunables](#awx-tunables)
-  - [Choose a deployment platform](#choose-a-deployment-platform)
-  - [Official vs Building Images](#official-vs-building-images)
-- [OpenShift](#openshift)
-  - [Prerequisites](#prerequisites-1)
-    - [Deploying to Minishift](#deploying-to-minishift)
-  - [Pre-build steps](#pre-build-steps)
-  - [PostgreSQL](#postgresql)
-  - [Start the build](#start-the-build)
-  - [Post build](#post-build)
-  - [Accessing AWX](#accessing-awx)
-- [Kubernetes](#kubernetes)
-  - [Prerequisites](#prerequisites-2)
-  - [Pre-build steps](#pre-build-steps-1)
-  - [Configuring Helm](#configuring-helm)
-  - [Start the build](#start-the-build-1)
-  - [Accessing AWX](#accessing-awx-1)
-  - [SSL Termination](#ssl-termination)
-- [Docker Compose](#docker-compose)
-  - [Prerequisites](#prerequisites-3)
-  - [Pre-build steps](#pre-build-steps-2)
-    - [Deploying to a remote host](#deploying-to-a-remote-host)
-    - [Inventory variables](#inventory-variables)
+- [Installing AWX](#installing-awx)
+  * [Getting started](#getting-started)
+    + [Clone the repo](#clone-the-repo)
+    + [AWX branding](#awx-branding)
+    + [Prerequisites](#prerequisites)
+    + [System Requirements](#system-requirements)
+    + [AWX Tunables](#awx-tunables)
+    + [Choose a deployment platform](#choose-a-deployment-platform)
+    + [Official vs Building Images](#official-vs-building-images)
+  * [Upgrading from previous versions](#upgrading-from-previous-versions)
+  * [OpenShift](#openshift)
+    + [Prerequisites](#prerequisites-1)
+    + [Pre-install steps](#pre-install-steps)
+      - [Deploying to Minishift](#deploying-to-minishift)
+      - [PostgreSQL](#postgresql)
+    + [Run the installer](#run-the-installer)
+    + [Post-install](#post-install)
+    + [Accessing AWX](#accessing-awx)
+  * [Kubernetes](#kubernetes)
+    + [Prerequisites](#prerequisites-2)
+    + [Pre-install steps](#pre-install-steps-1)
+    + [Configuring Helm](#configuring-helm)
+    + [Run the installer](#run-the-installer-1)
+    + [Post-install](#post-install-1)
+    + [Accessing AWX](#accessing-awx-1)
+    + [SSL Termination](#ssl-termination)
+  * [Docker-Compose](#docker-compose)
+    + [Prerequisites](#prerequisites-3)
+    + [Pre-install steps](#pre-install-steps-2)
+      - [Deploying to a remote host](#deploying-to-a-remote-host)
+      - [Inventory variables](#inventory-variables)
       - [Docker registry](#docker-registry)
-      - [PostgreSQL](#postgresql-1)
       - [Proxy settings](#proxy-settings)
-  - [Start the build](#start-the-build-2)
-  - [Post build](#post-build-2)
-  - [Accessing AWX](#accessing-awx-2)
+      - [PostgreSQL](#postgresql-1)
+    + [Run the installer](#run-the-installer-2)
+    + [Post-install](#post-install-2)
+    + [Accessing AWX](#accessing-awx-2)
 
+    
 ## Getting started
 
 ### Clone the repo
@@ -57,7 +61,7 @@ To install the assets, clone the `awx-logos` repo so that it is next to your `aw
 
 Before you can run a deployment, you'll need the following installed in your local environment:
 
-- [Ansible](http://docs.ansible.com/ansible/latest/intro_installation.html) Requires Version 2.4+
+- [Ansible](http://docs.ansible.com/ansible/latest/intro_installation.html) Requires Version 2.8+
 - [Docker](https://docs.docker.com/engine/installation/)
     + A recent version
 - [docker](https://pypi.org/project/docker/) Python module
@@ -114,6 +118,26 @@ If these variables are present then all deployments will use these hosted images
 
 > Multiple versions are provided. `latest` always pulls the most recent. You may also select version numbers at different granularities: 1, 1.0, 1.0.1, 1.0.0.123
 
+
+## Upgrading from previous versions
+
+Upgrading AWX involves rerunning the install playbook. Download a newer release from [https://github.com/ansible/awx/releases](https://github.com/ansible/awx/releases) and re-populate the inventory file with your customized variables.
+
+For convenience, you can create a file called `vars.yml`:
+
+```
+admin_password: 'adminpass'
+pg_password: 'pgpass'
+rabbitmq_password: 'rabbitpass'
+secret_key: 'mysupersecret'
+```
+
+And pass it to the installer:
+
+```
+$ ansible-playbook -i inventory install.yml -e @vars.yml
+```
+
 ## OpenShift
 
 ### Prerequisites
@@ -133,9 +157,9 @@ This can be tuned by overriding the variables found in [/installer/roles/kuberne
 
 For more detail on how resource requests are formed see: [https://docs.openshift.com/container-platform/latest/dev_guide/compute_resources.html#dev-compute-resources](https://docs.openshift.com/container-platform/latest/dev_guide/compute_resources.html#dev-compute-resources)
 
-### Pre-build steps
+### Pre-install steps
 
-Before starting the build process, review the [inventory](./installer/inventory) file, and uncomment and provide values for the following variables found in the `[all:vars]` section:
+Before starting the install, review the [inventory](./installer/inventory) file, and uncomment and provide values for the following variables found in the `[all:vars]` section:
 
 *openshift_host*
 
@@ -197,20 +221,20 @@ By default, AWX will deploy a PostgreSQL pod inside of your cluster. You will ne
 
 If you wish to use an external database, in the inventory file, set the value of `pg_hostname`, and update `pg_username`, `pg_password`, `pg_admin_password`, `pg_database`, and `pg_port` with the connection information. When setting `pg_hostname` the installer will assume you have configured the database in that location and will not launch the postgresql pod.
 
-### Start the build
+### Run the installer
 
-To start the build, you will pass two *extra* variables on the command line. The first is *openshift_password*, which is the password for the *openshift_user*, and the second is *docker_registry_password*, which is the password associated with *docker_registry_username*.
+To start the install, you will pass two *extra* variables on the command line. The first is *openshift_password*, which is the password for the *openshift_user*, and the second is *docker_registry_password*, which is the password associated with *docker_registry_username*.
 
 If you're using the OpenShift internal registry, then you'll pass an access token for the *docker_registry_password* value, rather than a password. The `oc whoami -t` command will generate the required token, as long as you're logged into the cluster via `oc cluster login`.
 
-To start the build and deployment, run the following (docker_registry_password is optional if using official images):
+Run the following command (docker_registry_password is optional if using official images):
 
 ```bash
-# Start the build and deployment
+# Start the install
 $ ansible-playbook -i inventory install.yml -e openshift_password=developer  -e docker_registry_password=$(oc whoami -t)
 ```
 
-### Post build
+### Post-install
 
 After the playbook run completes, check the status of the deployment by running `oc get pods`:
 
@@ -327,9 +351,9 @@ This can be tuned by overriding the variables found in [/installer/roles/kuberne
 
 For more detail on how resource requests are formed see: [https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/)
 
-### Pre-build steps
+### Pre-install steps
 
-Before starting the build process, review the [inventory](./installer/inventory) file, and uncomment and provide values for the following variables found in the `[all:vars]` section uncommenting when necessary. Make sure the openshift and standalone docker sections are commented out:
+Before starting the install process, review the [inventory](./installer/inventory) file, and uncomment and provide values for the following variables found in the `[all:vars]` section uncommenting when necessary. Make sure the openshift and standalone docker sections are commented out:
 
 *kubernetes_context*
 
@@ -349,7 +373,7 @@ If you want the AWX installer to manage creating the database pod (rather than i
 
 Newer Kubernetes clusters with RBAC enabled will need to make sure a service account is created, make sure to follow the instructions here [https://docs.helm.sh/using_helm/#role-based-access-control](https://docs.helm.sh/using_helm/#role-based-access-control)
 
-### Start the build
+### Run the installer
 
 After making changes to the `inventory` file use `ansible-playbook` to begin the install
 
@@ -357,7 +381,7 @@ After making changes to the `inventory` file use `ansible-playbook` to begin the
 $ ansible-playbook -i inventory install.yml
 ```
 
-### Post build
+### Post-install
 
 After the playbook run completes, check the status of the deployment by running `kubectl get pods --namespace awx` (replace awx with the namespace you used):
 
@@ -405,7 +429,7 @@ Unlike Openshift's `Route` the Kubernetes `Ingress` doesn't yet handle SSL termi
     + This also installs the `docker` Python module, which is incompatible with `docker-py`. If you have previously installed `docker-py`, please uninstall it.
 - [Docker Compose](https://docs.docker.com/compose/install/).
 
-### Pre-build steps
+### Pre-install steps
 
 #### Deploying to a remote host
 
@@ -436,7 +460,7 @@ If you choose to use the official images then the remote host will be the one to
 
 #### Inventory variables
 
-Before starting the build process, review the [inventory](./installer/inventory) file, and uncomment and provide values for the following variables found in the `[all:vars]` section:
+Before starting the install process, review the [inventory](./installer/inventory) file, and uncomment and provide values for the following variables found in the `[all:vars]` section:
 
 *postgres_data_dir*
 
@@ -511,9 +535,9 @@ AWX requires access to a PostgreSQL database, and by default, one will be create
 
 If you wish to use an external database, in the inventory file, set the value of `pg_hostname`, and update `pg_username`, `pg_password`, `pg_admin_password`, `pg_database`, and `pg_port` with the connection information.
 
-### Start the build
+### Run the installer
 
-If you are not pushing images to a Docker registry, start the build by running the following:
+If you are not pushing images to a Docker registry, start the install by running the following:
 
 ```bash
 # Set the working directory to installer
@@ -533,7 +557,7 @@ $ cd installer
 $ ansible-playbook -i inventory -e docker_registry_password=password install.yml
 ```
 
-### Post build
+### Post-install
 
 After the playbook run completes, Docker will report up to 5 running containers. If you chose to use an existing PostgresSQL database, then it will report 4. You can view the running containers using the `docker ps` command, as follows:
 
@@ -610,14 +634,3 @@ Added instance awx to tower
 The AWX web server is accessible on the deployment host, using the *host_port* value set in the *inventory* file. The default URL is [http://localhost](http://localhost).
 
 You will prompted with a login dialog. The default administrator username is `admin`, and the password is `password`.
-
-### Maintenance using docker-compose
-
-After the installation, maintenance operations with docker-compose can be done by using the  `docker-compose.yml` file created at the location pointed by `docker_compose_dir`.
-
-Among the possible operations, you may:
-
-- Stop AWX : `docker-compose stop`
-- Upgrade AWX : `docker-compose pull && docker-compose up --force-recreate`
-
-See the [docker-compose documentation](https://docs.docker.com/compose/) for details.

Makefile

@@ -26,6 +26,9 @@ DEV_DOCKER_TAG_BASE ?= gcr.io/ansible-tower-engineering
 # Python packages to install only from source (not from binary wheels)
 # Comma separated list
 SRC_ONLY_PKGS ?= cffi,pycparser,psycopg2,twilio
+# These should be upgraded in the AWX and Ansible venv before attempting
+# to install the actual requirements
+VENV_BOOTSTRAP ?= pip==19.3.1 setuptools==41.6.0
 
 # Determine appropriate shasum command
 UNAME_S := $(shell uname -s)
@@ -100,7 +103,7 @@ clean-languages:
 	find . -type f -regex ".*\.mo$$" -delete
 
 # Remove temporary build files, compiled Python files.
-clean: clean-ui clean-api clean-dist
+clean: clean-ui clean-api clean-awxkit clean-dist
 	rm -rf awx/public
 	rm -rf awx/lib/site-packages
 	rm -rf awx/job_status
@@ -116,6 +119,10 @@ clean-api:
 	find . -type d -name "__pycache__" -delete
 	rm -f awx/awx_test.sqlite3*
 	rm -rf requirements/vendor
+	rm -rf awx/projects
+
+clean-awxkit:
+	rm -rf awxkit/*.egg-info awxkit/.tox
 
 # convenience target to assert environment variables are defined
 guard-%:
@@ -126,16 +133,16 @@ guard-%:
 
 virtualenv: virtualenv_ansible virtualenv_awx
 
+# virtualenv_* targets do not use --system-site-packages to prevent bugs installing packages
+# but Ansible venvs are expected to have this, so that must be done after venv creation
 virtualenv_ansible:
 	if [ "$(VENV_BASE)" ]; then \
 		if [ ! -d "$(VENV_BASE)" ]; then \
 			mkdir $(VENV_BASE); \
 		fi; \
 		if [ ! -d "$(VENV_BASE)/ansible" ]; then \
-			virtualenv -p python --system-site-packages $(VENV_BASE)/ansible && \
-			$(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) --ignore-installed six packaging appdirs && \
-			$(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) --ignore-installed setuptools==36.0.1 && \
-			$(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) --ignore-installed pip==9.0.1; \
+			virtualenv -p python $(VENV_BASE)/ansible && \
+			$(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) $(VENV_BOOTSTRAP); \
 		fi; \
 	fi
 
@@ -145,36 +152,46 @@ virtualenv_ansible_py3:
 			mkdir $(VENV_BASE); \
 		fi; \
 		if [ ! -d "$(VENV_BASE)/ansible" ]; then \
-			$(PYTHON) -m venv --system-site-packages $(VENV_BASE)/ansible; \
+			virtualenv -p $(PYTHON) $(VENV_BASE)/ansible; \
+			$(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) $(VENV_BOOTSTRAP); \
 		fi; \
 	fi
 
+# flit is needed for offline install of certain packages, specifically ptyprocess
+# it is needed for setup, but not always recognized as a setup dependency
+# similar to pip, setuptools, and wheel, these are all needed here as a bootstrapping issues
 virtualenv_awx:
 	if [ "$(VENV_BASE)" ]; then \
 		if [ ! -d "$(VENV_BASE)" ]; then \
 			mkdir $(VENV_BASE); \
 		fi; \
 		if [ ! -d "$(VENV_BASE)/awx" ]; then \
-			$(PYTHON) -m venv --system-site-packages $(VENV_BASE)/awx; \
-			$(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) --ignore-installed docutils==0.14; \
+			virtualenv -p $(PYTHON) $(VENV_BASE)/awx; \
+			$(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) $(VENV_BOOTSTRAP) && \
+			$(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) flit; \
 		fi; \
 	fi
 
+# --ignore-install flag is not used because *.txt files should specify exact versions
 requirements_ansible: virtualenv_ansible
 	if [[ "$(PIP_OPTIONS)" == *"--no-index"* ]]; then \
-	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_local.txt | $(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) --ignore-installed -r /dev/stdin ; \
+	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_local.txt | $(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) -r /dev/stdin ; \
 	else \
-	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_git.txt | $(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) --ignore-installed -r /dev/stdin ; \
+	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_git.txt | $(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) -r /dev/stdin ; \
 	fi
 	$(VENV_BASE)/ansible/bin/pip uninstall --yes -r requirements/requirements_ansible_uninstall.txt
+	# Same effect as using --system-site-packages flag on venv creation
+	rm $(shell ls -d $(VENV_BASE)/ansible/lib/python* | head -n 1)/no-global-site-packages.txt
 
 requirements_ansible_py3: virtualenv_ansible_py3
 	if [[ "$(PIP_OPTIONS)" == *"--no-index"* ]]; then \
-	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_local.txt | $(VENV_BASE)/ansible/bin/pip3 install $(PIP_OPTIONS) --ignore-installed -r /dev/stdin ; \
+	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_local.txt | $(VENV_BASE)/ansible/bin/pip3 install $(PIP_OPTIONS) -r /dev/stdin ; \
 	else \
-	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_git.txt | $(VENV_BASE)/ansible/bin/pip3 install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) --ignore-installed -r /dev/stdin ; \
+	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_git.txt | $(VENV_BASE)/ansible/bin/pip3 install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) -r /dev/stdin ; \
 	fi
 	$(VENV_BASE)/ansible/bin/pip3 uninstall --yes -r requirements/requirements_ansible_uninstall.txt
+	# Same effect as using --system-site-packages flag on venv creation
+	rm $(shell ls -d $(VENV_BASE)/ansible/lib/python* | head -n 1)/no-global-site-packages.txt
 
 requirements_ansible_dev:
 	if [ "$(VENV_BASE)" ]; then \
@@ -182,13 +199,13 @@ requirements_ansible_dev:
 	fi
 
 # Install third-party requirements needed for AWX's environment.
+# this does not use system site packages intentionally
 requirements_awx: virtualenv_awx
 	if [[ "$(PIP_OPTIONS)" == *"--no-index"* ]]; then \
-	    cat requirements/requirements.txt requirements/requirements_local.txt | $(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) --ignore-installed -r /dev/stdin ; \
+	    cat requirements/requirements.txt requirements/requirements_local.txt | $(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) -r /dev/stdin ; \
 	else \
-	    cat requirements/requirements.txt requirements/requirements_git.txt | $(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) --ignore-installed -r /dev/stdin ; \
+	    cat requirements/requirements.txt requirements/requirements_git.txt | $(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) -r /dev/stdin ; \
 	fi
-	echo "include-system-site-packages = true" >> $(VENV_BASE)/awx/lib/python$(PYTHON_VERSION)/pyvenv.cfg
 	$(VENV_BASE)/awx/bin/pip uninstall --yes -r requirements/requirements_tower_uninstall.txt
 
 requirements_awx_dev:
@@ -364,7 +381,7 @@ check: flake8 pep8 # pyflakes pylint
 
 awx-link:
 	cp -R /tmp/awx.egg-info /awx_devel/ || true
-	sed -i "s/placeholder/$(shell git describe --long | sed 's/\./\\./g')/" /awx_devel/awx.egg-info/PKG-INFO
+	sed -i "s/placeholder/$(shell cat VERSION)/" /awx_devel/awx.egg-info/PKG-INFO
 	cp -f /tmp/awx.egg-link /venv/awx/lib/python$(PYTHON_VERSION)/site-packages/awx.egg-link
 
 TEST_DIRS ?= awx/main/tests/unit awx/main/tests/functional awx/conf/tests awx/sso/tests
@@ -381,7 +398,6 @@ test:
 prepare_collection_venv:
 	rm -rf $(COLLECTION_VENV)
 	mkdir $(COLLECTION_VENV)
-	ln -s /usr/lib/python2.7/site-packages/ansible $(COLLECTION_VENV)/ansible
 	$(VENV_BASE)/awx/bin/pip install --target=$(COLLECTION_VENV) git+https://github.com/ansible/tower-cli.git
 
 COLLECTION_TEST_DIRS ?= awx_collection/test/awx
@@ -392,7 +408,7 @@ test_collection:
 	@if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
 	fi; \
-	PYTHONPATH=$(COLLECTION_VENV):/awx_devel/awx_collection:$PYTHONPATH py.test $(COLLECTION_TEST_DIRS)
+	PYTHONPATH=$(COLLECTION_VENV):/awx_devel/awx_collection:$PYTHONPATH:/usr/lib/python3.6/site-packages py.test $(COLLECTION_TEST_DIRS)
 
 flake8_collection:
 	flake8 awx_collection/  # Different settings, in main exclude list
@@ -404,11 +420,15 @@ test_collection_sanity:
 	mkdir -p sanity/ansible_collections/awx
 	cp -Ra awx_collection sanity/ansible_collections/awx/awx  # symlinks do not work
 	cd sanity/ansible_collections/awx/awx && git init && git add . # requires both this file structure and a git repo, so there you go
-	cd sanity/ansible_collections/awx/awx && ansible-test sanity --test validate-modules
+	cd sanity/ansible_collections/awx/awx && ansible-test sanity
 
 build_collection:
 	ansible-playbook -i localhost, awx_collection/template_galaxy.yml -e collection_package=$(COLLECTION_PACKAGE) -e collection_namespace=$(COLLECTION_NAMESPACE) -e collection_version=$(VERSION)
-	ansible-galaxy collection build awx_collection --output-path=awx_collection
+	ansible-galaxy collection build awx_collection --force --output-path=awx_collection
+
+install_collection: build_collection
+	rm -rf ~/.ansible/collections/ansible_collections/awx/awx
+	ansible-galaxy collection install awx_collection/awx-awx-$(VERSION).tar.gz
 
 test_unit:
 	@if [ "$(VENV_BASE)" ]; then \
@@ -616,28 +636,34 @@ docker-auth:
 		echo "$(IMAGE_REPOSITORY_AUTH)" | docker login -u oauth2accesstoken --password-stdin $(IMAGE_REPOSITORY_BASE); \
 	fi;
 
+# This directory is bind-mounted inside of the development container and
+# needs to be pre-created for permissions to be set correctly. Otherwise,
+# Docker will create this directory as root.
+awx/projects:
+	@mkdir -p $@
+
 # Docker isolated rampart
-docker-compose-isolated:
+docker-compose-isolated: awx/projects
 	CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml -f tools/docker-isolated-override.yml up
 
 # Docker Compose Development environment
-docker-compose: docker-auth
+docker-compose: docker-auth awx/projects
 	CURRENT_UID=$(shell id -u) OS="$(shell docker info | grep 'Operating System')" TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml up --no-recreate awx
 
-docker-compose-cluster: docker-auth
+docker-compose-cluster: docker-auth awx/projects
 	CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose-cluster.yml up
 
-docker-compose-credential-plugins: docker-auth
+docker-compose-credential-plugins: docker-auth awx/projects
 	echo -e "\033[0;31mTo generate a CyberArk Conjur API key: docker exec -it tools_conjur_1 conjurctl account create quick-start\033[0m"
 	CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml -f tools/docker-credential-plugins-override.yml up --no-recreate awx
 
-docker-compose-test: docker-auth
+docker-compose-test: docker-auth awx/projects
 	cd tools && CURRENT_UID=$(shell id -u) OS="$(shell docker info | grep 'Operating System')" TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm --service-ports awx /bin/bash
 
-docker-compose-runtest:
+docker-compose-runtest: awx/projects
 	cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm --service-ports awx /start_tests.sh
 
-docker-compose-build-swagger:
+docker-compose-build-swagger: awx/projects
 	cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm --service-ports awx /start_tests.sh swagger
 
 detect-schema-change: genschema
@@ -645,7 +671,7 @@ detect-schema-change: genschema
 	# Ignore differences in whitespace with -b
 	diff -u -b reference-schema.json schema.json
 
-docker-compose-clean:
+docker-compose-clean: awx/projects
 	cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm -w /awx_devel --service-ports awx make clean
 	cd tools && TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose rm -sf
 
@@ -654,7 +680,6 @@ docker-compose-build: awx-devel-build
 # Base development image build
 awx-devel-build:
 	docker build -t ansible/awx_devel -f tools/docker-compose/Dockerfile \
-		--cache-from=$(DEV_DOCKER_TAG_BASE)/awx_devel:devel \
 		--cache-from=$(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG) .
 	docker tag ansible/awx_devel $(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG)
 	#docker push $(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG)
@@ -674,10 +699,10 @@ docker-clean:
 docker-refresh: docker-clean docker-compose
 
 # Docker Development Environment with Elastic Stack Connected
-docker-compose-elk: docker-auth
+docker-compose-elk: docker-auth awx/projects
 	CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml -f tools/elastic/docker-compose.logstash-link.yml -f tools/elastic/docker-compose.elastic-override.yml up --no-recreate
 
-docker-compose-cluster-elk: docker-auth
+docker-compose-cluster-elk: docker-auth awx/projects
 	TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose-cluster.yml -f tools/elastic/docker-compose.logstash-link-cluster.yml -f tools/elastic/docker-compose.elastic-override.yml up --no-recreate
 
 prometheus:

VERSION

@@ -1 +1 @@
-9.0.0
+9.1.1

awx/__init__.py

@@ -24,31 +24,18 @@ except ImportError: # pragma: no cover
 import hashlib
 
 try:
-    import django
-    from django.db.backends.base import schema
-    from django.db.backends.utils import names_digest
+    import django  # noqa: F401
     HAS_DJANGO = True
 except ImportError:
     HAS_DJANGO = False
+else:
+    from django.db.backends.base import schema
+    from django.db.backends.utils import names_digest
 
 
 if HAS_DJANGO is True:
-    # This line exists to make sure we don't regress on FIPS support if we
-    # upgrade Django; if you're upgrading Django and see this error,
-    # update the version check below, and confirm that FIPS still works.
-    # If operating in a FIPS environment, `hashlib.md5()` will raise a `ValueError`,
-    # but will support the `usedforsecurity` keyword on RHEL and Centos systems.
-
-    # Keep an eye on https://code.djangoproject.com/ticket/28401
-    target_version = '2.2.4'
-    if django.__version__ != target_version:
-        raise RuntimeError(
-            "Django version other than {target} detected: {current}. "
-            "Overriding `names_digest` is known to work for Django {target} "
-            "and may not work in other Django versions.".format(target=target_version,
-                                                                current=django.__version__)
-        )
 
+    # See upgrade blocker note in requirements/README.md
     try:
         names_digest('foo', 'bar', 'baz', length=8)
     except ValueError:
@@ -86,7 +73,14 @@ def oauth2_getattribute(self, attr):
     # Custom method to override
     # oauth2_provider.settings.OAuth2ProviderSettings.__getattribute__
     from django.conf import settings
-    val = settings.OAUTH2_PROVIDER.get(attr)
+    val = None
+    if 'migrate' not in sys.argv:
+        # certain Django OAuth Toolkit migrations actually reference
+        # setting lookups for references to model classes (e.g.,
+        # oauth2_settings.REFRESH_TOKEN_MODEL)
+        # If we're doing an OAuth2 setting lookup *while running* a migration,
+        # don't do our usual "Configure Tower in Tower" database setting lookup
+        val = settings.OAUTH2_PROVIDER.get(attr)
     if val is None:
         val = object.__getattribute__(self, attr)
     return val

awx/api/conf.py

@@ -62,3 +62,15 @@ register(
     category=_('Authentication'),
     category_slug='authentication',
 )
+register(
+    'LOGIN_REDIRECT_OVERRIDE',
+    field_class=fields.CharField,
+    allow_blank=True,
+    required=False,
+    default='',
+    label=_('Login redirect override URL'),
+    help_text=_('URL to which unauthorized users will be redirected to log in. '
+                'If blank, users will be sent to the Tower login page.'),
+    category=_('Authentication'),
+    category_slug='authentication',
+)

awx/api/serializers.py

@@ -141,6 +141,7 @@ SUMMARIZABLE_FK_FIELDS = {
     'target_credential': DEFAULT_SUMMARY_FIELDS + ('kind', 'cloud', 'credential_type_id'),
     'webhook_credential': DEFAULT_SUMMARY_FIELDS,
     'approved_or_denied_by': ('id', 'username', 'first_name', 'last_name'),
+    'credential_type': DEFAULT_SUMMARY_FIELDS,
 }
 
 
@@ -1472,7 +1473,7 @@ class ProjectUpdateSerializer(UnifiedJobSerializer, ProjectOptionsSerializer):
 
     class Meta:
         model = ProjectUpdate
-        fields = ('*', 'project', 'job_type', '-controller_node')
+        fields = ('*', 'project', 'job_type', 'job_tags', '-controller_node')
 
     def get_related(self, obj):
         res = super(ProjectUpdateSerializer, self).get_related(obj)
@@ -2456,12 +2457,18 @@ class CredentialTypeSerializer(BaseSerializer):
             raise PermissionDenied(
                 detail=_("Modifications not allowed for managed credential types")
             )
+
+        old_inputs = {}
+        if self.instance:
+            old_inputs = copy.deepcopy(self.instance.inputs)
+
+        ret = super(CredentialTypeSerializer, self).validate(attrs)
+
         if self.instance and self.instance.credentials.exists():
-            if 'inputs' in attrs and attrs['inputs'] != self.instance.inputs:
+            if 'inputs' in attrs and old_inputs != self.instance.inputs:
                 raise PermissionDenied(
                     detail= _("Modifications to inputs are not allowed for credential types that are in use")
                 )
-        ret = super(CredentialTypeSerializer, self).validate(attrs)
 
         if 'kind' in attrs and attrs['kind'] not in ('cloud', 'net'):
             raise serializers.ValidationError({
@@ -3867,26 +3874,6 @@ class JobEventSerializer(BaseSerializer):
         return data
 
 
-class JobEventWebSocketSerializer(JobEventSerializer):
-    created = serializers.SerializerMethodField()
-    modified = serializers.SerializerMethodField()
-    event_name = serializers.CharField(source='event')
-    group_name = serializers.SerializerMethodField()
-
-    class Meta:
-        model = JobEvent
-        fields = ('*', 'event_name', 'group_name',)
-
-    def get_created(self, obj):
-        return obj.created.isoformat()
-
-    def get_modified(self, obj):
-        return obj.modified.isoformat()
-
-    def get_group_name(self, obj):
-        return 'job_events'
-
-
 class ProjectUpdateEventSerializer(JobEventSerializer):
     stdout = serializers.SerializerMethodField()
     event_data = serializers.SerializerMethodField()
@@ -3918,26 +3905,6 @@ class ProjectUpdateEventSerializer(JobEventSerializer):
             return {}
 
 
-class ProjectUpdateEventWebSocketSerializer(ProjectUpdateEventSerializer):
-    created = serializers.SerializerMethodField()
-    modified = serializers.SerializerMethodField()
-    event_name = serializers.CharField(source='event')
-    group_name = serializers.SerializerMethodField()
-
-    class Meta:
-        model = ProjectUpdateEvent
-        fields = ('*', 'event_name', 'group_name',)
-
-    def get_created(self, obj):
-        return obj.created.isoformat()
-
-    def get_modified(self, obj):
-        return obj.modified.isoformat()
-
-    def get_group_name(self, obj):
-        return 'project_update_events'
-
-
 class AdHocCommandEventSerializer(BaseSerializer):
 
     event_display = serializers.CharField(source='get_event_display', read_only=True)
@@ -3969,26 +3936,6 @@ class AdHocCommandEventSerializer(BaseSerializer):
         return data
 
 
-class AdHocCommandEventWebSocketSerializer(AdHocCommandEventSerializer):
-    created = serializers.SerializerMethodField()
-    modified = serializers.SerializerMethodField()
-    event_name = serializers.CharField(source='event')
-    group_name = serializers.SerializerMethodField()
-
-    class Meta:
-        model = AdHocCommandEvent
-        fields = ('*', 'event_name', 'group_name',)
-
-    def get_created(self, obj):
-        return obj.created.isoformat()
-
-    def get_modified(self, obj):
-        return obj.modified.isoformat()
-
-    def get_group_name(self, obj):
-        return 'ad_hoc_command_events'
-
-
 class InventoryUpdateEventSerializer(AdHocCommandEventSerializer):
 
     class Meta:
@@ -4004,26 +3951,6 @@ class InventoryUpdateEventSerializer(AdHocCommandEventSerializer):
         return res
 
 
-class InventoryUpdateEventWebSocketSerializer(InventoryUpdateEventSerializer):
-    created = serializers.SerializerMethodField()
-    modified = serializers.SerializerMethodField()
-    event_name = serializers.CharField(source='event')
-    group_name = serializers.SerializerMethodField()
-
-    class Meta:
-        model = InventoryUpdateEvent
-        fields = ('*', 'event_name', 'group_name',)
-
-    def get_created(self, obj):
-        return obj.created.isoformat()
-
-    def get_modified(self, obj):
-        return obj.modified.isoformat()
-
-    def get_group_name(self, obj):
-        return 'inventory_update_events'
-
-
 class SystemJobEventSerializer(AdHocCommandEventSerializer):
 
     class Meta:
@@ -4039,26 +3966,6 @@ class SystemJobEventSerializer(AdHocCommandEventSerializer):
         return res
 
 
-class SystemJobEventWebSocketSerializer(SystemJobEventSerializer):
-    created = serializers.SerializerMethodField()
-    modified = serializers.SerializerMethodField()
-    event_name = serializers.CharField(source='event')
-    group_name = serializers.SerializerMethodField()
-
-    class Meta:
-        model = SystemJobEvent
-        fields = ('*', 'event_name', 'group_name',)
-
-    def get_created(self, obj):
-        return obj.created.isoformat()
-
-    def get_modified(self, obj):
-        return obj.modified.isoformat()
-
-    def get_group_name(self, obj):
-        return 'system_job_events'
-
-
 class JobLaunchSerializer(BaseSerializer):
 
     # Representational fields
@@ -4658,6 +4565,10 @@ class ScheduleSerializer(LaunchConfigurationBaseSerializer, SchedulePreviewSeria
 
     def get_summary_fields(self, obj):
         summary_fields = super(ScheduleSerializer, self).get_summary_fields(obj)
+
+        if isinstance(obj.unified_job_template, SystemJobTemplate):
+            summary_fields['unified_job_template']['job_type'] = obj.unified_job_template.job_type
+
         if 'inventory' in summary_fields:
             return summary_fields
 

awx/api/templates/api/inventory_source_cancel.md

@@ -1,7 +1,7 @@
 # Cancel Inventory Update
 
 Make a GET request to this resource to determine if the inventory update can be
-cancelled.  The response will include the following field:
+canceled.  The response will include the following field:
 
 * `can_cancel`: Indicates whether this update can be canceled (boolean,
   read-only)

awx/api/templates/api/job_cancel.md

@@ -1,7 +1,7 @@
 {% ifmeth GET %}
-# Determine if a Job can be cancelled
+# Determine if a Job can be canceled
 
-Make a GET request to this resource to determine if the job can be cancelled.
+Make a GET request to this resource to determine if the job can be canceled.
 The response will include the following field:
 
 * `can_cancel`: Indicates whether this job can be canceled (boolean, read-only)

awx/api/templates/api/project_update_cancel.md

@@ -1,7 +1,7 @@
 # Cancel Project Update
 
 Make a GET request to this resource to determine if the project update can be
-cancelled.  The response will include the following field:
+canceled.  The response will include the following field:
 
 * `can_cancel`: Indicates whether this update can be canceled (boolean,
   read-only)

awx/api/views/__init__.py

@@ -72,12 +72,11 @@ from awx.api.generics import (
     SubListDestroyAPIView
 )
 from awx.api.versioning import reverse
-from awx.conf.license import get_license
 from awx.main import models
 from awx.main.utils import (
     camelcase_to_underscore,
     extract_ansible_vars,
-    get_awx_version,
+    get_awx_http_client_headers,
     get_object_or_400,
     getattrd,
     get_pk_from_dict,
@@ -1386,6 +1385,7 @@ class CredentialExternalTest(SubDetailAPIView):
 
     model = models.Credential
     serializer_class = serializers.EmptySerializer
+    obj_permission_type = 'use'
 
     def post(self, request, *args, **kwargs):
         obj = self.get_object()
@@ -1643,18 +1643,6 @@ class HostInsights(GenericAPIView):
 
         return session
 
-    def _get_headers(self):
-        license = get_license(show_key=False).get('license_type', 'UNLICENSED')
-        headers = {
-            'Content-Type': 'application/json',
-            'User-Agent': '{} {} ({})'.format(
-                'AWX' if license == 'open' else 'Red Hat Ansible Tower',
-                get_awx_version(),
-                license
-            )
-        }
-
-        return headers
 
     def _get_platform_info(self, host, session, headers):
         url = '{}/api/inventory/v1/hosts?insights_id={}'.format(
@@ -1721,7 +1709,7 @@ class HostInsights(GenericAPIView):
         username = cred.get_input('username', default='')
         password = cred.get_input('password', default='')
         session = self._get_session(username, password)
-        headers = self._get_headers()
+        headers = get_awx_http_client_headers()
 
         data = self._get_insights(host, session, headers)
         return Response(data, status=status.HTTP_200_OK)
@@ -2561,7 +2549,7 @@ class JobTemplateSurveySpec(GenericAPIView):
                 if not isinstance(val, allow_types):
                     return Response(dict(error=_("'{field_name}' in survey question {idx} expected to be {type_label}.").format(
                         field_name=field_name, type_label=type_label, **context
-                    )))
+                    )), status=status.HTTP_400_BAD_REQUEST)
             if survey_item['variable'] in variable_set:
                 return Response(dict(error=_("'variable' '%(item)s' duplicated in survey question %(survey)s.") % {
                     'item': survey_item['variable'], 'survey': str(idx)}), status=status.HTTP_400_BAD_REQUEST)
@@ -2576,7 +2564,7 @@ class JobTemplateSurveySpec(GenericAPIView):
                     "'{survey_item[type]}' in survey question {idx} is not one of '{allowed_types}' allowed question types."
                 ).format(
                     allowed_types=', '.join(JobTemplateSurveySpec.ALLOWED_TYPES.keys()), **context
-                )))
+                )), status=status.HTTP_400_BAD_REQUEST)
             if 'default' in survey_item and survey_item['default'] != '':
                 if not isinstance(survey_item['default'], JobTemplateSurveySpec.ALLOWED_TYPES[qtype]):
                     type_label = 'string'
@@ -2594,7 +2582,7 @@ class JobTemplateSurveySpec(GenericAPIView):
                     if survey_item[key] is not None and (not isinstance(survey_item[key], int)):
                         return Response(dict(error=_(
                             "The {min_or_max} limit in survey question {idx} expected to be integer."
-                        ).format(min_or_max=key, **context)))
+                        ).format(min_or_max=key, **context)), status=status.HTTP_400_BAD_REQUEST)
             # if it's a multiselect or multiple choice, it must have coices listed
             # choices and defualts must come in as strings seperated by /n characters.
             if qtype == 'multiselect' or qtype == 'multiplechoice':
@@ -2604,7 +2592,7 @@ class JobTemplateSurveySpec(GenericAPIView):
                 else:
                     return Response(dict(error=_(
                         "Survey question {idx} of type {survey_item[type]} must specify choices.".format(**context)
-                    )))
+                    )), status=status.HTTP_400_BAD_REQUEST)
                 # If there is a default string split it out removing extra /n characters.
                 # Note: There can still be extra newline characters added in the API, these are sanitized out using .strip()
                 if 'default' in survey_item:
@@ -2618,11 +2606,11 @@ class JobTemplateSurveySpec(GenericAPIView):
                         if len(list_of_defaults) > 1:
                             return Response(dict(error=_(
                                 "Multiple Choice (Single Select) can only have one default value.".format(**context)
-                            )))
+                            )), status=status.HTTP_400_BAD_REQUEST)
                     if any(item not in survey_item['choices'] for item in list_of_defaults):
                         return Response(dict(error=_(
                             "Default choice must be answered from the choices listed.".format(**context)
-                        )))
+                        )), status=status.HTTP_400_BAD_REQUEST)
 
             # Process encryption substitution
             if ("default" in survey_item and isinstance(survey_item['default'], str) and

awx/api/views/root.py

@@ -60,6 +60,7 @@ class ApiRootView(APIView):
         data['oauth2'] = drf_reverse('api:oauth_authorization_root_view')
         data['custom_logo'] = settings.CUSTOM_LOGO
         data['custom_login_info'] = settings.CUSTOM_LOGIN_INFO
+        data['login_redirect_override'] = settings.LOGIN_REDIRECT_OVERRIDE
         return Response(data)
 
 

awx/conf/license.py

@@ -1,13 +1,12 @@
 # Copyright (c) 2016 Ansible, Inc.
 # All Rights Reserved.
 
-# Tower
-from awx.main.utils.common import get_licenser
 
 __all__ = ['get_license']
 
 
 def _get_validated_license_data():
+    from awx.main.utils.common import get_licenser
     return get_licenser().validate()
 
 

awx/conf/migrations/0004_v320_reencrypt.py

@@ -2,7 +2,6 @@
 from __future__ import unicode_literals
 
 from django.db import migrations
-from awx.conf.migrations import _reencrypt
 
 
 class Migration(migrations.Migration):
@@ -12,5 +11,8 @@ class Migration(migrations.Migration):
     ]
 
     operations = [
-        migrations.RunPython(_reencrypt.replace_aesecb_fernet),
+        # This list is intentionally empty.
+        # Tower 3.2 included several data migrations that are no longer
+        # necessary (this list is now empty because Tower 3.2 is past EOL and
+        # cannot be directly upgraded to modern versions of Tower)
     ]

awx/conf/migrations/_reencrypt.py

@@ -1,30 +1,13 @@
 import base64
 import hashlib
 
-from django.utils.encoding import smart_str
-
 from cryptography.hazmat.backends import default_backend
 from cryptography.hazmat.primitives.ciphers import Cipher
 from cryptography.hazmat.primitives.ciphers.algorithms import AES
 from cryptography.hazmat.primitives.ciphers.modes import ECB
 
-from awx.conf import settings_registry
 
-
-__all__ = ['replace_aesecb_fernet', 'get_encryption_key', 'encrypt_field',
-           'decrypt_value', 'decrypt_value', 'should_decrypt_field']
-
-
-def replace_aesecb_fernet(apps, schema_editor):
-    from awx.main.utils.encryption import encrypt_field
-    Setting = apps.get_model('conf', 'Setting')
-
-    for setting in Setting.objects.filter().order_by('pk'):
-        if settings_registry.is_setting_encrypted(setting.key):
-            if should_decrypt_field(setting.value):
-                setting.value = decrypt_field(setting, 'value')
-                setting.value = encrypt_field(setting, 'value')
-            setting.save()
+__all__ = ['get_encryption_key', 'decrypt_field']
 
 
 def get_encryption_key(field_name, pk=None):
@@ -76,38 +59,3 @@ def decrypt_field(instance, field_name, subfield=None):
     key = get_encryption_key(field_name, getattr(instance, 'pk', None))
 
     return decrypt_value(key, value)
-
-
-def encrypt_field(instance, field_name, ask=False, subfield=None, skip_utf8=False):
-    '''
-    Return content of the given instance and field name encrypted.
-    '''
-    value = getattr(instance, field_name)
-    if isinstance(value, dict) and subfield is not None:
-        value = value[subfield]
-    if not value or value.startswith('$encrypted$') or (ask and value == 'ASK'):
-        return value
-    if skip_utf8:
-        utf8 = False
-    else:
-        utf8 = type(value) == str
-    value = smart_str(value)
-    key = get_encryption_key(field_name, getattr(instance, 'pk', None))
-    encryptor = Cipher(AES(key), ECB(), default_backend()).encryptor()
-    block_size = 16
-    while len(value) % block_size != 0:
-        value += '\x00'
-    encrypted = encryptor.update(value) + encryptor.finalize()
-    b64data = base64.b64encode(encrypted)
-    tokens = ['$encrypted', 'AES', b64data]
-    if utf8:
-        # If the value to encrypt is utf-8, we need to add a marker so we
-        # know to decode the data when it's decrypted later
-        tokens.insert(1, 'UTF8')
-    return '$'.join(tokens)
-
-
-def should_decrypt_field(value):
-    if hasattr(value, 'startswith'):
-        return value.startswith('$encrypted$') and '$AESCBC$' not in value
-    return False

awx/conf/settings.py

@@ -28,6 +28,8 @@ from awx.conf import settings_registry
 from awx.conf.models import Setting
 from awx.conf.migrations._reencrypt import decrypt_field as old_decrypt_field
 
+import cachetools
+
 # FIXME: Gracefully handle when settings are accessed before the database is
 # ready (or during migrations).
 
@@ -136,6 +138,14 @@ def filter_sensitive(registry, key, value):
     return value
 
 
+# settings.__getattr__ is called *constantly*, and the LOG_AGGREGATOR_ ones are
+# so ubiquitous when external logging is enabled that they should kept in memory
+# with a short TTL to avoid even having to contact memcached
+# the primary use case for this optimization is the callback receiver
+# when external logging is enabled
+LOGGING_SETTINGS_CACHE = cachetools.TTLCache(maxsize=50, ttl=1)
+
+
 class EncryptedCacheProxy(object):
 
     def __init__(self, cache, registry, encrypter=None, decrypter=None):
@@ -437,11 +447,17 @@ class SettingsWrapper(UserSettingsHolder):
         return self._get_default('SETTINGS_MODULE')
 
     def __getattr__(self, name):
+        if name.startswith('LOG_AGGREGATOR_'):
+            cached = LOGGING_SETTINGS_CACHE.get(name)
+            if cached:
+                return cached
         value = empty
         if name in self.all_supported_settings:
             with _ctit_db_wrapper(trans_safe=True):
                 value = self._get_local(name)
         if value is not empty:
+            if name.startswith('LOG_AGGREGATOR_'):
+                LOGGING_SETTINGS_CACHE[name] = value
             return value
         value = self._get_default(name)
         # sometimes users specify RabbitMQ passwords that contain

awx/main/analytics/collectors.py

@@ -52,7 +52,7 @@ def config(since):
         'tower_version': get_awx_version(),
         'ansible_version': get_ansible_version(),
         'license_type': license_info.get('license_type', 'UNLICENSED'),
-        'free_instances': license_info.get('free instances', 0),
+        'free_instances': license_info.get('free_instances', 0),
         'license_expiry': license_info.get('time_remaining', 0),
         'pendo_tracking': settings.PENDO_TRACKING_STATE,
         'authentication_backends': settings.AUTHENTICATION_BACKENDS,
@@ -166,6 +166,8 @@ def instance_info(since, include_hostnames=False):
     instances = models.Instance.objects.values_list('hostname').values(
         'uuid', 'version', 'capacity', 'cpu', 'memory', 'managed_by_policy', 'hostname', 'last_isolated_check', 'enabled')
     for instance in instances:
+        consumed_capacity = sum(x.task_impact for x in models.UnifiedJob.objects.filter(execution_node=instance['hostname'],
+                                status__in=('running', 'waiting')))
         instance_info = {
             'uuid': instance['uuid'],
             'version': instance['version'],
@@ -174,7 +176,9 @@ def instance_info(since, include_hostnames=False):
             'memory': instance['memory'],
             'managed_by_policy': instance['managed_by_policy'],
             'last_isolated_check': _get_isolated_datetime(instance['last_isolated_check']),
-            'enabled': instance['enabled']
+            'enabled': instance['enabled'],
+            'consumed_capacity': consumed_capacity,
+            'remaining_capacity': instance['capacity'] - consumed_capacity
         }
         if include_hostnames is True:
             instance_info['hostname'] = instance['hostname']

awx/main/analytics/core.py

@@ -15,6 +15,7 @@ from awx.conf.license import get_license
 from awx.main.models import Job
 from awx.main.access import access_registry
 from awx.main.models.ha import TowerAnalyticsState
+from awx.main.utils import get_awx_http_client_headers
 
 
 __all__ = ['register', 'gather', 'ship', 'table_version']
@@ -165,11 +166,15 @@ def ship(path):
             return logger.error('REDHAT_PASSWORD is not set')
         with open(path, 'rb') as f:
             files = {'file': (os.path.basename(path), f, settings.INSIGHTS_AGENT_MIME)}
-            response = requests.post(url, 
-                                     files=files,
-                                     verify="/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem",
-                                     auth=(rh_user, rh_password),
-                                     timeout=(31, 31))
+            s = requests.Session()
+            s.headers = get_awx_http_client_headers()
+            s.headers.pop('Content-Type')
+            response = s.post(url, 
+                              files=files,
+                              verify="/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem",
+                              auth=(rh_user, rh_password),
+                              headers=s.headers,
+                              timeout=(31, 31))
             if response.status_code != 202:
                 return logger.exception('Upload failed with status {}, {}'.format(response.status_code,
                                                                                   response.text))

awx/main/analytics/metrics.py

@@ -46,6 +46,8 @@ INSTANCE_MEMORY = Gauge('awx_instance_memory', 'RAM (Kb) on each node in a Tower
 INSTANCE_INFO = Info('awx_instance', 'Info about each node in a Tower system', ['hostname', 'instance_uuid',])
 INSTANCE_LAUNCH_TYPE = Gauge('awx_instance_launch_type_total', 'Type of Job launched', ['node', 'launch_type',])
 INSTANCE_STATUS = Gauge('awx_instance_status_total', 'Status of Job launched', ['node', 'status',])
+INSTANCE_CONSUMED_CAPACITY = Gauge('awx_instance_consumed_capacity', 'Consumed capacity of each node in a Tower system', ['hostname', 'instance_uuid',])
+INSTANCE_REMAINING_CAPACITY = Gauge('awx_instance_remaining_capacity', 'Remaining capacity of each node in a Tower system', ['hostname', 'instance_uuid',])
 
 LICENSE_INSTANCE_TOTAL = Gauge('awx_license_instance_total', 'Total number of managed hosts provided by your license')
 LICENSE_INSTANCE_FREE = Gauge('awx_license_instance_free', 'Number of remaining managed hosts provided by your license')
@@ -104,6 +106,8 @@ def metrics():
         INSTANCE_CAPACITY.labels(hostname=hostname, instance_uuid=uuid).set(instance_data[uuid]['capacity'])
         INSTANCE_CPU.labels(hostname=hostname, instance_uuid=uuid).set(instance_data[uuid]['cpu'])
         INSTANCE_MEMORY.labels(hostname=hostname, instance_uuid=uuid).set(instance_data[uuid]['memory'])
+        INSTANCE_CONSUMED_CAPACITY.labels(hostname=hostname, instance_uuid=uuid).set(instance_data[uuid]['consumed_capacity'])
+        INSTANCE_REMAINING_CAPACITY.labels(hostname=hostname, instance_uuid=uuid).set(instance_data[uuid]['remaining_capacity'])
         INSTANCE_INFO.labels(hostname=hostname, instance_uuid=uuid).info({
             'enabled': str(instance_data[uuid]['enabled']),
             'last_isolated_check': getattr(instance_data[uuid], 'last_isolated_check', 'None'),

awx/main/conf.py

@@ -514,6 +514,17 @@ register(
     category_slug='jobs'
 )
 
+register(
+    'GALAXY_IGNORE_CERTS',
+    field_class=fields.BooleanField,
+    default=False,
+    label=_('Ignore Ansible Galaxy SSL Certificate Verification'),
+    help_text=_('If set to true, certificate validation will not be done when'
+                'installing content from any Galaxy server.'),
+    category=_('Jobs'),
+    category_slug='jobs'
+)
+
 register(
     'STDOUT_MAX_BYTES_DISPLAY',
     field_class=fields.IntegerField,

awx/main/dispatch/pool.py

@@ -123,8 +123,16 @@ class PoolWorker(object):
         # if any tasks were finished, removed them from the managed tasks for
         # this worker
         for uuid in finished:
-            self.messages_finished += 1
-            del self.managed_tasks[uuid]
+            try:
+                del self.managed_tasks[uuid]
+                self.messages_finished += 1
+            except KeyError:
+                # ansible _sometimes_ appears to send events w/ duplicate UUIDs;
+                # UUIDs for ansible events are *not* actually globally unique
+                # when this occurs, it's _fine_ to ignore this KeyError because
+                # the purpose of self.managed_tasks is to just track internal
+                # state of which events are *currently* being processed.
+                logger.warn('Event UUID {} appears to be have been duplicated.'.format(uuid))
 
     @property
     def current_task(self):
@@ -269,7 +277,7 @@ class WorkerPool(object):
                 logger.warn("could not write to queue %s" % preferred_queue)
                 logger.warn("detail: {}".format(tb))
             write_attempt_order.append(preferred_queue)
-        logger.warn("could not write payload to any queue, attempted order: {}".format(write_attempt_order))
+        logger.error("could not write payload to any queue, attempted order: {}".format(write_attempt_order))
         return None
 
     def stop(self, signum):

awx/main/dispatch/worker/base.py

@@ -119,6 +119,9 @@ class AWXConsumer(ConsumerMixin):
 
 class BaseWorker(object):
 
+    def read(self, queue):
+        return queue.get(block=True, timeout=1)
+
     def work_loop(self, queue, finished, idx, *args):
         ppid = os.getppid()
         signal_handler = WorkerSignalHandler()
@@ -128,7 +131,7 @@ class BaseWorker(object):
             if os.getppid() != ppid:
                 break
             try:
-                body = queue.get(block=True, timeout=1)
+                body = self.read(queue)
                 if body == 'QUIT':
                     break
             except QueueEmpty:

awx/main/dispatch/worker/callback.py

@@ -1,19 +1,26 @@
 import logging
 import time
 import traceback
+from queue import Empty as QueueEmpty
 
+from django.utils.timezone import now as tz_now
 from django.conf import settings
 from django.db import DatabaseError, OperationalError, connection as django_connection
-from django.db.utils import InterfaceError, InternalError
+from django.db.utils import InterfaceError, InternalError, IntegrityError
 
 from awx.main.consumers import emit_channel_notification
 from awx.main.models import (JobEvent, AdHocCommandEvent, ProjectUpdateEvent,
                              InventoryUpdateEvent, SystemJobEvent, UnifiedJob)
+from awx.main.models.events import emit_event_detail
 
 from .base import BaseWorker
 
 logger = logging.getLogger('awx.main.commands.run_callback_receiver')
 
+# the number of seconds to buffer events in memory before flushing
+# using JobEvent.objects.bulk_create()
+BUFFER_SECONDS = .1
+
 
 class CallbackBrokerWorker(BaseWorker):
     '''
@@ -26,89 +33,112 @@ class CallbackBrokerWorker(BaseWorker):
 
     MAX_RETRIES = 2
 
+    def __init__(self):
+        self.buff = {}
+
+    def read(self, queue):
+        try:
+            return queue.get(block=True, timeout=BUFFER_SECONDS)
+        except QueueEmpty:
+            return {'event': 'FLUSH'}
+
+    def flush(self, force=False):
+        now = tz_now()
+        if (
+            force or
+            any([len(events) >= 1000 for events in self.buff.values()])
+        ):
+            for cls, events in self.buff.items():
+                logger.debug(f'{cls.__name__}.objects.bulk_create({len(events)})')
+                for e in events:
+                    if not e.created:
+                        e.created = now
+                    e.modified = now
+                try:
+                    cls.objects.bulk_create(events)
+                except Exception as exc:
+                    # if an exception occurs, we should re-attempt to save the
+                    # events one-by-one, because something in the list is
+                    # broken/stale (e.g., an IntegrityError on a specific event)
+                    for e in events:
+                        try:
+                            if (
+                                isinstance(exc, IntegrityError),
+                                getattr(e, 'host_id', '')
+                            ):
+                                # this is one potential IntegrityError we can
+                                # work around - if the host disappears before
+                                # the event can be processed
+                                e.host_id = None
+                            e.save()
+                        except Exception:
+                            logger.exception('Database Error Saving Job Event')
+                for e in events:
+                    emit_event_detail(e)
+            self.buff = {}
+
     def perform_work(self, body):
         try:
-            event_map = {
-                'job_id': JobEvent,
-                'ad_hoc_command_id': AdHocCommandEvent,
-                'project_update_id': ProjectUpdateEvent,
-                'inventory_update_id': InventoryUpdateEvent,
-                'system_job_id': SystemJobEvent,
-            }
+            flush = body.get('event') == 'FLUSH'
+            if not flush:
+                event_map = {
+                    'job_id': JobEvent,
+                    'ad_hoc_command_id': AdHocCommandEvent,
+                    'project_update_id': ProjectUpdateEvent,
+                    'inventory_update_id': InventoryUpdateEvent,
+                    'system_job_id': SystemJobEvent,
+                }
 
-            if not any([key in body for key in event_map]):
-                raise Exception('Payload does not have a job identifier')
-
-            def _save_event_data():
+                job_identifier = 'unknown job'
                 for key, cls in event_map.items():
                     if key in body:
-                        cls.create_from_data(**body)
+                        job_identifier = body[key]
+                        break
 
-            job_identifier = 'unknown job'
-            job_key = 'unknown'
-            for key in event_map.keys():
-                if key in body:
-                    job_identifier = body[key]
-                    job_key = key
-                    break
-
-            if settings.DEBUG:
-                from pygments import highlight
-                from pygments.lexers import PythonLexer
-                from pygments.formatters import Terminal256Formatter
-                from pprint import pformat
                 if body.get('event') == 'EOF':
-                    event_thing = 'EOF event'
-                else:
-                    event_thing = 'event {}'.format(body.get('counter', 'unknown'))
-                logger.info('Callback worker received {} for {} {}'.format(
-                    event_thing, job_key[:-len('_id')], job_identifier
-                ))
-                logger.debug('Body: {}'.format(
-                    highlight(pformat(body, width=160), PythonLexer(), Terminal256Formatter(style='friendly'))
-                )[:1024 * 4])
+                    try:
+                        final_counter = body.get('final_counter', 0)
+                        logger.info('Event processing is finished for Job {}, sending notifications'.format(job_identifier))
+                        # EOF events are sent when stdout for the running task is
+                        # closed. don't actually persist them to the database; we
+                        # just use them to report `summary` websocket events as an
+                        # approximation for when a job is "done"
+                        emit_channel_notification(
+                            'jobs-summary',
+                            dict(group_name='jobs', unified_job_id=job_identifier, final_counter=final_counter)
+                        )
+                        # Additionally, when we've processed all events, we should
+                        # have all the data we need to send out success/failure
+                        # notification templates
+                        uj = UnifiedJob.objects.get(pk=job_identifier)
+                        if hasattr(uj, 'send_notification_templates'):
+                            retries = 0
+                            while retries < 5:
+                                if uj.finished:
+                                    uj.send_notification_templates('succeeded' if uj.status == 'successful' else 'failed')
+                                    break
+                                else:
+                                    # wait a few seconds to avoid a race where the
+                                    # events are persisted _before_ the UJ.status
+                                    # changes from running -> successful
+                                    retries += 1
+                                    time.sleep(1)
+                                    uj = UnifiedJob.objects.get(pk=job_identifier)
+                    except Exception:
+                        logger.exception('Worker failed to emit notifications: Job {}'.format(job_identifier))
+                    return
 
-            if body.get('event') == 'EOF':
-                try:
-                    final_counter = body.get('final_counter', 0)
-                    logger.info('Event processing is finished for Job {}, sending notifications'.format(job_identifier))
-                    # EOF events are sent when stdout for the running task is
-                    # closed. don't actually persist them to the database; we
-                    # just use them to report `summary` websocket events as an
-                    # approximation for when a job is "done"
-                    emit_channel_notification(
-                        'jobs-summary',
-                        dict(group_name='jobs', unified_job_id=job_identifier, final_counter=final_counter)
-                    )
-                    # Additionally, when we've processed all events, we should
-                    # have all the data we need to send out success/failure
-                    # notification templates
-                    uj = UnifiedJob.objects.get(pk=job_identifier)
-                    if hasattr(uj, 'send_notification_templates'):
-                        retries = 0
-                        while retries < 5:
-                            if uj.finished:
-                                uj.send_notification_templates('succeeded' if uj.status == 'successful' else 'failed')
-                                break
-                            else:
-                                # wait a few seconds to avoid a race where the
-                                # events are persisted _before_ the UJ.status
-                                # changes from running -> successful
-                                retries += 1
-                                time.sleep(1)
-                                uj = UnifiedJob.objects.get(pk=job_identifier)
-                except Exception:
-                    logger.exception('Worker failed to emit notifications: Job {}'.format(job_identifier))
-                return
+                event = cls.create_from_data(**body)
+                self.buff.setdefault(cls, []).append(event)
 
             retries = 0
             while retries <= self.MAX_RETRIES:
                 try:
-                    _save_event_data()
+                    self.flush(force=flush)
                     break
                 except (OperationalError, InterfaceError, InternalError):
                     if retries >= self.MAX_RETRIES:
-                        logger.exception('Worker could not re-establish database connectivity, giving up on event for Job {}'.format(job_identifier))
+                        logger.exception('Worker could not re-establish database connectivity, giving up on one or more events.')
                         return
                     delay = 60 * retries
                     logger.exception('Database Error Saving Job Event, retry #{i} in {delay} seconds:'.format(
@@ -119,7 +149,7 @@ class CallbackBrokerWorker(BaseWorker):
                     time.sleep(delay)
                     retries += 1
                 except DatabaseError:
-                    logger.exception('Database Error Saving Job Event for Job {}'.format(job_identifier))
+                    logger.exception('Database Error Saving Job Event')
                     break
         except Exception as exc:
             tb = traceback.format_exc()

awx/main/isolated/manager.py

@@ -32,15 +32,14 @@ def set_pythonpath(venv_libdir, env):
 
 class IsolatedManager(object):
 
-    def __init__(self, cancelled_callback=None, check_callback=None, pod_manager=None):
+    def __init__(self, canceled_callback=None, check_callback=None, pod_manager=None):
         """
-        :param cancelled_callback:  a callable - which returns `True` or `False`
+        :param canceled_callback:  a callable - which returns `True` or `False`
                                     - signifying if the job has been prematurely
-                                      cancelled
+                                      canceled
         """
-        self.cancelled_callback = cancelled_callback
+        self.canceled_callback = canceled_callback
         self.check_callback = check_callback
-        self.idle_timeout = max(60, 2 * settings.AWX_ISOLATED_CONNECTION_TIMEOUT)
         self.started_at = None
         self.captured_command_artifact = False
         self.instance = None
@@ -106,9 +105,8 @@ class IsolatedManager(object):
             'envvars': env,
             'finished_callback': finished_callback,
             'verbosity': verbosity,
-            'cancel_callback': self.cancelled_callback,
+            'cancel_callback': self.canceled_callback,
             'settings': {
-                'idle_timeout': self.idle_timeout,
                 'job_timeout': settings.AWX_ISOLATED_LAUNCH_TIMEOUT,
                 'pexpect_timeout': getattr(settings, 'PEXPECT_TIMEOUT', 5),
                 'suppress_ansible_output': True,
@@ -118,7 +116,7 @@ class IsolatedManager(object):
     def path_to(self, *args):
         return os.path.join(self.private_data_dir, *args)
 
-    def run_management_playbook(self, playbook, private_data_dir, **kw):
+    def run_management_playbook(self, playbook, private_data_dir, idle_timeout=None, **kw):
         iso_dir = tempfile.mkdtemp(
             prefix=playbook,
             dir=private_data_dir
@@ -126,6 +124,10 @@ class IsolatedManager(object):
         params = self.runner_params.copy()
         params['playbook'] = playbook
         params['private_data_dir'] = iso_dir
+        if idle_timeout:
+            params['settings']['idle_timeout'] = idle_timeout
+        else:
+            params['settings'].pop('idle_timeout', None)
         params.update(**kw)
         if all([
             getattr(settings, 'AWX_ISOLATED_KEY_GENERATION', False) is True,
@@ -177,6 +179,7 @@ class IsolatedManager(object):
         logger.debug('Starting job {} on isolated host with `run_isolated.yml` playbook.'.format(self.instance.id))
         runner_obj = self.run_management_playbook('run_isolated.yml',
                                                   self.private_data_dir,
+                                                  idle_timeout=max(60, 2 * settings.AWX_ISOLATED_CONNECTION_TIMEOUT),
                                                   extravars=extravars)
 
         if runner_obj.status == 'failed':
@@ -208,14 +211,14 @@ class IsolatedManager(object):
         dispatcher = CallbackQueueDispatcher()
 
         while status == 'failed':
-            canceled = self.cancelled_callback() if self.cancelled_callback else False
+            canceled = self.canceled_callback() if self.canceled_callback else False
             if not canceled and time.time() - last_check < interval:
-                # If the job isn't cancelled, but we haven't waited `interval` seconds, wait longer
+                # If the job isn't canceled, but we haven't waited `interval` seconds, wait longer
                 time.sleep(1)
                 continue
 
             if canceled:
-                logger.warning('Isolated job {} was manually cancelled.'.format(self.instance.id))
+                logger.warning('Isolated job {} was manually canceled.'.format(self.instance.id))
 
             logger.debug('Checking on isolated job {} with `check_isolated.yml`.'.format(self.instance.id))
             runner_obj = self.run_management_playbook('check_isolated.yml',

awx/main/management/commands/callback_stats.py

@@ -0,0 +1,37 @@
+import time
+import sys
+
+from django.db import connection
+from django.core.management.base import BaseCommand
+
+
+class Command(BaseCommand):
+
+    def handle(self, *args, **options):
+        with connection.cursor() as cursor:
+            clear = False
+            while True:
+                lines = []
+                for relation in (
+                    'main_jobevent', 'main_inventoryupdateevent',
+                    'main_projectupdateevent', 'main_adhoccommandevent'
+                ):
+                    lines.append(relation)
+                    for label, interval in (
+                        ('last minute:   ', '1 minute'),
+                        ('last 5 minutes:', '5 minutes'),
+                        ('last hour:     ', '1 hour'),
+                    ):
+                        cursor.execute(
+                            f"SELECT MAX(id) - MIN(id) FROM {relation} WHERE modified > now() - '{interval}'::interval;"
+                        )
+                        events = cursor.fetchone()[0] or 0
+                        lines.append(f'↳  {label} {events}')
+                    lines.append('')
+                if clear:
+                    for i in range(20):
+                        sys.stdout.write('\x1b[1A\x1b[2K')
+                for l in lines:
+                    print(l)
+                clear = True
+                time.sleep(.25)

awx/main/management/commands/inventory_import.py

@@ -28,6 +28,7 @@ from awx.main.models.inventory import (
     Host
 )
 from awx.main.utils.mem_inventory import MemInventory, dict_to_mem_data
+from awx.main.utils.safe_yaml import sanitize_jinja
 
 # other AWX imports
 from awx.main.models.rbac import batch_role_ancestor_rebuilding
@@ -795,6 +796,10 @@ class Command(BaseCommand):
             if self.instance_id_var:
                 instance_id = self._get_instance_id(mem_host.variables)
                 host_attrs['instance_id'] = instance_id
+            try:
+                sanitize_jinja(mem_host_name)
+            except ValueError as e:
+                raise ValueError(str(e) + ': {}'.format(mem_host_name))
             db_host = self.inventory.hosts.update_or_create(name=mem_host_name, defaults=host_attrs)[0]
             if enabled is False:
                 logger.debug('Host "%s" added (disabled)', mem_host_name)

awx/main/management/commands/provision_instance.py

@@ -1,6 +1,8 @@
 # Copyright (c) 2015 Ansible, Inc.
 # All Rights Reserved
 
+from uuid import uuid4
+
 from awx.main.models import Instance
 from django.conf import settings
 
@@ -22,6 +24,8 @@ class Command(BaseCommand):
     def add_arguments(self, parser):
         parser.add_argument('--hostname', dest='hostname', type=str,
                             help='Hostname used during provisioning')
+        parser.add_argument('--is-isolated', dest='is_isolated', action='store_true',
+                            help='Specify whether the instance is isolated')
 
     def _register_hostname(self, hostname):
         if not hostname:
@@ -37,7 +41,10 @@ class Command(BaseCommand):
     def handle(self, **options):
         if not options.get('hostname'):
             raise CommandError("Specify `--hostname` to use this command.")
-        self.uuid = settings.SYSTEM_UUID
+        if options['is_isolated']:
+            self.uuid = str(uuid4())
+        else:
+            self.uuid = settings.SYSTEM_UUID
         self.changed = False
         self._register_hostname(options.get('hostname'))
         if self.changed:

awx/main/management/commands/regenerate_secret_key.py

@@ -0,0 +1,129 @@
+import base64
+import json
+import os
+
+from django.core.management.base import BaseCommand
+from django.conf import settings
+from django.db import transaction
+from django.db.models.signals import post_save
+
+from awx.conf import settings_registry
+from awx.conf.models import Setting
+from awx.conf.signals import on_post_save_setting
+from awx.main.models import (
+    UnifiedJob, Credential, NotificationTemplate, Job, JobTemplate, WorkflowJob,
+    WorkflowJobTemplate, OAuth2Application
+)
+from awx.main.utils.encryption import (
+    encrypt_field, decrypt_field, encrypt_value, decrypt_value, get_encryption_key
+)
+
+
+class Command(BaseCommand):
+    """
+    Regenerate a new SECRET_KEY value and re-encrypt every secret in the
+    Tower database.
+    """
+
+    @transaction.atomic
+    def handle(self, **options):
+        self.old_key = settings.SECRET_KEY
+        self.new_key = base64.encodebytes(os.urandom(33)).decode().rstrip()
+        self._notification_templates()
+        self._credentials()
+        self._unified_jobs()
+        self._oauth2_app_secrets()
+        self._settings()
+        self._survey_passwords()
+        return self.new_key
+
+    def _notification_templates(self):
+        for nt in NotificationTemplate.objects.iterator():
+            CLASS_FOR_NOTIFICATION_TYPE = dict([(x[0], x[2]) for x in NotificationTemplate.NOTIFICATION_TYPES])
+            notification_class = CLASS_FOR_NOTIFICATION_TYPE[nt.notification_type]
+            for field in filter(lambda x: notification_class.init_parameters[x]['type'] == "password",
+                                notification_class.init_parameters):
+                nt.notification_configuration[field] = decrypt_field(nt, 'notification_configuration', subfield=field, secret_key=self.old_key)
+                nt.notification_configuration[field] = encrypt_field(nt, 'notification_configuration', subfield=field, secret_key=self.new_key)
+            nt.save()
+
+    def _credentials(self):
+        for credential in Credential.objects.iterator():
+            for field_name in credential.credential_type.secret_fields:
+                if field_name in credential.inputs:
+                    credential.inputs[field_name] = decrypt_field(
+                        credential,
+                        field_name,
+                        secret_key=self.old_key
+                    )
+                    credential.inputs[field_name] = encrypt_field(
+                        credential,
+                        field_name,
+                        secret_key=self.new_key
+                    )
+                credential.save()
+
+    def _unified_jobs(self):
+        for uj in UnifiedJob.objects.iterator():
+            if uj.start_args:
+                uj.start_args = decrypt_field(
+                    uj,
+                    'start_args',
+                    secret_key=self.old_key
+                )
+                uj.start_args = encrypt_field(uj, 'start_args', secret_key=self.new_key)
+                uj.save()
+
+    def _oauth2_app_secrets(self):
+        for app in OAuth2Application.objects.iterator():
+            raw = app.client_secret
+            app.client_secret = raw
+            encrypted = encrypt_value(raw, secret_key=self.new_key)
+            OAuth2Application.objects.filter(pk=app.pk).update(client_secret=encrypted)
+
+    def _settings(self):
+        # don't update memcached, the *actual* value isn't changing
+        post_save.disconnect(on_post_save_setting, sender=Setting)
+        for setting in Setting.objects.filter().order_by('pk'):
+            if settings_registry.is_setting_encrypted(setting.key):
+                setting.value = decrypt_field(setting, 'value', secret_key=self.old_key)
+                setting.value = encrypt_field(setting, 'value', secret_key=self.new_key)
+                setting.save()
+
+    def _survey_passwords(self):
+        for _type in (JobTemplate, WorkflowJobTemplate):
+            for jt in _type.objects.exclude(survey_spec={}):
+                changed = False
+                if jt.survey_spec.get('spec', []):
+                    for field in jt.survey_spec['spec']:
+                        if field.get('type') == 'password' and field.get('default', ''):
+                            raw = decrypt_value(
+                                get_encryption_key('value', None, secret_key=self.old_key),
+                                field['default']
+                            )
+                            field['default'] = encrypt_value(
+                                raw,
+                                pk=None,
+                                secret_key=self.new_key
+                            )
+                            changed = True
+                if changed:
+                    jt.save(update_fields=["survey_spec"])
+
+        for _type in (Job, WorkflowJob):
+            for job in _type.objects.exclude(survey_passwords={}).iterator():
+                changed = False
+                for key in job.survey_passwords:
+                    if key in job.extra_vars:
+                        extra_vars = json.loads(job.extra_vars)
+                        if not extra_vars.get(key):
+                            continue
+                        raw = decrypt_value(
+                            get_encryption_key('value', None, secret_key=self.old_key),
+                            extra_vars[key]
+                        )
+                        extra_vars[key] = encrypt_value(raw, pk=None, secret_key=self.new_key)
+                        job.extra_vars = json.dumps(extra_vars)
+                        changed = True
+                if changed:
+                    job.save(update_fields=["extra_vars"])

awx/main/management/commands/replay_job_events.py

@@ -9,6 +9,7 @@ import random
 from django.utils import timezone
 from django.core.management.base import BaseCommand
 
+from awx.main.models.events import emit_event_detail
 from awx.main.models import (
     UnifiedJob,
     Job,
@@ -17,14 +18,6 @@ from awx.main.models import (
     InventoryUpdate,
     SystemJob
 )
-from awx.main.consumers import emit_channel_notification
-from awx.api.serializers import (
-    JobEventWebSocketSerializer,
-    AdHocCommandEventWebSocketSerializer,
-    ProjectUpdateEventWebSocketSerializer,
-    InventoryUpdateEventWebSocketSerializer,
-    SystemJobEventWebSocketSerializer
-)
 
 
 class JobStatusLifeCycle():
@@ -96,21 +89,6 @@ class ReplayJobEvents(JobStatusLifeCycle):
             raise RuntimeError("No events for job id {}".format(job.id))
         return job_events, count
 
-    def get_serializer(self, job):
-        if type(job) is Job:
-            return JobEventWebSocketSerializer
-        elif type(job) is AdHocCommand:
-            return AdHocCommandEventWebSocketSerializer
-        elif type(job) is ProjectUpdate:
-            return ProjectUpdateEventWebSocketSerializer
-        elif type(job) is InventoryUpdate:
-            return InventoryUpdateEventWebSocketSerializer
-        elif type(job) is SystemJob:
-            return SystemJobEventWebSocketSerializer
-        else:
-            raise RuntimeError("Job is of type {} and replay is not yet supported.".format(type(job)))
-            sys.exit(1)
-
     def run(self, job_id, speed=1.0, verbosity=0, skip_range=[], random_seed=0, final_status_delay=0, debug=False):
         stats = {
             'events_ontime': {
@@ -136,7 +114,6 @@ class ReplayJobEvents(JobStatusLifeCycle):
         try:
             job = self.get_job(job_id)
             job_events, job_event_count = self.get_job_events(job)
-            serializer = self.get_serializer(job)
         except RuntimeError as e:
             print("{}".format(e.message))
             sys.exit(1)
@@ -162,8 +139,7 @@ class ReplayJobEvents(JobStatusLifeCycle):
                 stats['replay_start'] = self.replay_start
                 je_previous = je_current
 
-            je_serialized = serializer(je_current).data
-            emit_channel_notification('{}-{}'.format(je_serialized['group_name'], job.id), je_serialized)
+            emit_event_detail(je_current)
 
             replay_offset = self.replay_offset(je_previous.created, speed)
             recording_diff = (je_current.created - je_previous.created).total_seconds() * (1.0 / speed)

awx/main/middleware.py

@@ -62,6 +62,17 @@ class TimingMiddleware(threading.local, MiddlewareMixin):
         with open(filepath, 'w') as f:
             f.write('%s %s\n' % (request.method, request.get_full_path()))
             pstats.Stats(self.prof, stream=f).sort_stats('cumulative').print_stats()
+
+        if settings.AWX_REQUEST_PROFILE_WITH_DOT:
+            from gprof2dot import main as generate_dot
+            raw = os.path.join(self.dest, filename) + '.raw'
+            pstats.Stats(self.prof).dump_stats(raw)
+            generate_dot([
+                '-n', '2.5', '-f', 'pstats', '-o',
+                os.path.join( self.dest, filename).replace('.pstats', '.dot'),
+                raw
+            ])
+            os.remove(raw)
         return filepath
 
 

awx/main/migrations/0007_v320_data_migrations.py

@@ -7,12 +7,6 @@ from django.db import migrations, models
 
 # AWX
 from awx.main.migrations import ActivityStreamDisabledMigration
-from awx.main.migrations import _inventory_source as invsrc
-from awx.main.migrations import _migration_utils as migration_utils
-from awx.main.migrations import _reencrypt as reencrypt
-from awx.main.migrations import _scan_jobs as scan_jobs
-from awx.main.migrations import _credentialtypes as credentialtypes
-from awx.main.migrations import _azure_credentials as azurecreds
 import awx.main.fields
 
 
@@ -23,16 +17,8 @@ class Migration(ActivityStreamDisabledMigration):
     ]
 
     operations = [
-        # Inventory Refresh
-        migrations.RunPython(migration_utils.set_current_apps_for_migrations),
-        migrations.RunPython(invsrc.remove_rax_inventory_sources),
-        migrations.RunPython(azurecreds.remove_azure_credentials),
-        migrations.RunPython(invsrc.remove_azure_inventory_sources),
-        migrations.RunPython(invsrc.remove_inventory_source_with_no_inventory_link),
-        migrations.RunPython(invsrc.rename_inventory_sources),
-        migrations.RunPython(reencrypt.replace_aesecb_fernet),
-        migrations.RunPython(scan_jobs.migrate_scan_job_templates),
-
-        migrations.RunPython(credentialtypes.migrate_to_v2_credentials),
-        migrations.RunPython(credentialtypes.migrate_job_credentials),
+        # This list is intentionally empty.
+        # Tower 3.2 included several data migrations that are no longer
+        # necessary (this list is now empty because Tower 3.2 is past EOL and
+        # cannot be directly upgraded to modern versions of Tower)
     ]

awx/main/migrations/0010_v322_add_ovirt4_tower_inventory.py

@@ -15,8 +15,6 @@ class Migration(migrations.Migration):
     ]
 
     operations = [
-        migrations.RunPython(migration_utils.set_current_apps_for_migrations),
-        migrations.RunPython(credentialtypes.create_rhv_tower_credtype),
         migrations.AlterField(
             model_name='inventorysource',
             name='source',

awx/main/migrations/0011_v322_encrypt_survey_passwords.py

@@ -3,8 +3,6 @@ from __future__ import unicode_literals
 
 from django.db import migrations
 from awx.main.migrations import ActivityStreamDisabledMigration
-from awx.main.migrations import _reencrypt as reencrypt
-from awx.main.migrations import _migration_utils as migration_utils
 
 
 class Migration(ActivityStreamDisabledMigration):
@@ -14,6 +12,8 @@ class Migration(ActivityStreamDisabledMigration):
     ]
 
     operations = [
-        migrations.RunPython(migration_utils.set_current_apps_for_migrations),
-        migrations.RunPython(reencrypt.encrypt_survey_passwords),
+        # This list is intentionally empty.
+        # Tower 3.2 included several data migrations that are no longer
+        # necessary (this list is now empty because Tower 3.2 is past EOL and
+        # cannot be directly upgraded to modern versions of Tower)
     ]

awx/main/migrations/0012_v322_update_cred_types.py

@@ -1,10 +1,6 @@
 # -*- coding: utf-8 -*-
 from __future__ import unicode_literals
 
-# AWX
-from awx.main.migrations import _migration_utils as migration_utils
-from awx.main.migrations import _credentialtypes as credentialtypes
-
 from django.db import migrations
 
 
@@ -15,6 +11,8 @@ class Migration(migrations.Migration):
     ]
 
     operations = [
-        migrations.RunPython(migration_utils.set_current_apps_for_migrations),
-        migrations.RunPython(credentialtypes.add_azure_cloud_environment_field),
+        # This list is intentionally empty.
+        # Tower 3.2 included several data migrations that are no longer
+        # necessary (this list is now empty because Tower 3.2 is past EOL and
+        # cannot be directly upgraded to modern versions of Tower)
     ]

awx/main/migrations/0078_v360_clear_sessions_tokens_jt.py

@@ -19,11 +19,11 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='systemjob',
             name='job_type',
-            field=models.CharField(blank=True, choices=[('cleanup_jobs', 'Remove jobs older than a certain number of days'), ('cleanup_activitystream', 'Remove activity stream entries older than a certain number of days'), ('clearsessions', 'Removes expired browser sessions from the database'), ('cleartokens', 'Removes expired OAuth 2 access tokens and refresh tokens')], default='', max_length=32),
+            field=models.CharField(blank=True, choices=[('cleanup_jobs', 'Remove jobs older than a certain number of days'), ('cleanup_activitystream', 'Remove activity stream entries older than a certain number of days'), ('cleanup_sessions', 'Removes expired browser sessions from the database'), ('cleanup_tokens', 'Removes expired OAuth 2 access tokens and refresh tokens')], default='', max_length=32),
         ),
         migrations.AlterField(
             model_name='systemjobtemplate',
             name='job_type',
-            field=models.CharField(blank=True, choices=[('cleanup_jobs', 'Remove jobs older than a certain number of days'), ('cleanup_activitystream', 'Remove activity stream entries older than a certain number of days'), ('clearsessions', 'Removes expired browser sessions from the database'), ('cleartokens', 'Removes expired OAuth 2 access tokens and refresh tokens')], default='', max_length=32),
+            field=models.CharField(blank=True, choices=[('cleanup_jobs', 'Remove jobs older than a certain number of days'), ('cleanup_activitystream', 'Remove activity stream entries older than a certain number of days'), ('cleanup_sessions', 'Removes expired browser sessions from the database'), ('cleanup_tokens', 'Removes expired OAuth 2 access tokens and refresh tokens')], default='', max_length=32),
         ),
     ]

awx/main/migrations/0099_v361_license_cleanup.py

@@ -0,0 +1,21 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+from django.db import migrations
+
+
+def _cleanup_license_setting(apps, schema_editor):
+    Setting = apps.get_model('conf', 'Setting')
+    for license in Setting.objects.filter(key='LICENSE').all():
+        for k in ('rh_username', 'rh_password'):
+            license.value.pop(k, None)
+        license.save()
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('main', '0098_v360_rename_cyberark_aim_credential_type'),
+    ]
+
+    operations = [migrations.RunPython(_cleanup_license_setting)]

awx/main/migrations/0100_v370_projectupdate_job_tags.py

@@ -0,0 +1,18 @@
+# Generated by Django 2.2.4 on 2019-11-01 18:46
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('main', '0099_v361_license_cleanup'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='projectupdate',
+            name='job_tags',
+            field=models.CharField(blank=True, default='', help_text='Parts of the project update playbook that will be run.', max_length=1024),
+        ),
+    ]

awx/main/migrations/0101_v370_generate_new_uuids_for_iso_nodes.py

@@ -0,0 +1,24 @@
+# -*- coding: utf-8 -*-
+from uuid import uuid4
+
+from django.db import migrations
+
+from awx.main.models import Instance
+
+
+def _generate_new_uuid_for_iso_nodes(apps, schema_editor):
+    for instance in Instance.objects.all():
+        if instance.is_isolated():
+            instance.uuid = str(uuid4())
+            instance.save()
+    
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('main', '0100_v370_projectupdate_job_tags'),
+    ]
+
+    operations = [
+        migrations.RunPython(_generate_new_uuid_for_iso_nodes)
+    ]

awx/main/migrations/_azure_credentials.py

@@ -1,15 +0,0 @@
-import logging
-
-from django.db.models import Q
-
-logger = logging.getLogger('awx.main.migrations')
-
-
-def remove_azure_credentials(apps, schema_editor):
-    '''Azure is not supported as of 3.2 and greater. Instead, azure_rm is
-    supported.
-    '''
-    Credential = apps.get_model('main', 'Credential')
-    logger.debug("Removing all Azure Credentials from database.")
-    Credential.objects.filter(kind='azure').delete()
-

awx/main/migrations/_credentialtypes.py

@@ -1,6 +1,4 @@
-from awx.main import utils
 from awx.main.models import CredentialType
-from awx.main.utils.encryption import encrypt_field, decrypt_field
 from django.db.models import Q
 
 
@@ -61,16 +59,6 @@ def _disassociate_non_insights_projects(apps, cred):
     apps.get_model('main', 'Project').objects.filter(~Q(scm_type='insights') & Q(credential=cred)).update(credential=None)
 
 
-def migrate_to_v2_credentials(apps, schema_editor):
-    # TODO: remove once legacy/EOL'd Towers no longer support this upgrade path
-    pass
-
-
-def migrate_job_credentials(apps, schema_editor):
-    # TODO: remove once legacy/EOL'd Towers no longer support this upgrade path
-    pass
-
-
 def add_vault_id_field(apps, schema_editor):
     # this is no longer necessary; schemas are defined in code
     pass
@@ -81,21 +69,11 @@ def remove_vault_id_field(apps, schema_editor):
     pass
 
 
-def create_rhv_tower_credtype(apps, schema_editor):
-    # this is no longer necessary; schemas are defined in code
-    pass
-
-
 def add_tower_verify_field(apps, schema_editor):
     # this is no longer necessary; schemas are defined in code
     pass
 
 
-def add_azure_cloud_environment_field(apps, schema_editor):
-    # this is no longer necessary; schemas are defined in code
-    pass
-
-
 def remove_become_methods(apps, schema_editor):
     # this is no longer necessary; schemas are defined in code
     pass

awx/main/migrations/_inventory_source.py

@@ -1,6 +1,5 @@
 import logging
 
-from django.db.models import Q
 from django.utils.encoding import smart_text
 
 from awx.main.utils.common import parse_yaml_or_json
@@ -8,64 +7,6 @@ from awx.main.utils.common import parse_yaml_or_json
 logger = logging.getLogger('awx.main.migrations')
 
 
-def remove_manual_inventory_sources(apps, schema_editor):
-    '''Previously we would automatically create inventory sources after
-    Group creation and we would use the parent Group as our interface for the user.
-    During that process we would create InventorySource that had a source of "manual".
-    '''
-    # TODO: use this in the 3.3 data migrations
-    InventorySource = apps.get_model('main', 'InventorySource')
-    # see models/inventory.py SOURCE_CHOICES - ('', _('Manual'))
-    logger.debug("Removing all Manual InventorySource from database.")
-    InventorySource.objects.filter(source='').delete()
-
-
-def remove_rax_inventory_sources(apps, schema_editor):
-    '''Rackspace inventory sources are not supported since 3.2, remove them.
-    '''
-    InventorySource = apps.get_model('main', 'InventorySource')
-    logger.debug("Removing all Rackspace InventorySource from database.")
-    InventorySource.objects.filter(source='rax').delete()
-
-
-def rename_inventory_sources(apps, schema_editor):
-    '''Rename existing InventorySource entries using the following format.
-    {{ inventory_source.name }} - {{ inventory.module }} - {{ number }}
-    The number will be incremented for each InventorySource for the organization.
-    '''
-    Organization = apps.get_model('main', 'Organization')
-    InventorySource = apps.get_model('main', 'InventorySource')
-
-    for org in Organization.objects.iterator():
-        for i, invsrc in enumerate(InventorySource.objects.filter(Q(inventory__organization=org) |
-                                                                  Q(deprecated_group__inventory__organization=org)).distinct().all()):
-
-            inventory = invsrc.deprecated_group.inventory if invsrc.deprecated_group else invsrc.inventory
-            name = '{0} - {1} - {2}'.format(invsrc.name, inventory.name, i)
-            logger.debug("Renaming InventorySource({0}) {1} -> {2}".format(
-                invsrc.pk, invsrc.name, name
-            ))
-            invsrc.name = name
-            invsrc.save()
-
-
-def remove_inventory_source_with_no_inventory_link(apps, schema_editor):
-    '''If we cannot determine the Inventory for which an InventorySource exists
-    we can safely remove it.
-    '''
-    InventorySource = apps.get_model('main', 'InventorySource')
-    logger.debug("Removing all InventorySource that have no link to an Inventory from database.")
-    InventorySource.objects.filter(Q(inventory__organization=None) & Q(deprecated_group__inventory=None)).delete()
-
-
-def remove_azure_inventory_sources(apps, schema_editor):
-    '''Azure inventory sources are not supported since 3.2, remove them.
-    '''
-    InventorySource = apps.get_model('main', 'InventorySource')
-    logger.debug("Removing all Azure InventorySource from database.")
-    InventorySource.objects.filter(source='azure').delete()
-
-
 def _get_instance_id(from_dict, new_id, default=''):
     '''logic mostly duplicated with inventory_import command Command._get_instance_id
     frozen in time here, for purposes of migrations

awx/main/migrations/_reencrypt.py

@@ -1,79 +1,12 @@
 import logging
-import json
-from django.utils.translation import ugettext_lazy as _
 
 from awx.conf.migrations._reencrypt import (
     decrypt_field,
-    should_decrypt_field,
 )
-from awx.main.utils.encryption import encrypt_field
-
-from awx.main.notifications.email_backend import CustomEmailBackend
-from awx.main.notifications.slack_backend import SlackBackend
-from awx.main.notifications.twilio_backend import TwilioBackend
-from awx.main.notifications.pagerduty_backend import PagerDutyBackend
-from awx.main.notifications.hipchat_backend import HipChatBackend
-from awx.main.notifications.mattermost_backend import MattermostBackend
-from awx.main.notifications.webhook_backend import WebhookBackend
-from awx.main.notifications.irc_backend import IrcBackend
 
 logger = logging.getLogger('awx.main.migrations')
 
-__all__ = ['replace_aesecb_fernet']
-
-
-NOTIFICATION_TYPES = [('email', _('Email'), CustomEmailBackend),
-                      ('slack', _('Slack'), SlackBackend),
-                      ('twilio', _('Twilio'), TwilioBackend),
-                      ('pagerduty', _('Pagerduty'), PagerDutyBackend),
-                      ('hipchat', _('HipChat'), HipChatBackend),
-                      ('mattermost', _('Mattermost'), MattermostBackend),
-                      ('webhook', _('Webhook'), WebhookBackend),
-                      ('irc', _('IRC'), IrcBackend)]
-
-
-PASSWORD_FIELDS = ('password', 'security_token', 'ssh_key_data', 'ssh_key_unlock',
-                   'become_password', 'vault_password', 'secret', 'authorize_password')
-
-
-def replace_aesecb_fernet(apps, schema_editor):
-    _notification_templates(apps)
-    _credentials(apps)
-    _unified_jobs(apps)
-
-
-def _notification_templates(apps):
-    NotificationTemplate = apps.get_model('main', 'NotificationTemplate')
-    for nt in NotificationTemplate.objects.all():
-        CLASS_FOR_NOTIFICATION_TYPE = dict([(x[0], x[2]) for x in NOTIFICATION_TYPES])
-        notification_class = CLASS_FOR_NOTIFICATION_TYPE[nt.notification_type]
-        for field in filter(lambda x: notification_class.init_parameters[x]['type'] == "password",
-                            notification_class.init_parameters):
-            if should_decrypt_field(nt.notification_configuration[field]):
-                nt.notification_configuration[field] = decrypt_field(nt, 'notification_configuration', subfield=field)
-                nt.notification_configuration[field] = encrypt_field(nt, 'notification_configuration', subfield=field)
-        nt.save()
-
-
-def _credentials(apps):
-    for credential in apps.get_model('main', 'Credential').objects.all():
-        for field_name in PASSWORD_FIELDS:
-            value = getattr(credential, field_name)
-            if should_decrypt_field(value):
-                value = decrypt_field(credential, field_name)
-                setattr(credential, field_name, value)
-                setattr(credential, field_name, encrypt_field(credential, field_name))
-        credential.save()
-
-
-def _unified_jobs(apps):
-    UnifiedJob = apps.get_model('main', 'UnifiedJob')
-    for uj in UnifiedJob.objects.all():
-        if uj.start_args is not None:
-            if should_decrypt_field(uj.start_args):
-                uj.start_args = decrypt_field(uj, 'start_args')
-                uj.start_args = encrypt_field(uj, 'start_args')
-                uj.save()
+__all__ = []
 
 
 def blank_old_start_args(apps, schema_editor):
@@ -91,53 +24,3 @@ def blank_old_start_args(apps, schema_editor):
             logger.debug('Blanking job args for %s', uj.pk)
             uj.start_args = ''
             uj.save()
-
-
-def encrypt_survey_passwords(apps, schema_editor):
-    _encrypt_survey_passwords(
-        apps.get_model('main', 'Job'),
-        apps.get_model('main', 'JobTemplate'),
-        apps.get_model('main', 'WorkflowJob'),
-        apps.get_model('main', 'WorkflowJobTemplate'),
-    )
-
-
-def _encrypt_survey_passwords(Job, JobTemplate, WorkflowJob, WorkflowJobTemplate):
-    from awx.main.utils.encryption import encrypt_value
-    for _type in (JobTemplate, WorkflowJobTemplate):
-        for jt in _type.objects.exclude(survey_spec={}):
-            changed = False
-            if jt.survey_spec.get('spec', []):
-                for field in jt.survey_spec['spec']:
-                    if field.get('type') == 'password' and field.get('default', ''):
-                        default = field['default']
-                        if default.startswith('$encrypted$'):
-                            if default == '$encrypted$':
-                                # If you have a survey_spec with a literal
-                                # '$encrypted$' as the default, you have
-                                # encountered a known bug in awx/Tower
-                                # https://github.com/ansible/ansible-tower/issues/7800
-                                logger.error(
-                                    '{}.pk={} survey_spec has ambiguous $encrypted$ default for {}, needs attention...'.format(jt, jt.pk, field['variable'])
-                                )
-                                field['default'] = ''
-                                changed = True
-                            continue
-                        field['default'] = encrypt_value(field['default'], pk=None)
-                        changed = True
-            if changed:
-                jt.save()
-
-    for _type in (Job, WorkflowJob):
-        for job in _type.objects.defer('result_stdout_text').exclude(survey_passwords={}).iterator():
-            changed = False
-            for key in job.survey_passwords:
-                if key in job.extra_vars:
-                    extra_vars = json.loads(job.extra_vars)
-                    if not extra_vars.get(key, '') or extra_vars[key].startswith('$encrypted$'):
-                        continue
-                    extra_vars[key] = encrypt_value(extra_vars[key], pk=None)
-                    job.extra_vars = json.dumps(extra_vars)
-                    changed = True
-            if changed:
-                job.save()

awx/main/migrations/_scan_jobs.py

@@ -1,89 +1,9 @@
 import logging
 
-from django.utils.timezone import now
-from django.utils.text import slugify
-
-from awx.main.models.base import PERM_INVENTORY_SCAN, PERM_INVENTORY_DEPLOY
-from awx.main import utils
-
 
 logger = logging.getLogger('awx.main.migrations')
 
 
-def _create_fact_scan_project(ContentType, Project, org):
-    ct = ContentType.objects.get_for_model(Project)
-    name = u"Tower Fact Scan - {}".format(org.name if org else "No Organization")
-    proj = Project(name=name,
-                   scm_url='https://github.com/ansible/awx-facts-playbooks',
-                   scm_type='git',
-                   scm_update_on_launch=True,
-                   scm_update_cache_timeout=86400,
-                   organization=org,
-                   created=now(),
-                   modified=now(),
-                   polymorphic_ctype=ct)
-    proj.save()
-
-    slug_name = slugify(str(name)).replace(u'-', u'_')
-    proj.local_path = u'_%d__%s' % (int(proj.pk), slug_name)
-
-    proj.save()
-    return proj
-
-
-def _create_fact_scan_projects(ContentType, Project, orgs):
-    return {org.id : _create_fact_scan_project(ContentType, Project, org) for org in orgs}
-
-
-def _get_tower_scan_job_templates(JobTemplate):
-    return JobTemplate.objects.filter(job_type=PERM_INVENTORY_SCAN, project__isnull=True) \
-                              .prefetch_related('inventory__organization')
-
-
-def _get_orgs(Organization, job_template_ids):
-    return Organization.objects.filter(inventories__jobtemplates__in=job_template_ids).distinct()
-
-
-def _migrate_scan_job_templates(apps):
-    JobTemplate = apps.get_model('main', 'JobTemplate')
-    Organization = apps.get_model('main', 'Organization')
-    ContentType = apps.get_model('contenttypes', 'ContentType')
-    Project = apps.get_model('main', 'Project')
-
-    project_no_org = None
-
-    # A scan job template with a custom project will retain the custom project.
-    JobTemplate.objects.filter(job_type=PERM_INVENTORY_SCAN, project__isnull=False).update(use_fact_cache=True, job_type=PERM_INVENTORY_DEPLOY)
-
-    # Scan jobs templates using Tower's default scan playbook will now point at
-    # the same playbook but in a github repo.
-    jts = _get_tower_scan_job_templates(JobTemplate)
-    if jts.count() == 0:
-        return
-
-    orgs = _get_orgs(Organization, jts.values_list('id'))
-    if orgs.count() == 0:
-        return
-
-    org_proj_map = _create_fact_scan_projects(ContentType, Project, orgs)
-    for jt in jts:
-        if jt.inventory and jt.inventory.organization:
-            jt.project_id = org_proj_map[jt.inventory.organization.id].id
-        # Job Templates without an Organization; through related Inventory
-        else:
-            if not project_no_org:
-                project_no_org = _create_fact_scan_project(ContentType, Project, None)
-            jt.project_id = project_no_org.id
-        jt.job_type = PERM_INVENTORY_DEPLOY
-        jt.playbook = "scan_facts.yml"
-        jt.use_fact_cache = True
-        jt.save()
-
-
-def migrate_scan_job_templates(apps, schema_editor):
-    _migrate_scan_job_templates(apps)
-
-
 def remove_scan_type_nodes(apps, schema_editor):
     WorkflowJobTemplateNode = apps.get_model('main', 'WorkflowJobTemplateNode')
     WorkflowJobNode = apps.get_model('main', 'WorkflowJobNode')

awx/main/models/events.py

@@ -1,8 +1,9 @@
+# -*- coding: utf-8 -*-
+
 import datetime
 import logging
 from collections import defaultdict
 
-from django.conf import settings
 from django.db import models, DatabaseError
 from django.utils.dateparse import parse_datetime
 from django.utils.text import Truncator
@@ -11,9 +12,10 @@ from django.utils.translation import ugettext_lazy as _
 from django.utils.encoding import force_text
 
 from awx.api.versioning import reverse
+from awx.main import consumers
 from awx.main.fields import JSONField
 from awx.main.models.base import CreatedModifiedModel
-from awx.main.utils import ignore_inventory_computed_fields
+from awx.main.utils import ignore_inventory_computed_fields, camelcase_to_underscore
 
 analytics_logger = logging.getLogger('awx.analytics.job_events')
 
@@ -55,6 +57,51 @@ def create_host_status_counts(event_data):
     return dict(host_status_counts)
 
 
+def emit_event_detail(event):
+    cls = event.__class__
+    relation = {
+        JobEvent: 'job_id',
+        AdHocCommandEvent: 'ad_hoc_command_id',
+        ProjectUpdateEvent: 'project_update_id',
+        InventoryUpdateEvent: 'inventory_update_id',
+        SystemJobEvent: 'system_job_id',
+    }[cls]
+    url = ''
+    if isinstance(event, JobEvent):
+        url = '/api/v2/job_events/{}'.format(event.id)
+    if isinstance(event, AdHocCommandEvent):
+        url = '/api/v2/ad_hoc_command_events/{}'.format(event.id)
+    group = camelcase_to_underscore(cls.__name__) + 's'
+    timestamp = event.created.isoformat()
+    consumers.emit_channel_notification(
+        '-'.join([group, str(getattr(event, relation))]),
+        {
+            'id': event.id,
+            relation.replace('_id', ''): getattr(event, relation),
+            'created': timestamp,
+            'modified': timestamp,
+            'group_name': group,
+            'url': url,
+            'stdout': event.stdout,
+            'counter': event.counter,
+            'uuid': event.uuid,
+            'parent_uuid': getattr(event, 'parent_uuid', ''),
+            'start_line': event.start_line,
+            'end_line': event.end_line,
+            'event': event.event,
+            'event_data': getattr(event, 'event_data', {}),
+            'failed': event.failed,
+            'changed': event.changed,
+            'event_level': getattr(event, 'event_level', ''),
+            'play': getattr(event, 'play', ''),
+            'role': getattr(event, 'role', ''),
+            'task': getattr(event, 'task', ''),
+        }
+    )
+
+
+
+
 class BasePlaybookEvent(CreatedModifiedModel):
     '''
     An event/message logged from a playbook callback for each host.
@@ -63,7 +110,7 @@ class BasePlaybookEvent(CreatedModifiedModel):
     VALID_KEYS = [
         'event', 'event_data', 'playbook', 'play', 'role', 'task', 'created',
         'counter', 'uuid', 'stdout', 'parent_uuid', 'start_line', 'end_line',
-        'verbosity'
+        'host_id', 'host_name', 'verbosity',
     ]
 
     class Meta:
@@ -271,37 +318,67 @@ class BasePlaybookEvent(CreatedModifiedModel):
 
     def _update_from_event_data(self):
         # Update event model fields from event data.
-        updated_fields = set()
         event_data = self.event_data
         res = event_data.get('res', None)
         if self.event in self.FAILED_EVENTS and not event_data.get('ignore_errors', False):
             self.failed = True
-            updated_fields.add('failed')
         if isinstance(res, dict):
             if res.get('changed', False):
                 self.changed = True
-                updated_fields.add('changed')
         if self.event == 'playbook_on_stats':
             try:
                 failures_dict = event_data.get('failures', {})
                 dark_dict = event_data.get('dark', {})
                 self.failed = bool(sum(failures_dict.values()) +
                                    sum(dark_dict.values()))
-                updated_fields.add('failed')
                 changed_dict = event_data.get('changed', {})
                 self.changed = bool(sum(changed_dict.values()))
-                updated_fields.add('changed')
             except (AttributeError, TypeError):
                 pass
+
+            if isinstance(self, JobEvent):
+                hostnames = self._hostnames()
+                self._update_host_summary_from_stats(hostnames)
+                if self.job.inventory:
+                    try:
+                        self.job.inventory.update_computed_fields()
+                    except DatabaseError:
+                        logger.exception('Computed fields database error saving event {}'.format(self.pk))
+
+                # find parent links and progagate changed=T and failed=T
+                changed = self.job.job_events.filter(changed=True).exclude(parent_uuid=None).only('parent_uuid').values_list('parent_uuid', flat=True).distinct()  # noqa
+                failed = self.job.job_events.filter(failed=True).exclude(parent_uuid=None).only('parent_uuid').values_list('parent_uuid', flat=True).distinct()  # noqa
+
+                JobEvent.objects.filter(
+                    job_id=self.job_id, uuid__in=changed
+                ).update(changed=True)
+                JobEvent.objects.filter(
+                    job_id=self.job_id, uuid__in=failed
+                ).update(failed=True)
+
         for field in ('playbook', 'play', 'task', 'role'):
             value = force_text(event_data.get(field, '')).strip()
             if value != getattr(self, field):
                 setattr(self, field, value)
-                updated_fields.add(field)
-        return updated_fields
+        if isinstance(self, JobEvent):
+            analytics_logger.info(
+                'Event data saved.',
+                extra=dict(python_objects=dict(job_event=self))
+            )
 
     @classmethod
     def create_from_data(cls, **kwargs):
+        #
+        # ⚠️  D-D-D-DANGER ZONE ⚠️
+        # This function is called by the callback receiver *once* for *every
+        # event* emitted by Ansible as a playbook runs.  That means that
+        # changes to this function are _very_ susceptible to introducing
+        # performance regressions (which the user will experience as "my
+        # playbook stdout takes too long to show up"), *especially* code which
+        # might invoke additional database queries per event.
+        #
+        # Proceed with caution!
+        #
         pk = None
         for key in ('job_id', 'project_update_id'):
             if key in kwargs:
@@ -325,74 +402,16 @@ class BasePlaybookEvent(CreatedModifiedModel):
 
         sanitize_event_keys(kwargs, cls.VALID_KEYS)
         workflow_job_id = kwargs.pop('workflow_job_id', None)
-        job_event = cls.objects.create(**kwargs)
+        event = cls(**kwargs)
         if workflow_job_id:
-            setattr(job_event, 'workflow_job_id', workflow_job_id)
-        analytics_logger.info('Event data saved.', extra=dict(python_objects=dict(job_event=job_event)))
-        return job_event
+            setattr(event, 'workflow_job_id', workflow_job_id)
+        event._update_from_event_data()
+        return event
 
     @property
     def job_verbosity(self):
         return 0
 
-    def save(self, *args, **kwargs):
-        # If update_fields has been specified, add our field names to it,
-        # if it hasn't been specified, then we're just doing a normal save.
-        update_fields = kwargs.get('update_fields', [])
-        # Update model fields and related objects unless we're only updating
-        # failed/changed flags triggered from a child event.
-        from_parent_update = kwargs.pop('from_parent_update', False)
-        if not from_parent_update:
-            # Update model fields from event data.
-            updated_fields = self._update_from_event_data()
-            for field in updated_fields:
-                if field not in update_fields:
-                    update_fields.append(field)
-
-            # Update host related field from host_name.
-            if hasattr(self, 'job') and not self.host_id and self.host_name:
-                if self.job.inventory.kind == 'smart':
-                    # optimization to avoid calling inventory.hosts, which
-                    # can take a long time to run under some circumstances
-                    from awx.main.models.inventory import SmartInventoryMembership
-                    membership = SmartInventoryMembership.objects.filter(
-                        inventory=self.job.inventory, host__name=self.host_name
-                    ).first()
-                    if membership:
-                        host_id = membership.host_id
-                    else:
-                        host_id = None
-                else:
-                    host_qs = self.job.inventory.hosts.filter(name=self.host_name)
-                    host_id = host_qs.only('id').values_list('id', flat=True).first()
-                if host_id != self.host_id:
-                    self.host_id = host_id
-                    if 'host_id' not in update_fields:
-                        update_fields.append('host_id')
-        super(BasePlaybookEvent, self).save(*args, **kwargs)
-
-        # Update related objects after this event is saved.
-        if hasattr(self, 'job') and not from_parent_update:
-            if getattr(settings, 'CAPTURE_JOB_EVENT_HOSTS', False):
-                self._update_hosts()
-            if self.parent_uuid:
-                kwargs = {}
-                if self.changed is True:
-                    kwargs['changed'] = True
-                if self.failed is True:
-                    kwargs['failed'] = True
-                if kwargs:
-                    JobEvent.objects.filter(job_id=self.job_id, uuid=self.parent_uuid).update(**kwargs)
-
-            if self.event == 'playbook_on_stats':
-                hostnames = self._hostnames()
-                self._update_host_summary_from_stats(hostnames)
-                try:
-                    self.job.inventory.update_computed_fields()
-                except DatabaseError:
-                    logger.exception('Computed fields database error saving event {}'.format(self.pk))
-
-
 
 class JobEvent(BasePlaybookEvent):
     '''
@@ -456,38 +475,6 @@ class JobEvent(BasePlaybookEvent):
     def __str__(self):
         return u'%s @ %s' % (self.get_event_display2(), self.created.isoformat())
 
-    def _update_from_event_data(self):
-        # Update job event hostname
-        updated_fields = super(JobEvent, self)._update_from_event_data()
-        value = force_text(self.event_data.get('host', '')).strip()
-        if value != getattr(self, 'host_name'):
-            setattr(self, 'host_name', value)
-            updated_fields.add('host_name')
-        return updated_fields
-
-    def _update_hosts(self, extra_host_pks=None):
-        # Update job event hosts m2m from host_name, propagate to parent events.
-        extra_host_pks = set(extra_host_pks or [])
-        hostnames = set()
-        if self.host_name:
-            hostnames.add(self.host_name)
-        if self.event == 'playbook_on_stats':
-            try:
-                for v in self.event_data.values():
-                    hostnames.update(v.keys())
-            except AttributeError: # In case event_data or v isn't a dict.
-                pass
-        qs = self.job.inventory.hosts.all()
-        qs = qs.filter(models.Q(name__in=hostnames) | models.Q(pk__in=extra_host_pks))
-        qs = qs.exclude(job_events__pk=self.id).only('id')
-        for host in qs:
-            self.hosts.add(host)
-        if self.parent_uuid:
-            parent = JobEvent.objects.filter(uuid=self.parent_uuid)
-            if parent.exists():
-                parent = parent[0]
-                parent._update_hosts(qs.values_list('id', flat=True))
-
     def _hostnames(self):
         hostnames = set()
         try:
@@ -605,6 +592,17 @@ class BaseCommandEvent(CreatedModifiedModel):
 
     @classmethod
     def create_from_data(cls, **kwargs):
+        #
+        # ⚠️  D-D-D-DANGER ZONE ⚠️
+        # This function is called by the callback receiver *once* for *every
+        # event* emitted by Ansible as a playbook runs.  That means that
+        # changes to this function are _very_ susceptible to introducing
+        # performance regressions (which the user will experience as "my
+        # playbook stdout takes too long to show up"), *especially* code which
+        # might invoke additional database queries per event.
+        #
+        # Proceed with caution!
+        #
         # Convert the datetime for the event's creation
         # appropriately, and include a time zone for it.
         #
@@ -619,13 +617,8 @@ class BaseCommandEvent(CreatedModifiedModel):
             kwargs.pop('created', None)
 
         sanitize_event_keys(kwargs, cls.VALID_KEYS)
-        kwargs.pop('workflow_job_id', None)
-        event = cls.objects.create(**kwargs)
-        if isinstance(event, AdHocCommandEvent):
-            analytics_logger.info(
-                'Event data saved.',
-                extra=dict(python_objects=dict(job_event=event))
-            )
+        event = cls(**kwargs)
+        event._update_from_event_data()
         return event
 
     def get_event_display(self):
@@ -640,10 +633,15 @@ class BaseCommandEvent(CreatedModifiedModel):
     def get_host_status_counts(self):
         return create_host_status_counts(getattr(self, 'event_data', {}))
 
+    def _update_from_event_data(self):
+        pass
+
 
 class AdHocCommandEvent(BaseCommandEvent):
 
-    VALID_KEYS = BaseCommandEvent.VALID_KEYS + ['ad_hoc_command_id', 'event', 'workflow_job_id']
+    VALID_KEYS = BaseCommandEvent.VALID_KEYS + [
+        'ad_hoc_command_id', 'event', 'host_name', 'host_id', 'workflow_job_id'
+    ]
 
     class Meta:
         app_label = 'main'
@@ -719,34 +717,18 @@ class AdHocCommandEvent(BaseCommandEvent):
     def get_absolute_url(self, request=None):
         return reverse('api:ad_hoc_command_event_detail', kwargs={'pk': self.pk}, request=request)
 
-    def save(self, *args, **kwargs):
-        # If update_fields has been specified, add our field names to it,
-        # if it hasn't been specified, then we're just doing a normal save.
-        update_fields = kwargs.get('update_fields', [])
+    def _update_from_event_data(self):
         res = self.event_data.get('res', None)
         if self.event in self.FAILED_EVENTS:
             if not self.event_data.get('ignore_errors', False):
                 self.failed = True
-                if 'failed' not in update_fields:
-                    update_fields.append('failed')
         if isinstance(res, dict) and res.get('changed', False):
             self.changed = True
-            if 'changed' not in update_fields:
-                update_fields.append('changed')
-        self.host_name = self.event_data.get('host', '').strip()
-        if 'host_name' not in update_fields:
-            update_fields.append('host_name')
-        if not self.host_id and self.host_name:
-            host_qs = self.ad_hoc_command.inventory.hosts.filter(name=self.host_name)
-            try:
-                host_id = host_qs.only('id').values_list('id', flat=True)
-                if host_id.exists():
-                    self.host_id = host_id[0]
-                    if 'host_id' not in update_fields:
-                        update_fields.append('host_id')
-            except (IndexError, AttributeError):
-                pass
-        super(AdHocCommandEvent, self).save(*args, **kwargs)
+
+        analytics_logger.info(
+            'Event data saved.',
+            extra=dict(python_objects=dict(job_event=self))
+        )
 
 
 class InventoryUpdateEvent(BaseCommandEvent):

awx/main/models/ha.py

@@ -270,6 +270,11 @@ class InstanceGroup(HasPolicyEditsMixin, BaseModel, RelatedJobsMixin):
                                       .filter(capacity__gt=0, enabled=True)
                                       .values_list('hostname', flat=True)))
 
+    def set_default_policy_fields(self):
+        self.policy_instance_list = []
+        self.policy_instance_minimum = 0
+        self.policy_instance_percentage = 0
+
 
 class TowerScheduleState(SingletonModel):
     schedule_last_run = models.DateTimeField(auto_now_add=True)
@@ -289,6 +294,8 @@ def on_instance_group_saved(sender, instance, created=False, raw=False, **kwargs
     if created or instance.has_policy_changes():
         if not instance.is_containerized:
             schedule_policy_task()
+    elif created or instance.is_containerized:
+        instance.set_default_policy_fields()
 
 
 @receiver(post_save, sender=Instance)

awx/main/models/inventory.py

@@ -61,6 +61,7 @@ from awx.main.models.notifications import (
 )
 from awx.main.models.credential.injectors import _openstack_data
 from awx.main.utils import _inventory_updates, region_sorting, get_licenser
+from awx.main.utils.safe_yaml import sanitize_jinja
 
 
 __all__ = ['Inventory', 'Host', 'Group', 'InventorySource', 'InventoryUpdate',
@@ -754,6 +755,13 @@ class Host(CommonModelNameNotUnique, RelatedJobsMixin):
                 update_host_smart_inventory_memberships.delay()
             connection.on_commit(on_commit)
 
+    def clean_name(self):
+        try:
+            sanitize_jinja(self.name)
+        except ValueError as e:
+            raise ValidationError(str(e) + ": {}".format(self.name))
+        return self.name
+
     def save(self, *args, **kwargs):
         self._update_host_smart_inventory_memeberships()
         super(Host, self).save(*args, **kwargs)
@@ -2036,9 +2044,25 @@ class azure_rm(PluginFileInjector):
         for key, loc in old_filterables:
             value = source_vars.get(key, None)
             if value and isinstance(value, str):
-                user_filters.append('{} not in {}'.format(
-                    loc, value.split(',')
-                ))
+                # tags can be list of key:value pairs
+                #  e.g. 'Creator:jmarshall, peanutbutter:jelly'
+                # or tags can be a list of keys
+                #  e.g. 'Creator, peanutbutter'
+                if key == "tags":
+                    # grab each key value pair
+                    for kvpair in value.split(','):
+                        # split into key and value
+                        kv = kvpair.split(':')
+                        # filter out any host that does not have key
+                        # in their tags.keys() variable
+                        user_filters.append('"{}" not in tags.keys()'.format(kv[0].strip()))
+                        # if a value is provided, check that the key:value pair matches
+                        if len(kv) > 1:
+                            user_filters.append('tags["{}"] != "{}"'.format(kv[0].strip(), kv[1].strip()))
+                else:
+                    user_filters.append('{} not in {}'.format(
+                        loc, value.split(',')
+                    ))
         if user_filters:
             ret.setdefault('exclude_host_filters', [])
             ret['exclude_host_filters'].extend(user_filters)

awx/main/models/jobs.py

@@ -634,7 +634,7 @@ class Job(UnifiedJob, JobOptions, SurveyJobMixin, JobNotificationMixin, TaskMana
         else:
             # If for some reason we can't count the hosts then lets assume the impact as forks
             if self.inventory is not None:
-                count_hosts = self.inventory.hosts.count()
+                count_hosts = self.inventory.total_hosts
                 if self.job_slice_count > 1:
                     # Integer division intentional
                     count_hosts = (count_hosts + self.job_slice_count - self.job_slice_number) // self.job_slice_count
@@ -1106,8 +1106,8 @@ class SystemJobOptions(BaseModel):
     SYSTEM_JOB_TYPE = [
         ('cleanup_jobs', _('Remove jobs older than a certain number of days')),
         ('cleanup_activitystream', _('Remove activity stream entries older than a certain number of days')),
-        ('clearsessions', _('Removes expired browser sessions from the database')),
-        ('cleartokens', _('Removes expired OAuth 2 access tokens and refresh tokens'))
+        ('cleanup_sessions', _('Removes expired browser sessions from the database')),
+        ('cleanup_tokens', _('Removes expired OAuth 2 access tokens and refresh tokens'))
     ]
 
     class Meta:
@@ -1182,18 +1182,19 @@ class SystemJobTemplate(UnifiedJobTemplate, SystemJobOptions):
             for key in unallowed_vars:
                 rejected[key] = data.pop(key)
 
-        if 'days' in data:
-            try:
-                if type(data['days']) is bool:
-                    raise ValueError
-                if float(data['days']) != int(data['days']):
-                    raise ValueError
-                days = int(data['days'])
-                if days < 0:
-                    raise ValueError
-            except ValueError:
-                errors_list.append(_("days must be a positive integer."))
-                rejected['days'] = data.pop('days')
+        if self.job_type in ('cleanup_jobs', 'cleanup_activitystream'):
+            if 'days' in data:
+                try:
+                    if isinstance(data['days'], (bool, type(None))):
+                        raise ValueError
+                    if float(data['days']) != int(data['days']):
+                        raise ValueError
+                    days = int(data['days'])
+                    if days < 0:
+                        raise ValueError
+                except ValueError:
+                    errors_list.append(_("days must be a positive integer."))
+                    rejected['days'] = data.pop('days')
 
         if errors_list:
             errors['extra_vars'] = errors_list

awx/main/models/notifications.py

@@ -269,6 +269,7 @@ class JobNotificationMixin(object):
                             'timeout', 'use_fact_cache', 'launch_type', 'status', 'failed', 'started', 'finished',
                             'elapsed', 'job_explanation', 'execution_node', 'controller_node', 'allow_simultaneous',
                             'scm_revision', 'diff_mode', 'job_slice_number', 'job_slice_count', 'custom_virtualenv',
+                            'approval_status', 'approval_node_name', 'workflow_url',
                             {'host_status_counts': ['skipped', 'ok', 'changed', 'failures', 'dark']},
                             {'playbook_counts': ['play_count', 'task_count']},
                             {'summary_fields': [{'inventory': ['id', 'name', 'description', 'has_active_failures',
@@ -366,6 +367,9 @@ class JobNotificationMixin(object):
                            'verbosity': 0},
                    'job_friendly_name': 'Job',
                    'url': 'https://towerhost/#/jobs/playbook/1010',
+                   'approval_status': 'approved',
+                   'approval_node_name': 'Approve Me',
+                   'workflow_url': 'https://towerhost/#/workflows/1010',
                    'job_metadata': """{'url': 'https://towerhost/$/jobs/playbook/13',
  'traceback': '',
  'status': 'running',

awx/main/models/oauth.py

@@ -1,4 +1,5 @@
 # Python
+import logging
 import re
 
 # Django
@@ -22,6 +23,9 @@ DATA_URI_RE = re.compile(r'.*')  # FIXME
 __all__ = ['OAuth2AccessToken', 'OAuth2Application']
 
 
+logger = logging.getLogger('awx.main.models.oauth')
+
+
 class OAuth2Application(AbstractApplication):
 
     class Meta:
@@ -121,14 +125,22 @@ class OAuth2AccessToken(AbstractAccessToken):
         valid = super(OAuth2AccessToken, self).is_valid(scopes)
         if valid:
             self.last_used = now()
-            connection.on_commit(lambda: self.save(update_fields=['last_used']))
+
+            def _update_last_used():
+                if OAuth2AccessToken.objects.filter(pk=self.pk).exists():
+                    self.save(update_fields=['last_used'])
+            connection.on_commit(_update_last_used)
         return valid
 
-    def save(self, *args, **kwargs):
+    def validate_external_users(self):
         if self.user and settings.ALLOW_OAUTH2_FOR_EXTERNAL_USERS is False:
             external_account = get_external_account(self.user)
             if external_account is not None:
                 raise oauth2.AccessDeniedError(_(
                     'OAuth2 Tokens cannot be created by users associated with an external authentication provider ({})'
                 ).format(external_account))
+
+    def save(self, *args, **kwargs):
+        if not self.pk:
+            self.validate_external_users()
         super(OAuth2AccessToken, self).save(*args, **kwargs)

awx/main/models/projects.py

@@ -483,6 +483,12 @@ class ProjectUpdate(UnifiedJob, ProjectOptions, JobNotificationMixin, TaskManage
         choices=PROJECT_UPDATE_JOB_TYPE_CHOICES,
         default='check',
     )
+    job_tags = models.CharField(
+        max_length=1024,
+        blank=True,
+        default='',
+        help_text=_('Parts of the project update playbook that will be run.'),
+    )
     scm_revision = models.CharField(
         max_length=1024,
         blank=True,
@@ -587,3 +593,24 @@ class ProjectUpdate(UnifiedJob, ProjectOptions, JobNotificationMixin, TaskManage
         if not selected_groups:
             return self.global_instance_groups
         return selected_groups
+
+    def save(self, *args, **kwargs):
+        added_update_fields = []
+        if not self.job_tags:
+            job_tags = ['update_{}'.format(self.scm_type)]
+            if self.job_type == 'run':
+                job_tags.append('install_roles')
+                job_tags.append('install_collections')
+            self.job_tags = ','.join(job_tags)
+            added_update_fields.append('job_tags')
+        if self.scm_delete_on_update and 'delete' not in self.job_tags and self.job_type == 'check':
+            self.job_tags = ','.join([self.job_tags, 'delete'])
+            added_update_fields.append('job_tags')
+        elif (not self.scm_delete_on_update) and 'delete' in self.job_tags:
+            job_tags = self.job_tags.split(',')
+            job_tags.remove('delete')
+            self.job_tags = ','.join(job_tags)
+            added_update_fields.append('job_tags')
+        if 'update_fields' in kwargs:
+            kwargs['update_fields'].extend(added_update_fields)
+        return super(ProjectUpdate, self).save(*args, **kwargs)

awx/main/notifications/custom_notification_base.py

@@ -6,15 +6,24 @@ class CustomNotificationBase(object):
     DEFAULT_MSG = "{{ job_friendly_name }} #{{ job.id }} '{{ job.name }}' {{ job.status }}: {{ url }}"
     DEFAULT_BODY = "{{ job_friendly_name }} #{{ job.id }} had status {{ job.status }}, view details at {{ url }}\n\n{{ job_metadata }}"
 
+    DEFAULT_APPROVAL_RUNNING_MSG = 'The approval node "{{ approval_node_name }}" needs review. This node can be viewed at: {{ workflow_url }}'
+    DEFAULT_APPROVAL_RUNNING_BODY = ('The approval node "{{ approval_node_name }}" needs review. '
+                                     'This approval node can be viewed at: {{ workflow_url }}\n\n{{ job_metadata }}')
+
+    DEFAULT_APPROVAL_APPROVED_MSG = 'The approval node "{{ approval_node_name }}" was approved. {{ workflow_url }}'
+    DEFAULT_APPROVAL_APPROVED_BODY = 'The approval node "{{ approval_node_name }}" was approved. {{ workflow_url }}\n\n{{ job_metadata }}'
+
+    DEFAULT_APPROVAL_TIMEOUT_MSG = 'The approval node "{{ approval_node_name }}" has timed out. {{ workflow_url }}'
+    DEFAULT_APPROVAL_TIMEOUT_BODY = 'The approval node "{{ approval_node_name }}" has timed out. {{ workflow_url }}\n\n{{ job_metadata }}'
+
+    DEFAULT_APPROVAL_DENIED_MSG = 'The approval node "{{ approval_node_name }}" was denied. {{ workflow_url }}'
+    DEFAULT_APPROVAL_DENIED_BODY = 'The approval node "{{ approval_node_name }}" was denied. {{ workflow_url }}\n\n{{ job_metadata }}'
+
+
     default_messages = {"started": {"message": DEFAULT_MSG, "body": None},
                         "success": {"message": DEFAULT_MSG, "body": None},
                         "error": {"message": DEFAULT_MSG, "body": None},
-                        "workflow_approval": {"running": {"message": 'The approval node "{{ approval_node_name }}" needs review. '
-                                                                     'This node can be viewed at: {{ workflow_url }}',
-                                                                     "body": None},
-                                              "approved": {"message": 'The approval node "{{ approval_node_name }}" was approved. {{ workflow_url }}',
-                                                           "body": None},
-                                              "timed_out": {"message": 'The approval node "{{ approval_node_name }}" has timed out. {{ workflow_url }}',
-                                                            "body": None},
-                                              "denied": {"message": 'The approval node "{{ approval_node_name }}" was denied. {{ workflow_url }}',
-                                                         "body": None}}}
+                        "workflow_approval": {"running": {"message": DEFAULT_APPROVAL_RUNNING_MSG, "body": None},
+                                              "approved": {"message": DEFAULT_APPROVAL_APPROVED_MSG, "body": None},
+                                              "timed_out": {"message": DEFAULT_APPROVAL_TIMEOUT_MSG, "body": None},
+                                              "denied": {"message": DEFAULT_APPROVAL_DENIED_MSG, "body": None}}}

awx/main/notifications/email_backend.py

@@ -8,6 +8,18 @@ from awx.main.notifications.custom_notification_base import CustomNotificationBa
 DEFAULT_MSG = CustomNotificationBase.DEFAULT_MSG
 DEFAULT_BODY = CustomNotificationBase.DEFAULT_BODY
 
+DEFAULT_APPROVAL_RUNNING_MSG = CustomNotificationBase.DEFAULT_APPROVAL_RUNNING_MSG
+DEFAULT_APPROVAL_RUNNING_BODY = CustomNotificationBase.DEFAULT_APPROVAL_RUNNING_BODY
+
+DEFAULT_APPROVAL_APPROVED_MSG = CustomNotificationBase.DEFAULT_APPROVAL_APPROVED_MSG
+DEFAULT_APPROVAL_APPROVED_BODY = CustomNotificationBase.DEFAULT_APPROVAL_APPROVED_BODY
+
+DEFAULT_APPROVAL_TIMEOUT_MSG = CustomNotificationBase.DEFAULT_APPROVAL_TIMEOUT_MSG
+DEFAULT_APPROVAL_TIMEOUT_BODY = CustomNotificationBase.DEFAULT_APPROVAL_TIMEOUT_BODY
+
+DEFAULT_APPROVAL_DENIED_MSG = CustomNotificationBase.DEFAULT_APPROVAL_DENIED_MSG
+DEFAULT_APPROVAL_DENIED_BODY = CustomNotificationBase.DEFAULT_APPROVAL_DENIED_BODY
+
 
 class CustomEmailBackend(EmailBackend, CustomNotificationBase):
 
@@ -26,10 +38,10 @@ class CustomEmailBackend(EmailBackend, CustomNotificationBase):
     default_messages = {"started": {"message": DEFAULT_MSG, "body": DEFAULT_BODY},
                         "success": {"message": DEFAULT_MSG, "body": DEFAULT_BODY},
                         "error": {"message": DEFAULT_MSG, "body": DEFAULT_BODY},
-                        "workflow_approval": {"running": {"message": DEFAULT_MSG, "body": DEFAULT_BODY},
-                                              "approved": {"message": DEFAULT_MSG, "body": DEFAULT_BODY},
-                                              "timed_out": {"message": DEFAULT_MSG, "body": DEFAULT_BODY},
-                                              "denied": {"message": DEFAULT_MSG, "body": DEFAULT_BODY}}}
+                        "workflow_approval": {"running": {"message": DEFAULT_APPROVAL_RUNNING_MSG, "body": DEFAULT_APPROVAL_RUNNING_BODY},
+                                              "approved": {"message": DEFAULT_APPROVAL_APPROVED_MSG, "body": DEFAULT_APPROVAL_APPROVED_BODY},
+                                              "timed_out": {"message": DEFAULT_APPROVAL_TIMEOUT_MSG, "body": DEFAULT_APPROVAL_TIMEOUT_BODY},
+                                              "denied": {"message": DEFAULT_APPROVAL_DENIED_MSG, "body": DEFAULT_APPROVAL_DENIED_BODY}}}
 
     def format_body(self, body):
         # leave body unchanged (expect a string)

awx/main/scheduler/dependency_graph.py

@@ -15,8 +15,6 @@ class DependencyGraph(object):
     INVENTORY_UPDATES = 'inventory_updates'
 
     JOB_TEMPLATE_JOBS = 'job_template_jobs'
-    JOB_PROJECT_IDS = 'job_project_ids'
-    JOB_INVENTORY_IDS = 'job_inventory_ids'
 
     SYSTEM_JOB = 'system_job'
     INVENTORY_SOURCE_UPDATES = 'inventory_source_updates'
@@ -41,10 +39,6 @@ class DependencyGraph(object):
         Track runnable job related project and inventory to ensure updates
         don't run while a job needing those resources is running.
         '''
-        # project_id -> True / False
-        self.data[self.JOB_PROJECT_IDS] = {}
-        # inventory_id -> True / False
-        self.data[self.JOB_INVENTORY_IDS] = {}
 
         # inventory_source_id -> True / False
         self.data[self.INVENTORY_SOURCE_UPDATES] = {}
@@ -66,7 +60,7 @@ class DependencyGraph(object):
 
     def get_now(self):
         return tz_now()
-        
+
     def mark_system_job(self):
         self.data[self.SYSTEM_JOB] = False
 
@@ -80,20 +74,16 @@ class DependencyGraph(object):
         self.data[self.INVENTORY_SOURCE_UPDATES][inventory_source_id] = False
 
     def mark_job_template_job(self, job):
-        self.data[self.JOB_INVENTORY_IDS][job.inventory_id] = False
-        self.data[self.JOB_PROJECT_IDS][job.project_id] = False
         self.data[self.JOB_TEMPLATE_JOBS][job.job_template_id] = False
 
     def mark_workflow_job(self, job):
         self.data[self.WORKFLOW_JOB_TEMPLATES_JOBS][job.workflow_job_template_id] = False
 
     def can_project_update_run(self, job):
-        return self.data[self.JOB_PROJECT_IDS].get(job.project_id, True) and \
-            self.data[self.PROJECT_UPDATES].get(job.project_id, True)
+        return self.data[self.PROJECT_UPDATES].get(job.project_id, True)
 
     def can_inventory_update_run(self, job):
-        return self.data[self.JOB_INVENTORY_IDS].get(job.inventory_source.inventory_id, True) and \
-            self.data[self.INVENTORY_SOURCE_UPDATES].get(job.inventory_source_id, True)
+        return self.data[self.INVENTORY_SOURCE_UPDATES].get(job.inventory_source_id, True)
 
     def can_job_run(self, job):
         if self.data[self.PROJECT_UPDATES].get(job.project_id, True) is True and \

awx/main/scheduler/kubernetes.py

@@ -173,7 +173,7 @@ def generate_tmp_kube_config(credential, namespace):
         "current-context": host_input
     }
 
-    if credential.get_input('verify_ssl'):
+    if credential.get_input('verify_ssl') and 'ssl_ca_cert' in credential.inputs:
         config["clusters"][0]["cluster"]["certificate-authority-data"] = b64encode(
             credential.get_input('ssl_ca_cert').encode() # encode to bytes
         ).decode() # decode the base64 data into a str

awx/main/scheduler/task_manager.py

@@ -258,19 +258,24 @@ class TaskManager():
                 for group in InstanceGroup.objects.all():
                     if group.is_containerized or group.controller_id:
                         continue
-                    match = group.find_largest_idle_instance()
+                    match = group.fit_task_to_most_remaining_capacity_instance(task)
                     if match:
                         break
                 task.instance_group = rampart_group
-                if task.supports_isolation():
-                    task.controller_node = match.hostname
+                if match is None:
+                    logger.warn(
+                        'No available capacity to run containerized <{}>.'.format(task.log_format)
+                    )
                 else:
-                    # project updates and inventory updates don't *actually* run in pods,
-                    # so just pick *any* non-isolated, non-containerized host and use it
-                    # as the execution node
-                    task.execution_node = match.hostname
-                    logger.debug('Submitting containerized {} to queue {}.'.format(
-                                 task.log_format, task.execution_node))
+                    if task.supports_isolation():
+                        task.controller_node = match.hostname
+                    else:
+                        # project updates and inventory updates don't *actually* run in pods,
+                        # so just pick *any* non-isolated, non-containerized host and use it
+                        # as the execution node
+                        task.execution_node = match.hostname
+                        logger.debug('Submitting containerized {} to queue {}.'.format(
+                                     task.log_format, task.execution_node))
             else:
                 task.instance_group = rampart_group
                 if instance is not None:

awx/main/signals.py

@@ -30,12 +30,11 @@ from crum.signals import current_user_getter
 
 # AWX
 from awx.main.models import (
-    ActivityStream, AdHocCommandEvent, Group, Host, InstanceGroup, Inventory,
-    InventorySource, InventoryUpdateEvent, Job, JobEvent, JobHostSummary,
-    JobTemplate, OAuth2AccessToken, Organization, Project, ProjectUpdateEvent,
-    Role, SystemJob, SystemJobEvent, SystemJobTemplate, UnifiedJob,
-    UnifiedJobTemplate, User, UserSessionMembership, WorkflowJobTemplateNode,
-    WorkflowApproval, WorkflowApprovalTemplate, ROLE_SINGLETON_SYSTEM_ADMINISTRATOR
+    ActivityStream, Group, Host, InstanceGroup, Inventory, InventorySource,
+    Job, JobHostSummary, JobTemplate, OAuth2AccessToken, Organization, Project,
+    Role, SystemJob, SystemJobTemplate, UnifiedJob, UnifiedJobTemplate, User,
+    UserSessionMembership, WorkflowJobTemplateNode, WorkflowApproval,
+    WorkflowApprovalTemplate, ROLE_SINGLETON_SYSTEM_ADMINISTRATOR
 )
 from awx.main.constants import CENSOR_VALUE
 from awx.main.utils import model_instance_diff, model_to_dict, camelcase_to_underscore, get_current_apps
@@ -72,42 +71,6 @@ def get_current_user_or_none():
     return u
 
 
-def emit_event_detail(serializer, relation, **kwargs):
-    instance = kwargs['instance']
-    created = kwargs['created']
-    if created:
-        event_serializer = serializer(instance)
-        consumers.emit_channel_notification(
-            '-'.join([event_serializer.get_group_name(instance), str(getattr(instance, relation))]),
-            event_serializer.data
-        )
-
-
-def emit_job_event_detail(sender, **kwargs):
-    from awx.api import serializers
-    emit_event_detail(serializers.JobEventWebSocketSerializer, 'job_id', **kwargs)
-
-
-def emit_ad_hoc_command_event_detail(sender, **kwargs):
-    from awx.api import serializers
-    emit_event_detail(serializers.AdHocCommandEventWebSocketSerializer, 'ad_hoc_command_id', **kwargs)
-
-
-def emit_project_update_event_detail(sender, **kwargs):
-    from awx.api import serializers
-    emit_event_detail(serializers.ProjectUpdateEventWebSocketSerializer, 'project_update_id', **kwargs)
-
-
-def emit_inventory_update_event_detail(sender, **kwargs):
-    from awx.api import serializers
-    emit_event_detail(serializers.InventoryUpdateEventWebSocketSerializer, 'inventory_update_id', **kwargs)
-
-
-def emit_system_job_event_detail(sender, **kwargs):
-    from awx.api import serializers
-    emit_event_detail(serializers.SystemJobEventWebSocketSerializer, 'system_job_id', **kwargs)
-
-
 def emit_update_inventory_computed_fields(sender, **kwargs):
     logger.debug("In update inventory computed fields")
     if getattr(_inventory_updates, 'is_updating', False):
@@ -258,11 +221,6 @@ connect_computed_field_signals()
 
 post_save.connect(save_related_job_templates, sender=Project)
 post_save.connect(save_related_job_templates, sender=Inventory)
-post_save.connect(emit_job_event_detail, sender=JobEvent)
-post_save.connect(emit_ad_hoc_command_event_detail, sender=AdHocCommandEvent)
-post_save.connect(emit_project_update_event_detail, sender=ProjectUpdateEvent)
-post_save.connect(emit_inventory_update_event_detail, sender=InventoryUpdateEvent)
-post_save.connect(emit_system_job_event_detail, sender=SystemJobEvent)
 m2m_changed.connect(rebuild_role_ancestor_list, Role.parents.through)
 m2m_changed.connect(rbac_activity_stream, Role.members.through)
 m2m_changed.connect(rbac_activity_stream, Role.parents.through)

awx/main/tasks.py

@@ -71,7 +71,7 @@ from awx.main.utils import (get_ssh_version, update_scm_url,
                             ignore_inventory_group_removal, extract_ansible_vars, schedule_task_manager,
                             get_awx_version)
 from awx.main.utils.ansible import read_ansible_config
-from awx.main.utils.common import get_ansible_version, _get_ansible_version, get_custom_venv_choices
+from awx.main.utils.common import _get_ansible_version, get_custom_venv_choices
 from awx.main.utils.safe_yaml import safe_dump, sanitize_jinja
 from awx.main.utils.reload import stop_local_services
 from awx.main.utils.pglock import advisory_lock
@@ -703,6 +703,7 @@ class BaseTask(object):
     def __init__(self):
         self.cleanup_paths = []
         self.parent_workflow_job_id = None
+        self.host_map = {}
 
     def update_model(self, pk, _attempt=0, **updates):
         """Reload the model instance from the database and update the
@@ -1001,11 +1002,17 @@ class BaseTask(object):
         return False
 
     def build_inventory(self, instance, private_data_dir):
-        script_params = dict(hostvars=True)
+        script_params = dict(hostvars=True, towervars=True)
         if hasattr(instance, 'job_slice_number'):
             script_params['slice_number'] = instance.job_slice_number
             script_params['slice_count'] = instance.job_slice_count
         script_data = instance.inventory.get_script_data(**script_params)
+        # maintain a list of host_name --> host_id
+        # so we can associate emitted events to Host objects
+        self.host_map = {
+            hostname: hv.pop('remote_tower_id', '')
+            for hostname, hv in script_data.get('_meta', {}).get('hostvars', {}).items()
+        }
         json_data = json.dumps(script_data)
         handle, path = tempfile.mkstemp(dir=private_data_dir)
         f = os.fdopen(handle, 'w')
@@ -1114,6 +1121,15 @@ class BaseTask(object):
                 event_data.pop('parent_uuid', None)
         if self.parent_workflow_job_id:
             event_data['workflow_job_id'] = self.parent_workflow_job_id
+        if self.host_map:
+            host = event_data.get('event_data', {}).get('host', '').strip()
+            if host:
+                event_data['host_name'] = host
+                if host in self.host_map:
+                    event_data['host_id'] = self.host_map[host]
+            else:
+                event_data['host_name'] = ''
+                event_data['host_id'] = ''
         should_write_event = False
         event_data.setdefault(self.event_data_key, self.instance.id)
         self.dispatcher.dispatch(event_data)
@@ -1336,7 +1352,7 @@ class BaseTask(object):
 
                 ansible_runner.utils.dump_artifacts(params)
                 isolated_manager_instance = isolated_manager.IsolatedManager(
-                    cancelled_callback=lambda: self.update_model(self.instance.pk).cancel_flag,
+                    canceled_callback=lambda: self.update_model(self.instance.pk).cancel_flag,
                     check_callback=self.check_handler,
                     pod_manager=pod_manager
                 )
@@ -1734,14 +1750,16 @@ class RunJob(BaseTask):
 
         project_path = job.project.get_project_path(check_if_exists=False)
         job_revision = job.project.scm_revision
-        needs_sync = True
+        sync_needs = []
+        all_sync_needs = ['update_{}'.format(job.project.scm_type), 'install_roles', 'install_collections']
         if not job.project.scm_type:
-            # manual projects are not synced, user has responsibility for that
-            needs_sync = False
+            pass # manual projects are not synced, user has responsibility for that
         elif not os.path.exists(project_path):
             logger.debug('Performing fresh clone of {} on this instance.'.format(job.project))
+            sync_needs = all_sync_needs
         elif not job.project.scm_revision:
             logger.debug('Revision not known for {}, will sync with remote'.format(job.project))
+            sync_needs = all_sync_needs
         elif job.project.scm_type == 'git':
             git_repo = git.Repo(project_path)
             try:
@@ -1752,23 +1770,27 @@ class RunJob(BaseTask):
                 if desired_revision == current_revision:
                     job_revision = desired_revision
                     logger.info('Skipping project sync for {} because commit is locally available'.format(job.log_format))
-                    needs_sync = False
+                else:
+                    sync_needs = all_sync_needs
             except (ValueError, BadGitName):
                 logger.debug('Needed commit for {} not in local source tree, will sync with remote'.format(job.log_format))
+                sync_needs = all_sync_needs
+        else:
+            sync_needs = all_sync_needs
         # Galaxy requirements are not supported for manual projects
-        if not needs_sync and job.project.scm_type:
+        if not sync_needs and job.project.scm_type:
             # see if we need a sync because of presence of roles
             galaxy_req_path = os.path.join(project_path, 'roles', 'requirements.yml')
             if os.path.exists(galaxy_req_path):
                 logger.debug('Running project sync for {} because of galaxy role requirements.'.format(job.log_format))
-                needs_sync = True
+                sync_needs.append('install_roles')
 
             galaxy_collections_req_path = os.path.join(project_path, 'collections', 'requirements.yml')
             if os.path.exists(galaxy_collections_req_path):
                 logger.debug('Running project sync for {} because of galaxy collections requirements.'.format(job.log_format))
-                needs_sync = True
+                sync_needs.append('install_collections')
 
-        if needs_sync:
+        if sync_needs:
             pu_ig = job.instance_group
             pu_en = job.execution_node
             if job.is_isolated() is True:
@@ -1778,6 +1800,7 @@ class RunJob(BaseTask):
             sync_metafields = dict(
                 launch_type="sync",
                 job_type='run',
+                job_tags=','.join(sync_needs),
                 status='running',
                 instance_group = pu_ig,
                 execution_node=pu_en,
@@ -1785,6 +1808,8 @@ class RunJob(BaseTask):
             )
             if job.scm_branch and job.scm_branch != job.project.scm_branch:
                 sync_metafields['scm_branch'] = job.scm_branch
+            if 'update_' not in sync_metafields['job_tags']:
+                sync_metafields['scm_revision'] = job_revision
             local_project_sync = job.project.create_project_update(_eager_fields=sync_metafields)
             # save the associated job before calling run() so that a
             # cancel() call on the job can cancel the project update
@@ -1929,7 +1954,8 @@ class RunProjectUpdate(BaseTask):
         env['TMP'] = settings.AWX_PROOT_BASE_PATH
         env['PROJECT_UPDATE_ID'] = str(project_update.pk)
         env['ANSIBLE_CALLBACK_PLUGINS'] = self.get_path_to('..', 'plugins', 'callback')
-        env['ANSIBLE_GALAXY_IGNORE'] = True
+        if settings.GALAXY_IGNORE_CERTS:
+            env['ANSIBLE_GALAXY_IGNORE'] = True
         # Set up the public Galaxy server, if enabled
         if settings.PUBLIC_GALAXY_ENABLED:
             galaxy_servers = [settings.PUBLIC_GALAXY_SERVER]
@@ -2008,8 +2034,8 @@ class RunProjectUpdate(BaseTask):
         args = []
         if getattr(settings, 'PROJECT_UPDATE_VVV', False):
             args.append('-vvv')
-        else:
-            args.append('-v')
+        if project_update.job_tags:
+            args.extend(['-t', project_update.job_tags])
         return args
 
     def build_extra_vars_file(self, project_update, private_data_dir):
@@ -2023,28 +2049,16 @@ class RunProjectUpdate(BaseTask):
             scm_branch = project_update.project.scm_revision
         elif not scm_branch:
             scm_branch = {'hg': 'tip'}.get(project_update.scm_type, 'HEAD')
-        if project_update.job_type == 'check':
-            roles_enabled = False
-            collections_enabled = False
-        else:
-            roles_enabled = getattr(settings, 'AWX_ROLES_ENABLED', True)
-            collections_enabled = getattr(settings, 'AWX_COLLECTIONS_ENABLED', True)
-            # collections were introduced in Ansible version 2.8
-            if Version(get_ansible_version()) <= Version('2.8'):
-                collections_enabled = False
         extra_vars.update({
             'project_path': project_update.get_project_path(check_if_exists=False),
             'insights_url': settings.INSIGHTS_URL_BASE,
             'awx_license_type': get_license(show_key=False).get('license_type', 'UNLICENSED'),
             'awx_version': get_awx_version(),
-            'scm_type': project_update.scm_type,
             'scm_url': scm_url,
             'scm_branch': scm_branch,
             'scm_clean': project_update.scm_clean,
-            'scm_delete_on_update': project_update.scm_delete_on_update if project_update.job_type == 'check' else False,
-            'scm_full_checkout': True if project_update.job_type == 'run' else False,
-            'roles_enabled': roles_enabled,
-            'collections_enabled': collections_enabled,
+            'roles_enabled': settings.AWX_ROLES_ENABLED,
+            'collections_enabled': settings.AWX_COLLECTIONS_ENABLED,
         })
         if project_update.job_type != 'check' and self.job_private_data_dir:
             extra_vars['collections_destination'] = os.path.join(self.job_private_data_dir, 'requirements_collections')
@@ -2156,7 +2170,7 @@ class RunProjectUpdate(BaseTask):
             try:
                 instance.refresh_from_db(fields=['cancel_flag'])
                 if instance.cancel_flag:
-                    logger.debug("ProjectUpdate({0}) was cancelled".format(instance.pk))
+                    logger.debug("ProjectUpdate({0}) was canceled".format(instance.pk))
                     return
                 fcntl.lockf(self.lock_fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
                 break
@@ -2185,7 +2199,10 @@ class RunProjectUpdate(BaseTask):
             project_path = instance.project.get_project_path(check_if_exists=False)
             if os.path.exists(project_path):
                 git_repo = git.Repo(project_path)
-                self.original_branch = git_repo.active_branch
+                if git_repo.head.is_detached:
+                    self.original_branch = git_repo.head.commit
+                else:
+                    self.original_branch = git_repo.active_branch
 
     @staticmethod
     def make_local_copy(project_path, destination_folder, scm_type, scm_revision):
@@ -2217,26 +2234,29 @@ class RunProjectUpdate(BaseTask):
             copy_tree(project_path, destination_folder)
 
     def post_run_hook(self, instance, status):
-        if self.job_private_data_dir:
-            # copy project folder before resetting to default branch
-            # because some git-tree-specific resources (like submodules) might matter
-            self.make_local_copy(
-                instance.get_project_path(check_if_exists=False), os.path.join(self.job_private_data_dir, 'project'),
-                instance.scm_type, self.playbook_new_revision
-            )
-            if self.original_branch:
-                # for git project syncs, non-default branches can be problems
-                # restore to branch the repo was on before this run
-                try:
-                    self.original_branch.checkout()
-                except Exception:
-                    # this could have failed due to dirty tree, but difficult to predict all cases
-                    logger.exception('Failed to restore project repo to prior state after {}'.format(instance.log_format))
-        self.release_lock(instance)
+        # To avoid hangs, very important to release lock even if errors happen here
+        try:
+            if self.playbook_new_revision:
+                instance.scm_revision = self.playbook_new_revision
+                instance.save(update_fields=['scm_revision'])
+            if self.job_private_data_dir:
+                # copy project folder before resetting to default branch
+                # because some git-tree-specific resources (like submodules) might matter
+                self.make_local_copy(
+                    instance.get_project_path(check_if_exists=False), os.path.join(self.job_private_data_dir, 'project'),
+                    instance.scm_type, instance.scm_revision
+                )
+                if self.original_branch:
+                    # for git project syncs, non-default branches can be problems
+                    # restore to branch the repo was on before this run
+                    try:
+                        self.original_branch.checkout()
+                    except Exception:
+                        # this could have failed due to dirty tree, but difficult to predict all cases
+                        logger.exception('Failed to restore project repo to prior state after {}'.format(instance.log_format))
+        finally:
+            self.release_lock(instance)
         p = instance.project
-        if self.playbook_new_revision:
-            instance.scm_revision = self.playbook_new_revision
-            instance.save(update_fields=['scm_revision'])
         if instance.job_type == 'check' and status not in ('failed', 'canceled',):
             if self.playbook_new_revision:
                 p.scm_revision = self.playbook_new_revision
@@ -2497,6 +2517,7 @@ class RunInventoryUpdate(BaseTask):
                 _eager_fields=dict(
                     launch_type="sync",
                     job_type='run',
+                    job_tags='update_{},install_collections'.format(source_project.scm_type),  # roles are never valid for inventory
                     status='running',
                     execution_node=inventory_update.execution_node,
                     instance_group = inventory_update.instance_group,
@@ -2732,10 +2753,11 @@ class RunSystemJob(BaseTask):
                 json_vars = {}
             else:
                 json_vars = json.loads(system_job.extra_vars)
-            if 'days' in json_vars:
-                args.extend(['--days', str(json_vars.get('days', 60))])
-            if 'dry_run' in json_vars and json_vars['dry_run']:
-                args.extend(['--dry-run'])
+            if system_job.job_type in ('cleanup_jobs', 'cleanup_activitystream'):
+                if 'days' in json_vars:
+                    args.extend(['--days', str(json_vars.get('days', 60))])
+                if 'dry_run' in json_vars and json_vars['dry_run']:
+                    args.extend(['--dry-run'])
             if system_job.job_type == 'cleanup_jobs':
                 args.extend(['--jobs', '--project-updates', '--inventory-updates',
                              '--management-jobs', '--ad-hoc-commands', '--workflow-jobs',

awx/main/tests/data/ansible_utils/playbooks/valid/hello_world.yaml

@@ -1,7 +1,7 @@
+---
 - name: Hello World Sample
   hosts: all
   tasks:
     - name: Hello Message
       debug:
         msg: "Hello World!"
-

awx/main/tests/data/ansible_utils/playbooks/valid/hello_world.yml

@@ -1,7 +1,7 @@
+---
 - name: Hello World Sample
   hosts: all
   tasks:
     - name: Hello Message
       debug:
         msg: "Hello World!"
-

awx/main/tests/data/ansible_utils/playbooks/valid/hosts.yml

@@ -1 +1,2 @@
+---
 - hosts: all

awx/main/tests/data/ansible_utils/playbooks/valid/import.yml

@@ -1 +1,2 @@
+---
 - import_playbook: foo

awx/main/tests/data/ansible_utils/playbooks/valid/include.yml

@@ -1 +1,2 @@
+---
 - include: foo

awx/main/tests/data/inventory/plugins/azure_rm/files/azure_rm.yml

@@ -3,6 +3,10 @@ conditional_groups:
 default_host_filters: []
 exclude_host_filters:
 - resource_group not in ['foo_resources', 'bar_resources']
+- '"Creator" not in tags.keys()'
+- tags["Creator"] != "jmarshall"
+- '"peanutbutter" not in tags.keys()'
+- tags["peanutbutter"] != "jelly"
 - location not in ['southcentralus', 'westus']
 fail_on_template_errors: false
 hostvar_expressions:

awx/main/tests/data/inventory/scripts/azure_rm/files/file_reference

@@ -7,4 +7,5 @@ locations = southcentralus,westus
 base_source_var = value_of_var
 use_private_ip = True
 resource_groups = foo_resources,bar_resources
+tags = Creator:jmarshall, peanutbutter:jelly
 

awx/main/tests/functional/analytics/test_metrics.py

@@ -25,6 +25,8 @@ EXPECTED_VALUES = {
     'awx_custom_virtualenvs_total':0.0,
     'awx_running_jobs_total':0.0,
     'awx_instance_capacity':100.0,
+    'awx_instance_consumed_capacity':0.0,
+    'awx_instance_remaining_capacity':100.0,
     'awx_instance_cpu':0.0,
     'awx_instance_memory':0.0,
     'awx_instance_info':1.0,

awx/main/tests/functional/api/test_credential.py

@@ -1439,3 +1439,15 @@ def test_create_credential_with_invalid_url_xfail(post, organization, admin, url
     assert response.status_code == status
     if status != 201:
         assert response.data['inputs']['server_url'] == [msg]
+
+
+@pytest.mark.django_db
+def test_external_credential_rbac_test_endpoint(post, alice, external_credential):
+    url = reverse('api:credential_external_test', kwargs={'pk': external_credential.pk})
+    data = {'metadata': {'key': 'some_key'}}
+
+    external_credential.read_role.members.add(alice)
+    assert post(url, data, alice).status_code == 403
+
+    external_credential.use_role.members.add(alice)
+    assert post(url, data, alice).status_code == 202

awx/main/tests/functional/api/test_credential_type.py

@@ -85,14 +85,35 @@ def test_update_credential_type_in_use_xfail(patch, delete, admin):
     Credential(credential_type=_type, name='My Custom Cred').save()
 
     url = reverse('api:credential_type_detail', kwargs={'pk': _type.pk})
-    response = patch(url, {'name': 'Some Other Name'}, admin)
-    assert response.status_code == 200
+    patch(url, {'name': 'Some Other Name'}, admin, expect=200)
 
     url = reverse('api:credential_type_detail', kwargs={'pk': _type.pk})
-    response = patch(url, {'inputs': {}}, admin)
-    assert response.status_code == 403
+    response = patch(url, {'inputs': {}}, admin, expect=403)
+    assert response.data['detail'] == 'Modifications to inputs are not allowed for credential types that are in use'
 
-    assert delete(url, admin).status_code == 403
+    response = delete(url, admin, expect=403)
+    assert response.data['detail'] == 'Credential types that are in use cannot be deleted'
+
+
+@pytest.mark.django_db
+def test_update_credential_type_unvalidated_inputs(post, patch, admin):
+    simple_inputs = {'fields': [
+        {'id': 'api_token', 'label': 'fooo'}
+    ]}
+    response = post(
+        url=reverse('api:credential_type_list'),
+        data={'name': 'foo', 'kind': 'cloud', 'inputs': simple_inputs},
+        user=admin,
+        expect=201
+    )
+    # validation adds the type field to the input
+    _type = CredentialType.objects.get(pk=response.data['id'])
+    Credential(credential_type=_type, name='My Custom Cred').save()
+
+    # should not raise an error because we should only compare
+    # post-validation values to other post-validation values
+    url = reverse('api:credential_type_detail', kwargs={'pk': _type.id})
+    patch(url, {'inputs': simple_inputs}, admin, expect=200)
 
 
 @pytest.mark.django_db
@@ -460,3 +481,12 @@ def test_create_with_undefined_template_variable_xfail(post, admin):
     }, admin)
     assert response.status_code == 400
     assert "'api_tolkien' is undefined" in json.dumps(response.data)
+
+
+@pytest.mark.django_db
+def test_credential_type_rbac_external_test(post, alice, admin, credentialtype_external):
+    # only admins may use the credential type test endpoint
+    url = reverse('api:credential_type_external_test', kwargs={'pk': credentialtype_external.pk})
+    data = {'inputs': {}, 'metadata': {}}
+    assert post(url, data, admin).status_code == 202
+    assert post(url, data, alice).status_code == 403

awx/main/tests/functional/api/test_events.py

@@ -15,7 +15,7 @@ def test_job_events_sublist_truncation(get, organization_factory, job_template_f
                               inventory='test_inv', project='test_proj').job_template
     job = jt.create_unified_job()
     JobEvent.create_from_data(job_id=job.pk, uuid='abc123', event='runner_on_start',
-                              stdout='a' * 1025)
+                              stdout='a' * 1025).save()
 
     url = reverse('api:job_job_events_list', kwargs={'pk': job.pk})
     if not truncate:
@@ -35,7 +35,7 @@ def test_ad_hoc_events_sublist_truncation(get, organization_factory, job_templat
     adhoc = AdHocCommand()
     adhoc.save()
     AdHocCommandEvent.create_from_data(ad_hoc_command_id=adhoc.pk, uuid='abc123', event='runner_on_start',
-                                       stdout='a' * 1025)
+                                       stdout='a' * 1025).save()
 
     url = reverse('api:ad_hoc_command_ad_hoc_command_events_list', kwargs={'pk': adhoc.pk})
     if not truncate:

awx/main/tests/functional/api/test_instance_group.py

@@ -274,3 +274,21 @@ def test_instance_group_update_fields(patch, instance, instance_group, admin, co
     assert ["Containerized instances may not be managed via the API"] == resp.data['policy_instance_minimum']
     resp = patch(cg_url, {'policy_instance_list':[instance.hostname]}, admin)
     assert ["Containerized instances may not be managed via the API"] == resp.data['policy_instance_list']
+
+
+@pytest.mark.django_db
+def test_containerized_group_default_fields(instance_group, kube_credential):
+    ig = InstanceGroup(name="test_policy_field_defaults")
+    ig.policy_instance_list = [1]
+    ig.policy_instance_minimum = 5
+    ig.policy_instance_percentage = 5
+    ig.save()
+    assert ig.policy_instance_list == [1]
+    assert ig.policy_instance_minimum == 5
+    assert ig.policy_instance_percentage == 5
+    ig.credential = kube_credential
+    ig.save()
+    assert ig.policy_instance_list == []
+    assert ig.policy_instance_minimum == 0
+    assert ig.policy_instance_percentage == 0
+    

awx/main/tests/functional/api/test_oauth.py

@@ -1,6 +1,8 @@
 import pytest
 import base64
+import contextlib
 import json
+from unittest import mock
 
 from django.db import connection
 from django.test.utils import override_settings
@@ -14,6 +16,18 @@ from awx.sso.models import UserEnterpriseAuth
 from oauth2_provider.models import RefreshToken
 
 
+@contextlib.contextmanager
+def immediate_on_commit():
+    """
+    Context manager executing transaction.on_commit() hooks immediately as
+    if the connection was in auto-commit mode.
+    """
+    def on_commit(func):
+        func()
+    with mock.patch('django.db.connection.on_commit', side_effect=on_commit) as patch:
+        yield patch
+
+
 @pytest.mark.django_db
 def test_personal_access_token_creation(oauth_application, post, alice):
     url = drf_reverse('api:oauth_authorization_root_view') + 'token/'
@@ -54,6 +68,41 @@ def test_token_creation_disabled_for_external_accounts(oauth_application, post,
             assert AccessToken.objects.count() == 0
 
 
+@pytest.mark.django_db
+def test_existing_token_enabled_for_external_accounts(oauth_application, get, post, admin):
+    UserEnterpriseAuth(user=admin, provider='radius').save()
+    url = drf_reverse('api:oauth_authorization_root_view') + 'token/'
+    with override_settings(RADIUS_SERVER='example.org', ALLOW_OAUTH2_FOR_EXTERNAL_USERS=True):
+        resp = post(
+            url,
+            data='grant_type=password&username=admin&password=admin&scope=read',
+            content_type='application/x-www-form-urlencoded',
+            HTTP_AUTHORIZATION='Basic ' + smart_str(base64.b64encode(smart_bytes(':'.join([
+                oauth_application.client_id, oauth_application.client_secret
+            ])))),
+            status=201
+        )
+        token = json.loads(resp.content)['access_token']
+        assert AccessToken.objects.count() == 1
+
+        with immediate_on_commit():
+            resp = get(
+                drf_reverse('api:user_me_list', kwargs={'version': 'v2'}),
+                HTTP_AUTHORIZATION='Bearer ' + token,
+                status=200
+            )
+            assert json.loads(resp.content)['results'][0]['username'] == 'admin'
+
+    with override_settings(RADIUS_SERVER='example.org', ALLOW_OAUTH2_FOR_EXTERNAL_USER=False):
+        with immediate_on_commit():
+            resp = get(
+                drf_reverse('api:user_me_list', kwargs={'version': 'v2'}),
+                HTTP_AUTHORIZATION='Bearer ' + token,
+                status=200
+            )
+            assert json.loads(resp.content)['results'][0]['username'] == 'admin'
+
+
 @pytest.mark.django_db
 def test_pat_creation_no_default_scope(oauth_application, post, admin):
     # tests that the default scope is overriden

awx/main/tests/functional/api/test_settings.py

@@ -5,6 +5,7 @@
 # Python
 import pytest
 import os
+import time
 
 from django.conf import settings
 from kombu.utils.url import parse_url
@@ -276,6 +277,7 @@ def test_logging_aggregrator_connection_test_valid(mocker, get, post, admin):
 def test_logging_aggregrator_connection_test_with_masked_password(mocker, patch, post, admin):
     url = reverse('api:setting_singleton_detail', kwargs={'category_slug': 'logging'})
     patch(url, user=admin, data={'LOG_AGGREGATOR_PASSWORD': 'password123'}, expect=200)
+    time.sleep(1)  # log settings are cached slightly
 
     with mock.patch.object(AWXProxyHandler, 'perform_test') as perform_test:
         url = reverse('api:setting_logging_test')

awx/main/tests/functional/api/test_survey_spec.py

@@ -219,6 +219,30 @@ def test_survey_spec_passwords_with_default_required(job_template_factory, post,
         assert launch_value not in json.loads(job.extra_vars).values()
 
 
+@pytest.mark.django_db
+def test_survey_spec_default_not_allowed(job_template, post, admin_user):
+    survey_input_data = {
+        'description': 'A survey',
+        'spec': [{
+            'question_name': 'You must choose wisely',
+            'variable': 'your_choice',
+            'default': 'blue',
+            'required': False,
+            'type': 'multiplechoice',
+            "choices": ["red", "green", "purple"]
+        }],
+        'name': 'my survey'
+    }
+    r = post(
+        url=reverse(
+            'api:job_template_survey_spec',
+            kwargs={'pk': job_template.id}
+        ),
+        data=survey_input_data, user=admin_user, expect=400
+    )
+    assert r.data['error'] == 'Default choice must be answered from the choices listed.'
+
+
 @pytest.mark.django_db
 @pytest.mark.parametrize('default, status', [
     ('SUPERSECRET', 200),

awx/main/tests/functional/commands/test_secret_key_regeneration.py

@@ -0,0 +1,173 @@
+import json
+
+from cryptography.fernet import InvalidToken
+from django.test.utils import override_settings
+from django.conf import settings
+import pytest
+
+from awx.main import models
+from awx.conf.models import Setting
+from awx.main.management.commands import regenerate_secret_key
+from awx.main.utils.encryption import encrypt_field, decrypt_field, encrypt_value
+
+
+PREFIX = '$encrypted$UTF8$AESCBC$'
+
+
+@pytest.mark.django_db
+class TestKeyRegeneration:
+
+    def test_encrypted_ssh_password(self, credential):
+        # test basic decryption
+        assert credential.inputs['password'].startswith(PREFIX)
+        assert credential.get_input('password') == 'secret'
+
+        # re-key the credential
+        new_key = regenerate_secret_key.Command().handle()
+        new_cred = models.Credential.objects.get(pk=credential.pk)
+        assert credential.inputs['password'] != new_cred.inputs['password']
+
+        # verify that the old SECRET_KEY doesn't work
+        with pytest.raises(InvalidToken):
+            new_cred.get_input('password')
+
+        # verify that the new SECRET_KEY *does* work
+        with override_settings(SECRET_KEY=new_key):
+            assert new_cred.get_input('password') == 'secret'
+
+    def test_encrypted_setting_values(self):
+        # test basic decryption
+        settings.REDHAT_PASSWORD = 'sensitive'
+        s = Setting.objects.filter(key='REDHAT_PASSWORD').first()
+        assert s.value.startswith(PREFIX)
+        assert settings.REDHAT_PASSWORD == 'sensitive'
+
+        # re-key the setting value
+        new_key = regenerate_secret_key.Command().handle()
+        new_setting = Setting.objects.filter(key='REDHAT_PASSWORD').first()
+        assert s.value != new_setting.value
+
+        # wipe out the local cache so the value is pulled from the DB again
+        settings.cache.delete('REDHAT_PASSWORD')
+
+        # verify that the old SECRET_KEY doesn't work
+        with pytest.raises(InvalidToken):
+            settings.REDHAT_PASSWORD
+
+        # verify that the new SECRET_KEY *does* work
+        with override_settings(SECRET_KEY=new_key):
+            assert settings.REDHAT_PASSWORD == 'sensitive'
+
+    def test_encrypted_notification_secrets(self, notification_template_with_encrypt):
+        # test basic decryption
+        nt = notification_template_with_encrypt
+        nc = nt.notification_configuration
+        assert nc['token'].startswith(PREFIX)
+
+        Slack = nt.CLASS_FOR_NOTIFICATION_TYPE[nt.notification_type]
+        class TestBackend(Slack):
+
+            def __init__(self, *args, **kw):
+                assert kw['token'] == 'token'
+
+            def send_messages(self, messages):
+                pass
+
+        nt.CLASS_FOR_NOTIFICATION_TYPE['test'] = TestBackend
+        nt.notification_type = 'test'
+        nt.send('Subject', 'Body')
+
+        # re-key the notification config
+        new_key = regenerate_secret_key.Command().handle()
+        new_nt = models.NotificationTemplate.objects.get(pk=nt.pk)
+        assert nt.notification_configuration['token'] != new_nt.notification_configuration['token']
+
+        # verify that the old SECRET_KEY doesn't work
+        with pytest.raises(InvalidToken):
+            new_nt.CLASS_FOR_NOTIFICATION_TYPE['test'] = TestBackend
+            new_nt.notification_type = 'test'
+            new_nt.send('Subject', 'Body')
+
+        # verify that the new SECRET_KEY *does* work
+        with override_settings(SECRET_KEY=new_key):
+            new_nt.send('Subject', 'Body')
+
+    def test_job_start_args(self, job_factory):
+        # test basic decryption
+        job = job_factory()
+        job.start_args = json.dumps({'foo': 'bar'})
+        job.start_args = encrypt_field(job, field_name='start_args')
+        job.save()
+        assert job.start_args.startswith(PREFIX)
+
+        # re-key the start_args
+        new_key = regenerate_secret_key.Command().handle()
+        new_job = models.Job.objects.get(pk=job.pk)
+        assert new_job.start_args != job.start_args
+
+        # verify that the old SECRET_KEY doesn't work
+        with pytest.raises(InvalidToken):
+            decrypt_field(new_job, field_name='start_args')
+
+        # verify that the new SECRET_KEY *does* work
+        with override_settings(SECRET_KEY=new_key):
+            assert json.loads(
+                decrypt_field(new_job, field_name='start_args')
+            ) == {'foo': 'bar'}
+
+    @pytest.mark.parametrize('cls', ('JobTemplate', 'WorkflowJobTemplate'))
+    def test_survey_spec(self, inventory, project, survey_spec_factory, cls):
+        params = {}
+        if cls == 'JobTemplate':
+            params['inventory'] = inventory
+            params['project'] = project
+        # test basic decryption
+        jt = getattr(models, cls).objects.create(
+            name='Example Template',
+            survey_spec=survey_spec_factory([{
+                'variable': 'secret_key',
+                'default': encrypt_value('donttell', pk=None),
+                'type': 'password'
+            }]),
+            survey_enabled=True,
+            **params
+        )
+        job = jt.create_unified_job()
+        assert jt.survey_spec['spec'][0]['default'].startswith(PREFIX)
+        assert job.survey_passwords == {'secret_key': '$encrypted$'}
+        assert json.loads(job.decrypted_extra_vars())['secret_key'] == 'donttell'
+
+        # re-key the extra_vars
+        new_key = regenerate_secret_key.Command().handle()
+        new_job = models.UnifiedJob.objects.get(pk=job.pk)
+        assert new_job.extra_vars != job.extra_vars
+
+        # verify that the old SECRET_KEY doesn't work
+        with pytest.raises(InvalidToken):
+            new_job.decrypted_extra_vars()
+
+        # verify that the new SECRET_KEY *does* work
+        with override_settings(SECRET_KEY=new_key):
+            assert json.loads(
+                new_job.decrypted_extra_vars()
+            )['secret_key'] == 'donttell'
+
+    def test_oauth2_application_client_secret(self, oauth_application):
+        # test basic decryption
+        secret = oauth_application.client_secret
+        assert len(secret) == 128
+
+        # re-key the client_secret
+        new_key = regenerate_secret_key.Command().handle()
+
+        # verify that the old SECRET_KEY doesn't work
+        with pytest.raises(InvalidToken):
+            models.OAuth2Application.objects.get(
+                pk=oauth_application.pk
+            ).client_secret
+
+        # verify that the new SECRET_KEY *does* work
+        with override_settings(SECRET_KEY=new_key):
+            assert models.OAuth2Application.objects.get(
+                pk=oauth_application.pk
+            ).client_secret == secret

awx/main/tests/functional/conftest.py

@@ -122,6 +122,22 @@ def project_playbooks():
     mocked.start()
 
 
+@pytest.fixture
+def run_computed_fields_right_away(request):
+
+    def run_me(inventory_id, should_update_hosts=True):
+        i = Inventory.objects.get(id=inventory_id)
+        i.update_computed_fields(update_hosts=should_update_hosts)
+
+    mocked = mock.patch(
+        'awx.main.signals.update_inventory_computed_fields.delay',
+        new=run_me
+    )
+    mocked.start()
+
+    request.addfinalizer(mocked.stop)
+
+
 @pytest.fixture
 @mock.patch.object(Project, "update", lambda self, **kwargs: None)
 def project(instance, organization):
@@ -280,14 +296,21 @@ def credentialtype_external():
         }],
         'required': ['url', 'token', 'key'],
     }
-    external_type = CredentialType(
-        kind='external',
-        managed_by_tower=True,
-        name='External Service',
-        inputs=external_type_inputs
-    )
-    external_type.save()
-    return external_type
+
+    class MockPlugin(object):
+        def backend(self, **kwargs):
+            return 'secret'
+
+    with mock.patch('awx.main.models.credential.CredentialType.plugin', new_callable=PropertyMock) as mock_plugin:
+        mock_plugin.return_value = MockPlugin()
+        external_type = CredentialType(
+            kind='external',
+            managed_by_tower=True,
+            name='External Service',
+            inputs=external_type_inputs
+        )
+        external_type.save()
+        yield external_type
 
 
 @pytest.fixture

awx/main/tests/functional/models/test_activity_stream.py

@@ -296,3 +296,15 @@ def test_cluster_node_long_node_name(inventory, project):
     # node name is very long, we just want to make sure it does not error
     entry = ActivityStream.objects.filter(job=job).first()
     assert entry.action_node.startswith('ffffff')
+
+
+@pytest.mark.django_db
+def test_credential_defaults_idempotency():
+    CredentialType.setup_tower_managed_defaults()
+    old_inputs = CredentialType.objects.get(name='Ansible Tower', kind='cloud').inputs
+    prior_count = ActivityStream.objects.count()
+    # this is commonly re-ran in migrations, and no changes should be shown
+    # because inputs and injectors are not actually tracked in the database
+    CredentialType.setup_tower_managed_defaults()
+    assert CredentialType.objects.get(name='Ansible Tower', kind='cloud').inputs == old_inputs
+    assert ActivityStream.objects.count() == prior_count

awx/main/tests/functional/models/test_events.py

@@ -1,18 +1,15 @@
 from unittest import mock
 import pytest
 
-from awx.main.models import (Job, JobEvent, ProjectUpdate, ProjectUpdateEvent,
-                             AdHocCommand, AdHocCommandEvent, InventoryUpdate,
-                             InventorySource, InventoryUpdateEvent, SystemJob,
-                             SystemJobEvent)
+from awx.main.models import Job, JobEvent
 
 
 @pytest.mark.django_db
-@mock.patch('awx.main.consumers.emit_channel_notification')
+@mock.patch('awx.main.models.events.emit_event_detail')
 def test_parent_changed(emit):
     j = Job()
     j.save()
-    JobEvent.create_from_data(job_id=j.pk, uuid='abc123', event='playbook_on_task_start')
+    JobEvent.create_from_data(job_id=j.pk, uuid='abc123', event='playbook_on_task_start').save()
     assert JobEvent.objects.count() == 1
     for e in JobEvent.objects.all():
         assert e.changed is False
@@ -24,19 +21,26 @@ def test_parent_changed(emit):
         event_data={
             'res': {'changed': ['localhost']}
         }
-    )
-    assert JobEvent.objects.count() == 2
-    for e in JobEvent.objects.all():
+    ).save()
+    # the `playbook_on_stats` event is where we update the parent changed linkage
+    JobEvent.create_from_data(
+        job_id=j.pk,
+        parent_uuid='abc123',
+        event='playbook_on_stats'
+    ).save()
+    events = JobEvent.objects.filter(event__in=['playbook_on_task_start', 'runner_on_ok'])
+    assert events.count() == 2
+    for e in events.all():
         assert e.changed is True
 
 
 @pytest.mark.django_db
 @pytest.mark.parametrize('event', JobEvent.FAILED_EVENTS)
-@mock.patch('awx.main.consumers.emit_channel_notification')
+@mock.patch('awx.main.models.events.emit_event_detail')
 def test_parent_failed(emit, event):
     j = Job()
     j.save()
-    JobEvent.create_from_data(job_id=j.pk, uuid='abc123', event='playbook_on_task_start')
+    JobEvent.create_from_data(job_id=j.pk, uuid='abc123', event='playbook_on_task_start').save()
     assert JobEvent.objects.count() == 1
     for e in JobEvent.objects.all():
         assert e.failed is False
@@ -45,69 +49,15 @@ def test_parent_failed(emit, event):
         job_id=j.pk,
         parent_uuid='abc123',
         event=event
-    )
-    assert JobEvent.objects.count() == 2
-    for e in JobEvent.objects.all():
+    ).save()
+
+    # the `playbook_on_stats` event is where we update the parent failed linkage
+    JobEvent.create_from_data(
+        job_id=j.pk,
+        parent_uuid='abc123',
+        event='playbook_on_stats'
+    ).save()
+    events = JobEvent.objects.filter(event__in=['playbook_on_task_start', event])
+    assert events.count() == 2
+    for e in events.all():
         assert e.failed is True
-
-
-@pytest.mark.django_db
-@mock.patch('awx.main.consumers.emit_channel_notification')
-def test_job_event_websocket_notifications(emit):
-    j = Job(id=123)
-    j.save()
-    JobEvent.create_from_data(job_id=j.pk)
-    assert len(emit.call_args_list) == 1
-    topic, payload = emit.call_args_list[0][0]
-    assert topic == 'job_events-123'
-    assert payload['job'] == 123
-
-
-@pytest.mark.django_db
-@mock.patch('awx.main.consumers.emit_channel_notification')
-def test_ad_hoc_event_websocket_notifications(emit):
-    ahc = AdHocCommand(id=123)
-    ahc.save()
-    AdHocCommandEvent.create_from_data(ad_hoc_command_id=ahc.pk)
-    assert len(emit.call_args_list) == 1
-    topic, payload = emit.call_args_list[0][0]
-    assert topic == 'ad_hoc_command_events-123'
-    assert payload['ad_hoc_command'] == 123
-
-
-@pytest.mark.django_db
-@mock.patch('awx.main.consumers.emit_channel_notification')
-def test_project_update_event_websocket_notifications(emit, project):
-    pu = ProjectUpdate(id=123, project=project)
-    pu.save()
-    ProjectUpdateEvent.create_from_data(project_update_id=pu.pk)
-    assert len(emit.call_args_list) == 1
-    topic, payload = emit.call_args_list[0][0]
-    assert topic == 'project_update_events-123'
-    assert payload['project_update'] == 123
-
-
-@pytest.mark.django_db
-@mock.patch('awx.main.consumers.emit_channel_notification')
-def test_inventory_update_event_websocket_notifications(emit, inventory):
-    source = InventorySource()
-    source.save()
-    iu = InventoryUpdate(id=123, inventory_source=source)
-    iu.save()
-    InventoryUpdateEvent.create_from_data(inventory_update_id=iu.pk)
-    assert len(emit.call_args_list) == 1
-    topic, payload = emit.call_args_list[0][0]
-    assert topic == 'inventory_update_events-123'
-    assert payload['inventory_update'] == 123
-
-
-@pytest.mark.django_db
-@mock.patch('awx.main.consumers.emit_channel_notification')
-def test_system_job_event_websocket_notifications(emit, inventory):
-    j = SystemJob(id=123)
-    j.save()
-    SystemJobEvent.create_from_data(system_job_id=j.pk)
-    assert len(emit.call_args_list) == 1
-    topic, payload = emit.call_args_list[0][0]
-    assert topic == 'system_job_events-123'
-    assert payload['system_job'] == 123

awx/main/tests/functional/models/test_notifications.py

@@ -88,6 +88,9 @@ class TestJobNotificationMixin(object):
                                  'verbosity': int},
                          'job_friendly_name': str,
                          'job_metadata': str,
+                         'approval_status': str,
+                         'approval_node_name': str,
+                         'workflow_url': str,
                          'url': str}
 
 

awx/main/tests/functional/models/test_unified_job.py

@@ -281,15 +281,18 @@ class TestTaskImpact:
             return job
         return r
 
-    def test_limit_task_impact(self, job_host_limit):
+    def test_limit_task_impact(self, job_host_limit, run_computed_fields_right_away):
         job = job_host_limit(5, 2)
+        job.inventory.refresh_from_db()  # FIXME: computed fields operates on reloaded inventory
+        assert job.inventory.total_hosts == 5
         assert job.task_impact == 2 + 1  # forks becomes constraint
 
-    def test_host_task_impact(self, job_host_limit):
+    def test_host_task_impact(self, job_host_limit, run_computed_fields_right_away):
         job = job_host_limit(3, 5)
+        job.inventory.refresh_from_db()  # FIXME: computed fields operates on reloaded inventory
         assert job.task_impact == 3 + 1  # hosts becomes constraint
 
-    def test_shard_task_impact(self, slice_job_factory):
+    def test_shard_task_impact(self, slice_job_factory, run_computed_fields_right_away):
         # factory creates on host per slice
         workflow_job = slice_job_factory(3, jt_kwargs={'forks': 50}, spawn=True)
         # arrange the jobs by their number
@@ -308,4 +311,5 @@ class TestTaskImpact:
             len(jobs[0].inventory.get_script_data(slice_number=i + 1, slice_count=3)['all']['hosts'])
             for i in range(3)
         ] == [2, 1, 1]
+        jobs[0].inventory.refresh_from_db()  # FIXME: computed fields operates on reloaded inventory
         assert [job.task_impact for job in jobs] == [3, 2, 2]

awx/main/tests/functional/task_management/test_scheduler.py

@@ -4,6 +4,7 @@ import json
 from datetime import timedelta
 
 from awx.main.scheduler import TaskManager
+from awx.main.scheduler.dependency_graph import DependencyGraph
 from awx.main.utils import encrypt_field
 from awx.main.models import WorkflowJobTemplate, JobTemplate
 
@@ -326,3 +327,59 @@ def test_shared_dependencies_launch(default_instance_group, job_template_factory
     iu = [x for x in ii.inventory_updates.all()]
     assert len(pu) == 1
     assert len(iu) == 1
+
+
+@pytest.mark.django_db
+def test_job_not_blocking_project_update(default_instance_group, job_template_factory):
+    objects = job_template_factory('jt', organization='org1', project='proj',
+                                   inventory='inv', credential='cred',
+                                   jobs=["job"])
+    job = objects.jobs["job"]
+    job.instance_group = default_instance_group
+    job.status = "running"
+    job.save()
+
+    with mock.patch("awx.main.scheduler.TaskManager.start_task"):
+        task_manager = TaskManager()
+        task_manager._schedule()
+
+        proj = objects.project
+        project_update = proj.create_project_update()
+        project_update.instance_group = default_instance_group
+        project_update.status = "pending"
+        project_update.save()
+        assert not task_manager.is_job_blocked(project_update)
+
+        dependency_graph = DependencyGraph(None)
+        dependency_graph.add_job(job)
+        assert not dependency_graph.is_job_blocked(project_update)
+
+
+@pytest.mark.django_db
+def test_job_not_blocking_inventory_update(default_instance_group, job_template_factory, inventory_source_factory):
+    objects = job_template_factory('jt', organization='org1', project='proj',
+                                   inventory='inv', credential='cred',
+                                   jobs=["job"])
+    job = objects.jobs["job"]
+    job.instance_group = default_instance_group
+    job.status = "running"
+    job.save()
+
+    with mock.patch("awx.main.scheduler.TaskManager.start_task"):
+        task_manager = TaskManager()
+        task_manager._schedule()
+
+        inv = objects.inventory
+        inv_source = inventory_source_factory("ec2")
+        inv_source.source = "ec2"
+        inv.inventory_sources.add(inv_source)
+        inventory_update = inv_source.create_inventory_update()
+        inventory_update.instance_group = default_instance_group
+        inventory_update.status = "pending"
+        inventory_update.save()
+
+        assert not task_manager.is_job_blocked(inventory_update)
+
+        dependency_graph = DependencyGraph(None)
+        dependency_graph.add_job(job)
+        assert not dependency_graph.is_job_blocked(inventory_update)

awx/main/tests/functional/test_inventory_source_injectors.py

@@ -54,7 +54,8 @@ INI_TEST_VARS = {
     },
     'azure_rm': {
         'use_private_ip': True,
-        'resource_groups': 'foo_resources,bar_resources'
+        'resource_groups': 'foo_resources,bar_resources',
+        'tags': 'Creator:jmarshall, peanutbutter:jelly'
     },
     'satellite6': {
         'satellite6_group_patterns': 'foo_group_patterns',

awx/main/tests/functional/test_inventory_source_migration.py

@@ -2,52 +2,10 @@ import pytest
 from unittest import mock
 
 from awx.main.migrations import _inventory_source as invsrc
-from awx.main.models import InventorySource
 
 from django.apps import apps
 
 
-@pytest.mark.django_db
-def test_inv_src_manual_removal(inventory_source):
-    inventory_source.source = ''
-    inventory_source.save()
-
-    assert InventorySource.objects.filter(pk=inventory_source.pk).exists()
-    invsrc.remove_manual_inventory_sources(apps, None)
-    assert not InventorySource.objects.filter(pk=inventory_source.pk).exists()
-
-
-@pytest.mark.django_db
-def test_rax_inv_src_removal(inventory_source):
-    inventory_source.source = 'rax'
-    inventory_source.save()
-
-    assert InventorySource.objects.filter(pk=inventory_source.pk).exists()
-    invsrc.remove_rax_inventory_sources(apps, None)
-    assert not InventorySource.objects.filter(pk=inventory_source.pk).exists()
-
-
-@pytest.mark.django_db
-def test_inv_src_rename(inventory_source_factory):
-    inv_src01 = inventory_source_factory('t1')
-
-    invsrc.rename_inventory_sources(apps, None)
-
-    inv_src01.refresh_from_db()
-    # inv-is-t1 is generated in the inventory_source_factory
-    assert inv_src01.name == 't1 - inv-is-t1 - 0'
-
-
-@pytest.mark.django_db
-def test_azure_inv_src_removal(inventory_source):
-    inventory_source.source = 'azure'
-    inventory_source.save()
-
-    assert InventorySource.objects.filter(pk=inventory_source.pk).exists()
-    invsrc.remove_azure_inventory_sources(apps, None)
-    assert not InventorySource.objects.filter(pk=inventory_source.pk).exists()
-
-
 @pytest.mark.parametrize('vars,id_var,result', [
     ({'foo': {'bar': '1234'}}, 'foo.bar', '1234'),
     ({'cat': 'meow'}, 'cat', 'meow'),

awx/main/tests/functional/test_scan_jobs_migration.py

@@ -1,100 +0,0 @@
-# -*- coding: utf-8 -*-
-
-# Copyright (c) 2017 Ansible, Inc.
-# All Rights Reserved.
-import pytest
-
-from django.apps import apps
-
-from awx.main.models.base import PERM_INVENTORY_SCAN, PERM_INVENTORY_DEPLOY
-from awx.main.models import (
-    JobTemplate,
-    Project,
-    Inventory,
-    Organization,
-)
-
-from awx.main.migrations._scan_jobs import _migrate_scan_job_templates
-
-
-@pytest.fixture
-def organizations():
-    return [Organization.objects.create(name=u"org-\xe9-{}".format(x)) for x in range(3)]
-
-
-@pytest.fixture
-def inventories(organizations):
-    return [Inventory.objects.create(name=u"inv-\xe9-{}".format(x), 
-                                     organization=organizations[x]) for x in range(3)]
-
-
-@pytest.fixture
-def job_templates_scan(inventories):
-    return [JobTemplate.objects.create(name=u"jt-\xe9-scan-{}".format(x), 
-                                       job_type=PERM_INVENTORY_SCAN,
-                                       inventory=inventories[x]) for x in range(3)]
-
-
-@pytest.fixture
-def job_templates_deploy(inventories):
-    return [JobTemplate.objects.create(name=u"jt-\xe9-deploy-{}".format(x), 
-                                       job_type=PERM_INVENTORY_DEPLOY,
-                                       inventory=inventories[x]) for x in range(3)]
-
-
-@pytest.fixture
-def project_custom(organizations):
-    return Project.objects.create(name=u"proj-\xe9-scan_custom", 
-                                  scm_url='https://giggity.com',
-                                  organization=organizations[0])
-
-
-@pytest.fixture
-def job_templates_custom_scan_project(project_custom):
-    return [JobTemplate.objects.create(name=u"jt-\xe9-scan-custom-{}".format(x), 
-                                       project=project_custom, 
-                                       job_type=PERM_INVENTORY_SCAN) for x in range(3)]
-
-
-@pytest.fixture
-def job_template_scan_no_org():
-    return JobTemplate.objects.create(name=u"jt-\xe9-scan-no-org",
-                                      job_type=PERM_INVENTORY_SCAN)
-
-
-@pytest.mark.django_db
-def test_scan_jobs_migration(job_templates_scan, job_templates_deploy, job_templates_custom_scan_project, project_custom, job_template_scan_no_org):
-    _migrate_scan_job_templates(apps)
-
-    # Ensure there are no scan job templates after the migration
-    assert 0 == JobTemplate.objects.filter(job_type=PERM_INVENTORY_SCAN).count()
-
-    # Ensure special No Organization proj created
-    # And No Organization project is associated with correct jt
-    proj = Project.objects.get(name="Tower Fact Scan - No Organization")
-    assert proj.id == JobTemplate.objects.get(id=job_template_scan_no_org.id).project.id
-
-    # Ensure per-org projects were created
-    projs = Project.objects.filter(name__startswith="Tower Fact Scan")
-    assert projs.count() == 4
-
-    # Ensure scan job templates with Tower project are migrated
-    for i, jt_old in enumerate(job_templates_scan):
-        jt = JobTemplate.objects.get(id=jt_old.id)
-        assert PERM_INVENTORY_DEPLOY == jt.job_type
-        assert jt.use_fact_cache is True
-        assert projs[i] == jt.project
-
-    # Ensure scan job templates with custom projects are migrated
-    for jt_old in job_templates_custom_scan_project:
-        jt = JobTemplate.objects.get(id=jt_old.id)
-        assert PERM_INVENTORY_DEPLOY == jt.job_type
-        assert jt.use_fact_cache is True
-        assert project_custom == jt.project
-
-    # Ensure other job template aren't touched
-    for jt_old in job_templates_deploy:
-        jt = JobTemplate.objects.get(id=jt_old.id)
-        assert PERM_INVENTORY_DEPLOY == jt.job_type
-        assert jt.project is None
-

awx/main/tests/unit/commands/test_replay_job_events.py

@@ -60,7 +60,7 @@ class TestReplayJobEvents():
         r.emit_job_status = lambda job, status: True
         return r
 
-    @mock.patch('awx.main.management.commands.replay_job_events.emit_channel_notification', lambda *a, **kw: None)
+    @mock.patch('awx.main.management.commands.replay_job_events.emit_event_detail', lambda *a, **kw: None)
     def test_sleep(self, mocker, replayer):
         replayer.run(3, 1)
 
@@ -74,7 +74,7 @@ class TestReplayJobEvents():
             mock.call(0.000001),
         ])
 
-    @mock.patch('awx.main.management.commands.replay_job_events.emit_channel_notification', lambda *a, **kw: None)
+    @mock.patch('awx.main.management.commands.replay_job_events.emit_event_detail', lambda *a, **kw: None)
     def test_speed(self, mocker, replayer):
         replayer.run(3, 2)
 

awx/main/tests/unit/models/test_events.py

@@ -1,6 +1,5 @@
 from datetime import datetime
 from django.utils.timezone import utc
-from unittest import mock
 import pytest
 
 from awx.main.models import (JobEvent, ProjectUpdateEvent, AdHocCommandEvent,
@@ -18,16 +17,11 @@ from awx.main.models import (JobEvent, ProjectUpdateEvent, AdHocCommandEvent,
     datetime(2018, 1, 1).isoformat(), datetime(2018, 1, 1)
 ])
 def test_event_parse_created(job_identifier, cls, created):
-    with mock.patch.object(cls, 'objects') as manager:
-        cls.create_from_data(**{
-            job_identifier: 123,
-            'created': created
-        })
-        expected_created = datetime(2018, 1, 1).replace(tzinfo=utc)
-        manager.create.assert_called_with(**{
-            job_identifier: 123,
-            'created': expected_created
-        })
+    event = cls.create_from_data(**{
+        job_identifier: 123,
+        'created': created
+    })
+    assert event.created == datetime(2018, 1, 1).replace(tzinfo=utc)
 
 
 @pytest.mark.parametrize('job_identifier, cls', [
@@ -38,24 +32,20 @@ def test_event_parse_created(job_identifier, cls, created):
     ['system_job_id', SystemJobEvent],
 ])
 def test_playbook_event_strip_invalid_keys(job_identifier, cls):
-    with mock.patch.object(cls, 'objects') as manager:
-        cls.create_from_data(**{
-            job_identifier: 123,
-            'extra_key': 'extra_value'
-        })
-        manager.create.assert_called_with(**{job_identifier: 123})
+    event = cls.create_from_data(**{
+        job_identifier: 123,
+        'extra_key': 'extra_value'
+    })
+    assert getattr(event, job_identifier) == 123
+    assert not hasattr(event, 'extra_key')
 
 
 @pytest.mark.parametrize('field', [
     'play', 'role', 'task', 'playbook'
 ])
 def test_really_long_event_fields(field):
-    with mock.patch.object(JobEvent, 'objects') as manager:
-        JobEvent.create_from_data(**{
-            'job_id': 123,
-            'event_data': {field: 'X' * 4096}
-        })
-        manager.create.assert_called_with(**{
-            'job_id': 123,
-            'event_data': {field: 'X' * 1023 + '…'}
-        })
+    event = JobEvent.create_from_data(**{
+        'job_id': 123,
+        'event_data': {field: 'X' * 4096}
+    })
+    assert event.event_data[field] == 'X' * 1023 + '…'

awx/main/tests/unit/models/test_system_jobs.py

@@ -12,7 +12,9 @@ from awx.main.models import SystemJobTemplate
     {"days": 13435},
 ])
 def test_valid__clean_extra_data_system_jobs(extra_data):
-    accepted, rejected, errors = SystemJobTemplate().accept_or_ignore_variables(extra_data)
+    accepted, rejected, errors = SystemJobTemplate(
+        job_type='cleanup_jobs'
+    ).accept_or_ignore_variables(extra_data)
     assert not rejected
     assert not errors
 
@@ -32,12 +34,14 @@ def test_valid__clean_extra_data_system_jobs(extra_data):
     {"days": "foobar"},
 ])
 def test_invalid__extra_data_system_jobs(extra_data):
-    accepted, rejected, errors = SystemJobTemplate().accept_or_ignore_variables(extra_data)
+    accepted, rejected, errors = SystemJobTemplate(
+        job_type='cleanup_jobs'
+    ).accept_or_ignore_variables(extra_data)
     assert str(errors['extra_vars'][0]) == u'days must be a positive integer.'
 
 
 def test_unallowed_system_job_data():
-    sjt = SystemJobTemplate()
+    sjt = SystemJobTemplate(job_type='cleanup_jobs')
     accepted, ignored, errors = sjt.accept_or_ignore_variables({
         'days': 34,
         'foobar': 'baz'
@@ -54,7 +58,7 @@ def test_reject_other_prommpts():
 
 
 def test_reject_some_accept_some():
-    sjt = SystemJobTemplate()
+    sjt = SystemJobTemplate(job_type='cleanup_jobs')
     accepted, ignored, errors = sjt._accept_or_ignore_job_kwargs(limit="", extra_vars={
         'days': 34,
         'foobar': 'baz'

awx/main/utils/common.py

@@ -36,11 +36,14 @@ from django.utils.encoding import smart_str
 from django.utils.text import slugify
 from django.apps import apps
 
+# AWX
+from awx.conf.license import get_license
+
 logger = logging.getLogger('awx.main.utils')
 
 __all__ = [
     'get_object_or_400', 'camelcase_to_underscore', 'underscore_to_camelcase', 'memoize',
-    'memoize_delete', 'get_ansible_version', 'get_ssh_version', 'get_licenser',
+    'memoize_delete', 'get_ansible_version', 'get_ssh_version', 'get_licenser', 'get_awx_http_client_headers',
     'get_awx_version', 'update_scm_url', 'get_type_for_model', 'get_model_for_type',
     'copy_model_by_class', 'region_sorting', 'copy_m2m_relationships',
     'prefetch_page_capabilities', 'to_python_boolean', 'ignore_inventory_computed_fields',
@@ -212,6 +215,19 @@ def get_awx_version():
         return __version__
 
 
+def get_awx_http_client_headers():
+    license = get_license(show_key=False).get('license_type', 'UNLICENSED')
+    headers = {
+        'Content-Type': 'application/json',
+        'User-Agent': '{} {} ({})'.format(
+            'AWX' if license == 'open' else 'Red Hat Ansible Tower',
+            get_awx_version(),
+            license
+        )
+    }
+    return headers
+
+
 class StubLicense(object):
 
     features = {
@@ -363,7 +379,12 @@ def get_allowed_fields(obj, serializer_mapping):
         'oauth2accesstoken': ['last_used'],
         'oauth2application': ['client_secret']
     }
-    field_blacklist = ACTIVITY_STREAM_FIELD_EXCLUSIONS.get(obj._meta.model_name, [])
+    model_name = obj._meta.model_name
+    field_blacklist = ACTIVITY_STREAM_FIELD_EXCLUSIONS.get(model_name, [])
+    # see definition of from_db for CredentialType
+    # injection logic of any managed types are incompatible with activity stream
+    if model_name == 'credentialtype' and obj.managed_by_tower and obj.namespace:
+        field_blacklist.extend(['inputs', 'injectors'])
     if field_blacklist:
         allowed_fields = [f for f in allowed_fields if f not in field_blacklist]
     return allowed_fields

awx/main/utils/encryption.py

@@ -1,3 +1,5 @@
+# -*- coding: utf-8 -*-
+
 import base64
 import hashlib
 import logging
@@ -35,7 +37,7 @@ class Fernet256(Fernet):
         self._backend = backend
 
 
-def get_encryption_key(field_name, pk=None):
+def get_encryption_key(field_name, pk=None, secret_key=None):
     '''
     Generate key for encrypted password based on field name,
     ``settings.SECRET_KEY``, and instance pk (if available).
@@ -46,19 +48,58 @@ def get_encryption_key(field_name, pk=None):
     '''
     from django.conf import settings
     h = hashlib.sha512()
-    h.update(smart_bytes(settings.SECRET_KEY))
+    h.update(smart_bytes(secret_key or settings.SECRET_KEY))
     if pk is not None:
         h.update(smart_bytes(str(pk)))
     h.update(smart_bytes(field_name))
     return base64.urlsafe_b64encode(h.digest())
 
 
-def encrypt_value(value, pk=None):
+def encrypt_value(value, pk=None, secret_key=None):
+    #
+    # ⚠️  D-D-D-DANGER ZONE ⚠️
+    #
+    # !!! BEFORE USING THIS FUNCTION PLEASE READ encrypt_field !!!
+    #
     TransientField = namedtuple('TransientField', ['pk', 'value'])
-    return encrypt_field(TransientField(pk=pk, value=value), 'value')
+    return encrypt_field(TransientField(pk=pk, value=value), 'value', secret_key=secret_key)
 
 
-def encrypt_field(instance, field_name, ask=False, subfield=None):
+def encrypt_field(instance, field_name, ask=False, subfield=None, secret_key=None):
+    #
+    # ⚠️  D-D-D-DANGER ZONE ⚠️
+    #
+    # !!! PLEASE READ BEFORE USING THIS FUNCTION ANYWHERE !!!
+    #
+    # You should know that this function is used in various places throughout
+    # AWX for symmetric encryption - generally it's used to encrypt sensitive
+    # values that we store in the AWX database (such as SSH private keys for
+    # credentials).
+    #
+    # If you're reading this function's code because you're thinking about
+    # using it to encrypt *something new*, please remember that AWX has
+    # official support for *regenerating* the SECRET_KEY (on which the
+    # symmetric key is based):
+    #
+    # $ awx-manage regenerate_secret_key
+    # $ setup.sh -k
+    #
+    # ...so you'll need to *also* add code to support the
+    # migration/re-encryption of these values (the code in question lives in
+    # `awx.main.management.commands.regenerate_secret_key`):
+    #
+    # For example, if you find that you're adding a new database column that is
+    # encrypted, in addition to calling `encrypt_field` in the appropriate
+    # places, you would also need to update the `awx-manage regenerate_secret_key`
+    # so that values are properly migrated when the SECRET_KEY changes.
+    #
+    # This process *generally* involves adding Python code to the
+    # `regenerate_secret_key` command, i.e.,
+    #
+    # 1.  Query the database for existing encrypted values on the appropriate object(s)
+    # 2.  Decrypting them using the *old* SECRET_KEY
+    # 3.  Storing newly encrypted values using the *newly generated* SECRET_KEY
+    #
     '''
     Return content of the given instance and field name encrypted.
     '''
@@ -76,7 +117,11 @@ def encrypt_field(instance, field_name, ask=False, subfield=None):
     value = smart_str(value)
     if not value or value.startswith('$encrypted$') or (ask and value == 'ASK'):
         return value
-    key = get_encryption_key(field_name, getattr(instance, 'pk', None))
+    key = get_encryption_key(
+        field_name,
+        getattr(instance, 'pk', None),
+        secret_key=secret_key
+    )
     f = Fernet256(key)
     encrypted = f.encrypt(smart_bytes(value))
     b64data = smart_str(base64.b64encode(encrypted))
@@ -99,7 +144,7 @@ def decrypt_value(encryption_key, value):
     return smart_str(value)
 
 
-def decrypt_field(instance, field_name, subfield=None):
+def decrypt_field(instance, field_name, subfield=None, secret_key=None):
     '''
     Return content of the given instance and field name decrypted.
     '''
@@ -115,7 +160,11 @@ def decrypt_field(instance, field_name, subfield=None):
     value = smart_str(value)
     if not value or not value.startswith('$encrypted$'):
         return value
-    key = get_encryption_key(field_name, getattr(instance, 'pk', None))
+    key = get_encryption_key(
+        field_name,
+        getattr(instance, 'pk', None),
+        secret_key=secret_key
+    )
 
     try:
         return smart_str(decrypt_value(key, value))

awx/main/views.py

@@ -4,7 +4,7 @@
 import json
 
 # Django
-from django.http import HttpResponse
+from django.http import HttpResponse, HttpResponseRedirect
 from django.shortcuts import render
 from django.utils.html import format_html
 from django.utils.translation import ugettext_lazy as _
@@ -97,3 +97,7 @@ def handle_csp_violation(request):
     logger = logging.getLogger('awx')
     logger.error(json.loads(request.body))
     return HttpResponse(content=None)
+
+
+def handle_login_redirect(request):
+    return HttpResponseRedirect("/#/login")

awx/playbooks/check_isolated.yml

@@ -18,8 +18,8 @@
         src: "{{src}}/artifacts/"
         dest: "{{src}}/artifacts/"
         mode: pull
-        delete: yes
-        recursive: yes
+        delete: true
+        recursive: true
       when: ansible_kubectl_config is not defined
 
     - name: Copy daemon log from the isolated host
@@ -34,9 +34,9 @@
         src: "{{src}}/artifacts/"
         dest: "{{src}}/artifacts/"
         mode: pull
-        delete: yes
-        recursive: yes
-        set_remote_user: no
+        delete: true
+        recursive: true
+        set_remote_user: false
         rsync_opts:
           - "--rsh=$RSH"
       environment:
@@ -49,7 +49,7 @@
         src: "{{src}}/daemon.log"
         dest: "{{src}}/daemon.log"
         mode: pull
-        set_remote_user: no
+        set_remote_user: false
         rsync_opts:
           - "--rsh=$RSH"
       environment:

awx/playbooks/clean_isolated.yml

@@ -12,14 +12,16 @@
 
     - name: cancel the job
       command: "ansible-runner stop {{private_data_dir}}"
-      ignore_errors: yes
+      ignore_errors: true
 
     - name: remove build artifacts
-      file: path="{{item}}" state=absent
+      file:
+        path: '{{item}}'
+        state: absent
       register: result
       with_items: "{{cleanup_dirs}}"
       until: result is succeeded
-      ignore_errors: yes
+      ignore_errors: true
       retries: 3
       delay: 5
 

awx/playbooks/project_update.yml

@@ -1,160 +1,173 @@
 ---
 # The following variables will be set by the runner of this playbook:
 # project_path: PROJECTS_DIR/_local_path_
-# scm_type: git|hg|svn|insights
 # scm_url: https://server/repo
 # insights_url: Insights service URL (from configuration)
 # scm_branch: branch/tag/revision (HEAD if unset)
 # scm_clean: true/false
-# scm_delete_on_update: true/false
-# scm_full_checkout: true (if for a job template run), false (if retrieving revision)
 # scm_username: username (only for svn/insights)
 # scm_password: password (only for svn/insights)
 # scm_accept_hostkey: true/false (only for git, set automatically)
 # scm_refspec: a refspec to fetch in addition to obtaining version
-# roles_enabled: Allow us to pull roles from a requirements.yml file
+# roles_enabled: Value of the global setting to enable roles downloading
+# collections_enabled: Value of the global setting to enable collections downloading
 # roles_destination: Path to save roles from galaxy to
+# collections_destination: Path to save collections from galaxy to
 # awx_version: Current running version of the awx or tower as a string
 # awx_license_type: "open" for AWX; else presume Tower
 
-- hosts: all
+- hosts: localhost
   gather_facts: false
+  connection: local
+  name: Update source tree if necessary
   tasks:
 
     - name: delete project directory before update
       file:
         path: "{{project_path|quote}}"
         state: absent
-      when: scm_delete_on_update|default('')
-      delegate_to: localhost
+      tags:
+        - delete
 
     - block:
-      - name: update project using git
-        git:
-          dest: "{{project_path|quote}}"
-          repo: "{{scm_url}}"
-          version: "{{scm_branch|quote}}"
-          refspec: "{{scm_refspec|default(omit)}}"
-          force: "{{scm_clean}}"
-          accept_hostkey: "{{scm_accept_hostkey|default(omit)}}"
-        register: git_result
+        - name: update project using git
+          git:
+            dest: "{{project_path|quote}}"
+            repo: "{{scm_url}}"
+            version: "{{scm_branch|quote}}"
+            refspec: "{{scm_refspec|default(omit)}}"
+            force: "{{scm_clean}}"
+            accept_hostkey: "{{scm_accept_hostkey|default(omit)}}"
+          register: git_result
 
-      - name: Set the git repository version
-        set_fact:
-          scm_version: "{{ git_result['after'] }}"
-        when: "'after' in git_result"
-      when: scm_type == 'git'
-      delegate_to: localhost
+        - name: Set the git repository version
+          set_fact:
+            scm_version: "{{ git_result['after'] }}"
+          when: "'after' in git_result"
+      tags:
+        - update_git
 
     - block:
-      - name: update project using hg
-        hg:
-          dest: "{{project_path|quote}}"
-          repo: "{{scm_url|quote}}"
-          revision: "{{scm_branch|quote}}"
-          force: "{{scm_clean}}"
-        register: hg_result
+        - name: update project using hg
+          hg:
+            dest: "{{project_path|quote}}"
+            repo: "{{scm_url|quote}}"
+            revision: "{{scm_branch|quote}}"
+            force: "{{scm_clean}}"
+          register: hg_result
 
-      - name: Set the hg repository version
-        set_fact:
-          scm_version: "{{ hg_result['after'] }}"
-        when: "'after' in hg_result"
+        - name: Set the hg repository version
+          set_fact:
+            scm_version: "{{ hg_result['after'] }}"
+          when: "'after' in hg_result"
 
-      - name: parse hg version string properly
-        set_fact:
-          scm_version: "{{scm_version|regex_replace('^([A-Za-z0-9]+).*$', '\\1')}}"
-      when: scm_type == 'hg'
-      delegate_to: localhost
+        - name: parse hg version string properly
+          set_fact:
+            scm_version: "{{scm_version|regex_replace('^([A-Za-z0-9]+).*$', '\\1')}}"
+      tags:
+        - update_hg
 
     - block:
-      - name: update project using svn
-        subversion:
-          dest: "{{project_path|quote}}"
-          repo: "{{scm_url|quote}}"
-          revision: "{{scm_branch|quote}}"
-          force: "{{scm_clean}}"
-          username: "{{scm_username|default(omit)}}"
-          password: "{{scm_password|default(omit)}}"
-        environment:
-          LC_ALL: 'en_US.UTF-8'
-        register: svn_result
+        - name: update project using svn
+          subversion:
+            dest: "{{project_path|quote}}"
+            repo: "{{scm_url|quote}}"
+            revision: "{{scm_branch|quote}}"
+            force: "{{scm_clean}}"
+            username: "{{scm_username|default(omit)}}"
+            password: "{{scm_password|default(omit)}}"
+          environment:
+            LC_ALL: 'en_US.UTF-8'
+          register: svn_result
 
-      - name: Set the svn repository version
-        set_fact:
-          scm_version: "{{ svn_result['after'] }}"
-        when: "'after' in svn_result"
+        - name: Set the svn repository version
+          set_fact:
+            scm_version: "{{ svn_result['after'] }}"
+          when: "'after' in svn_result"
 
-      - name: parse subversion version string properly
-        set_fact:
-          scm_version: "{{scm_version|regex_replace('^.*Revision: ([0-9]+).*$', '\\1')}}"
-      when: scm_type == 'svn'
-      delegate_to: localhost
+        - name: parse subversion version string properly
+          set_fact:
+            scm_version: "{{scm_version|regex_replace('^.*Revision: ([0-9]+).*$', '\\1')}}"
+      tags:
+        - update_svn
 
     - block:
-      - name: Ensure the project directory is present
-        file:
-          dest: "{{project_path|quote}}"
-          state: directory
+        - name: Ensure the project directory is present
+          file:
+            dest: "{{project_path|quote}}"
+            state: directory
 
-      - name: Fetch Insights Playbook(s)
-        insights:
-          insights_url: "{{insights_url}}"
-          username: "{{scm_username}}"
-          password: "{{scm_password}}"
-          project_path: "{{project_path}}"
-          awx_license_type: "{{awx_license_type}}"
-          awx_version: "{{awx_version}}"
-        register: results
-
-      - name: Save Insights Version
-        set_fact:
-          scm_version: "{{results.version}}"
-        when: results is defined
-      when: scm_type == 'insights'
-      delegate_to: localhost
+        - name: Fetch Insights Playbook(s)
+          insights:
+            insights_url: "{{insights_url}}"
+            username: "{{scm_username}}"
+            password: "{{scm_password}}"
+            project_path: "{{project_path}}"
+            awx_license_type: "{{awx_license_type}}"
+            awx_version: "{{awx_version}}"
+          register: results
 
+        - name: Save Insights Version
+          set_fact:
+            scm_version: "{{results.version}}"
+          when: results is defined
+      tags:
+        - update_insights
 
     - name: Repository Version
-      debug: msg="Repository Version {{ scm_version }}"
-      when: scm_version is defined
+      debug:
+        msg: "Repository Version {{ scm_version }}"
+      tags:
+        - update_git
+        - update_hg
+        - update_svn
+        - update_insights
 
-- hosts: all
+- hosts: localhost
   gather_facts: false
+  connection: local
+  name: Install content with ansible-galaxy command if necessary
   tasks:
 
     - block:
-      - name: detect requirements.yml
-        stat: path={{project_path|quote}}/roles/requirements.yml
-        register: doesRequirementsExist
+        - name: detect requirements.yml
+          stat:
+            path: '{{project_path|quote}}/roles/requirements.yml'
+          register: doesRequirementsExist
 
-      - name: fetch galaxy roles from requirements.yml
-        command: ansible-galaxy install -r requirements.yml -p {{roles_destination|quote}}{{ ' -' + 'v' * ansible_verbosity if ansible_verbosity else '' }}
-        args:
-          chdir: "{{project_path|quote}}/roles"
-        register: galaxy_result
-        when: doesRequirementsExist.stat.exists
-        changed_when: "'was installed successfully' in galaxy_result.stdout"
-        environment:
-          ANSIBLE_FORCE_COLOR: False
+        - name: fetch galaxy roles from requirements.yml
+          command: ansible-galaxy install -r requirements.yml -p {{roles_destination|quote}}{{ ' -' + 'v' * ansible_verbosity if ansible_verbosity else '' }}
+          args:
+            chdir: "{{project_path|quote}}/roles"
+          register: galaxy_result
+          when: doesRequirementsExist.stat.exists
+          changed_when: "'was installed successfully' in galaxy_result.stdout"
+          environment:
+            ANSIBLE_FORCE_COLOR: false
 
       when: roles_enabled|bool
-      delegate_to: localhost
+      tags:
+        - install_roles
 
     - block:
-      - name: detect collections/requirements.yml
-        stat: path={{project_path|quote}}/collections/requirements.yml
-        register: doesCollectionRequirementsExist
+        - name: detect collections/requirements.yml
+          stat:
+            path: '{{project_path|quote}}/collections/requirements.yml'
+          register: doesCollectionRequirementsExist
 
-      - name: fetch galaxy collections from collections/requirements.yml
-        command: ansible-galaxy collection install -r requirements.yml -p {{collections_destination|quote}}{{ ' -' + 'v' * ansible_verbosity if ansible_verbosity else '' }}
-        args:
-          chdir: "{{project_path|quote}}/collections"
-        register: galaxy_collection_result
-        when: doesCollectionRequirementsExist.stat.exists
-        changed_when: "'Installing ' in galaxy_collection_result.stdout"
-        environment:
-          ANSIBLE_FORCE_COLOR: False
-          ANSIBLE_COLLECTIONS_PATHS: "{{ collections_destination }}"
+        - name: fetch galaxy collections from collections/requirements.yml
+          command: ansible-galaxy collection install -r requirements.yml -p {{collections_destination|quote}}{{ ' -' + 'v' * ansible_verbosity if ansible_verbosity else '' }}
+          args:
+            chdir: "{{project_path|quote}}/collections"
+          register: galaxy_collection_result
+          when: doesCollectionRequirementsExist.stat.exists
+          changed_when: "'Installing ' in galaxy_collection_result.stdout"
+          environment:
+            ANSIBLE_FORCE_COLOR: false
+            ANSIBLE_COLLECTIONS_PATHS: "{{ collections_destination }}"
 
-      when: collections_enabled|bool
-      delegate_to: localhost
+      when:
+        - "ansible_version.full is version_compare('2.8', '>=')"
+        - collections_enabled|bool
+      tags:
+        - install_collections

awx/playbooks/run_isolated.yml

@@ -13,17 +13,17 @@
   tasks:
     - name: synchronize job environment with isolated host
       synchronize:
-        copy_links: yes
+        copy_links: true
         src: "{{ src }}"
         dest: "{{ dest }}"
       when: ansible_kubectl_config is not defined
 
     - name: synchronize job environment with remote job container
       synchronize:
-        copy_links: yes
+        copy_links: true
         src: "{{ src }}"
         dest: "{{ dest }}"
-        set_remote_user: no
+        set_remote_user: false
         rsync_opts:
           - "--rsh=$RSH"
       environment:
@@ -51,4 +51,4 @@
         content: "{{secret}}"
         path: "{{src}}/env/ssh_key"
       when: key.stat.exists
-      no_log: True
+      no_log: true

awx/playbooks/scan_facts.yml

@@ -1,3 +1,4 @@
+---
 - hosts: all
   vars:
     scan_use_checksum: false
@@ -33,4 +34,3 @@
         get_checksum: '{{ scan_use_checksum }}'
         recursive: '{{ scan_use_recursive }}'
       when: scan_file_paths is defined and ansible_os_family == "Windows"
-

awx/plugins/inventory/foreman.ini.example

@@ -123,6 +123,12 @@ user = foreman
 password = secret
 ssl_verify = True
 
+# Foreman 1.24 introduces a new reports API to improve performance of the inventory script.
+# Note: This requires foreman_ansible plugin installed.
+# Set to False if you want to use the old API. Defaults to True.
+
+use_reports_api = True
+
 # Retrieve only hosts from the organization "Web Engineering".
 # host_filters = organization="Web Engineering"
 
@@ -130,15 +136,42 @@ ssl_verify = True
 # also in the host collection "Apache Servers".
 # host_filters = organization="Web Engineering" and host_collection="Apache Servers"
 
+
+# Foreman Inventory report related configuration options.
+# Configs that default to True :
+# want_organization , want_location, want_ipv4, want_host_group, want_subnet, want_smart_proxies, want_facts
+# Configs that default to False :
+# want_ipv6, want_subnet_v6, want_content_facet_attributes, want_host_params
+
+[report]
+want_organization = True
+want_location = True
+want_ipv4 = True
+want_ipv6 = False
+want_host_group = True
+want_subnet = True
+want_subnet_v6 = False
+want_smart_proxies = True
+want_content_facet_attributes = False
+want_host_params = False
+
+# use this config to determine if facts are to be fetched in the report and stored on the hosts.
+# want_facts = False
+
+# Upon receiving a request to return inventory report, Foreman schedules a report generation job.
+# The script then polls the report_data endpoint repeatedly to check if the job is complete and retrieves data  
+# poll_interval allows to define the polling interval between 2 calls to the report_data endpoint while polling.
+# Defaults to 10 seconds
+
+poll_interval = 10
+
 [ansible]
 group_patterns = ["{app}-{tier}-{color}",
 	          "{app}-{color}",
 	          "{app}",
 		  "{tier}"]
-group_prefix = foreman_
 
-# Whether to fetch facts from Foreman and store them on the host
-want_facts = True
+group_prefix = foreman_
 
 # Whether to create Ansible groups for host collections. Only tested
 # with Katello (Red Hat Satellite). Disabled by default to not break

awx/plugins/inventory/foreman.py

@@ -34,7 +34,7 @@ import copy
 import os
 import re
 import sys
-from time import time
+from time import time, sleep
 from collections import defaultdict
 from distutils.version import LooseVersion, StrictVersion
 
@@ -92,6 +92,72 @@ class ForemanInventory(object):
             print("Error parsing configuration: %s" % e, file=sys.stderr)
             return False
 
+        # Inventory Report Related
+        try:
+            self.foreman_use_reports_api = config.getboolean('foreman', 'use_reports_api')
+        except (ConfigParser.NoOptionError, ConfigParser.NoSectionError):
+            self.foreman_use_reports_api = True
+
+        try:
+            self.want_organization = config.getboolean('report', 'want_organization')
+        except (ConfigParser.NoOptionError, ConfigParser.NoSectionError):
+            self.want_organization = True
+
+        try:
+            self.want_location = config.getboolean('report', 'want_location')
+        except (ConfigParser.NoOptionError, ConfigParser.NoSectionError):
+            self.want_location = True
+
+        try:
+            self.want_IPv4 = config.getboolean('report', 'want_ipv4')
+        except (ConfigParser.NoOptionError, ConfigParser.NoSectionError):
+            self.want_IPv4 = True
+
+        try:
+            self.want_IPv6 = config.getboolean('report', 'want_ipv6')
+        except (ConfigParser.NoOptionError, ConfigParser.NoSectionError):
+            self.want_IPv6 = False
+
+        try:
+            self.want_host_group = config.getboolean('report', 'want_host_group')
+        except (ConfigParser.NoOptionError, ConfigParser.NoSectionError):
+            self.want_host_group = True
+
+        try:
+            self.want_host_params = config.getboolean('report', 'want_host_params')
+        except (ConfigParser.NoOptionError, ConfigParser.NoSectionError):
+            self.want_host_params = False
+
+        try:
+            self.want_subnet = config.getboolean('report', 'want_subnet')
+        except (ConfigParser.NoOptionError, ConfigParser.NoSectionError):
+            self.want_subnet = True
+
+        try:
+            self.want_subnet_v6 = config.getboolean('report', 'want_subnet_v6')
+        except (ConfigParser.NoOptionError, ConfigParser.NoSectionError):
+            self.want_subnet_v6 = False
+
+        try:
+            self.want_smart_proxies = config.getboolean('report', 'want_smart_proxies')
+        except (ConfigParser.NoOptionError, ConfigParser.NoSectionError):
+            self.want_smart_proxies = True
+
+        try:
+            self.want_content_facet_attributes = config.getboolean('report', 'want_content_facet_attributes')
+        except (ConfigParser.NoOptionError, ConfigParser.NoSectionError):
+            self.want_content_facet_attributes = False
+
+        try:
+            self.report_want_facts = config.getboolean('report', 'want_facts')
+        except (ConfigParser.NoOptionError, ConfigParser.NoSectionError):
+            self.report_want_facts = True
+
+        try:
+            self.poll_interval = config.getint('report', 'poll_interval')
+        except (ConfigParser.NoOptionError, ConfigParser.NoSectionError):
+            self.poll_interval = 10
+
         # Ansible related
         try:
             group_patterns = config.get('ansible', 'group_patterns')
@@ -110,6 +176,8 @@ class ForemanInventory(object):
         except (ConfigParser.NoOptionError, ConfigParser.NoSectionError):
             self.want_facts = True
 
+        self.want_facts = self.want_facts and self.report_want_facts
+
         try:
             self.want_hostcollections = config.getboolean('ansible', 'want_hostcollections')
         except (ConfigParser.NoOptionError, ConfigParser.NoSectionError):
@@ -203,6 +271,52 @@ class ForemanInventory(object):
                 break
         return results
 
+    def _use_inventory_report(self):
+        if not self.foreman_use_reports_api:
+            return False
+        status_url = "%s/api/v2/status" % self.foreman_url
+        result = self._get_json(status_url)
+        foreman_version = (LooseVersion(result.get('version')) >= LooseVersion('1.24.0'))
+        return foreman_version
+
+    def _fetch_params(self):
+        options, params = ("no", "yes"), dict()
+        params["Organization"] = options[self.want_organization]
+        params["Location"] = options[self.want_location]
+        params["IPv4"] = options[self.want_IPv4]
+        params["IPv6"] = options[self.want_IPv6]
+        params["Facts"] = options[self.want_facts]
+        params["Host Group"] = options[self.want_host_group]
+        params["Host Collections"] = options[self.want_hostcollections]
+        params["Subnet"] = options[self.want_subnet]
+        params["Subnet v6"] = options[self.want_subnet_v6]
+        params["Smart Proxies"] = options[self.want_smart_proxies]
+        params["Content Attributes"] = options[self.want_content_facet_attributes]
+        params["Host Parameters"] = options[self.want_host_params]
+        if self.host_filters:
+            params["Hosts"] = self.host_filters
+        return params
+
+    def _post_request(self):
+        url = "%s/ansible/api/v2/ansible_inventories/schedule" % self.foreman_url
+        session = self._get_session()
+        params = {'input_values': self._fetch_params()}
+        ret = session.post(url, json=params)
+        if not ret:
+            raise Exception("Error scheduling inventory report on foreman. Please check foreman logs!")
+        url = "{0}/{1}".format(self.foreman_url, ret.json().get('data_url'))
+        response = session.get(url)
+        while response:
+            if response.status_code != 204:
+                break
+            else:
+                sleep(self.poll_interval)
+                response = session.get(url)
+        if not response:
+            raise Exception("Error receiving inventory report from foreman. Please check foreman logs!")
+        else:
+            return response.json()
+
     def _get_hosts(self):
         url = "%s/api/v2/hosts" % self.foreman_url
 
@@ -276,6 +390,97 @@ class ForemanInventory(object):
 
     def update_cache(self, scan_only_new_hosts=False):
         """Make calls to foreman and save the output in a cache"""
+        use_inventory_report = self._use_inventory_report()
+        if use_inventory_report:
+            self._update_cache_inventory(scan_only_new_hosts)
+        else:
+            self._update_cache_host_api(scan_only_new_hosts)
+
+    def _update_cache_inventory(self, scan_only_new_hosts):
+        self.groups = dict()
+        self.hosts = dict()
+        try:
+            inventory_report_response = self._post_request()
+        except Exception:
+            self._update_cache_host_api(scan_only_new_hosts)
+            return
+        host_data = json.loads(inventory_report_response)
+        for host in host_data:
+            if not(host) or (host["name"] in self.cache.keys() and scan_only_new_hosts):
+                continue
+            dns_name = host['name']
+
+            host_params = host.pop('host_parameters', {})
+            fact_list = host.pop('facts', {})
+            content_facet_attributes = host.get('content_attributes', {}) or {}
+
+            # Create ansible groups for hostgroup
+            group = 'host_group'
+            val = host.get(group)
+            if val:
+                safe_key = self.to_safe('%s%s_%s' % (
+                    to_text(self.group_prefix),
+                    group,
+                    to_text(val).lower()
+                ))
+                self.inventory[safe_key].append(dns_name)
+
+            # Create ansible groups for environment, location and organization
+            for group in ['environment', 'location', 'organization']:
+                val = host.get('%s' % group)
+                if val:
+                    safe_key = self.to_safe('%s%s_%s' % (
+                        to_text(self.group_prefix),
+                        group,
+                        to_text(val).lower()
+                    ))
+                    self.inventory[safe_key].append(dns_name)
+
+            for group in ['lifecycle_environment', 'content_view']:
+                val = content_facet_attributes.get('%s_name' % group)
+                if val:
+                    safe_key = self.to_safe('%s%s_%s' % (
+                        to_text(self.group_prefix),
+                        group,
+                        to_text(val).lower()
+                    ))
+                    self.inventory[safe_key].append(dns_name)
+
+            params = host_params
+
+            # Ansible groups by parameters in host groups and Foreman host
+            # attributes.
+            groupby = dict()
+            for k, v in params.items():
+                groupby[k] = self.to_safe(to_text(v))
+
+            # The name of the ansible groups is given by group_patterns:
+            for pattern in self.group_patterns:
+                try:
+                    key = pattern.format(**groupby)
+                    self.inventory[key].append(dns_name)
+                except KeyError:
+                    pass  # Host not part of this group
+
+            if self.want_hostcollections:
+                hostcollections = host.get('host_collections')
+
+                if hostcollections:
+                    # Create Ansible groups for host collections
+                    for hostcollection in hostcollections:
+                        safe_key = self.to_safe('%shostcollection_%s' % (self.group_prefix, hostcollection.lower()))
+                        self.inventory[safe_key].append(dns_name)
+
+                self.hostcollections[dns_name] = hostcollections
+
+            self.cache[dns_name] = host
+            self.params[dns_name] = params
+            self.facts[dns_name] = fact_list
+            self.inventory['all'].append(dns_name)
+        self._write_cache()
+
+    def _update_cache_host_api(self, scan_only_new_hosts):
+        """Make calls to foreman and save the output in a cache"""
 
         self.groups = dict()
         self.hosts = dict()

awx/plugins/inventory/openstack.yml

@@ -1,3 +1,4 @@
+---
 clouds:
   vexxhost:
     profile: vexxhost
@@ -19,6 +20,6 @@ clouds:
       password: stack
       project_name: stack
 ansible:
-  use_hostnames: True
-  expand_hostvars: False
-  fail_on_errors: True
+  use_hostnames: true
+  expand_hostvars: false
+  fail_on_errors: true

awx/settings/defaults.py

@@ -310,6 +310,9 @@ REST_FRAMEWORK = {
     'VIEW_DESCRIPTION_FUNCTION': 'awx.api.generics.get_view_description',
     'NON_FIELD_ERRORS_KEY': '__all__',
     'DEFAULT_VERSION': 'v2',
+    # For swagger schema generation
+    # see https://github.com/encode/django-rest-framework/pull/6532
+    'DEFAULT_SCHEMA_CLASS': 'rest_framework.schemas.AutoSchema',
     #'URL_FORMAT_OVERRIDE': None,
 }
 
@@ -373,6 +376,10 @@ TACACSPLUS_AUTH_PROTOCOL = 'ascii'
 # Note: This setting may be overridden by database settings.
 AUTH_BASIC_ENABLED = True
 
+# If set, specifies a URL that unauthenticated users will be redirected to
+# when trying to access a UI page that requries authentication.
+LOGIN_REDIRECT_OVERRIDE = ''
+
 # If set, serve only minified JS for UI.
 USE_MINIFIED_JS = False
 
@@ -569,9 +576,6 @@ ANSIBLE_INVENTORY_UNPARSED_FAILED = True
 # Additional environment variables to be passed to the ansible subprocesses
 AWX_TASK_ENV = {}
 
-# Flag to enable/disable updating hosts M2M when saving job events.
-CAPTURE_JOB_EVENT_HOSTS = False
-
 # Rebuild Host Smart Inventory memberships.
 AWX_REBUILD_SMART_MEMBERSHIP = False
 
@@ -605,6 +609,9 @@ PUBLIC_GALAXY_SERVER = {
     'url': 'https://galaxy.ansible.com'
 }
 
+# Applies to any galaxy server
+GALAXY_IGNORE_CERTS = False
+
 # List of dicts of fallback (additional) Galaxy servers.  If configured, these
 # will be higher precedence than public Galaxy, but lower than primary Galaxy.
 # Available options: 'id', 'url', 'username', 'password', 'token', 'auth_url'
@@ -1201,6 +1208,19 @@ SILENCED_SYSTEM_CHECKS = ['models.E006']
 # Use middleware to get request statistics
 AWX_REQUEST_PROFILE = False
 
+#
+# Optionally, AWX can generate DOT graphs
+# (http://www.graphviz.org/doc/info/lang.html) for per-request profiling
+# via gprof2dot (https://github.com/jrfonseca/gprof2dot)
+#
+# If you set this to True, you must `/var/lib/awx/venv/awx/bin/pip install gprof2dot`
+# .dot files will be saved in `/var/log/tower/profile/` and can be converted e.g.,
+#
+# ~ yum install graphviz
+# ~ dot -o profile.png -Tpng /var/log/tower/profile/some-profile-data.dot
+#
+AWX_REQUEST_PROFILE_WITH_DOT = False
+
 # Delete temporary directories created to store playbook run-time
 AWX_CLEANUP_PATHS = True
 

awx/settings/development.py

@@ -52,6 +52,9 @@ COLOR_LOGS = True
 # Pipe management playbook output to console
 LOGGING['loggers']['awx.isolated.manager.playbooks']['propagate'] = True  # noqa
 
+# celery is annoyingly loud when docker containers start
+LOGGING['loggers'].pop('celery', None)  # noqa
+
 ALLOWED_HOSTS = ['*']
 
 mimetypes.add_type("image/svg+xml", ".svg", True)