Merge pull request #7176 from nixocio/ui_issue_5819

Add support Prompt on Launch for Workflow Job Template

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul

.gitignore

@@ -31,9 +31,11 @@ awx/ui/templates/ui/installing.html
 awx/ui_next/node_modules/
 awx/ui_next/coverage/
 awx/ui_next/build/locales/_build
+rsyslog.pid
 /tower-license
 /tower-license/**
 tools/prometheus/data
+tools/docker-compose/Dockerfile
 
 # Tower setup playbook testing
 setup/test/roles/postgresql

CHANGELOG.md

@@ -2,6 +2,74 @@
 
 This is a list of high-level changes for each release of AWX. A full list of commits can be found at `https://github.com/ansible/awx/releases/tag/<version>`.
 
+## 12.0.0 (Jun 9, 2020)
+- Removed memcached as a dependency of AWX (https://github.com/ansible/awx/pull/7240) 
+- Moved to a single container image build instead of separate awx_web and awx_task images. The container image is just `awx` (https://github.com/ansible/awx/pull/7228)
+- Official AWX container image builds now use a two-stage container build process that notably reduces the size of our published images (https://github.com/ansible/awx/pull/7017)
+- Removed support for HipChat notifications ([EoL announcement](https://www.atlassian.com/partnerships/slack/faq#faq-98b17ca3-247f-423b-9a78-70a91681eff0)); all previously-created HipChat notification templates will be deleted due to this removal.
+- Fixed a bug which broke AWX installations with oc version 4.3 (https://github.com/ansible/awx/pull/6948/files)
+- Fixed a performance issue that caused notable delay of stdout processing for playbooks run against large numbers of hosts (https://github.com/ansible/awx/issues/6991)
+- Fixed a bug that caused CyberArk AIM credential plugin looks to hang forever in some environments (https://github.com/ansible/awx/issues/6986)
+- Fixed a bug that caused ANY/ALL converage settings not to properly save when editing approval nodes in the UI (https://github.com/ansible/awx/issues/6998)
+- Fixed a bug that broke support for the satellite6_group_prefix source variable (https://github.com/ansible/awx/issues/7031)
+- Fixed a bug that prevented changes to workflow node convergence settings when approval nodes were in use (https://github.com/ansible/awx/issues/7063)
+- Fixed a bug that caused notifications to fail on newer version of Mattermost (https://github.com/ansible/awx/issues/7264)
+- Fixed a bug (by upgrading to 0.8.1 of the foreman collection) that prevented host_filters from working properly with Foreman-based inventory (https://github.com/ansible/awx/issues/7225)
+- Fixed a bug that prevented the usage of the Conjur credential plugin with secrets that contain spaces (https://github.com/ansible/awx/issues/7191)
+- Fixed a bug in awx-manage run_wsbroadcast --status in kubernetes (https://github.com/ansible/awx/pull/7009)
+- Fixed a bug that broke notification toggles for system jobs in the UI (https://github.com/ansible/awx/pull/7042)
+- Fixed a bug that broke local pip installs of awxkit (https://github.com/ansible/awx/issues/7107)
+- Fixed a bug that prevented PagerDuty notifications from sending for workflow job template approvals (https://github.com/ansible/awx/issues/7094)
+- Fixed a bug that broke external log aggregation support for URL paths that include the = character (such as the tokens for SumoLogic) (https://github.com/ansible/awx/issues/7139)
+- Fixed a bug that prevented organization admins from removing labels from workflow job templates (https://github.com/ansible/awx/pull/7143)
+
+## 11.2.0 (Apr 29, 2020)
+
+- Inventory updates now use collection-based plugins by default (in Ansible 2.9+):
+    - amazon.aws.aws_ec2
+    - community.vmware.vmware_vm_inventory
+    - azure.azcollection.azure_rm
+    - google.cloud.gcp_compute
+    - theforeman.foreman.foreman
+    - openstack.cloud.openstack
+    - ovirt.ovirt_collection.ovirt
+    - awx.awx.tower
+- Added support for Approle and LDAP/AD mechanisms to the Hashicorp Vault credential plugin (https://github.com/ansible/awx/issues/5076)
+- Added Project (Domain Name) support for the OpenStack Keystone v3 API (https://github.com/ansible/awx/issues/6831)
+- Added a new setting for raising log verbosity for rsyslogd (https://github.com/ansible/awx/pull/6818)
+- Added the ability to monitor stdout in the CLI for running jobs and workflow jobs (https://github.com/ansible/awx/issues/6165)
+- Fixed a bug which prevented the AWX CLI from properly installing with newer versions of pip (https://github.com/ansible/awx/issues/6870)
+- Fixed a bug which broke AWX's external logging support when configured with HTTPS endpoints that utilize self-signed certificates (https://github.com/ansible/awx/issues/6851)
+- Fixed a local docker installer bug that mistakenly attempted to upgrade PostgreSQL when an external pg_hostname is specified (https://github.com/ansible/awx/pull/5398)
+- Fixed a race condition that caused task container crashes when pods are quickly brought down and back up (https://github.com/ansible/awx/issues/6750)
+- Fixed a bug that caused 404 errors when attempting to view the second page of the workflow approvals view (https://github.com/ansible/awx/issues/6803)
+- Fixed a bug that prevented the use of ANSIBLE_SSH_ARGS for ad-hoc-commands (https://github.com/ansible/awx/pull/6811)
+- Fixed a bug that broke AWX installs/upgrades on Red Hat OpenShift (https://github.com/ansible/awx/issues/6791)
+
+
+## 11.1.0 (Apr 22, 2020)
+- Changed rsyslogd to persist queued events to disk (to prevent a risk of out-of-memory errors) (https://github.com/ansible/awx/issues/6746)
+- Added the ability to configure the destination and maximum disk size of rsyslogd spool (in the event of a log aggregator outage) (https://github.com/ansible/awx/pull/6763)
+- Added the ability to discover playbooks in project clones from symlinked directories (https://github.com/ansible/awx/pull/6773)
+- Fixed a bug that caused certain log aggregator settings to break logging integration (https://github.com/ansible/awx/issues/6760)
+- Fixed a bug that caused playbook execution in container groups to sometimes unexpectedly deadlock (https://github.com/ansible/awx/issues/6692)
+- Improved stability of the new redis clustering implementation (https://github.com/ansible/awx/pull/6739 https://github.com/ansible/awx/pull/6720)
+- Improved stability of the new rsyslogd-based logging implementation (https://github.com/ansible/awx/pull/6796)
+
+## 11.0.0 (Apr 16, 2020)
+- As of AWX 11.0.0, Kubernetes-based deployments use a Deployment rather than a StatefulSet.
+- Reimplemented external logging support using rsyslogd to improve reliability and address a number of issues (https://github.com/ansible/awx/issues/5155)
+- Changed activity stream logs to include summary fields for related objects (https://github.com/ansible/awx/issues/1761)
+- Added code to more gracefully attempt to reconnect to redis if it restarts/becomes unavailable (https://github.com/ansible/awx/pull/6670)
+- Fixed a bug that caused REFRESH_TOKEN_EXPIRE_SECONDS to not properly be respected for OAuth2.0 refresh tokens generated by AWX (https://github.com/ansible/awx/issues/6630)
+- Fixed a bug that broke schedules containing RRULES with very old DTSTART dates (https://github.com/ansible/awx/pull/6550)
+- Fixed a bug that broke installs on older versions of Ansible packaged with certain Linux distributions (https://github.com/ansible/awx/issues/5501)
+- Fixed a bug that caused the activity stream to sometimes report the incorrect actor when associating user membership on SAML login (https://github.com/ansible/awx/pull/6525)
+- Fixed a bug in AWX's Grafana notification support when annotation tags are omitted (https://github.com/ansible/awx/issues/6580)
+- Fixed a bug that prevented some users from searching for Source Control credentials in the AWX user interface (https://github.com/ansible/awx/issues/6600)
+- Fixed a bug that prevented disassociating orphaned users from credentials (https://github.com/ansible/awx/pull/6554)
+- Updated Twisted to address CVE-2020-10108 and CVE-2020-10109.
+
 ## 10.0.0 (Mar 30, 2020)
 - As of AWX 10.0.0, the official AWX CLI no longer supports Python 2 (it requires at least Python 3.6) (https://github.com/ansible/awx/pull/6327)
 - AWX no longer relies on RabbitMQ; Redis is added as a new dependency (https://github.com/ansible/awx/issues/5443)
@@ -56,7 +124,7 @@ This is a list of high-level changes for each release of AWX. A full list of com
 - Updated the bundled version of openstacksdk to address a known issue https://github.com/ansible/awx/issues/5821
 - Updated the bundled vmware_inventory plugin to the latest version to address a bug https://github.com/ansible/awx/pull/5668
 - Fixed a bug that can cause inventory updates to fail to properly save their output when run within a workflow https://github.com/ansible/awx/pull/5666
-- Removed a number of pre-computed fields from the Host and Group models to improve AWX performance.  As part of this change, inventory group UIs throughout the interface no longer display status icons https://github.com/ansible/awx/pull/5448 
+- Removed a number of pre-computed fields from the Host and Group models to improve AWX performance.  As part of this change, inventory group UIs throughout the interface no longer display status icons https://github.com/ansible/awx/pull/5448
 
 ## 9.1.1 (Jan 14, 2020)
 
@@ -95,7 +163,7 @@ This is a list of high-level changes for each release of AWX. A full list of com
 - Fixed a bug in the CLI which incorrectly parsed launch time arguments for `awx job_templates launch` and `awx workflow_job_templates launch` (https://github.com/ansible/awx/issues/5093).
 - Fixed a bug that caused inventory updates using "sourced from a project" to stop working (https://github.com/ansible/awx/issues/4750).
 - Fixed a bug that caused Slack notifications to sometimes show the wrong bot avatar (https://github.com/ansible/awx/pull/5125).
-- Fixed a bug that prevented the use of digits in Tower's URL settings (https://github.com/ansible/awx/issues/5081).
+- Fixed a bug that prevented the use of digits in AWX's URL settings (https://github.com/ansible/awx/issues/5081).
 
 ## 8.0.0 (Oct 21, 2019)
 

CONTRIBUTING.md

@@ -157,8 +157,7 @@ If you start a second terminal session, you can take a look at the running conta
 $ docker ps
 CONTAINER ID        IMAGE                                              COMMAND                  CREATED             STATUS              PORTS                                                                                                                                                              NAMES
 44251b476f98        gcr.io/ansible-tower-engineering/awx_devel:devel   "/entrypoint.sh /bin…"   27 seconds ago      Up 23 seconds       0.0.0.0:6899->6899/tcp, 0.0.0.0:7899-7999->7899-7999/tcp, 0.0.0.0:8013->8013/tcp, 0.0.0.0:8043->8043/tcp, 0.0.0.0:8080->8080/tcp, 22/tcp, 0.0.0.0:8888->8888/tcp   tools_awx_run_9e820694d57e
-b049a43817b4        memcached:alpine                                   "docker-entrypoint.s…"   28 seconds ago      Up 26 seconds       0.0.0.0:11211->11211/tcp                                                                                                                                           tools_memcached_1
-40de380e3c2e        redis:latest                                       "docker-entrypoint.s…"   28 seconds ago      Up 26 seconds       0.0.0.0:6379->6379/tcp                                                                                                                                             tools_redis_1
+40de380e3c2e        redis:latest                                       "docker-entrypoint.s…"   28 seconds ago      Up 26 seconds
 b66a506d3007        postgres:10                                        "docker-entrypoint.s…"   28 seconds ago      Up 26 seconds       0.0.0.0:5432->5432/tcp                                                                                                                                             tools_postgres_1
 ```
 **NOTE**
@@ -215,18 +214,23 @@ Using `docker exec`, this will create a session in the running *awx* container,
 If you want to start and use the development environment, you'll first need to bootstrap it by running the following command:
 
 ```bash
-(container)# /bootstrap_development.sh
+(container)# /usr/bin/bootstrap_development.sh
 ```
 
-The above will do all the setup tasks, including running database migrations, so it may take a couple minutes.
+The above will do all the setup tasks, including running database migrations, so it may take a couple minutes. Once it's done it
+will drop you back to the shell.
 
-Now you can start each service individually, or start all services in a pre-configured tmux session like so:
+In order to launch all developer services:
 
 ```bash
-(container)# cd /awx_devel
-(container)# make server
+(container)# /usr/bin/launch_awx.sh
 ```
 
+`launch_awx.sh` also calls `bootstrap_development.sh` so if all you are doing is launching the supervisor to start all services, you don't
+need to call `bootstrap_development.sh` first.
+
+
+
 ### Post Build Steps
 
 Before you can log in and use the system, you will need to create an admin user. Optionally, you may also want to load some demo data.

INSTALL.md

@@ -10,7 +10,6 @@ This document provides a guide for installing AWX.
     + [AWX branding](#awx-branding)
     + [Prerequisites](#prerequisites)
     + [System Requirements](#system-requirements)
-    + [AWX Tunables](#awx-tunables)
     + [Choose a deployment platform](#choose-a-deployment-platform)
     + [Official vs Building Images](#official-vs-building-images)
   * [Upgrading from previous versions](#upgrading-from-previous-versions)
@@ -49,7 +48,17 @@ This document provides a guide for installing AWX.
 
 ### Clone the repo
 
-If you have not already done so, you will need to clone, or create a local copy, of the [AWX repo](https://github.com/ansible/awx). For more on how to clone the repo, view [git clone help](https://git-scm.com/docs/git-clone).
+If you have not already done so, you will need to clone, or create a local copy, of the [AWX repo](https://github.com/ansible/awx). We generally recommend that you view the releases page:
+
+https://github.com/ansible/awx/releases
+
+...and clone the latest stable release, e.g.,
+
+`git clone -b x.y.z https://github.com/ansible/awx.git`
+
+Please note that deploying from `HEAD` (or the latest commit) is **not** stable, and that if you want to do this, you should proceed at your own risk (also, see the section #official-vs-building-images for building your own image).
+
+For more on how to clone the repo, view [git clone help](https://git-scm.com/docs/git-clone).
 
 Once you have a local copy, run commands within the root of the project tree.
 
@@ -71,8 +80,11 @@ Before you can run a deployment, you'll need the following installed in your loc
     + We use this module instead of `docker-py` because it is what the `docker-compose` Python module requires.
 - [GNU Make](https://www.gnu.org/software/make/)
 - [Git](https://git-scm.com/) Requires Version 1.8.4+
+- Python 3.6+
 - [Node 10.x LTS version](https://nodejs.org/en/download/)
+    + This is only required if you're [building your own container images](#official-vs-building-images) with `use_container_for_build=false`
 - [NPM 6.x LTS](https://docs.npmjs.com/)
+    + This is only required if you're [building your own container images](#official-vs-building-images) with `use_container_for_build=false`
 
 ### System Requirements
 
@@ -82,11 +94,7 @@ The system that runs the AWX service will need to satisfy the following requirem
 - At least 2 cpu cores
 - At least 20GB of space
 - Running Docker, Openshift, or Kubernetes
-- If you choose to use an external PostgreSQL database, please note that the minimum version is 9.6+.
-
-### AWX Tunables
-
-**TODO** add tunable bits
+- If you choose to use an external PostgreSQL database, please note that the minimum version is 10+.
 
 ### Choose a deployment platform
 
@@ -101,7 +109,7 @@ In the sections below, you'll find deployment details and instructions for each
 
 ### Official vs Building Images
 
-When installing AWX you have the option of building your own images or using the images provided on DockerHub (see [awx_web](https://hub.docker.com/r/ansible/awx_web/) and [awx_task](https://hub.docker.com/r/ansible/awx_task/))
+When installing AWX you have the option of building your own image or using the image provided on DockerHub (see [awx](https://hub.docker.com/r/ansible/awx/))
 
 This is controlled by the following variables in the `inventory` file
 
@@ -114,12 +122,16 @@ If these variables are present then all deployments will use these hosted images
 
 *dockerhub_base*
 
-> The base location on DockerHub where the images are hosted (by default this pulls container images named `ansible/awx_web:tag` and `ansible/awx_task:tag`)
+> The base location on DockerHub where the images are hosted (by default this pulls a container image named `ansible/awx:tag`)
 
 *dockerhub_version*
 
 > Multiple versions are provided. `latest` always pulls the most recent. You may also select version numbers at different granularities: 1, 1.0, 1.0.1, 1.0.0.123
 
+*use_container_for_build*
+
+> Use a local distribution build container image for building the AWX package. This is helpful if you don't want to bother installing the build-time dependencies as it is taken care of already.
+
 
 ## Upgrading from previous versions
 
@@ -143,7 +155,7 @@ $ ansible-playbook -i inventory install.yml -e @vars.yml
 
 ### Prerequisites
 
-To complete a deployment to OpenShift, you will obviously need access to an OpenShift cluster. For demo and testing purposes, you can use [Minishift](https://github.com/minishift/minishift) to create a single node cluster running inside a virtual machine.
+To complete a deployment to OpenShift, you will need access to an OpenShift cluster. For demo and testing purposes, you can use [Minishift](https://github.com/minishift/minishift) to create a single node cluster running inside a virtual machine.
 
 When using OpenShift for deploying AWX make sure you have correct privileges to add the security context 'privileged', otherwise the installation will fail. The privileged context is needed because of the use of [the bubblewrap tool](https://github.com/containers/bubblewrap) to add an additional layer of security when using containers.
 
@@ -469,15 +481,15 @@ Before starting the install process, review the [inventory](./installer/inventor
 
 *host_port*
 
-> Provide a port number that can be mapped from the Docker daemon host to the web server running inside the AWX container. Defaults to *80*.
+> Provide a port number that can be mapped from the Docker daemon host to the web server running inside the AWX container. If undefined no port will be exposed. Defaults to *80*.
 
 *host_port_ssl*
 
-> Provide a port number that can be mapped from the Docker daemon host to the web server running inside the AWX container for SSL support. Defaults to *443*, only works if you also set `ssl_certificate` (see below).
+> Provide a port number that can be mapped from the Docker daemon host to the web server running inside the AWX container for SSL support. If undefined no port will be exposed. Defaults to *443*, only works if you also set `ssl_certificate` (see below).
 
 *ssl_certificate*
 
-> Optionally, provide the path to a file that contains a certificate and its private key.
+> Optionally, provide the path to a file that contains a certificate and its private key. This needs to be a .pem-file
 
 *docker_compose_dir*
 

MANIFEST.in

@@ -10,6 +10,7 @@ recursive-include awx/playbooks *.yml
 recursive-include awx/lib/site-packages *
 recursive-include awx/plugins *.ps1
 recursive-include requirements *.txt
+recursive-include requirements *.yml
 recursive-include config *
 recursive-include docs/licenses *
 recursive-exclude awx devonly.py*

Makefile

@@ -6,26 +6,27 @@ PACKER ?= packer
 PACKER_BUILD_OPTS ?= -var 'official=$(OFFICIAL)' -var 'aw_repo_url=$(AW_REPO_URL)'
 NODE ?= node
 NPM_BIN ?= npm
+CHROMIUM_BIN=/tmp/chrome-linux/chrome
 DEPS_SCRIPT ?= packaging/bundle/deps.py
 GIT_BRANCH ?= $(shell git rev-parse --abbrev-ref HEAD)
 MANAGEMENT_COMMAND ?= awx-manage
 IMAGE_REPOSITORY_AUTH ?=
 IMAGE_REPOSITORY_BASE ?= https://gcr.io
 VERSION := $(shell cat VERSION)
+PYCURL_SSL_LIBRARY ?= openssl
 
 # NOTE: This defaults the container image version to the branch that's active
 COMPOSE_TAG ?= $(GIT_BRANCH)
 COMPOSE_HOST ?= $(shell hostname)
 
 VENV_BASE ?= /venv
-COLLECTION_VENV ?= /awx_devel/awx_collection_test_venv
 SCL_PREFIX ?=
 CELERY_SCHEDULE_FILE ?= /var/lib/awx/beat.db
 
 DEV_DOCKER_TAG_BASE ?= gcr.io/ansible-tower-engineering
 # Python packages to install only from source (not from binary wheels)
 # Comma separated list
-SRC_ONLY_PKGS ?= cffi,pycparser,psycopg2,twilio
+SRC_ONLY_PKGS ?= cffi,pycparser,psycopg2,twilio,pycurl
 # These should be upgraded in the AWX and Ansible venv before attempting
 # to install the actual requirements
 VENV_BOOTSTRAP ?= pip==19.3.1 setuptools==41.6.0
@@ -174,9 +175,9 @@ virtualenv_awx:
 # --ignore-install flag is not used because *.txt files should specify exact versions
 requirements_ansible: virtualenv_ansible
 	if [[ "$(PIP_OPTIONS)" == *"--no-index"* ]]; then \
-	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_local.txt | $(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) -r /dev/stdin ; \
+	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_local.txt | PYCURL_SSL_LIBRARY=$(PYCURL_SSL_LIBRARY) $(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) -r /dev/stdin ; \
 	else \
-	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_git.txt | $(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) -r /dev/stdin ; \
+	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_git.txt | PYCURL_SSL_LIBRARY=$(PYCURL_SSL_LIBRARY) $(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) -r /dev/stdin ; \
 	fi
 	$(VENV_BASE)/ansible/bin/pip uninstall --yes -r requirements/requirements_ansible_uninstall.txt
 	# Same effect as using --system-site-packages flag on venv creation
@@ -184,9 +185,9 @@ requirements_ansible: virtualenv_ansible
 
 requirements_ansible_py3: virtualenv_ansible_py3
 	if [[ "$(PIP_OPTIONS)" == *"--no-index"* ]]; then \
-	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_local.txt | $(VENV_BASE)/ansible/bin/pip3 install $(PIP_OPTIONS) -r /dev/stdin ; \
+	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_local.txt | PYCURL_SSL_LIBRARY=$(PYCURL_SSL_LIBRARY) $(VENV_BASE)/ansible/bin/pip3 install $(PIP_OPTIONS) -r /dev/stdin ; \
 	else \
-	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_git.txt | $(VENV_BASE)/ansible/bin/pip3 install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) -r /dev/stdin ; \
+	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_git.txt | PYCURL_SSL_LIBRARY=$(PYCURL_SSL_LIBRARY) $(VENV_BASE)/ansible/bin/pip3 install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) -r /dev/stdin ; \
 	fi
 	$(VENV_BASE)/ansible/bin/pip3 uninstall --yes -r requirements/requirements_ansible_uninstall.txt
 	# Same effect as using --system-site-packages flag on venv creation
@@ -210,7 +211,11 @@ requirements_awx: virtualenv_awx
 requirements_awx_dev:
 	$(VENV_BASE)/awx/bin/pip install -r requirements/requirements_dev.txt
 
-requirements: requirements_ansible requirements_awx
+requirements_collections:
+	mkdir -p $(COLLECTION_BASE)
+	ansible-galaxy collection install -r requirements/collections_requirements.yml -p $(COLLECTION_BASE)
+
+requirements: requirements_ansible requirements_awx requirements_collections
 
 requirements_dev: requirements_awx requirements_ansible_py3 requirements_awx_dev requirements_ansible_dev
 
@@ -350,8 +355,7 @@ swagger: reports
 check: flake8 pep8 # pyflakes pylint
 
 awx-link:
-	cp -R /tmp/awx.egg-info /awx_devel/ || true
-	sed -i "s/placeholder/$(shell cat VERSION)/" /awx_devel/awx.egg-info/PKG-INFO
+	[ -d "/awx_devel/awx.egg-info" ] || python3 /awx_devel/setup.py egg_info_dev
 	cp -f /tmp/awx.egg-link /venv/awx/lib/python$(PYTHON_VERSION)/site-packages/awx.egg-link
 
 TEST_DIRS ?= awx/main/tests/unit awx/main/tests/functional awx/conf/tests awx/sso/tests
@@ -362,14 +366,10 @@ test:
 		. $(VENV_BASE)/awx/bin/activate; \
 	fi; \
 	PYTHONDONTWRITEBYTECODE=1 py.test -p no:cacheprovider -n auto $(TEST_DIRS)
+	cmp VERSION awxkit/VERSION || "VERSION and awxkit/VERSION *must* match"
 	cd awxkit && $(VENV_BASE)/awx/bin/tox -re py2,py3
 	awx-manage check_migrations --dry-run --check  -n 'vNNN_missing_migration_file'
 
-prepare_collection_venv:
-	rm -rf $(COLLECTION_VENV)
-	mkdir $(COLLECTION_VENV)
-	$(VENV_BASE)/awx/bin/pip install --target=$(COLLECTION_VENV) git+https://github.com/ansible/tower-cli.git
-
 COLLECTION_TEST_DIRS ?= awx_collection/test/awx
 COLLECTION_TEST_TARGET ?=
 COLLECTION_PACKAGE ?= awx
@@ -380,12 +380,16 @@ test_collection:
 	@if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
 	fi; \
-	PYTHONPATH=$(COLLECTION_VENV):$PYTHONPATH:/usr/lib/python3.6/site-packages py.test $(COLLECTION_TEST_DIRS)
+	PYTHONPATH=$(PYTHONPATH):$(VENV_BASE)/awx/lib/python3.6/site-packages:/usr/lib/python3.6/site-packages py.test $(COLLECTION_TEST_DIRS)
+	# The python path needs to be modified so that the tests can find Ansible within the container
+	# First we will use anything expility set as PYTHONPATH
+	# Second we will load any libraries out of the virtualenv (if it's unspecified that should be ok because python should not load out of an empty directory)
+	# Finally we will add the system path so that the tests can find the ansible libraries
 
 flake8_collection:
 	flake8 awx_collection/  # Different settings, in main exclude list
 
-test_collection_all: prepare_collection_venv test_collection flake8_collection
+test_collection_all: test_collection flake8_collection
 
 # WARNING: symlinking a collection is fundamentally unstable
 # this is for rapid development iteration with playbooks, do not use with other test targets
@@ -395,7 +399,7 @@ symlink_collection:
 	ln -s $(shell pwd)/awx_collection $(COLLECTION_INSTALL)
 
 build_collection:
-	ansible-playbook -i localhost, awx_collection/template_galaxy.yml -e collection_package=$(COLLECTION_PACKAGE) -e collection_namespace=$(COLLECTION_NAMESPACE) -e collection_version=$(VERSION)
+	ansible-playbook -i localhost, awx_collection/tools/template_galaxy.yml -e collection_package=$(COLLECTION_PACKAGE) -e collection_namespace=$(COLLECTION_NAMESPACE) -e collection_version=$(VERSION) -e '{"awx_template_version":false}'
 	ansible-galaxy collection build awx_collection --force --output-path=awx_collection
 
 install_collection: build_collection
@@ -549,11 +553,13 @@ jshint: $(UI_DEPS_FLAG_FILE)
 	$(NPM_BIN) run --prefix awx/ui jshint
 	$(NPM_BIN) run --prefix awx/ui lint
 
-ui-zuul-lint-and-test: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) run --prefix awx/ui jshint
-	$(NPM_BIN) run --prefix awx/ui lint
-	$(NPM_BIN) --prefix awx/ui run test:ci
-	$(NPM_BIN) --prefix awx/ui run unit
+ui-zuul-lint-and-test:
+	CHROMIUM_BIN=$(CHROMIUM_BIN) ./awx/ui/build/zuul_download_chromium.sh
+	CHROMIUM_BIN=$(CHROMIUM_BIN) PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=1 $(NPM_BIN) --unsafe-perm --prefix awx/ui ci --no-save awx/ui
+	CHROMIUM_BIN=$(CHROMIUM_BIN) $(NPM_BIN) run --prefix awx/ui jshint
+	CHROMIUM_BIN=$(CHROMIUM_BIN) $(NPM_BIN) run --prefix awx/ui lint
+	CHROME_BIN=$(CHROMIUM_BIN) $(NPM_BIN) --prefix awx/ui run test:ci
+	CHROME_BIN=$(CHROMIUM_BIN) $(NPM_BIN) --prefix awx/ui run unit
 
 # END UI TASKS
 # --------------------------------------
@@ -642,7 +648,7 @@ docker-compose-runtest: awx/projects
 	cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm --service-ports awx /start_tests.sh
 
 docker-compose-build-swagger: awx/projects
-	cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm --service-ports awx /start_tests.sh swagger
+	cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm --service-ports --no-deps awx /start_tests.sh swagger
 
 detect-schema-change: genschema
 	curl https://s3.amazonaws.com/awx-public-ci-files/schema.json -o reference-schema.json
@@ -650,29 +656,28 @@ detect-schema-change: genschema
 	diff -u -b reference-schema.json schema.json
 
 docker-compose-clean: awx/projects
-	cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm -w /awx_devel --service-ports awx make clean
 	cd tools && TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose rm -sf
 
-docker-compose-build: awx-devel-build
-
 # Base development image build
-awx-devel-build:
+docker-compose-build:
+	ansible localhost -m template -a "src=installer/roles/image_build/templates/Dockerfile.j2 dest=tools/docker-compose/Dockerfile" -e build_dev=True
 	docker build -t ansible/awx_devel -f tools/docker-compose/Dockerfile \
 		--cache-from=$(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG) .
 	docker tag ansible/awx_devel $(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG)
 	#docker push $(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG)
 
 # For use when developing on "isolated" AWX deployments
-docker-compose-isolated-build: awx-devel-build
+docker-compose-isolated-build: docker-compose-build
 	docker build -t ansible/awx_isolated -f tools/docker-isolated/Dockerfile .
 	docker tag ansible/awx_isolated $(DEV_DOCKER_TAG_BASE)/awx_isolated:$(COMPOSE_TAG)
 	#docker push $(DEV_DOCKER_TAG_BASE)/awx_isolated:$(COMPOSE_TAG)
 
-MACHINE?=default
 docker-clean:
-	eval $$(docker-machine env $(MACHINE))
 	$(foreach container_id,$(shell docker ps -f name=tools_awx -aq),docker stop $(container_id); docker rm -f $(container_id);)
-	-docker images | grep "awx_devel" | awk '{print $$1 ":" $$2}' | xargs docker rmi
+	docker images | grep "awx_devel" | awk '{print $$1 ":" $$2}' | xargs docker rmi
+
+docker-clean-volumes: docker-compose-clean
+	docker volume rm tools_awx_db
 
 docker-refresh: docker-clean docker-compose
 
@@ -686,9 +691,6 @@ docker-compose-cluster-elk: docker-auth awx/projects
 prometheus:
 	docker run -u0 --net=tools_default --link=`docker ps | egrep -o "tools_awx(_run)?_([^ ]+)?"`:awxweb --volume `pwd`/tools/prometheus:/prometheus --name prometheus -d -p 0.0.0.0:9090:9090 prom/prometheus --web.enable-lifecycle --config.file=/prometheus/prometheus.yml
 
-minishift-dev:
-	ansible-playbook -i localhost, -e devtree_directory=$(CURDIR) tools/clusterdevel/start_minishift_dev.yml
-
 clean-elk:
 	docker stop tools_kibana_1
 	docker stop tools_logstash_1

VERSION

@@ -1 +1 @@
-10.0.0
+12.0.0

awx/__init__.py

@@ -30,6 +30,7 @@ except ImportError:
     HAS_DJANGO = False
 else:
     from django.db.backends.base import schema
+    from django.db.models import indexes
     from django.db.backends.utils import names_digest
 
 
@@ -50,6 +51,7 @@ if HAS_DJANGO is True:
             return h.hexdigest()[:length]
 
         schema.names_digest = names_digest
+        indexes.names_digest = names_digest
 
 
 def find_commands(management_dir):

awx/api/generics.py

@@ -45,7 +45,10 @@ from awx.main.utils import (
     get_search_fields,
     getattrd,
     get_object_or_400,
-    decrypt_field
+    decrypt_field,
+    get_awx_version,
+    get_licenser,
+    StubLicense
 )
 from awx.main.utils.db import get_all_field_names
 from awx.api.serializers import ResourceAccessListElementSerializer, CopySerializer, UserSerializer
@@ -197,6 +200,8 @@ class APIView(views.APIView):
                 logger.warning(status_msg)
         response = super(APIView, self).finalize_response(request, response, *args, **kwargs)
         time_started = getattr(self, 'time_started', None)
+        response['X-API-Product-Version'] = get_awx_version()
+        response['X-API-Product-Name'] = 'AWX' if isinstance(get_licenser(), StubLicense) else 'Red Hat Ansible Tower'
         response['X-API-Node'] = settings.CLUSTER_HOST_ID
         if time_started:
             time_elapsed = time.time() - self.time_started

awx/api/metadata.py

@@ -2,6 +2,7 @@
 # All Rights Reserved.
 
 from collections import OrderedDict
+from uuid import UUID
 
 # Django
 from django.core.exceptions import PermissionDenied
@@ -86,6 +87,8 @@ class Metadata(metadata.SimpleMetadata):
         # FIXME: Still isn't showing all default values?
         try:
             default = field.get_default()
+            if type(default) is UUID:
+                default = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
             if field.field_name == 'TOWER_URL_BASE' and default == 'https://towerhost':
                 default = '{}://{}'.format(self.request.scheme, self.request.get_host())
             field_info['default'] = default

awx/api/serializers.py

@@ -806,7 +806,9 @@ class UnifiedJobSerializer(BaseSerializer):
                 td = now() - obj.started
                 ret['elapsed'] = (td.microseconds + (td.seconds + td.days * 24 * 3600) * 10 ** 6) / (10 ** 6 * 1.0)
             ret['elapsed'] = float(ret['elapsed'])
-
+        # Because this string is saved in the db in the source language,
+        # it must be marked for translation after it is pulled from the db, not when set
+        ret['job_explanation'] = _(obj.job_explanation)
         return ret
 
 
@@ -2034,11 +2036,6 @@ class InventorySourceSerializer(UnifiedJobTemplateSerializer, InventorySourceOpt
             res['credentials'] = self.reverse('api:inventory_source_credentials_list', kwargs={'pk': obj.pk})
         return res
 
-    def get_group(self, obj):  # TODO: remove in 3.3
-        if obj.deprecated_group:
-            return obj.deprecated_group.id
-        return None
-
     def build_relational_field(self, field_name, relation_info):
         field_class, field_kwargs = super(InventorySourceSerializer, self).build_relational_field(field_name, relation_info)
         # SCM Project and inventory are read-only unless creating a new inventory.
@@ -2311,6 +2308,7 @@ class RoleSerializer(BaseSerializer):
             content_model = obj.content_type.model_class()
             ret['summary_fields']['resource_type'] = get_type_for_model(content_model)
             ret['summary_fields']['resource_type_display_name'] = content_model._meta.verbose_name.title()
+            ret['summary_fields']['resource_id'] = obj.object_id
 
         return ret
 
@@ -2761,16 +2759,11 @@ class JobOptionsSerializer(LabelsListMixin, BaseSerializer):
         if obj.organization_id:
             res['organization'] = self.reverse('api:organization_detail', kwargs={'pk': obj.organization_id})
         if isinstance(obj, UnifiedJobTemplate):
-            res['extra_credentials'] = self.reverse(
-                'api:job_template_extra_credentials_list',
-                kwargs={'pk': obj.pk}
-            )
             res['credentials'] = self.reverse(
                 'api:job_template_credentials_list',
                 kwargs={'pk': obj.pk}
             )
         elif isinstance(obj, UnifiedJob):
-            res['extra_credentials'] = self.reverse('api:job_extra_credentials_list', kwargs={'pk': obj.pk})
             res['credentials'] = self.reverse('api:job_credentials_list', kwargs={'pk': obj.pk})
 
         return res
@@ -2939,7 +2932,6 @@ class JobTemplateSerializer(JobTemplateMixin, UnifiedJobTemplateSerializer, JobO
         summary_fields = super(JobTemplateSerializer, self).get_summary_fields(obj)
         all_creds = []
         # Organize credential data into multitude of deprecated fields
-        extra_creds = []
         if obj.pk:
             for cred in obj.credentials.all():
                 summarized_cred = {
@@ -2950,10 +2942,6 @@ class JobTemplateSerializer(JobTemplateMixin, UnifiedJobTemplateSerializer, JobO
                     'cloud': cred.credential_type.kind == 'cloud'
                 }
                 all_creds.append(summarized_cred)
-                if cred.credential_type.kind in ('cloud', 'net'):
-                    extra_creds.append(summarized_cred)
-        if self.is_detail_view:
-            summary_fields['extra_credentials'] = extra_creds
         summary_fields['credentials'] = all_creds
         return summary_fields
 
@@ -3028,7 +3016,6 @@ class JobSerializer(UnifiedJobSerializer, JobOptionsSerializer):
         summary_fields = super(JobSerializer, self).get_summary_fields(obj)
         all_creds = []
         # Organize credential data into multitude of deprecated fields
-        extra_creds = []
         if obj.pk:
             for cred in obj.credentials.all():
                 summarized_cred = {
@@ -3039,10 +3026,6 @@ class JobSerializer(UnifiedJobSerializer, JobOptionsSerializer):
                     'cloud': cred.credential_type.kind == 'cloud'
                 }
                 all_creds.append(summarized_cred)
-                if cred.credential_type.kind in ('cloud', 'net'):
-                    extra_creds.append(summarized_cred)
-        if self.is_detail_view:
-            summary_fields['extra_credentials'] = extra_creds
         summary_fields['credentials'] = all_creds
         return summary_fields
 
@@ -3616,9 +3599,11 @@ class LaunchConfigurationBaseSerializer(BaseSerializer):
         elif self.instance:
             ujt = self.instance.unified_job_template
         if ujt is None:
-            if 'workflow_job_template' in attrs:
-                return {'workflow_job_template': attrs['workflow_job_template']}
-            return {}
+            ret = {}
+            for fd in ('workflow_job_template', 'identifier', 'all_parents_must_converge'):
+                if fd in attrs:
+                    ret[fd] = attrs[fd]
+            return ret
 
         # build additional field survey_passwords to track redacted variables
         password_dict = {}
@@ -3671,7 +3656,7 @@ class LaunchConfigurationBaseSerializer(BaseSerializer):
                         attrs.get('survey_passwords', {}).pop(key, None)
                     else:
                         errors.setdefault('extra_vars', []).append(
-                            _('"$encrypted$ is a reserved keyword, may not be used for {var_name}."'.format(key))
+                            _('"$encrypted$ is a reserved keyword, may not be used for {}."'.format(key))
                         )
 
         # Launch configs call extra_vars extra_data for historical reasons
@@ -3902,15 +3887,23 @@ class ProjectUpdateEventSerializer(JobEventSerializer):
         return UriCleaner.remove_sensitive(obj.stdout)
 
     def get_event_data(self, obj):
-        try:
-            return json.loads(
-                UriCleaner.remove_sensitive(
-                    json.dumps(obj.event_data)
+        # the project update playbook uses the git, hg, or svn modules
+        # to clone repositories, and those modules are prone to printing
+        # raw SCM URLs in their stdout (which *could* contain passwords)
+        # attempt to detect and filter HTTP basic auth passwords in the stdout
+        # of these types of events
+        if obj.event_data.get('task_action') in ('git', 'hg', 'svn'):
+            try:
+                return json.loads(
+                    UriCleaner.remove_sensitive(
+                        json.dumps(obj.event_data)
+                    )
                 )
-            )
-        except Exception:
-            logger.exception("Failed to sanitize event_data")
-            return {}
+            except Exception:
+                logger.exception("Failed to sanitize event_data")
+                return {}
+        else:
+            return obj.event_data
 
 
 class AdHocCommandEventSerializer(BaseSerializer):
@@ -4539,6 +4532,8 @@ class SchedulePreviewSerializer(BaseSerializer):
         try:
             Schedule.rrulestr(rrule_value)
         except Exception as e:
+            import traceback
+            logger.error(traceback.format_exc())
             raise serializers.ValidationError(_("rrule parsing failed validation: {}").format(e))
         return value
 

awx/api/templates/api/setting_logging_test.md

@@ -1 +1,2 @@
 # Test Logging Configuration
+

awx/api/urls/oauth2_root.py

@@ -1,11 +1,15 @@
 # Copyright (c) 2017 Ansible, Inc.
 # All Rights Reserved.
+from datetime import timedelta
 
+from django.utils.timezone import now
+from django.conf import settings
 from django.conf.urls import url
 
 from oauthlib import oauth2
 from oauth2_provider import views
 
+from awx.main.models import RefreshToken
 from awx.api.views import (
     ApiOAuthAuthorizationRootView,
 )
@@ -14,6 +18,21 @@ from awx.api.views import (
 class TokenView(views.TokenView):
 
     def create_token_response(self, request):
+        # Django OAuth2 Toolkit has a bug whereby refresh tokens are *never*
+        # properly expired (ugh):
+        #
+        # https://github.com/jazzband/django-oauth-toolkit/issues/746
+        #
+        # This code detects and auto-expires them on refresh grant
+        # requests.
+        if request.POST.get('grant_type') == 'refresh_token' and 'refresh_token' in request.POST:
+            refresh_token = RefreshToken.objects.filter(
+                token=request.POST['refresh_token']
+            ).first()
+            if refresh_token:
+                expire_seconds = settings.OAUTH2_PROVIDER.get('REFRESH_TOKEN_EXPIRE_SECONDS', 0)
+                if refresh_token.created + timedelta(seconds=expire_seconds) < now():
+                    return request.build_absolute_uri(), {}, 'The refresh token has expired.', '403'
         try:
             return super(TokenView, self).create_token_response(request)
         except oauth2.AccessDeniedError as e:

awx/api/urls/urls.py

@@ -23,9 +23,7 @@ from awx.api.views import (
     UnifiedJobList,
     HostAnsibleFactsDetail,
     JobCredentialsList,
-    JobExtraCredentialsList,
     JobTemplateCredentialsList,
-    JobTemplateExtraCredentialsList,
     SchedulePreview,
     ScheduleZoneInfo,
     OAuth2ApplicationList,
@@ -83,9 +81,7 @@ v2_urls = [
     url(r'^credential_types/', include(credential_type_urls)),
     url(r'^credential_input_sources/', include(credential_input_source_urls)),
     url(r'^hosts/(?P<pk>[0-9]+)/ansible_facts/$', HostAnsibleFactsDetail.as_view(), name='host_ansible_facts_detail'),
-    url(r'^jobs/(?P<pk>[0-9]+)/extra_credentials/$', JobExtraCredentialsList.as_view(), name='job_extra_credentials_list'),
     url(r'^jobs/(?P<pk>[0-9]+)/credentials/$', JobCredentialsList.as_view(), name='job_credentials_list'),
-    url(r'^job_templates/(?P<pk>[0-9]+)/extra_credentials/$', JobTemplateExtraCredentialsList.as_view(), name='job_template_extra_credentials_list'),
     url(r'^job_templates/(?P<pk>[0-9]+)/credentials/$', JobTemplateCredentialsList.as_view(), name='job_template_credentials_list'),
     url(r'^schedules/preview/$', SchedulePreview.as_view(), name='schedule_rrule'),
     url(r'^schedules/zoneinfo/$', ScheduleZoneInfo.as_view(), name='schedule_zoneinfo'),

awx/api/views/__init__.py

@@ -12,7 +12,7 @@ import socket
 import sys
 import time
 from base64 import b64encode
-from collections import OrderedDict, Iterable
+from collections import OrderedDict
 
 
 # Django
@@ -1092,7 +1092,7 @@ class UserRolesList(SubListAttachDetachAPIView):
 
         credential_content_type = ContentType.objects.get_for_model(models.Credential)
         if role.content_type == credential_content_type:
-            if role.content_object.organization and user not in role.content_object.organization.member_role:
+            if 'disassociate' not in request.data and role.content_object.organization and user not in role.content_object.organization.member_role:
                 data = dict(msg=_("You cannot grant credential access to a user not in the credentials' organization"))
                 return Response(data, status=status.HTTP_400_BAD_REQUEST)
 
@@ -2337,70 +2337,24 @@ class JobTemplateLaunch(RetrieveAPIView):
         old field structure to launch endpoint
         TODO: delete this method with future API version changes
         '''
-        ignored_fields = {}
         modern_data = data.copy()
 
         id_fd = '{}_id'.format('inventory')
         if 'inventory' not in modern_data and id_fd in modern_data:
             modern_data['inventory'] = modern_data[id_fd]
 
-        # Automatically convert legacy launch credential arguments into a list of `.credentials`
-        if 'credentials' in modern_data and 'extra_credentials' in modern_data:
-            raise ParseError({"error": _(
-                "'credentials' cannot be used in combination with 'extra_credentials'."
-            )})
-
-        if 'extra_credentials' in modern_data:
-            # make a list of the current credentials
-            existing_credentials = obj.credentials.all()
-            template_credentials = list(existing_credentials)  # save copy of existing
-            new_credentials = []
-            if 'extra_credentials' in modern_data:
-                existing_credentials = [
-                    cred for cred in existing_credentials
-                    if cred.credential_type.kind not in ('cloud', 'net')
-                ]
-                prompted_value = modern_data.pop('extra_credentials')
-
-                # validate type, since these are not covered by a serializer
-                if not isinstance(prompted_value, Iterable):
-                    msg = _(
-                        "Incorrect type. Expected a list received {}."
-                    ).format(prompted_value.__class__.__name__)
-                    raise ParseError({'extra_credentials': [msg], 'credentials': [msg]})
-
-                # add the deprecated credential specified in the request
-                if not isinstance(prompted_value, Iterable) or isinstance(prompted_value, str):
-                    prompted_value = [prompted_value]
-
-                # If user gave extra_credentials, special case to use exactly
-                # the given list without merging with JT credentials
-                if prompted_value:
-                    obj._deprecated_credential_launch = True  # signal to not merge credentials
-                new_credentials.extend(prompted_value)
-
-            # combine the list of "new" and the filtered list of "old"
-            new_credentials.extend([cred.pk for cred in existing_credentials])
-            if new_credentials:
-                # If provided list doesn't contain the pre-existing credentials
-                # defined on the template, add them back here
-                for cred_obj in template_credentials:
-                    if cred_obj.pk not in new_credentials:
-                        new_credentials.append(cred_obj.pk)
-                modern_data['credentials'] = new_credentials
-
         # credential passwords were historically provided as top-level attributes
         if 'credential_passwords' not in modern_data:
             modern_data['credential_passwords'] = data.copy()
 
-        return (modern_data, ignored_fields)
+        return modern_data
 
 
     def post(self, request, *args, **kwargs):
         obj = self.get_object()
 
         try:
-            modern_data, ignored_fields = self.modernize_launch_payload(
+            modern_data = self.modernize_launch_payload(
                 data=request.data, obj=obj
             )
         except ParseError as exc:
@@ -2410,8 +2364,6 @@ class JobTemplateLaunch(RetrieveAPIView):
         if not serializer.is_valid():
             return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)
 
-        ignored_fields.update(serializer._ignored_fields)
-
         if not request.user.can_access(models.JobLaunchConfig, 'add', serializer.validated_data, template=obj):
             raise PermissionDenied()
 
@@ -2427,11 +2379,11 @@ class JobTemplateLaunch(RetrieveAPIView):
             data = OrderedDict()
             if isinstance(new_job, models.WorkflowJob):
                 data['workflow_job'] = new_job.id
-                data['ignored_fields'] = self.sanitize_for_response(ignored_fields)
+                data['ignored_fields'] = self.sanitize_for_response(serializer._ignored_fields)
                 data.update(serializers.WorkflowJobSerializer(new_job, context=self.get_serializer_context()).to_representation(new_job))
             else:
                 data['job'] = new_job.id
-                data['ignored_fields'] = self.sanitize_for_response(ignored_fields)
+                data['ignored_fields'] = self.sanitize_for_response(serializer._ignored_fields)
                 data.update(serializers.JobSerializer(new_job, context=self.get_serializer_context()).to_representation(new_job))
             headers = {'Location': new_job.get_absolute_url(request)}
             return Response(data, status=status.HTTP_201_CREATED, headers=headers)
@@ -2711,22 +2663,6 @@ class JobTemplateCredentialsList(SubListCreateAttachDetachAPIView):
         return super(JobTemplateCredentialsList, self).is_valid_relation(parent, sub, created)
 
 
-class JobTemplateExtraCredentialsList(JobTemplateCredentialsList):
-
-    deprecated = True
-
-    def get_queryset(self):
-        sublist_qs = super(JobTemplateExtraCredentialsList, self).get_queryset()
-        sublist_qs = sublist_qs.filter(credential_type__kind__in=['cloud', 'net'])
-        return sublist_qs
-
-    def is_valid_relation(self, parent, sub, created=False):
-        valid = super(JobTemplateExtraCredentialsList, self).is_valid_relation(parent, sub, created)
-        if sub.credential_type.kind not in ('cloud', 'net'):
-            return {'error': _('Extra credentials must be network or cloud.')}
-        return valid
-
-
 class JobTemplateLabelList(DeleteLastUnattachLabelMixin, SubListCreateAttachDetachAPIView):
 
     model = models.Label
@@ -3543,16 +3479,6 @@ class JobCredentialsList(SubListAPIView):
     relationship = 'credentials'
 
 
-class JobExtraCredentialsList(JobCredentialsList):
-
-    deprecated = True
-
-    def get_queryset(self):
-        sublist_qs = super(JobExtraCredentialsList, self).get_queryset()
-        sublist_qs = sublist_qs.filter(credential_type__kind__in=['cloud', 'net'])
-        return sublist_qs
-
-
 class JobLabelList(SubListAPIView):
 
     model = models.Label
@@ -4415,7 +4341,7 @@ class RoleUsersList(SubListAttachDetachAPIView):
 
         credential_content_type = ContentType.objects.get_for_model(models.Credential)
         if role.content_type == credential_content_type:
-            if role.content_object.organization and user not in role.content_object.organization.member_role:
+            if 'disassociate' not in request.data and role.content_object.organization and user not in role.content_object.organization.member_role:
                 data = dict(msg=_("You cannot grant credential access to a user not in the credentials' organization"))
                 return Response(data, status=status.HTTP_400_BAD_REQUEST)
 

awx/asgi.py

@@ -4,12 +4,11 @@ import os
 import logging
 import django
 from awx import __version__ as tower_version
-
 # Prepare the AWX environment.
 from awx import prepare_env, MODE
+from channels.routing import get_default_application  # noqa
 prepare_env() # NOQA
 
-from channels.routing import get_default_application
 
 
 """

awx/conf/fields.py

@@ -172,9 +172,9 @@ class URLField(CharField):
                         netloc = '{}:{}'.format(netloc, url_parts.port)
                     if url_parts.username:
                         if url_parts.password:
-                            netloc = '{}:{}@{}' % (url_parts.username, url_parts.password, netloc)
+                            netloc = '{}:{}@{}'.format(url_parts.username, url_parts.password, netloc)
                         else:
-                            netloc = '{}@{}' % (url_parts.username, netloc)
+                            netloc = '{}@{}'.format(url_parts.username, netloc)
                     value = urlparse.urlunsplit([url_parts.scheme, netloc, url_parts.path, url_parts.query, url_parts.fragment])
             except Exception:
                 raise  # If something fails here, just fall through and let the validators check it.

awx/conf/settings.py

@@ -11,7 +11,7 @@ from django.conf import settings, UserSettingsHolder
 from django.core.cache import cache as django_cache
 from django.core.exceptions import ImproperlyConfigured
 from django.db import transaction, connection
-from django.db.utils import Error as DBError
+from django.db.utils import Error as DBError, ProgrammingError
 from django.utils.functional import cached_property
 
 # Django REST Framework
@@ -74,10 +74,19 @@ def _ctit_db_wrapper(trans_safe=False):
                     logger.debug('Obtaining database settings in spite of broken transaction.')
                     transaction.set_rollback(False)
         yield
-    except DBError:
+    except DBError as exc:
         if trans_safe:
             if 'migrate' not in sys.argv and 'check_migrations' not in sys.argv:
-                logger.exception('Database settings are not available, using defaults.')
+                level = logger.exception
+                if isinstance(exc, ProgrammingError):
+                    if 'relation' in str(exc) and 'does not exist' in str(exc):
+                        # this generally means we can't fetch Tower configuration
+                        # because the database hasn't actually finished migrating yet;
+                        # this is usually a sign that a service in a container (such as ws_broadcast)
+                        # has come up *before* the database has finished migrating, and
+                        # especially that the conf.settings table doesn't exist yet
+                        level = logger.debug
+                level('Database settings are not available, using defaults.')
         else:
             logger.exception('Error modifying something related to database settings.')
     finally:
@@ -410,7 +419,7 @@ class SettingsWrapper(UserSettingsHolder):
         field = self.registry.get_setting_field(name)
         if field.read_only:
             logger.warning('Attempt to set read only setting "%s".', name)
-            raise ImproperlyConfigured('Setting "%s" is read only.'.format(name))
+            raise ImproperlyConfigured('Setting "{}" is read only.'.format(name))
 
         try:
             data = field.to_representation(value)
@@ -441,7 +450,7 @@ class SettingsWrapper(UserSettingsHolder):
         field = self.registry.get_setting_field(name)
         if field.read_only:
             logger.warning('Attempt to delete read only setting "%s".', name)
-            raise ImproperlyConfigured('Setting "%s" is read only.'.format(name))
+            raise ImproperlyConfigured('Setting "{}" is read only.'.format(name))
         for setting in Setting.objects.filter(key=name, user__isnull=True):
             setting.delete()
             # pre_delete handler will delete from cache.

awx/conf/tests/functional/test_api.py

@@ -325,17 +325,3 @@ def test_setting_singleton_delete_no_read_only_fields(api_request, dummy_setting
         )
         assert response.data['FOO_BAR'] == 23
 
-
-@pytest.mark.django_db
-def test_setting_logging_test(api_request):
-    with mock.patch('awx.conf.views.AWXProxyHandler.perform_test') as mock_func:
-        api_request(
-            'post',
-            reverse('api:setting_logging_test'),
-            data={'LOG_AGGREGATOR_HOST': 'http://foobar', 'LOG_AGGREGATOR_TYPE': 'logstash'}
-        )
-        call = mock_func.call_args_list[0]
-        args, kwargs = call
-        given_settings = kwargs['custom_settings']
-        assert given_settings.LOG_AGGREGATOR_HOST == 'http://foobar'
-        assert given_settings.LOG_AGGREGATOR_TYPE == 'logstash'

awx/conf/views.py

@@ -3,15 +3,20 @@
 
 # Python
 import collections
+import logging
+import subprocess
 import sys
+import socket
+from socket import SHUT_RDWR
 
 # Django
+from django.db import connection
 from django.conf import settings
 from django.http import Http404
 from django.utils.translation import ugettext_lazy as _
 
 # Django REST Framework
-from rest_framework.exceptions import PermissionDenied, ValidationError
+from rest_framework.exceptions import PermissionDenied
 from rest_framework.response import Response
 from rest_framework import serializers
 from rest_framework import status
@@ -26,7 +31,6 @@ from awx.api.generics import (
 from awx.api.permissions import IsSuperUser
 from awx.api.versioning import reverse
 from awx.main.utils import camelcase_to_underscore
-from awx.main.utils.handlers import AWXProxyHandler, LoggingConnectivityException
 from awx.main.tasks import handle_setting_changes
 from awx.conf.models import Setting
 from awx.conf.serializers import SettingCategorySerializer, SettingSingletonSerializer
@@ -127,7 +131,8 @@ class SettingSingletonDetail(RetrieveUpdateDestroyAPIView):
                 setting.save(update_fields=['value'])
                 settings_change_list.append(key)
         if settings_change_list:
-            handle_setting_changes.delay(settings_change_list)
+            connection.on_commit(lambda: handle_setting_changes.delay(settings_change_list))
+
 
     def destroy(self, request, *args, **kwargs):
         instance = self.get_object()
@@ -142,7 +147,7 @@ class SettingSingletonDetail(RetrieveUpdateDestroyAPIView):
             setting.delete()
             settings_change_list.append(setting.key)
         if settings_change_list:
-            handle_setting_changes.delay(settings_change_list)
+            connection.on_commit(lambda: handle_setting_changes.delay(settings_change_list))
 
         # When TOWER_URL_BASE is deleted from the API, reset it to the hostname
         # used to make the request as a default.
@@ -161,40 +166,53 @@ class SettingLoggingTest(GenericAPIView):
     filter_backends = []
 
     def post(self, request, *args, **kwargs):
-        defaults = dict()
-        for key in settings_registry.get_registered_settings(category_slug='logging'):
-            try:
-                defaults[key] = settings_registry.get_setting_field(key).get_default()
-            except serializers.SkipField:
-                defaults[key] = None
-        obj = type('Settings', (object,), defaults)()
-        serializer = self.get_serializer(obj, data=request.data)
-        serializer.is_valid(raise_exception=True)
-        # Special validation specific to logging test.
-        errors = {}
-        for key in ['LOG_AGGREGATOR_TYPE', 'LOG_AGGREGATOR_HOST']:
-            if not request.data.get(key, ''):
-                errors[key] = 'This field is required.'
-        if errors:
-            raise ValidationError(errors)
-
-        if request.data.get('LOG_AGGREGATOR_PASSWORD', '').startswith('$encrypted$'):
-            serializer.validated_data['LOG_AGGREGATOR_PASSWORD'] = getattr(
-                settings, 'LOG_AGGREGATOR_PASSWORD', ''
-            )
+        # Error if logging is not enabled
+        enabled = getattr(settings, 'LOG_AGGREGATOR_ENABLED', False)
+        if not enabled:
+            return Response({'error': 'Logging not enabled'}, status=status.HTTP_409_CONFLICT)
+        
+        # Send test message to configured logger based on db settings
+        try:
+            default_logger = settings.LOG_AGGREGATOR_LOGGERS[0]
+            if default_logger != 'awx':
+                default_logger = f'awx.analytics.{default_logger}'
+        except IndexError:
+            default_logger = 'awx'
+        logging.getLogger(default_logger).error('AWX Connection Test Message')
+        
+        hostname = getattr(settings, 'LOG_AGGREGATOR_HOST', None)
+        protocol = getattr(settings, 'LOG_AGGREGATOR_PROTOCOL', None)
 
         try:
-            class MockSettings:
-                pass
-            mock_settings = MockSettings()
-            for k, v in serializer.validated_data.items():
-                setattr(mock_settings, k, v)
-            AWXProxyHandler().perform_test(custom_settings=mock_settings)
-            if mock_settings.LOG_AGGREGATOR_PROTOCOL.upper() == 'UDP':
-                return Response(status=status.HTTP_201_CREATED)
-        except LoggingConnectivityException as e:
-            return Response({'error': str(e)}, status=status.HTTP_500_INTERNAL_SERVER_ERROR)
-        return Response(status=status.HTTP_200_OK)
+            subprocess.check_output(
+                ['rsyslogd', '-N1', '-f', '/var/lib/awx/rsyslog/rsyslog.conf'],
+                stderr=subprocess.STDOUT
+            )
+        except subprocess.CalledProcessError as exc:
+            return Response({'error': exc.output}, status=status.HTTP_400_BAD_REQUEST)
+        
+        # Check to ensure port is open at host
+        if protocol in ['udp', 'tcp']:
+            port = getattr(settings, 'LOG_AGGREGATOR_PORT', None)
+            # Error if port is not set when using UDP/TCP
+            if not port:
+                return Response({'error': 'Port required for ' + protocol}, status=status.HTTP_400_BAD_REQUEST)
+        else:
+            # if http/https by this point, domain is reacheable
+            return Response(status=status.HTTP_202_ACCEPTED)
+
+        if protocol == 'udp':
+            s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+        else:
+            s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        try:
+            s.settimeout(.5)
+            s.connect((hostname, int(port)))
+            s.shutdown(SHUT_RDWR)
+            s.close()
+            return Response(status=status.HTTP_202_ACCEPTED)
+        except Exception as e:
+            return Response({'error': str(e)}, status=status.HTTP_400_BAD_REQUEST)
 
 
 # Create view functions for all of the class-based views to simplify inclusion

awx/main/access.py

@@ -11,7 +11,6 @@ from functools import reduce
 from django.conf import settings
 from django.db.models import Q, Prefetch
 from django.contrib.auth.models import User
-from django.contrib.contenttypes.models import ContentType
 from django.utils.translation import ugettext_lazy as _
 from django.core.exceptions import ObjectDoesNotExist
 
@@ -405,14 +404,6 @@ class BaseAccess(object):
                 # Cannot copy manual project without errors
                 user_capabilities[display_method] = False
                 continue
-            elif display_method in ['start', 'schedule'] and isinstance(obj, Group):  # TODO: remove in 3.3
-                try:
-                    if obj.deprecated_inventory_source and not obj.deprecated_inventory_source._can_update():
-                        user_capabilities[display_method] = False
-                        continue
-                except Group.deprecated_inventory_source.RelatedObjectDoesNotExist:
-                    user_capabilities[display_method] = False
-                    continue
             elif display_method in ['start', 'schedule'] and isinstance(obj, (Project)):
                 if obj.scm_type == '':
                     user_capabilities[display_method] = False
@@ -504,7 +495,7 @@ class NotificationAttachMixin(BaseAccess):
             # due to this special case, we use symmetrical logic with attach permission
             return self._can_attach(notification_template=sub_obj, resource_obj=obj)
         return super(NotificationAttachMixin, self).can_unattach(
-            obj, sub_obj, relationship, relationship, data=data
+            obj, sub_obj, relationship, data=data
         )
 
 
@@ -650,8 +641,8 @@ class UserAccess(BaseAccess):
                 # in these cases only superusers can modify orphan users
                 return False
             return not obj.roles.all().exclude(
-                content_type=ContentType.objects.get_for_model(User)
-            ).filter(ancestors__in=self.user.roles.all()).exists()
+                ancestors__in=self.user.roles.all()
+            ).exists()
         else:
             return self.is_all_org_admin(obj)
 
@@ -1434,7 +1425,7 @@ class JobTemplateAccess(NotificationAttachMixin, BaseAccess):
         Users who are able to create deploy jobs can also run normal and check (dry run) jobs.
         '''
         if not data:  # So the browseable API will work
-            return Organization.accessible_objects(self.user, 'job_template_admin_role').exists()
+            return Project.accessible_objects(self.user, 'use_role').exists()
 
         # if reference_obj is provided, determine if it can be copied
         reference_obj = data.get('reference_obj', None)
@@ -1503,11 +1494,6 @@ class JobTemplateAccess(NotificationAttachMixin, BaseAccess):
         if data is None:
             return True
 
-        # standard type of check for organization - cannot change the value
-        # unless posessing the respective job_template_admin_role, otherwise non-blocking
-        if not self.check_related('organization', Organization, data, obj=obj, role_field='job_template_admin_role'):
-            return False
-
         data = dict(data)
 
         if self.changes_are_non_sensitive(obj, data):

awx/main/analytics/broadcast_websocket.py

@@ -11,6 +11,7 @@ from prometheus_client import (
     Counter,
     Enum,
     CollectorRegistry,
+    parser,
 )
 
 from django.conf import settings
@@ -30,6 +31,11 @@ def now_seconds():
     return dt_to_seconds(datetime.datetime.now())
 
 
+def safe_name(s):
+    # Replace all non alpha-numeric characters with _
+    return re.sub('[^0-9a-zA-Z]+', '_', s)
+
+
 # Second granularity; Per-minute
 class FixedSlidingWindow():
     def __init__(self, start_time=None):
@@ -38,8 +44,8 @@ class FixedSlidingWindow():
 
     def cleanup(self, now_bucket=None):
         now_bucket = now_bucket or now_seconds()
-        if self.start_time + 60 <= now_bucket:
-            self.start_time = now_bucket + 60 + 1
+        if self.start_time + 60 < now_bucket:
+            self.start_time = now_bucket - 60
 
             # Delete old entries
             for k in list(self.buckets.keys()):
@@ -47,16 +53,15 @@ class FixedSlidingWindow():
                     del self.buckets[k]
 
     def record(self, ts=None):
-        ts = ts or datetime.datetime.now()
-        now_bucket = int((ts - datetime.datetime(1970,1,1)).total_seconds())
+        now_bucket = ts or dt_to_seconds(datetime.datetime.now())
 
         val = self.buckets.get(now_bucket, 0)
         self.buckets[now_bucket] = val + 1
 
         self.cleanup(now_bucket)
 
-    def render(self):
-        self.cleanup()
+    def render(self, ts=None):
+        self.cleanup(now_bucket=ts)
         return sum(self.buckets.values()) or 0
 
 
@@ -99,7 +104,8 @@ class BroadcastWebsocketStatsManager():
         Stringified verion of all the stats
         '''
         redis_conn = redis.Redis.from_url(settings.BROKER_URL)
-        return redis_conn.get(BROADCAST_WEBSOCKET_REDIS_KEY_NAME)
+        stats_str = redis_conn.get(BROADCAST_WEBSOCKET_REDIS_KEY_NAME) or b''
+        return parser.text_string_to_metric_families(stats_str.decode('UTF-8'))
 
 
 class BroadcastWebsocketStats():
@@ -109,8 +115,8 @@ class BroadcastWebsocketStats():
         self._registry = CollectorRegistry()
 
         # TODO: More robust replacement
-        self.name = self.safe_name(self._local_hostname)
-        self.remote_name = self.safe_name(self._remote_hostname)
+        self.name = safe_name(self._local_hostname)
+        self.remote_name = safe_name(self._remote_hostname)
 
         self._messages_received_total = Counter(f'awx_{self.remote_name}_messages_received_total',
                                                 'Number of messages received, to be forwarded, by the broadcast websocket system',
@@ -122,6 +128,7 @@ class BroadcastWebsocketStats():
                                 'Websocket broadcast connection',
                                 states=['disconnected', 'connected'],
                                 registry=self._registry)
+        self._connection.state('disconnected')
         self._connection_start = Gauge(f'awx_{self.remote_name}_connection_start',
                                        'Time the connection was established',
                                        registry=self._registry)
@@ -131,10 +138,6 @@ class BroadcastWebsocketStats():
                                                    registry=self._registry)
         self._internal_messages_received_per_minute = FixedSlidingWindow()
 
-    def safe_name(self, s):
-        # Replace all non alpha-numeric characters with _
-        return re.sub('[^0-9a-zA-Z]+', '_', s)
-
     def unregister(self):
         self._registry.unregister(f'awx_{self.remote_name}_messages_received')
         self._registry.unregister(f'awx_{self.remote_name}_connection')

awx/main/analytics/collectors.py

@@ -122,22 +122,27 @@ def cred_type_counts(since):
     return counts
     
     
-@register('inventory_counts', '1.0')
+@register('inventory_counts', '1.2')
 def inventory_counts(since):
     counts = {}
     for inv in models.Inventory.objects.filter(kind='').annotate(num_sources=Count('inventory_sources', distinct=True), 
                                                                  num_hosts=Count('hosts', distinct=True)).only('id', 'name', 'kind'):
+        source_list = []
+        for source in inv.inventory_sources.filter().annotate(num_hosts=Count('hosts', distinct=True)).values('name','source', 'num_hosts'):
+            source_list.append(source)
         counts[inv.id] = {'name': inv.name,
                           'kind': inv.kind,
                           'hosts': inv.num_hosts,
-                          'sources': inv.num_sources
+                          'sources': inv.num_sources,
+                          'source_list': source_list
                           }
 
     for smart_inv in models.Inventory.objects.filter(kind='smart'):
         counts[smart_inv.id] = {'name': smart_inv.name,
                                 'kind': smart_inv.kind,
-                                'num_hosts': smart_inv.hosts.count(),
-                                'num_sources': smart_inv.inventory_sources.count()
+                                'hosts': smart_inv.hosts.count(),
+                                'sources': 0,
+                                'source_list': []
                                 }
     return counts
 
@@ -222,10 +227,12 @@ def query_info(since, collection_type):
 
 
 # Copies Job Events from db to a .csv to be shipped
-@table_version('events_table.csv', '1.0')
+@table_version('events_table.csv', '1.1')
 @table_version('unified_jobs_table.csv', '1.0')
 @table_version('unified_job_template_table.csv', '1.0')
-def copy_tables(since, full_path):
+@table_version('workflow_job_node_table.csv', '1.0')
+@table_version('workflow_job_template_node_table.csv', '1.0')
+def copy_tables(since, full_path, subset=None):
     def _copy_table(table, query, path):
         file_path = os.path.join(path, table + '_table.csv')
         file = open(file_path, 'w', encoding='utf-8')
@@ -249,10 +256,16 @@ def copy_tables(since, full_path):
                               main_jobevent.job_id, 
                               main_jobevent.host_id, 
                               main_jobevent.host_name
+                              , CAST(main_jobevent.event_data::json->>'start' AS TIMESTAMP WITH TIME ZONE) AS start,
+                              CAST(main_jobevent.event_data::json->>'end' AS TIMESTAMP WITH TIME ZONE) AS end,
+                              main_jobevent.event_data::json->'duration' AS duration,
+                              main_jobevent.event_data::json->'res'->'warnings' AS warnings,
+                              main_jobevent.event_data::json->'res'->'deprecations' AS deprecations
                               FROM main_jobevent 
                               WHERE main_jobevent.created > {}
                               ORDER BY main_jobevent.id ASC) TO STDOUT WITH CSV HEADER'''.format(since.strftime("'%Y-%m-%d %H:%M:%S'"))
-    _copy_table(table='events', query=events_query, path=full_path)
+    if not subset or 'events' in subset:
+        _copy_table(table='events', query=events_query, path=full_path)
 
     unified_job_query = '''COPY (SELECT main_unifiedjob.id,
                                  main_unifiedjob.polymorphic_ctype_id,
@@ -276,11 +289,12 @@ def copy_tables(since, full_path):
                                  main_unifiedjob.instance_group_id
                                  FROM main_unifiedjob
                                  JOIN django_content_type ON main_unifiedjob.polymorphic_ctype_id = django_content_type.id
-                                 JOIN main_organization ON main_organization.id = main_unifiedjob.organization_id
-                                 WHERE main_unifiedjob.created > {} 
-                                 AND main_unifiedjob.launch_type != 'sync'
+                                 LEFT JOIN main_organization ON main_organization.id = main_unifiedjob.organization_id
+                                 WHERE (main_unifiedjob.created > {0} OR main_unifiedjob.finished > {0})
+                                       AND main_unifiedjob.launch_type != 'sync'
                                  ORDER BY main_unifiedjob.id ASC) TO STDOUT WITH CSV HEADER'''.format(since.strftime("'%Y-%m-%d %H:%M:%S'"))    
-    _copy_table(table='unified_jobs', query=unified_job_query, path=full_path)
+    if not subset or 'unified_jobs' in subset:
+        _copy_table(table='unified_jobs', query=unified_job_query, path=full_path)
 
     unified_job_template_query = '''COPY (SELECT main_unifiedjobtemplate.id, 
                                  main_unifiedjobtemplate.polymorphic_ctype_id,
@@ -299,6 +313,71 @@ def copy_tables(since, full_path):
                                  main_unifiedjobtemplate.status 
                                  FROM main_unifiedjobtemplate, django_content_type
                                  WHERE main_unifiedjobtemplate.polymorphic_ctype_id = django_content_type.id
-                                 ORDER BY main_unifiedjobtemplate.id ASC) TO STDOUT WITH CSV HEADER'''.format(since.strftime("'%Y-%m-%d %H:%M:%S'"))    
-    _copy_table(table='unified_job_template', query=unified_job_template_query, path=full_path)
+                                 ORDER BY main_unifiedjobtemplate.id ASC) TO STDOUT WITH CSV HEADER'''  
+    if not subset or 'unified_job_template' in subset:
+        _copy_table(table='unified_job_template', query=unified_job_template_query, path=full_path)
+
+    workflow_job_node_query = '''COPY (SELECT main_workflowjobnode.id,
+                                 main_workflowjobnode.created,
+                                 main_workflowjobnode.modified, 
+                                 main_workflowjobnode.job_id, 
+                                 main_workflowjobnode.unified_job_template_id, 
+                                 main_workflowjobnode.workflow_job_id, 
+                                 main_workflowjobnode.inventory_id, 
+                                 success_nodes.nodes AS success_nodes,
+                                 failure_nodes.nodes AS failure_nodes,
+                                 always_nodes.nodes AS always_nodes,
+                                 main_workflowjobnode.do_not_run, 
+                                 main_workflowjobnode.all_parents_must_converge
+                                 FROM main_workflowjobnode
+                                 LEFT JOIN (
+                                     SELECT from_workflowjobnode_id, ARRAY_AGG(to_workflowjobnode_id) AS nodes
+                                     FROM main_workflowjobnode_success_nodes
+                                     GROUP BY from_workflowjobnode_id
+                                 ) success_nodes ON main_workflowjobnode.id = success_nodes.from_workflowjobnode_id
+                                 LEFT JOIN (
+                                     SELECT from_workflowjobnode_id, ARRAY_AGG(to_workflowjobnode_id) AS nodes
+                                     FROM main_workflowjobnode_failure_nodes
+                                     GROUP BY from_workflowjobnode_id
+                                 ) failure_nodes ON main_workflowjobnode.id = failure_nodes.from_workflowjobnode_id
+                                 LEFT JOIN (
+                                     SELECT from_workflowjobnode_id, ARRAY_AGG(to_workflowjobnode_id) AS nodes
+                                     FROM main_workflowjobnode_always_nodes
+                                     GROUP BY from_workflowjobnode_id
+                                 ) always_nodes ON main_workflowjobnode.id = always_nodes.from_workflowjobnode_id
+                                 WHERE main_workflowjobnode.modified > {}
+                                 ORDER BY main_workflowjobnode.id ASC) TO STDOUT WITH CSV HEADER'''.format(since.strftime("'%Y-%m-%d %H:%M:%S'"))    
+    if not subset or 'workflow_job_node' in subset:
+        _copy_table(table='workflow_job_node', query=workflow_job_node_query, path=full_path)
+
+    workflow_job_template_node_query = '''COPY (SELECT main_workflowjobtemplatenode.id, 
+                                 main_workflowjobtemplatenode.created,
+                                 main_workflowjobtemplatenode.modified, 
+                                 main_workflowjobtemplatenode.unified_job_template_id, 
+                                 main_workflowjobtemplatenode.workflow_job_template_id, 
+                                 main_workflowjobtemplatenode.inventory_id, 
+                                 success_nodes.nodes AS success_nodes,
+                                 failure_nodes.nodes AS failure_nodes,
+                                 always_nodes.nodes AS always_nodes,
+                                 main_workflowjobtemplatenode.all_parents_must_converge
+                                 FROM main_workflowjobtemplatenode
+                                 LEFT JOIN (
+                                     SELECT from_workflowjobtemplatenode_id, ARRAY_AGG(to_workflowjobtemplatenode_id) AS nodes
+                                     FROM main_workflowjobtemplatenode_success_nodes
+                                     GROUP BY from_workflowjobtemplatenode_id
+                                 ) success_nodes ON main_workflowjobtemplatenode.id = success_nodes.from_workflowjobtemplatenode_id
+                                 LEFT JOIN (
+                                     SELECT from_workflowjobtemplatenode_id, ARRAY_AGG(to_workflowjobtemplatenode_id) AS nodes
+                                     FROM main_workflowjobtemplatenode_failure_nodes
+                                     GROUP BY from_workflowjobtemplatenode_id
+                                 ) failure_nodes ON main_workflowjobtemplatenode.id = failure_nodes.from_workflowjobtemplatenode_id
+                                 LEFT JOIN (
+                                     SELECT from_workflowjobtemplatenode_id, ARRAY_AGG(to_workflowjobtemplatenode_id) AS nodes
+                                     FROM main_workflowjobtemplatenode_always_nodes
+                                     GROUP BY from_workflowjobtemplatenode_id
+                                 ) always_nodes ON main_workflowjobtemplatenode.id = always_nodes.from_workflowjobtemplatenode_id
+                                 ORDER BY main_workflowjobtemplatenode.id ASC) TO STDOUT WITH CSV HEADER'''   
+    if not subset or 'workflow_job_template_node' in subset:
+        _copy_table(table='workflow_job_template_node', query=workflow_job_template_node_query, path=full_path)
+
     return

awx/main/conf.py

@@ -246,18 +246,6 @@ register(
     category_slug='jobs',
 )
 
-register(
-    'AWX_ISOLATED_VERBOSITY',
-    field_class=fields.IntegerField,
-    min_value=0,
-    max_value=5,
-    label=_('Verbosity level for isolated node management tasks'),
-    help_text=_('This can be raised to aid in debugging connection issues for isolated task execution'),
-    category=_('Jobs'),
-    category_slug='jobs',
-    default=0
-)
-
 register(
     'AWX_ISOLATED_CHECK_INTERVAL',
     field_class=fields.IntegerField,
@@ -435,6 +423,19 @@ register(
     category_slug='jobs',
 )
 
+register(
+    'AWX_SHOW_PLAYBOOK_LINKS',
+    field_class=fields.BooleanField,
+    default=False,
+    label=_('Follow symlinks'),
+    help_text=_(
+        'Follow symbolic links when scanning for playbooks. Be aware that setting this to True can lead '
+        'to infinite recursion if a link points to a parent directory of itself.'
+    ),
+    category=_('Jobs'),
+    category_slug='jobs',
+)
+
 register(
     'PRIMARY_GALAXY_URL',
     field_class=fields.URLField,
@@ -667,7 +668,7 @@ register(
     allow_blank=True,
     default='',
     label=_('Logging Aggregator Username'),
-    help_text=_('Username for external log aggregator (if required).'),
+    help_text=_('Username for external log aggregator (if required; HTTP/s only).'),
     category=_('Logging'),
     category_slug='logging',
     required=False,
@@ -679,7 +680,7 @@ register(
     default='',
     encrypted=True,
     label=_('Logging Aggregator Password/Token'),
-    help_text=_('Password or authentication token for external log aggregator (if required).'),
+    help_text=_('Password or authentication token for external log aggregator (if required; HTTP/s only).'),
     category=_('Logging'),
     category_slug='logging',
     required=False,
@@ -778,26 +779,40 @@ register(
     category_slug='logging',
 )
 register(
-    'LOG_AGGREGATOR_AUDIT',
+    'LOG_AGGREGATOR_MAX_DISK_USAGE_GB',
+    field_class=fields.IntegerField,
+    default=1,
+    min_value=1,
+    label=_('Maximum disk persistance for external log aggregation (in GB)'),
+    help_text=_('Amount of data to store (in gigabytes) during an outage of '
+                'the external log aggregator (defaults to 1). '
+                'Equivalent to the rsyslogd queue.maxdiskspace setting.'),
+    category=_('Logging'),
+    category_slug='logging',
+)
+register(
+    'LOG_AGGREGATOR_MAX_DISK_USAGE_PATH',
+    field_class=fields.CharField,
+    default='/var/lib/awx',
+    label=_('File system location for rsyslogd disk persistence'),
+    help_text=_('Location to persist logs that should be retried after an outage '
+                'of the external log aggregator (defaults to /var/lib/awx). '
+                'Equivalent to the rsyslogd queue.spoolDirectory setting.'),
+    category=_('Logging'),
+    category_slug='logging',
+)
+register(
+    'LOG_AGGREGATOR_RSYSLOGD_DEBUG',
     field_class=fields.BooleanField,
-    allow_null=True,
     default=False,
-    label=_('Enabled external log aggregation auditing'),
-    help_text=_('When enabled, all external logs emitted by Tower will also be written to /var/log/tower/external.log.  This is an experimental setting intended to be used for debugging external log aggregation issues (and may be subject to change in the future).'),  # noqa
+    label=_('Enable rsyslogd debugging'),
+    help_text=_('Enabled high verbosity debugging for rsyslogd.  '
+                'Useful for debugging connection issues for external log aggregation.'),
     category=_('Logging'),
     category_slug='logging',
 )
 
 
-register(
-    'BROKER_DURABILITY',
-    field_class=fields.BooleanField,
-    label=_('Message Durability'),
-    help_text=_('When set (the default), underlying queues will be persisted to disk.  Disable this to enable higher message bus throughput.'),
-    category=_('System'),
-    category_slug='system',
-)
-
 
 register(
     'AUTOMATION_ANALYTICS_LAST_GATHER',

awx/main/constants.py

@@ -38,7 +38,7 @@ ENV_BLACKLIST = frozenset((
     'AD_HOC_COMMAND_ID', 'REST_API_URL', 'REST_API_TOKEN', 'MAX_EVENT_RES',
     'CALLBACK_QUEUE', 'CALLBACK_CONNECTION', 'CACHE',
     'JOB_CALLBACK_DEBUG', 'INVENTORY_HOSTVARS',
-    'AWX_HOST', 'PROJECT_REVISION'
+    'AWX_HOST', 'PROJECT_REVISION', 'SUPERVISOR_WEB_CONFIG_PATH'
 ))
 
 # loggers that may be called in process of emitting a log

awx/main/consumers.py

@@ -1,6 +1,6 @@
 import json
 import logging
-import datetime
+import time
 import hmac
 import asyncio
 
@@ -29,7 +29,7 @@ class WebsocketSecretAuthHelper:
 
     @classmethod
     def construct_secret(cls):
-        nonce_serialized = "{}".format(int((datetime.datetime.utcnow() - datetime.datetime.fromtimestamp(0)).total_seconds()))
+        nonce_serialized = f"{int(time.time())}"
         payload_dict = {
             'secret': settings.BROADCAST_WEBSOCKET_SECRET,
             'nonce': nonce_serialized
@@ -70,10 +70,12 @@ class WebsocketSecretAuthHelper:
             raise ValueError("Invalid secret")
 
         # Avoid timing attack and check the nonce after all the heavy lifting
-        now = datetime.datetime.utcnow()
-        nonce_parsed = datetime.datetime.fromtimestamp(int(nonce_parsed))
-        if (now - nonce_parsed).total_seconds() > nonce_tolerance:
-            raise ValueError("Potential replay attack or machine(s) time out of sync.")
+        now = int(time.time())
+        nonce_parsed = int(nonce_parsed)
+        nonce_diff = now - nonce_parsed
+        if abs(nonce_diff) > nonce_tolerance:
+            logger.warn(f"Potential replay attack or machine(s) time out of sync by {nonce_diff} seconds.")
+            raise ValueError("Potential replay attack or machine(s) time out of sync by {nonce_diff} seconds.")
 
         return True
 
@@ -93,19 +95,17 @@ class BroadcastConsumer(AsyncJsonWebsocketConsumer):
         try:
             WebsocketSecretAuthHelper.is_authorized(self.scope)
         except Exception:
-            # TODO: log ip of connected client
-            logger.warn("Broadcast client failed to authorize.")
+            logger.warn(f"client '{self.channel_name}' failed to authorize against the broadcast endpoint.")
             await self.close()
             return
 
-        # TODO: log ip of connected client
-        logger.info(f"Broadcast client connected.")
         await self.accept()
         await self.channel_layer.group_add(settings.BROADCAST_WEBSOCKET_GROUP_NAME, self.channel_name)
+        logger.info(f"client '{self.channel_name}' joined the broadcast group.")
 
     async def disconnect(self, code):
-        # TODO: log ip of disconnected client
-        logger.info("Client disconnected")
+        logger.info(f"client '{self.channel_name}' disconnected from the broadcast group.")
+        await self.channel_layer.group_discard(settings.BROADCAST_WEBSOCKET_GROUP_NAME, self.channel_name)
 
     async def internal_message(self, event):
         await self.send(event['text'])
@@ -130,6 +130,14 @@ class EventConsumer(AsyncJsonWebsocketConsumer):
             await self.send_json({"close": True})
             await self.close()
 
+    async def disconnect(self, code):
+        current_groups = set(self.scope['session'].pop('groups') if 'groups' in self.scope['session'] else [])
+        for group_name in current_groups:
+            await self.channel_layer.group_discard(
+                group_name,
+                self.channel_name,
+            )
+
     @database_sync_to_async
     def user_can_see_object_id(self, user_access, oid):
         # At this point user is a channels.auth.UserLazyObject object
@@ -187,7 +195,6 @@ class EventConsumer(AsyncJsonWebsocketConsumer):
                     group_name,
                     self.channel_name
                 )
-            logger.debug(f"Channel {self.channel_name} left groups {old_groups} and joined {new_groups_exclusive}")
             self.scope['session']['groups'] = new_groups
             await self.send_json({
                 "groups_current": list(new_groups),
@@ -213,31 +220,6 @@ def _dump_payload(payload):
         return None
 
 
-async def emit_channel_notification_async(group, payload):
-    from awx.main.wsbroadcast import wrap_broadcast_msg # noqa
-
-    payload_dumped = _dump_payload(payload)
-    if payload_dumped is None:
-        return
-
-    channel_layer = get_channel_layer()
-    await channel_layer.group_send(
-        group,
-        {
-            "type": "internal.message",
-            "text": payload_dumped
-        },
-    )
-
-    await channel_layer.group_send(
-        settings.BROADCAST_WEBSOCKET_GROUP_NAME,
-        {
-            "type": "internal.message",
-            "text": wrap_broadcast_msg(group, payload_dumped),
-        },
-    )
-
-
 def emit_channel_notification(group, payload):
     from awx.main.wsbroadcast import wrap_broadcast_msg # noqa
 

awx/main/credential_plugins/aim.py

@@ -1,15 +1,10 @@
-from .plugin import CredentialPlugin
+from .plugin import CredentialPlugin, CertFiles
 
 from urllib.parse import quote, urlencode, urljoin
 
 from django.utils.translation import ugettext_lazy as _
 import requests
 
-# AWX
-from awx.main.utils import (
-    create_temporary_fifo,
-)
-
 aim_inputs = {
     'fields': [{
         'id': 'url',
@@ -81,21 +76,13 @@ def aim_backend(**kwargs):
     request_qs = '?' + urlencode(query_params, quote_via=quote)
     request_url = urljoin(url, '/'.join(['AIMWebService', 'api', 'Accounts']))
 
-    cert = None
-    if client_cert and client_key:
-        cert = (
-            create_temporary_fifo(client_cert.encode()),
-            create_temporary_fifo(client_key.encode())
+    with CertFiles(client_cert, client_key) as cert:
+        res = requests.get(
+            request_url + request_qs,
+            timeout=30,
+            cert=cert,
+            verify=verify,
         )
-    elif client_cert:
-        cert = create_temporary_fifo(client_cert.encode())
-
-    res = requests.get(
-        request_url + request_qs,
-        timeout=30,
-        cert=cert,
-        verify=verify,
-    )
     res.raise_for_status()
     return res.json()['Content']
 

awx/main/credential_plugins/conjur.py

@@ -1,16 +1,11 @@
-from .plugin import CredentialPlugin
+from .plugin import CredentialPlugin, CertFiles
 
 import base64
-from urllib.parse import urljoin, quote_plus
+from urllib.parse import urljoin, quote
 
 from django.utils.translation import ugettext_lazy as _
 import requests
 
-# AWX
-from awx.main.utils import (
-    create_temporary_fifo,
-)
-
 
 conjur_inputs = {
     'fields': [{
@@ -55,9 +50,9 @@ conjur_inputs = {
 def conjur_backend(**kwargs):
     url = kwargs['url']
     api_key = kwargs['api_key']
-    account = quote_plus(kwargs['account'])
-    username = quote_plus(kwargs['username'])
-    secret_path = quote_plus(kwargs['secret_path'])
+    account = quote(kwargs['account'], safe='') 
+    username = quote(kwargs['username'], safe='')
+    secret_path = quote(kwargs['secret_path'], safe='')
     version = kwargs.get('secret_version')
     cacert = kwargs.get('cacert', None)
 
@@ -65,22 +60,20 @@ def conjur_backend(**kwargs):
         'headers': {'Content-Type': 'text/plain'},
         'data': api_key
     }
-    if cacert:
-        auth_kwargs['verify'] = create_temporary_fifo(cacert.encode())
 
-    # https://www.conjur.org/api.html#authentication-authenticate-post
-    resp = requests.post(
-        urljoin(url, '/'.join(['authn', account, username, 'authenticate'])),
-        **auth_kwargs
-    )
+    with CertFiles(cacert) as cert:
+        # https://www.conjur.org/api.html#authentication-authenticate-post
+        auth_kwargs['verify'] = cert
+        resp = requests.post(
+            urljoin(url, '/'.join(['authn', account, username, 'authenticate'])),
+            **auth_kwargs
+        )
     resp.raise_for_status()
     token = base64.b64encode(resp.content).decode('utf-8')
 
     lookup_kwargs = {
         'headers': {'Authorization': 'Token token="{}"'.format(token)},
     }
-    if cacert:
-        lookup_kwargs['verify'] = create_temporary_fifo(cacert.encode())
 
     # https://www.conjur.org/api.html#secrets-retrieve-a-secret-get
     path = urljoin(url, '/'.join([
@@ -92,7 +85,9 @@ def conjur_backend(**kwargs):
     if version:
         path = '?'.join([path, version])
 
-    resp = requests.get(path, timeout=30, **lookup_kwargs)
+    with CertFiles(cacert) as cert:
+        lookup_kwargs['verify'] = cert
+        resp = requests.get(path, timeout=30, **lookup_kwargs)
     resp.raise_for_status()
     return resp.text
 

awx/main/credential_plugins/hashivault.py

@@ -3,16 +3,11 @@ import os
 import pathlib
 from urllib.parse import urljoin
 
-from .plugin import CredentialPlugin
+from .plugin import CredentialPlugin, CertFiles
 
 import requests
 from django.utils.translation import ugettext_lazy as _
 
-# AWX
-from awx.main.utils import (
-    create_temporary_fifo,
-)
-
 base_inputs = {
     'fields': [{
         'id': 'url',
@@ -32,14 +27,33 @@ base_inputs = {
         'type': 'string',
         'multiline': True,
         'help_text': _('The CA certificate used to verify the SSL certificate of the Vault server')
-    }],
+    }, {
+        'id': 'role_id',
+        'label': _('AppRole role_id'),
+        'type': 'string',
+        'multiline': False,
+        'help_text': _('The Role ID for AppRole Authentication')
+    }, {
+        'id': 'secret_id',
+        'label': _('AppRole secret_id'),
+        'type': 'string',
+        'multiline': False,
+        'secret': True,
+        'help_text': _('The Secret ID for AppRole Authentication')
+    }
+    ],
     'metadata': [{
         'id': 'secret_path',
         'label': _('Path to Secret'),
         'type': 'string',
         'help_text': _('The path to the secret stored in the secret backend e.g, /some/secret/')
+    },{
+        'id': 'auth_path',
+        'label': _('Path to Auth'),
+        'type': 'string',
+        'help_text': _('The path where the Authentication method is mounted e.g, approle')
     }],
-    'required': ['url', 'token', 'secret_path'],
+    'required': ['url', 'secret_path'],
 }
 
 hashi_kv_inputs = copy.deepcopy(base_inputs)
@@ -88,8 +102,42 @@ hashi_ssh_inputs['metadata'] = [{
 hashi_ssh_inputs['required'].extend(['public_key', 'role'])
 
 
+def handle_auth(**kwargs):
+    token = None
+
+    if kwargs.get('token'):
+        token = kwargs['token']
+    elif kwargs.get('role_id') and kwargs.get('secret_id'):
+        token = approle_auth(**kwargs)
+    else:
+        raise Exception('Either token or AppRole parameters must be set')
+
+    return token
+
+
+def approle_auth(**kwargs):
+    role_id = kwargs['role_id']
+    secret_id = kwargs['secret_id']
+    auth_path = kwargs.get('auth_path') or 'approle'
+
+    url = urljoin(kwargs['url'], 'v1')
+    cacert = kwargs.get('cacert', None)
+
+    request_kwargs = {'timeout': 30}
+    # AppRole Login
+    request_kwargs['json'] = {'role_id': role_id, 'secret_id': secret_id}
+    sess = requests.Session()
+    request_url = '/'.join([url, 'auth', auth_path, 'login']).rstrip('/')
+    with CertFiles(cacert) as cert:
+        request_kwargs['verify'] = cert
+        resp = sess.post(request_url, **request_kwargs)
+    resp.raise_for_status()
+    token = resp.json()['auth']['client_token']
+    return token
+
+
 def kv_backend(**kwargs):
-    token = kwargs['token']
+    token = handle_auth(**kwargs)
     url = kwargs['url']
     secret_path = kwargs['secret_path']
     secret_backend = kwargs.get('secret_backend', None)
@@ -98,8 +146,6 @@ def kv_backend(**kwargs):
     api_version = kwargs['api_version']
 
     request_kwargs = {'timeout': 30}
-    if cacert:
-        request_kwargs['verify'] = create_temporary_fifo(cacert.encode())
 
     sess = requests.Session()
     sess.headers['Authorization'] = 'Bearer {}'.format(token)
@@ -126,7 +172,9 @@ def kv_backend(**kwargs):
             path_segments = [secret_path]
 
     request_url = urljoin(url, '/'.join(['v1'] + path_segments)).rstrip('/')
-    response = sess.get(request_url, **request_kwargs)
+    with CertFiles(cacert) as cert:
+        request_kwargs['verify'] = cert
+        response = sess.get(request_url, **request_kwargs)
     response.raise_for_status()
 
     json = response.json()
@@ -144,15 +192,13 @@ def kv_backend(**kwargs):
 
 
 def ssh_backend(**kwargs):
-    token = kwargs['token']
+    token = handle_auth(**kwargs)
     url = urljoin(kwargs['url'], 'v1')
     secret_path = kwargs['secret_path']
     role = kwargs['role']
     cacert = kwargs.get('cacert', None)
 
     request_kwargs = {'timeout': 30}
-    if cacert:
-        request_kwargs['verify'] = create_temporary_fifo(cacert.encode())
 
     request_kwargs['json'] = {'public_key': kwargs['public_key']}
     if kwargs.get('valid_principals'):
@@ -164,7 +210,10 @@ def ssh_backend(**kwargs):
     sess.headers['X-Vault-Token'] = token
     # https://www.vaultproject.io/api/secret/ssh/index.html#sign-ssh-key
     request_url = '/'.join([url, secret_path, 'sign', role]).rstrip('/')
-    resp = sess.post(request_url, **request_kwargs)
+
+    with CertFiles(cacert) as cert:
+        request_kwargs['verify'] = cert
+        resp = sess.post(request_url, **request_kwargs)
 
     resp.raise_for_status()
     return resp.json()['data']['signed_key']

awx/main/credential_plugins/plugin.py

@@ -1,3 +1,45 @@
+import os
+import tempfile
+
 from collections import namedtuple
 
 CredentialPlugin = namedtuple('CredentialPlugin', ['name', 'inputs', 'backend'])
+
+
+class CertFiles():
+    """
+    A context manager used for writing a certificate and (optional) key
+    to $TMPDIR, and cleaning up afterwards.
+
+    This is particularly useful as a shared resource for credential plugins
+    that want to pull cert/key data out of the database and persist it
+    temporarily to the file system so that it can loaded into the openssl
+    certificate chain (generally, for HTTPS requests plugins make via the
+    Python requests library)
+
+    with CertFiles(cert_data, key_data) as cert:
+        # cert is string representing a path to the cert or pemfile
+        # temporarily written to disk
+        requests.post(..., cert=cert)
+    """
+
+    certfile = None
+
+    def __init__(self, cert, key=None):
+        self.cert = cert
+        self.key = key
+
+    def __enter__(self):
+        if not self.cert:
+            return None
+        self.certfile = tempfile.NamedTemporaryFile('wb', delete=False)
+        self.certfile.write(self.cert.encode())
+        if self.key:
+            self.certfile.write(b'\n')
+            self.certfile.write(self.key.encode())
+        self.certfile.flush()
+        return str(self.certfile.name)
+
+    def __exit__(self, *args):
+        if self.certfile and os.path.exists(self.certfile.name):
+            os.remove(self.certfile.name)

awx/main/dispatch/periodic.py

@@ -22,7 +22,7 @@ class Scheduler(Scheduler):
 
         def run():
             ppid = os.getppid()
-            logger.warn(f'periodic beat started')
+            logger.warn('periodic beat started')
             while True:
                 if os.getppid() != ppid:
                     # if the parent PID changes, this process has been orphaned

awx/main/dispatch/worker/base.py

@@ -8,6 +8,7 @@ import sys
 import redis
 import json
 import psycopg2
+import time
 from uuid import UUID
 from queue import Empty as QueueEmpty
 
@@ -34,6 +35,7 @@ class WorkerSignalHandler:
 
     def __init__(self):
         self.kill_now = False
+        signal.signal(signal.SIGTERM, signal.SIG_DFL)
         signal.signal(signal.SIGINT, self.exit_gracefully)
 
     def exit_gracefully(self, *args, **kwargs):
@@ -116,13 +118,23 @@ class AWXConsumerRedis(AWXConsumerBase):
         super(AWXConsumerRedis, self).run(*args, **kwargs)
         self.worker.on_start()
 
-        queue = redis.Redis.from_url(settings.BROKER_URL)
+        time_to_sleep = 1
         while True:
-            res = queue.blpop(self.queues)
-            res = json.loads(res[1])
-            self.process_task(res)
-            if self.should_stop:
-                return
+            queue = redis.Redis.from_url(settings.BROKER_URL)
+            while True:
+                try:
+                    res = queue.blpop(self.queues)
+                    time_to_sleep = 1
+                    res = json.loads(res[1])
+                    self.process_task(res)
+                except redis.exceptions.RedisError:
+                    time_to_sleep = min(time_to_sleep * 2, 30)
+                    logger.exception(f"encountered an error communicating with redis. Reconnect attempt in {time_to_sleep} seconds")
+                    time.sleep(time_to_sleep)
+                except (json.JSONDecodeError, KeyError):
+                    logger.exception("failed to decode JSON message from redis")
+                if self.should_stop:
+                    return
 
 
 class AWXConsumerPG(AWXConsumerBase):

awx/main/dispatch/worker/callback.py

@@ -91,7 +91,7 @@ class CallbackBrokerWorker(BaseWorker):
                     for e in events:
                         try:
                             if (
-                                isinstance(exc, IntegrityError),
+                                isinstance(exc, IntegrityError) and
                                 getattr(e, 'host_id', '')
                             ):
                                 # this is one potential IntegrityError we can

awx/main/isolated/manager.py

@@ -58,7 +58,7 @@ class IsolatedManager(object):
                 os.chmod(temp.name, stat.S_IRUSR | stat.S_IWUSR | stat.S_IXUSR)
             for host in hosts:
                 inventory['all']['hosts'][host] = {
-                    "ansible_connection": "kubectl",
+                    "ansible_connection": "community.kubernetes.kubectl",
                     "ansible_kubectl_config": path,
                 }
         else:
@@ -74,6 +74,7 @@ class IsolatedManager(object):
         env['ANSIBLE_RETRY_FILES_ENABLED'] = 'False'
         env['ANSIBLE_HOST_KEY_CHECKING'] = str(settings.AWX_ISOLATED_HOST_KEY_CHECKING)
         env['ANSIBLE_LIBRARY'] = os.path.join(os.path.dirname(awx.__file__), 'plugins', 'isolated')
+        env['ANSIBLE_COLLECTIONS_PATHS'] = settings.AWX_ANSIBLE_COLLECTIONS_PATHS
         set_pythonpath(os.path.join(settings.ANSIBLE_VENV_PATH, 'lib'), env)
 
         def finished_callback(runner_obj):
@@ -109,7 +110,6 @@ class IsolatedManager(object):
             'cancel_callback': self.canceled_callback,
             'settings': {
                 'job_timeout': settings.AWX_ISOLATED_LAUNCH_TIMEOUT,
-                'pexpect_timeout': getattr(settings, 'PEXPECT_TIMEOUT', 5),
                 'suppress_ansible_output': True,
             },
         }
@@ -268,13 +268,6 @@ class IsolatedManager(object):
         # in the final sync
         self.consume_events()
 
-        # emit an EOF event
-        event_data = {
-            'event': 'EOF',
-            'final_counter': len(self.handled_events)
-        }
-        self.event_handler(event_data)
-
         return status, rc
 
     def consume_events(self):
@@ -287,7 +280,7 @@ class IsolatedManager(object):
         if os.path.exists(events_path):
             for event in set(os.listdir(events_path)) - self.handled_events:
                 path = os.path.join(events_path, event)
-                if os.path.exists(path):
+                if os.path.exists(path) and os.path.isfile(path):
                     try:
                         event_data = json.load(
                             open(os.path.join(events_path, event), 'r')
@@ -420,8 +413,4 @@ class IsolatedManager(object):
         status, rc = self.dispatch(playbook, module, module_args)
         if status == 'successful':
             status, rc = self.check()
-        else:
-            # emit an EOF event
-            event_data = {'event': 'EOF', 'final_counter': 0}
-            self.event_handler(event_data)
         return status, rc

awx/main/management/commands/callback_stats.py

@@ -34,7 +34,7 @@ class Command(BaseCommand):
                 if clear:
                     for i in range(12):
                         sys.stdout.write('\x1b[1A\x1b[2K')
-                for l in lines:
-                    print(l)
+                for line in lines:
+                    print(line)
                 clear = True
                 time.sleep(.25)

awx/main/management/commands/inventory_import.py

@@ -169,7 +169,7 @@ class AnsibleInventoryLoader(object):
             self.tmp_private_dir = build_proot_temp_dir()
             logger.debug("Using fresh temporary directory '{}' for isolation.".format(self.tmp_private_dir))
             kwargs['proot_temp_dir'] = self.tmp_private_dir
-            kwargs['proot_show_paths'] = [functioning_dir(self.source)]
+            kwargs['proot_show_paths'] = [functioning_dir(self.source), settings.AWX_ANSIBLE_COLLECTIONS_PATHS]
         logger.debug("Running from `{}` working directory.".format(cwd))
 
         if self.venv_path != settings.ANSIBLE_VENV_PATH:
@@ -496,12 +496,6 @@ class Command(BaseCommand):
             group_names = all_group_names[offset:(offset + self._batch_size)]
             for group_pk in groups_qs.filter(name__in=group_names).values_list('pk', flat=True):
                 del_group_pks.discard(group_pk)
-        if self.inventory_source.deprecated_group_id in del_group_pks:  # TODO: remove in 3.3
-            logger.warning(
-                'Group "%s" from v1 API is not deleted by overwrite',
-                self.inventory_source.deprecated_group.name
-            )
-            del_group_pks.discard(self.inventory_source.deprecated_group_id)
         # Now delete all remaining groups in batches.
         all_del_pks = sorted(list(del_group_pks))
         for offset in range(0, len(all_del_pks), self._batch_size):
@@ -534,12 +528,6 @@ class Command(BaseCommand):
         # Set of all host pks managed by this inventory source
         all_source_host_pks = self._existing_host_pks()
         for db_group in db_groups.all():
-            if self.inventory_source.deprecated_group_id == db_group.id:  # TODO: remove in 3.3
-                logger.debug(
-                    'Group "%s" from v1 API child group/host connections preserved',
-                    db_group.name
-                )
-                continue
             # Delete child group relationships not present in imported data.
             db_children = db_group.children
             db_children_name_pk_map = dict(db_children.values_list('name', 'pk'))
@@ -661,11 +649,12 @@ class Command(BaseCommand):
             if group_name in existing_group_names:
                 continue
             mem_group = self.all_group.all_groups[group_name]
+            group_desc = mem_group.variables.pop('_awx_description', 'imported')
             group = self.inventory.groups.update_or_create(
                 name=group_name,
                 defaults={
                     'variables':json.dumps(mem_group.variables),
-                    'description':'imported'
+                    'description':group_desc
                 }
             )[0]
             logger.debug('Group "%s" added', group.name)
@@ -788,8 +777,9 @@ class Command(BaseCommand):
         # Create any new hosts.
         for mem_host_name in sorted(mem_host_names_to_update):
             mem_host = self.all_group.all_hosts[mem_host_name]
-            host_attrs = dict(variables=json.dumps(mem_host.variables),
-                              description='imported')
+            import_vars = mem_host.variables
+            host_desc = import_vars.pop('_awx_description', 'imported')
+            host_attrs = dict(variables=json.dumps(import_vars), description=host_desc)
             enabled = self._get_enabled(mem_host.variables)
             if enabled is not None:
                 host_attrs['enabled'] = enabled
@@ -1090,7 +1080,7 @@ class Command(BaseCommand):
                             if settings.SQL_DEBUG:
                                 logger.warning('update computed fields took %d queries',
                                                len(connection.queries) - queries_before2)
-                        # Check if the license is valid. 
+                        # Check if the license is valid.
                         # If the license is not valid, a CommandError will be thrown,
                         # and inventory update will be marked as invalid.
                         # with transaction.atomic() will roll back the changes.

awx/main/management/commands/register_queue.py

@@ -16,31 +16,24 @@ class InstanceNotFound(Exception):
         super(InstanceNotFound, self).__init__(*args, **kwargs)
 
 
-class Command(BaseCommand):
+class RegisterQueue:
+    def __init__(self, queuename, controller, instance_percent, inst_min, hostname_list):
+        self.instance_not_found_err = None
+        self.queuename = queuename
+        self.controller = controller
+        self.instance_percent = instance_percent
+        self.instance_min = inst_min
+        self.hostname_list = hostname_list
 
-    def add_arguments(self, parser):
-        parser.add_argument('--queuename', dest='queuename', type=str,
-                            help='Queue to create/update')
-        parser.add_argument('--hostnames', dest='hostnames', type=str,
-                            help='Comma-Delimited Hosts to add to the Queue (will not remove already assigned instances)')
-        parser.add_argument('--controller', dest='controller', type=str,
-                            default='', help='The controlling group (makes this an isolated group)')
-        parser.add_argument('--instance_percent', dest='instance_percent', type=int, default=0,
-                            help='The percentage of active instances that will be assigned to this group'),
-        parser.add_argument('--instance_minimum', dest='instance_minimum', type=int, default=0,
-                            help='The minimum number of instance that will be retained for this group from available instances')
-
-
-    def get_create_update_instance_group(self, queuename, instance_percent, instance_min):
+    def get_create_update_instance_group(self):
         created = False
         changed = False
-
-        (ig, created) = InstanceGroup.objects.get_or_create(name=queuename)
-        if ig.policy_instance_percentage != instance_percent:
-            ig.policy_instance_percentage = instance_percent
+        (ig, created) = InstanceGroup.objects.get_or_create(name=self.queuename)
+        if ig.policy_instance_percentage != self.instance_percent:
+            ig.policy_instance_percentage = self.instance_percent
             changed = True
-        if ig.policy_instance_minimum != instance_min:
-            ig.policy_instance_minimum = instance_min
+        if ig.policy_instance_minimum != self.instance_min:
+            ig.policy_instance_minimum = self.instance_min
             changed = True
 
         if changed:
@@ -48,12 +41,12 @@ class Command(BaseCommand):
 
         return (ig, created, changed)
 
-    def update_instance_group_controller(self, ig, controller):
+    def update_instance_group_controller(self, ig):
         changed = False
         control_ig = None
 
-        if controller:
-            control_ig = InstanceGroup.objects.filter(name=controller).first()
+        if self.controller:
+            control_ig = InstanceGroup.objects.filter(name=self.controller).first()
 
         if control_ig and ig.controller_id != control_ig.pk:
             ig.controller = control_ig
@@ -62,10 +55,10 @@ class Command(BaseCommand):
 
         return (control_ig, changed)
 
-    def add_instances_to_group(self, ig, hostname_list):
+    def add_instances_to_group(self, ig):
         changed = False
 
-        instance_list_unique = set([x.strip() for x in hostname_list if x])
+        instance_list_unique = set([x.strip() for x in self.hostname_list if x])
         instances = []
         for inst_name in instance_list_unique:
             instance = Instance.objects.filter(hostname=inst_name)
@@ -86,43 +79,61 @@ class Command(BaseCommand):
 
         return (instances, changed)
 
-    def handle(self, **options):
-        instance_not_found_err = None
-        queuename = options.get('queuename')
-        if not queuename:
-            raise CommandError("Specify `--queuename` to use this command.")
-        ctrl = options.get('controller')
-        inst_per = options.get('instance_percent')
-        inst_min = options.get('instance_minimum')
-        hostname_list = []
-        if options.get('hostnames'):
-            hostname_list = options.get('hostnames').split(",")
-
+    def register(self):
         with advisory_lock('cluster_policy_lock'):
             with transaction.atomic():
                 changed2 = False
                 changed3 = False
-                (ig, created, changed1) = self.get_create_update_instance_group(queuename, inst_per, inst_min)
+                (ig, created, changed1) = self.get_create_update_instance_group()
                 if created:
                     print("Creating instance group {}".format(ig.name))
                 elif not created:
                     print("Instance Group already registered {}".format(ig.name))
 
-                if ctrl:
-                    (ig_ctrl, changed2) = self.update_instance_group_controller(ig, ctrl)
+                if self.controller:
+                    (ig_ctrl, changed2) = self.update_instance_group_controller(ig)
                     if changed2:
-                        print("Set controller group {} on {}.".format(ctrl, queuename))
+                        print("Set controller group {} on {}.".format(self.controller, self.queuename))
 
                 try:
-                    (instances, changed3) = self.add_instances_to_group(ig, hostname_list)
+                    (instances, changed3) = self.add_instances_to_group(ig)
                     for i in instances:
                         print("Added instance {} to {}".format(i.hostname, ig.name))
                 except InstanceNotFound as e:
-                    instance_not_found_err = e
+                    self.instance_not_found_err = e
 
         if any([changed1, changed2, changed3]):
             print('(changed: True)')
 
-        if instance_not_found_err:
-            print(instance_not_found_err.message)
+
+class Command(BaseCommand):
+
+    def add_arguments(self, parser):
+        parser.add_argument('--queuename', dest='queuename', type=str,
+                            help='Queue to create/update')
+        parser.add_argument('--hostnames', dest='hostnames', type=str,
+                            help='Comma-Delimited Hosts to add to the Queue (will not remove already assigned instances)')
+        parser.add_argument('--controller', dest='controller', type=str,
+                            default='', help='The controlling group (makes this an isolated group)')
+        parser.add_argument('--instance_percent', dest='instance_percent', type=int, default=0,
+                            help='The percentage of active instances that will be assigned to this group'),
+        parser.add_argument('--instance_minimum', dest='instance_minimum', type=int, default=0,
+                            help='The minimum number of instance that will be retained for this group from available instances')
+
+
+    def handle(self, **options):
+        queuename = options.get('queuename')
+        if not queuename:
+            raise CommandError("Specify `--queuename` to use this command.")
+        ctrl = options.get('controller')
+        inst_per = options.get('instance_percent')
+        instance_min = options.get('instance_minimum')
+        hostname_list = []
+        if options.get('hostnames'):
+            hostname_list = options.get('hostnames').split(",")
+
+        rq = RegisterQueue(queuename, ctrl, inst_per, instance_min, hostname_list)
+        rq.register()
+        if rq.instance_not_found_err:
+            print(rq.instance_not_found_err.message)
             sys.exit(1)

awx/main/management/commands/run_dispatcher.py

@@ -7,7 +7,6 @@ from django.core.cache import cache as django_cache
 from django.core.management.base import BaseCommand
 from django.db import connection as django_connection
 
-from awx.main.utils.handlers import AWXProxyHandler
 from awx.main.dispatch import get_local_queuename, reaper
 from awx.main.dispatch.control import Control
 from awx.main.dispatch.pool import AutoscalePool
@@ -56,11 +55,6 @@ class Command(BaseCommand):
         reaper.reap()
         consumer = None
 
-        # don't ship external logs inside the dispatcher's parent process
-        # this exists to work around a race condition + deadlock bug on fork
-        # in cpython itself:
-        # https://bugs.python.org/issue37429
-        AWXProxyHandler.disable()
         try:
             queues = ['tower_broadcast_all', get_local_queuename()]
             consumer = AWXConsumerPG(

awx/main/management/commands/run_wsbroadcast.py

@@ -2,9 +2,21 @@
 # All Rights Reserved.
 import logging
 import asyncio
+import datetime
+import re
+import redis
+import time
+from datetime import datetime as dt
 
 from django.core.management.base import BaseCommand
+from django.db import connection
+from django.db.models import Q
+from django.db.migrations.executor import MigrationExecutor
 
+from awx.main.analytics.broadcast_websocket import (
+    BroadcastWebsocketStatsManager,
+    safe_name,
+)
 from awx.main.wsbroadcast import BroadcastWebsocketManager
 
 
@@ -14,7 +26,137 @@ logger = logging.getLogger('awx.main.wsbroadcast')
 class Command(BaseCommand):
     help = 'Launch the websocket broadcaster'
 
+    def add_arguments(self, parser):
+        parser.add_argument('--status', dest='status', action='store_true',
+                            help='print the internal state of any running broadcast websocket')
+
+    @classmethod
+    def display_len(cls, s):
+        return len(re.sub('\x1b.*?m', '', s))
+
+    @classmethod
+    def _format_lines(cls, host_stats, padding=5):
+        widths = [0 for i in host_stats[0]]
+        for entry in host_stats:
+            for i, e in enumerate(entry):
+                if Command.display_len(e) > widths[i]:
+                    widths[i] = Command.display_len(e)
+        paddings = [padding for i in widths]
+
+        lines = []
+        for entry in host_stats:
+            line = ""
+            for pad, width, value in zip(paddings, widths, entry):
+                if len(value) > Command.display_len(value):
+                    width += len(value) - Command.display_len(value)
+                total_width = width + pad
+                line += f'{value:{total_width}}'
+            lines.append(line)
+        return lines
+
+    @classmethod
+    def get_connection_status(cls, me, hostnames, data):
+        host_stats = [('hostname', 'state', 'start time', 'duration (sec)')]
+        for h in hostnames:
+            connection_color = '91'    # red
+            h_safe = safe_name(h)
+            prefix = f'awx_{h_safe}'
+            connection_state = data.get(f'{prefix}_connection', 'N/A')
+            connection_started = 'N/A'
+            connection_duration = 'N/A'
+            if connection_state is None:
+                connection_state = 'unknown'
+            if connection_state == 'connected':
+                connection_color = '92' # green
+                connection_started = data.get(f'{prefix}_connection_start', 'Error')
+                if connection_started != 'Error':
+                    connection_started = datetime.datetime.fromtimestamp(connection_started)
+                    connection_duration = int((dt.now() - connection_started).total_seconds())
+
+            connection_state = f'\033[{connection_color}m{connection_state}\033[0m'
+
+            host_stats.append((h, connection_state, str(connection_started), str(connection_duration)))
+
+        return host_stats
+
+    @classmethod
+    def get_connection_stats(cls, me, hostnames, data):
+        host_stats = [('hostname', 'total', 'per minute')]
+        for h in hostnames:
+            h_safe = safe_name(h)
+            prefix = f'awx_{h_safe}'
+            messages_total = data.get(f'{prefix}_messages_received', '0')
+            messages_per_minute = data.get(f'{prefix}_messages_received_per_minute', '0')
+
+            host_stats.append((h, str(int(messages_total)), str(int(messages_per_minute))))
+
+        return host_stats
+
     def handle(self, *arg, **options):
+        # it's necessary to delay this import in case
+        # database migrations are still running
+        from awx.main.models.ha import Instance
+
+        executor = MigrationExecutor(connection)
+        migrating = bool(executor.migration_plan(executor.loader.graph.leaf_nodes()))
+        registered = False
+
+        if not migrating:
+            try:
+                Instance.objects.me()
+                registered = True
+            except RuntimeError:
+                pass
+
+        if migrating or not registered:
+            # In containerized deployments, migrations happen in the task container,
+            # and the services running there don't start until migrations are
+            # finished.
+            # *This* service runs in the web container, and it's possible that it can
+            # start _before_ migrations are finished, thus causing issues with the ORM
+            # queries it makes (specifically, conf.settings queries).
+            # This block is meant to serve as a sort of bail-out for the situation
+            # where migrations aren't yet finished (similar to the migration
+            # detection middleware that the uwsgi processes have) or when instance
+            # registration isn't done yet
+            logger.error('AWX is currently installing/upgrading.  Trying again in 5s...')
+            time.sleep(5)
+            return
+
+        if options.get('status'):
+            try:
+                stats_all = BroadcastWebsocketStatsManager.get_stats_sync()
+            except redis.exceptions.ConnectionError as e:
+                print(f"Unable to get Broadcast Websocket Status. Failed to connect to redis {e}")
+                return
+
+            data = {}
+            for family in stats_all:
+                if family.type == 'gauge' and len(family.samples) > 1:
+                    for sample in family.samples:
+                        if sample.value >= 1:
+                            data[family.name] = sample.labels[family.name]
+                            break
+                else:
+                    data[family.name] = family.samples[0].value
+
+            me = Instance.objects.me()
+            hostnames = [i.hostname for i in Instance.objects.exclude(Q(hostname=me.hostname) | Q(rampart_groups__controller__isnull=False))]
+
+            host_stats = Command.get_connection_status(me, hostnames, data)
+            lines = Command._format_lines(host_stats)
+
+            print(f'Broadcast websocket connection status from "{me.hostname}" to:')
+            print('\n'.join(lines))
+
+            host_stats = Command.get_connection_stats(me, hostnames, data)
+            lines = Command._format_lines(host_stats)
+
+            print(f'\nBroadcast websocket connection stats from "{me.hostname}" to:')
+            print('\n'.join(lines))
+
+            return
+
         try:
             broadcast_websocket_mgr = BroadcastWebsocketManager()
             task = broadcast_websocket_mgr.start()

awx/main/managers.py

@@ -121,6 +121,17 @@ class InstanceManager(models.Manager):
         if not hostname:
             hostname = settings.CLUSTER_HOST_ID
         with advisory_lock('instance_registration_%s' % hostname):
+            if settings.AWX_AUTO_DEPROVISION_INSTANCES:
+                # detect any instances with the same IP address.
+                # if one exists, set it to None
+                inst_conflicting_ip = self.filter(ip_address=ip_address).exclude(hostname=hostname)
+                if inst_conflicting_ip.exists():
+                    for other_inst in inst_conflicting_ip:
+                        other_hostname = other_inst.hostname
+                        other_inst.ip_address = None
+                        other_inst.save(update_fields=['ip_address'])
+                        logger.warning("IP address {0} conflict detected, ip address unset for host {1}.".format(ip_address, other_hostname))
+
             instance = self.filter(hostname=hostname)
             if instance.exists():
                 instance = instance.get()
@@ -138,8 +149,11 @@ class InstanceManager(models.Manager):
 
     def get_or_register(self):
         if settings.AWX_AUTO_DEPROVISION_INSTANCES:
+            from awx.main.management.commands.register_queue import RegisterQueue
             pod_ip = os.environ.get('MY_POD_IP')
-            return self.register(ip_address=pod_ip)
+            registered = self.register(ip_address=pod_ip)
+            RegisterQueue('tower', None, 100, 0, []).register()
+            return registered
         else:
             return (False, self.me())
 

awx/main/middleware.py

@@ -12,23 +12,19 @@ import urllib.parse
 
 from django.conf import settings
 from django.contrib.auth.models import User
-from django.db.models.signals import post_save
 from django.db.migrations.executor import MigrationExecutor
-from django.db import IntegrityError, connection
-from django.utils.functional import curry
+from django.db import connection
 from django.shortcuts import get_object_or_404, redirect
 from django.apps import apps
 from django.utils.deprecation import MiddlewareMixin
 from django.utils.translation import ugettext_lazy as _
 from django.urls import reverse, resolve
 
-from awx.main.models import ActivityStream
 from awx.main.utils.named_url_graph import generate_graph, GraphNode
 from awx.conf import fields, register
 
 
 logger = logging.getLogger('awx.main.middleware')
-analytics_logger = logging.getLogger('awx.analytics.activity_stream')
 perf_logger = logging.getLogger('awx.analytics.performance')
 
 
@@ -76,61 +72,6 @@ class TimingMiddleware(threading.local, MiddlewareMixin):
         return filepath
 
 
-class ActivityStreamMiddleware(threading.local, MiddlewareMixin):
-
-    def __init__(self, get_response=None):
-        self.disp_uid = None
-        self.instance_ids = []
-        super().__init__(get_response)
-
-    def process_request(self, request):
-        if hasattr(request, 'user') and request.user.is_authenticated:
-            user = request.user
-        else:
-            user = None
-
-        set_actor = curry(self.set_actor, user)
-        self.disp_uid = str(uuid.uuid1())
-        self.instance_ids = []
-        post_save.connect(set_actor, sender=ActivityStream, dispatch_uid=self.disp_uid, weak=False)
-
-    def process_response(self, request, response):
-        drf_request = getattr(request, 'drf_request', None)
-        drf_user = getattr(drf_request, 'user', None)
-        if self.disp_uid is not None:
-            post_save.disconnect(dispatch_uid=self.disp_uid)
-
-        for instance in ActivityStream.objects.filter(id__in=self.instance_ids):
-            if drf_user and drf_user.id:
-                instance.actor = drf_user
-                try:
-                    instance.save(update_fields=['actor'])
-                    analytics_logger.info('Activity Stream update entry for %s' % str(instance.object1),
-                                          extra=dict(changes=instance.changes, relationship=instance.object_relationship_type,
-                                          actor=drf_user.username, operation=instance.operation,
-                                          object1=instance.object1, object2=instance.object2))
-                except IntegrityError:
-                    logger.debug("Integrity Error saving Activity Stream instance for id : " + str(instance.id))
-            # else:
-            #     obj1_type_actual = instance.object1_type.split(".")[-1]
-            #     if obj1_type_actual in ("InventoryUpdate", "ProjectUpdate", "Job") and instance.id is not None:
-            #         instance.delete()
-
-        self.instance_ids = []
-        return response
-
-    def set_actor(self, user, sender, instance, **kwargs):
-        if sender == ActivityStream:
-            if isinstance(user, User) and instance.actor is None:
-                user = User.objects.filter(id=user.id)
-                if user.exists():
-                    user = user[0]
-                    instance.actor = user
-            else:
-                if instance.id not in self.instance_ids:
-                    self.instance_ids.append(instance.id)
-
-
 class SessionTimeoutMiddleware(MiddlewareMixin):
     """
     Resets the session timeout for both the UI and the actual session for the API

awx/main/migrations/0006_v320_release.py

@@ -464,7 +464,7 @@ class Migration(migrations.Migration):
         migrations.AddField(
             model_name='unifiedjob',
             name='instance_group',
-            field=models.ForeignKey(on_delete=models.SET_NULL, default=None, blank=True, to='main.InstanceGroup', help_text='The Rampart/Instance group the job was run under', null=True),
+            field=models.ForeignKey(on_delete=models.SET_NULL, default=None, blank=True, to='main.InstanceGroup', help_text='The Instance group the job was run under', null=True),
         ),
         migrations.AddField(
             model_name='unifiedjobtemplate',

awx/main/migrations/0032_v330_polymorphic_delete.py

@@ -16,6 +16,6 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='unifiedjob',
             name='instance_group',
-            field=models.ForeignKey(blank=True, default=None, help_text='The Rampart/Instance group the job was run under', null=True, on_delete=awx.main.utils.polymorphic.SET_NULL, to='main.InstanceGroup'),
+            field=models.ForeignKey(blank=True, default=None, help_text='The Instance group the job was run under', null=True, on_delete=awx.main.utils.polymorphic.SET_NULL, to='main.InstanceGroup'),
         ),
     ]

awx/main/migrations/0114_v370_remove_deprecated_manual_inventory_sources.py

@@ -0,0 +1,39 @@
+# Generated by Django 2.2.11 on 2020-04-03 00:11
+
+from django.db import migrations, models
+
+
+def remove_manual_inventory_sources(apps, schema_editor):
+    '''Previously we would automatically create inventory sources after
+    Group creation and we would use the parent Group as our interface for the user.
+    During that process we would create InventorySource that had a source of "manual".
+    '''
+    InventoryUpdate = apps.get_model('main', 'InventoryUpdate')
+    InventoryUpdate.objects.filter(source='').delete()
+    InventorySource = apps.get_model('main', 'InventorySource')
+    InventorySource.objects.filter(source='').delete()
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('main', '0113_v370_event_bigint'),
+    ]
+
+    operations = [
+        migrations.RemoveField(
+            model_name='inventorysource',
+            name='deprecated_group',
+        ),
+        migrations.RunPython(remove_manual_inventory_sources),
+        migrations.AlterField(
+            model_name='inventorysource',
+            name='source',
+            field=models.CharField(choices=[('file', 'File, Directory or Script'), ('scm', 'Sourced from a Project'), ('ec2', 'Amazon EC2'), ('gce', 'Google Compute Engine'), ('azure_rm', 'Microsoft Azure Resource Manager'), ('vmware', 'VMware vCenter'), ('satellite6', 'Red Hat Satellite 6'), ('cloudforms', 'Red Hat CloudForms'), ('openstack', 'OpenStack'), ('rhv', 'Red Hat Virtualization'), ('tower', 'Ansible Tower'), ('custom', 'Custom Script')], default=None, max_length=32),
+        ),
+        migrations.AlterField(
+            model_name='inventoryupdate',
+            name='source',
+            field=models.CharField(choices=[('file', 'File, Directory or Script'), ('scm', 'Sourced from a Project'), ('ec2', 'Amazon EC2'), ('gce', 'Google Compute Engine'), ('azure_rm', 'Microsoft Azure Resource Manager'), ('vmware', 'VMware vCenter'), ('satellite6', 'Red Hat Satellite 6'), ('cloudforms', 'Red Hat CloudForms'), ('openstack', 'OpenStack'), ('rhv', 'Red Hat Virtualization'), ('tower', 'Ansible Tower'), ('custom', 'Custom Script')], default=None, max_length=32),
+        ),
+    ]

awx/main/migrations/0115_v370_schedule_set_null.py

@@ -0,0 +1,24 @@
+# Generated by Django 2.2.11 on 2020-05-04 02:26
+
+import awx.main.utils.polymorphic
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('main', '0114_v370_remove_deprecated_manual_inventory_sources'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='unifiedjob',
+            name='schedule',
+            field=models.ForeignKey(default=None, editable=False, null=True, on_delete=awx.main.utils.polymorphic.SET_NULL, to='main.Schedule'),
+        ),
+        migrations.AlterField(
+            model_name='unifiedjobtemplate',
+            name='next_schedule',
+            field=models.ForeignKey(default=None, editable=False, null=True, on_delete=awx.main.utils.polymorphic.SET_NULL, related_name='unifiedjobtemplate_as_next_schedule+', to='main.Schedule'),
+        ),
+    ]

awx/main/migrations/0116_v400_remove_hipchat_notifications.py

@@ -0,0 +1,34 @@
+# Generated by Django 2.2.11 on 2020-05-19 02:27
+
+from django.db import migrations, models
+
+
+def remove_hipchat_notifications(apps, schema_editor):
+    '''
+    HipChat notifications are no longer in service, remove any that are found.
+    '''
+    Notification = apps.get_model('main', 'Notification')
+    Notification.objects.filter(notification_type='hipchat').delete()
+    NotificationTemplate = apps.get_model('main', 'NotificationTemplate')
+    NotificationTemplate.objects.filter(notification_type='hipchat').delete()
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('main', '0115_v370_schedule_set_null'),
+    ]
+
+    operations = [
+        migrations.RunPython(remove_hipchat_notifications),
+        migrations.AlterField(
+            model_name='notification',
+            name='notification_type',
+            field=models.CharField(choices=[('email', 'Email'), ('grafana', 'Grafana'), ('irc', 'IRC'), ('mattermost', 'Mattermost'), ('pagerduty', 'Pagerduty'), ('rocketchat', 'Rocket.Chat'), ('slack', 'Slack'), ('twilio', 'Twilio'), ('webhook', 'Webhook')], max_length=32),
+        ),
+        migrations.AlterField(
+            model_name='notificationtemplate',
+            name='notification_type',
+            field=models.CharField(choices=[('email', 'Email'), ('grafana', 'Grafana'), ('irc', 'IRC'), ('mattermost', 'Mattermost'), ('pagerduty', 'Pagerduty'), ('rocketchat', 'Rocket.Chat'), ('slack', 'Slack'), ('twilio', 'Twilio'), ('webhook', 'Webhook')], max_length=32),
+        ),
+    ]

awx/main/models/__init__.py

@@ -3,7 +3,7 @@
 
 # Django
 from django.conf import settings # noqa
-from django.db import connection, ProgrammingError
+from django.db import connection
 from django.db.models.signals import pre_delete  # noqa
 
 # AWX
@@ -91,14 +91,13 @@ def enforce_bigint_pk_migration():
         'main_systemjobevent'
     ):
         with connection.cursor() as cursor:
-            try:
-                cursor.execute(f'SELECT MAX(id) FROM _old_{tblname}')
-                if cursor.fetchone():
-                    from awx.main.tasks import migrate_legacy_event_data
-                    migrate_legacy_event_data.apply_async([tblname])
-            except ProgrammingError:
-                # the table is gone (migration is unnecessary)
-                pass
+            cursor.execute(
+                'SELECT 1 FROM information_schema.tables WHERE table_name=%s',
+                (f'_old_{tblname}',)
+            )
+            if bool(cursor.rowcount):
+                from awx.main.tasks import migrate_legacy_event_data
+                migrate_legacy_event_data.apply_async([tblname])
 
 
 def cleanup_created_modified_by(sender, **kwargs):

awx/main/models/credential/__init__.py

@@ -799,6 +799,10 @@ ManagedCredentialType(
             'id': 'project',
             'label': ugettext_noop('Project (Tenant Name)'),
             'type': 'string',
+        }, {
+            'id': 'project_domain_name',
+            'label': ugettext_noop('Project (Domain Name)'),
+            'type': 'string',
         }, {
             'id': 'domain',
             'label': ugettext_noop('Domain Name'),

awx/main/models/credential/injectors.py

@@ -77,6 +77,8 @@ def _openstack_data(cred):
                           username=cred.get_input('username', default=''),
                           password=cred.get_input('password', default=''),
                           project_name=cred.get_input('project', default=''))
+    if cred.has_input('project_domain_name'):
+        openstack_auth['project_domain_name'] = cred.get_input('project_domain_name', default='')
     if cred.has_input('domain'):
         openstack_auth['domain_name'] = cred.get_input('domain', default='')
     verify_state = cred.get_input('verify_ssl', default=True)

awx/main/models/events.py

@@ -7,7 +7,7 @@ from collections import defaultdict
 from django.db import models, DatabaseError, connection
 from django.utils.dateparse import parse_datetime
 from django.utils.text import Truncator
-from django.utils.timezone import utc
+from django.utils.timezone import utc, now
 from django.utils.translation import ugettext_lazy as _
 from django.utils.encoding import force_text
 
@@ -407,11 +407,14 @@ class BasePlaybookEvent(CreatedModifiedModel):
         except (KeyError, ValueError):
             kwargs.pop('created', None)
 
+        host_map = kwargs.pop('host_map', {})
+
         sanitize_event_keys(kwargs, cls.VALID_KEYS)
         workflow_job_id = kwargs.pop('workflow_job_id', None)
         event = cls(**kwargs)
         if workflow_job_id:
             setattr(event, 'workflow_job_id', workflow_job_id)
+        setattr(event, 'host_map', host_map)
         event._update_from_event_data()
         return event
 
@@ -484,29 +487,45 @@ class JobEvent(BasePlaybookEvent):
             if not self.job or not self.job.inventory:
                 logger.info('Event {} missing job or inventory, host summaries not updated'.format(self.pk))
                 return
-            qs = self.job.inventory.hosts.filter(name__in=hostnames)
             job = self.job
+
+            from awx.main.models import Host, JobHostSummary  # circular import
+            all_hosts = Host.objects.filter(
+                pk__in=self.host_map.values()
+            ).only('id')
+            existing_host_ids = set(h.id for h in all_hosts)
+
+            summaries = dict()
             for host in hostnames:
+                host_id = self.host_map.get(host, None)
+                if host_id not in existing_host_ids:
+                    host_id = None
                 host_stats = {}
                 for stat in ('changed', 'dark', 'failures', 'ignored', 'ok', 'processed', 'rescued', 'skipped'):
                     try:
                         host_stats[stat] = self.event_data.get(stat, {}).get(host, 0)
                     except AttributeError:  # in case event_data[stat] isn't a dict.
                         pass
-                if qs.filter(name=host).exists():
-                    host_actual = qs.get(name=host)
-                    host_summary, created = job.job_host_summaries.get_or_create(host=host_actual, host_name=host_actual.name, defaults=host_stats)
-                else:
-                    host_summary, created = job.job_host_summaries.get_or_create(host_name=host, defaults=host_stats)
+                summary = JobHostSummary(
+                    created=now(), modified=now(), job_id=job.id, host_id=host_id, host_name=host, **host_stats
+                )
+                summary.failed = bool(summary.dark or summary.failures)
+                summaries[(host_id, host)] = summary
+
+            JobHostSummary.objects.bulk_create(summaries.values())
+
+            # update the last_job_id and last_job_host_summary_id
+            # in single queries
+            host_mapping = dict(
+                (summary['host_id'], summary['id'])
+                for summary in JobHostSummary.objects.filter(job_id=job.id).values('id', 'host_id')
+            )
+            for h in all_hosts:
+                h.last_job_id = job.id
+                if h.id in host_mapping:
+                    h.last_job_host_summary_id = host_mapping[h.id]
+            Host.objects.bulk_update(all_hosts, ['last_job_id', 'last_job_host_summary_id'])
 
-                if not created:
-                    update_fields = []
-                    for stat, value in host_stats.items():
-                        if getattr(host_summary, stat) != value:
-                            setattr(host_summary, stat, value)
-                            update_fields.append(stat)
-                    if update_fields:
-                        host_summary.save(update_fields=update_fields)
 
     @property
     def job_verbosity(self):

awx/main/models/inventory.py

@@ -4,6 +4,7 @@
 # Python
 import datetime
 import time
+import json
 import logging
 import re
 import copy
@@ -821,7 +822,6 @@ class InventorySourceOptions(BaseModel):
     injectors = dict()
 
     SOURCE_CHOICES = [
-        ('', _('Manual')),
         ('file', _('File, Directory or Script')),
         ('scm', _('Sourced from a Project')),
         ('ec2', _('Amazon EC2')),
@@ -932,8 +932,8 @@ class InventorySourceOptions(BaseModel):
     source = models.CharField(
         max_length=32,
         choices=SOURCE_CHOICES,
-        blank=True,
-        default='',
+        blank=False,
+        default=None,
     )
     source_path = models.CharField(
         max_length=1024,
@@ -1237,14 +1237,6 @@ class InventorySource(UnifiedJobTemplate, InventorySourceOptions, CustomVirtualE
         on_delete=models.CASCADE,
     )
 
-    deprecated_group = models.OneToOneField(
-        'Group',
-        related_name='deprecated_inventory_source',
-        null=True,
-        default=None,
-        on_delete=models.CASCADE,
-    )
-
     source_project = models.ForeignKey(
         'Project',
         related_name='scm_inventory_sources',
@@ -1414,16 +1406,6 @@ class InventorySource(UnifiedJobTemplate, InventorySourceOptions, CustomVirtualE
                     started=list(started_notification_templates),
                     success=list(success_notification_templates))
 
-    def clean_source(self):  # TODO: remove in 3.3
-        source = self.source
-        if source and self.deprecated_group:
-            qs = self.deprecated_group.inventory_sources.filter(source__in=CLOUD_INVENTORY_SOURCES)
-            existing_sources = qs.exclude(pk=self.pk)
-            if existing_sources.count():
-                s = u', '.join([x.deprecated_group.name for x in existing_sources])
-                raise ValidationError(_('Unable to configure this item for cloud sync. It is already managed by %s.') % s)
-        return source
-
     def clean_update_on_project_update(self):
         if self.update_on_project_update is True and \
                 self.source == 'scm' and \
@@ -1512,8 +1494,6 @@ class InventoryUpdate(UnifiedJob, InventorySourceOptions, JobNotificationMixin,
         if self.inventory_source.inventory is not None:
             websocket_data.update(dict(inventory_id=self.inventory_source.inventory.pk))
 
-        if self.inventory_source.deprecated_group is not None:  # TODO: remove in 3.3
-            websocket_data.update(dict(group_id=self.inventory_source.deprecated_group.id))
         return websocket_data
 
     def get_absolute_url(self, request=None):
@@ -1633,6 +1613,11 @@ class PluginFileInjector(object):
     # base injector should be one of None, "managed", or "template"
     # this dictates which logic to borrow from playbook injectors
     base_injector = None
+    # every source should have collection, but these are set here
+    # so that a source without a collection will have null values
+    namespace = None
+    collection = None
+    collection_migration = '2.9'  # Starting with this version, we use collections
 
     def __init__(self, ansible_version):
         # This is InventoryOptions instance, could be source or inventory update
@@ -1659,7 +1644,11 @@ class PluginFileInjector(object):
         """
         if self.plugin_name is None:
             raise NotImplementedError('At minimum the plugin name is needed for inventory plugin use.')
-        return {'plugin': self.plugin_name}
+        if self.initial_version is None or Version(self.ansible_version) >= Version(self.collection_migration):
+            proper_name = f'{self.namespace}.{self.collection}.{self.plugin_name}'
+        else:
+            proper_name = self.plugin_name
+        return {'plugin': proper_name}
 
     def inventory_contents(self, inventory_update, private_data_dir):
         """Returns a string that is the content for the inventory file for the inventory plugin
@@ -1714,7 +1703,10 @@ class PluginFileInjector(object):
         return injected_env
 
     def get_plugin_env(self, inventory_update, private_data_dir, private_data_files):
-        return self._get_shared_env(inventory_update, private_data_dir, private_data_files)
+        env = self._get_shared_env(inventory_update, private_data_dir, private_data_files)
+        if self.initial_version is None or Version(self.ansible_version) >= Version(self.collection_migration):
+            env['ANSIBLE_COLLECTIONS_PATHS'] = settings.AWX_ANSIBLE_COLLECTIONS_PATHS
+        return env
 
     def get_script_env(self, inventory_update, private_data_dir, private_data_files):
         injected_env = self._get_shared_env(inventory_update, private_data_dir, private_data_files)
@@ -1759,6 +1751,8 @@ class azure_rm(PluginFileInjector):
     initial_version = '2.8'  # Driven by unsafe group names issue, hostvars, host names
     ini_env_reference = 'AZURE_INI_PATH'
     base_injector = 'managed'
+    namespace = 'azure'
+    collection = 'azcollection'
 
     def get_plugin_env(self, *args, **kwargs):
         ret = super(azure_rm, self).get_plugin_env(*args, **kwargs)
@@ -1890,9 +1884,11 @@ class azure_rm(PluginFileInjector):
 class ec2(PluginFileInjector):
     plugin_name = 'aws_ec2'
     # blocked by https://github.com/ansible/ansible/issues/54059
-    # initial_version = '2.8'  # Driven by unsafe group names issue, parent_group templating, hostvars
+    initial_version = '2.9'  # Driven by unsafe group names issue, parent_group templating, hostvars
     ini_env_reference = 'EC2_INI_PATH'
     base_injector = 'managed'
+    namespace = 'amazon'
+    collection = 'aws'
 
     def get_plugin_env(self, *args, **kwargs):
         ret = super(ec2, self).get_plugin_env(*args, **kwargs)
@@ -2032,6 +2028,9 @@ class ec2(PluginFileInjector):
                 grouping_data['key'] += ' | regex_replace("{rx}", "_")'.format(rx=legacy_regex)
         # end compatibility content
 
+        if source_vars.get('iam_role_arn', None):
+            ret['iam_role_arn'] = source_vars['iam_role_arn']
+
         # This was an allowed ec2.ini option, also plugin option, so pass through
         if source_vars.get('boto_profile', None):
             ret['boto_profile'] = source_vars['boto_profile']
@@ -2040,6 +2039,10 @@ class ec2(PluginFileInjector):
             # Using the plugin, but still want dashes whitelisted
             ret['use_contrib_script_compatible_sanitization'] = True
 
+        if source_vars.get('nested_groups') is False:
+            for this_keyed_group in keyed_groups:
+                this_keyed_group.pop('parent_group', None)
+
         if keyed_groups:
             ret['keyed_groups'] = keyed_groups
 
@@ -2051,18 +2054,35 @@ class ec2(PluginFileInjector):
         compose_dict.update(self._compat_compose_vars())
         # plugin provides "aws_ec2", but not this which the script gave
         ret['groups'] = {'ec2': True}
-        # public_ip as hostname is non-default plugin behavior, script behavior
-        ret['hostnames'] = [
-            'network-interface.addresses.association.public-ip',
-            'dns-name',
-            'private-dns-name'
-        ]
+        if source_vars.get('hostname_variable') is not None:
+            hnames = []
+            for expr in source_vars.get('hostname_variable').split(','):
+                if expr == 'public_dns_name':
+                    hnames.append('dns-name')
+                elif not expr.startswith('tag:') and '_' in expr:
+                    hnames.append(expr.replace('_', '-'))
+                else:
+                    hnames.append(expr)
+            ret['hostnames'] = hnames
+        else:
+            # public_ip as hostname is non-default plugin behavior, script behavior
+            ret['hostnames'] = [
+                'network-interface.addresses.association.public-ip',
+                'dns-name',
+                'private-dns-name'
+            ]
         # The script returned only running state by default, the plugin does not
         # https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html#options
         # options: pending | running | shutting-down | terminated | stopping | stopped
         inst_filters['instance-state-name'] = ['running']
         # end compatibility content
 
+        if source_vars.get('destination_variable') or source_vars.get('vpc_destination_variable'):
+            for fd in ('destination_variable', 'vpc_destination_variable'):
+                if source_vars.get(fd):
+                    compose_dict['ansible_host'] = source_vars.get(fd)
+                    break
+
         if compose_dict:
             ret['compose'] = compose_dict
 
@@ -2129,6 +2149,8 @@ class gce(PluginFileInjector):
     initial_version = '2.8'  # Driven by unsafe group names issue, hostvars
     ini_env_reference = 'GCE_INI_PATH'
     base_injector = 'managed'
+    namespace = 'google'
+    collection = 'cloud'
 
     def get_plugin_env(self, *args, **kwargs):
         ret = super(gce, self).get_plugin_env(*args, **kwargs)
@@ -2229,14 +2251,118 @@ class gce(PluginFileInjector):
 
 
 class vmware(PluginFileInjector):
-    # plugin_name = 'vmware_vm_inventory'  # FIXME: implement me
+    plugin_name = 'vmware_vm_inventory'
+    initial_version = '2.9'
     ini_env_reference = 'VMWARE_INI_PATH'
     base_injector = 'managed'
+    namespace = 'community'
+    collection = 'vmware'
 
     @property
     def script_name(self):
         return 'vmware_inventory.py'  # exception
 
+
+    def inventory_as_dict(self, inventory_update, private_data_dir):
+        ret = super(vmware, self).inventory_as_dict(inventory_update, private_data_dir)
+        ret['strict'] = False
+        # Documentation of props, see
+        # https://github.com/ansible/ansible/blob/devel/docs/docsite/rst/scenario_guides/vmware_scenarios/vmware_inventory_vm_attributes.rst
+        UPPERCASE_PROPS = [
+            "availableField",
+            "configIssue",
+            "configStatus",
+            "customValue",  # optional
+            "datastore",
+            "effectiveRole",
+            "guestHeartbeatStatus",  # optional
+            "layout",  # optional
+            "layoutEx",  # optional
+            "name",
+            "network",
+            "overallStatus",
+            "parentVApp",  # optional
+            "permission",
+            "recentTask",
+            "resourcePool",
+            "rootSnapshot",
+            "snapshot",  # optional
+            "triggeredAlarmState",
+            "value"
+        ]
+        NESTED_PROPS = [
+            "capability",
+            "config",
+            "guest",
+            "runtime",
+            "storage",
+            "summary",  # repeat of other properties
+        ]
+        ret['properties'] = UPPERCASE_PROPS + NESTED_PROPS
+        ret['compose'] = {'ansible_host': 'guest.ipAddress'}  # default value
+        ret['compose']['ansible_ssh_host'] = ret['compose']['ansible_host']
+        # the ansible_uuid was unique every host, every import, from the script
+        ret['compose']['ansible_uuid'] = '99999999 | random | to_uuid'
+        for prop in UPPERCASE_PROPS:
+            if prop == prop.lower():
+                continue
+            ret['compose'][prop.lower()] = prop
+        ret['with_nested_properties'] = True
+        # ret['property_name_format'] = 'lower_case'  # only dacrystal/topic/vmware-inventory-plugin-property-format
+
+        # process custom options
+        vmware_opts = dict(inventory_update.source_vars_dict.items())
+        if inventory_update.instance_filters:
+            vmware_opts.setdefault('host_filters', inventory_update.instance_filters)
+        if inventory_update.group_by:
+            vmware_opts.setdefault('groupby_patterns', inventory_update.group_by)
+
+        alias_pattern = vmware_opts.get('alias_pattern')
+        if alias_pattern:
+            ret.setdefault('hostnames', [])
+            for alias in alias_pattern.split(','):  # make best effort
+                striped_alias = alias.replace('{', '').replace('}', '').strip()  # make best effort
+                if not striped_alias:
+                    continue
+                ret['hostnames'].append(striped_alias)
+
+        host_pattern = vmware_opts.get('host_pattern')  # not working in script
+        if host_pattern:
+            stripped_hp = host_pattern.replace('{', '').replace('}', '').strip()  # make best effort
+            ret['compose']['ansible_host'] = stripped_hp
+            ret['compose']['ansible_ssh_host'] = stripped_hp
+
+        host_filters = vmware_opts.get('host_filters')
+        if host_filters:
+            ret.setdefault('filters', [])
+            for hf in host_filters.split(','):
+                striped_hf = hf.replace('{', '').replace('}', '').strip()  # make best effort
+                if not striped_hf:
+                    continue
+                ret['filters'].append(striped_hf)
+        else:
+            # default behavior filters by power state
+            ret['filters'] = ['runtime.powerState == "poweredOn"']
+
+        groupby_patterns = vmware_opts.get('groupby_patterns')
+        ret.setdefault('keyed_groups', [])
+        if groupby_patterns:
+            for pattern in groupby_patterns.split(','):
+                stripped_pattern = pattern.replace('{', '').replace('}', '').strip()  # make best effort
+                ret['keyed_groups'].append({
+                    'prefix': '', 'separator': '',
+                    'key': stripped_pattern
+                })
+        else:
+            # default groups from script
+            for entry in ('config.guestId', '"templates" if config.template else "guests"'):
+                ret['keyed_groups'].append({
+                    'prefix': '', 'separator': '',
+                    'key': entry
+                })
+
+        return ret
+
     def build_script_private_data(self, inventory_update, private_data_dir):
         cp = configparser.RawConfigParser()
         credential = inventory_update.get_cloud_credential()
@@ -2267,6 +2393,8 @@ class openstack(PluginFileInjector):
     plugin_name = 'openstack'
     # minimum version of 2.7.8 may be theoretically possible
     initial_version = '2.8'  # Driven by consistency with other sources
+    namespace = 'openstack'
+    collection = 'cloud'
 
     @property
     def script_name(self):
@@ -2318,7 +2446,10 @@ class openstack(PluginFileInjector):
         return self.build_script_private_data(inventory_update, private_data_dir, mk_cache=False)
 
     def get_plugin_env(self, inventory_update, private_data_dir, private_data_files):
-        return self.get_script_env(inventory_update, private_data_dir, private_data_files)
+        env = super(openstack, self).get_plugin_env(inventory_update, private_data_dir, private_data_files)
+        script_env = self.get_script_env(inventory_update, private_data_dir, private_data_files)
+        env.update(script_env)
+        return env
 
     def inventory_as_dict(self, inventory_update, private_data_dir):
         def use_host_name_for_name(a_bool_maybe):
@@ -2330,12 +2461,10 @@ class openstack(PluginFileInjector):
             else:
                 return 'uuid'
 
-        ret = dict(
-            plugin=self.plugin_name,
-            fail_on_errors=True,
-            expand_hostvars=True,
-            inventory_hostname=use_host_name_for_name(False),
-        )
+        ret = super(openstack, self).inventory_as_dict(inventory_update, private_data_dir)
+        ret['fail_on_errors'] = True
+        ret['expand_hostvars'] = True
+        ret['inventory_hostname'] = use_host_name_for_name(False)
         # Note: mucking with defaults will break import integrity
         # For the plugin, we need to use the same defaults as the old script
         # or else imports will conflict. To find script defaults you have
@@ -2360,19 +2489,43 @@ class openstack(PluginFileInjector):
 class rhv(PluginFileInjector):
     """ovirt uses the custom credential templating, and that is all
     """
-    # plugin_name = 'FIXME'  # contribute inventory plugin to Ansible
+    plugin_name = 'ovirt'
     base_injector = 'template'
+    initial_version = '2.9'
+    namespace = 'ovirt'
+    collection = 'ovirt'
 
     @property
     def script_name(self):
         return 'ovirt4.py'  # exception
 
+    def inventory_as_dict(self, inventory_update, private_data_dir):
+        ret = super(rhv, self).inventory_as_dict(inventory_update, private_data_dir)
+        ret['ovirt_insecure'] = False  # Default changed from script
+        # TODO: process strict option upstream
+        ret['compose'] = {
+            'ansible_host': '(devices.values() | list)[0][0] if devices else None'
+        }
+        ret['keyed_groups'] = []
+        for key in ('cluster', 'status'):
+            ret['keyed_groups'].append({'prefix': key, 'separator': '_', 'key': key})
+        ret['keyed_groups'].append({'prefix': 'tag', 'separator': '_', 'key': 'tags'})
+        ret['ovirt_hostname_preference'] = ['name', 'fqdn']
+        source_vars = inventory_update.source_vars_dict
+        for key, value in source_vars.items():
+            if key == 'plugin':
+                continue
+            ret[key] = value
+        return ret
+
 
 class satellite6(PluginFileInjector):
     plugin_name = 'foreman'
     ini_env_reference = 'FOREMAN_INI_PATH'
-    # initial_version = '2.8'  # FIXME: turn on after plugin is validated
+    initial_version = '2.9'
     # No base injector, because this does not work in playbooks. Bug??
+    namespace = 'theforeman'
+    collection = 'foreman'
 
     @property
     def script_name(self):
@@ -2434,18 +2587,123 @@ class satellite6(PluginFileInjector):
         # this assumes that this is merged
         # https://github.com/ansible/ansible/pull/52693
         credential = inventory_update.get_cloud_credential()
-        ret = {}
+        ret = super(satellite6, self).get_plugin_env(inventory_update, private_data_dir, private_data_files)
         if credential:
             ret['FOREMAN_SERVER'] = credential.get_input('host', default='')
             ret['FOREMAN_USER'] = credential.get_input('username', default='')
             ret['FOREMAN_PASSWORD'] = credential.get_input('password', default='')
         return ret
 
+    def inventory_as_dict(self, inventory_update, private_data_dir):
+        ret = super(satellite6, self).inventory_as_dict(inventory_update, private_data_dir)
+
+        group_patterns = '[]'
+        group_prefix = 'foreman_'
+        want_hostcollections = False
+        want_ansible_ssh_host = False
+        want_facts = True
+
+        foreman_opts = inventory_update.source_vars_dict.copy()
+        for k, v in foreman_opts.items():
+            if k == 'satellite6_group_patterns' and isinstance(v, str):
+                group_patterns = v
+            elif k == 'satellite6_group_prefix' and isinstance(v, str):
+                group_prefix = v
+            elif k == 'satellite6_want_hostcollections' and isinstance(v, bool):
+                want_hostcollections = v
+            elif k == 'satellite6_want_ansible_ssh_host' and isinstance(v, bool):
+                want_ansible_ssh_host = v
+            elif k == 'satellite6_want_facts' and isinstance(v, bool):
+                want_facts = v
+            else:
+                ret[k] = str(v)
+
+        # Compatibility content
+        group_by_hostvar = {
+            "environment":           {"prefix": "{}environment_".format(group_prefix),
+                                      "separator": "",
+                                      "key": "foreman['environment_name'] | lower | regex_replace(' ', '') | "
+                                             "regex_replace('[^A-Za-z0-9_]', '_') | regex_replace('none', '')"},
+            "location":              {"prefix": "{}location_".format(group_prefix),
+                                      "separator": "",
+                                      "key": "foreman['location_name'] | lower | regex_replace(' ', '') | regex_replace('[^A-Za-z0-9_]', '_')"},
+            "organization":          {"prefix": "{}organization_".format(group_prefix),
+                                      "separator": "",
+                                      "key": "foreman['organization_name'] | lower | regex_replace(' ', '') | regex_replace('[^A-Za-z0-9_]', '_')"},
+            "lifecycle_environment": {"prefix": "{}lifecycle_environment_".format(group_prefix),
+                                      "separator": "",
+                                      "key": "foreman['content_facet_attributes']['lifecycle_environment_name'] | "
+                                             "lower | regex_replace(' ', '') | regex_replace('[^A-Za-z0-9_]', '_')"},
+            "content_view":          {"prefix": "{}content_view_".format(group_prefix),
+                                      "separator": "",
+                                      "key": "foreman['content_facet_attributes']['content_view_name'] | "
+                                             "lower | regex_replace(' ', '') | regex_replace('[^A-Za-z0-9_]', '_')"}
+        }
+
+        ret['legacy_hostvars'] = True  # convert hostvar structure to the form used by the script
+        ret['want_params'] = True
+        ret['group_prefix'] = group_prefix
+        ret['want_hostcollections'] = want_hostcollections
+        ret['want_facts'] = want_facts
+
+        if want_ansible_ssh_host:
+            ret['compose'] = {'ansible_ssh_host': "foreman['ip6'] | default(foreman['ip'], true)"}
+        ret['keyed_groups'] = [group_by_hostvar[grouping_name] for grouping_name in group_by_hostvar]
+
+        def form_keyed_group(group_pattern):
+            """
+            Converts foreman group_pattern to
+            inventory plugin keyed_group
+
+            e.g. {app_param}-{tier_param}-{dc_param}
+                 becomes
+                 "%s-%s-%s" | format(app_param, tier_param, dc_param)
+            """
+            if type(group_pattern) is not str:
+                return None
+            params = re.findall('{[^}]*}', group_pattern)
+            if len(params) == 0:
+                return None
+
+            param_names = []
+            for p in params:
+                param_names.append(p[1:-1].strip())  # strip braces and space
+
+            # form keyed_group key by
+            # replacing curly braces with '%s'
+            # (for use with jinja's format filter)
+            key = group_pattern
+            for p in params:
+                key = key.replace(p, '%s', 1)
+
+            # apply jinja filter to key
+            key = '"{}" | format({})'.format(key, ', '.join(param_names))
+
+            keyed_group = {'key': key,
+                           'separator': ''}
+            return keyed_group
+
+        try:
+            group_patterns = json.loads(group_patterns)
+
+            if type(group_patterns) is list:
+                for group_pattern in group_patterns:
+                    keyed_group = form_keyed_group(group_pattern)
+                    if keyed_group:
+                        ret['keyed_groups'].append(keyed_group)
+        except json.JSONDecodeError:
+            logger.warning('Could not parse group_patterns. Expected JSON-formatted string, found: {}'
+                           .format(group_patterns))
+
+        return ret
+
 
 class cloudforms(PluginFileInjector):
     # plugin_name = 'FIXME'  # contribute inventory plugin to Ansible
     ini_env_reference = 'CLOUDFORMS_INI_PATH'
     # Also no base_injector because this does not work in playbooks
+    # namespace = ''  # does not have a collection
+    # collection = ''
 
     def build_script_private_data(self, inventory_update, private_data_dir):
         cp = configparser.RawConfigParser()
@@ -2481,6 +2739,8 @@ class tower(PluginFileInjector):
     plugin_name = 'tower'
     base_injector = 'template'
     initial_version = '2.8'  # Driven by "include_metadata" hostvars
+    namespace = 'awx'
+    collection = 'awx'
 
     def get_script_env(self, inventory_update, private_data_dir, private_data_files):
         env = super(tower, self).get_script_env(inventory_update, private_data_dir, private_data_files)
@@ -2489,6 +2749,7 @@ class tower(PluginFileInjector):
         return env
 
     def inventory_as_dict(self, inventory_update, private_data_dir):
+        ret = super(tower, self).inventory_as_dict(inventory_update, private_data_dir)
         # Credentials injected as env vars, same as script
         try:
             # plugin can take an actual int type
@@ -2496,11 +2757,9 @@ class tower(PluginFileInjector):
         except ValueError:
             # inventory_id could be a named URL
             identifier = iri_to_uri(inventory_update.instance_filters)
-        return {
-            'plugin': self.plugin_name,
-            'inventory_id': identifier,
-            'include_metadata': True  # used for license check
-        }
+        ret['inventory_id'] = identifier
+        ret['include_metadata'] = True  # used for license check
+        return ret
 
 
 for cls in PluginFileInjector.__subclasses__():

awx/main/models/jobs.py

@@ -439,13 +439,9 @@ class JobTemplate(UnifiedJobTemplate, JobOptions, SurveyJobTemplateMixin, Resour
             field = self._meta.get_field(field_name)
             if isinstance(field, models.ManyToManyField):
                 old_value = set(old_value.all())
-                if getattr(self, '_deprecated_credential_launch', False):
-                    # TODO: remove this code branch when support for `extra_credentials` goes away
-                    new_value = set(kwargs[field_name])
-                else:
-                    new_value = set(kwargs[field_name]) - old_value
-                    if not new_value:
-                        continue
+                new_value = set(kwargs[field_name]) - old_value
+                if not new_value:
+                    continue
 
             if new_value == old_value:
                 # no-op case: Fields the same as template's value
@@ -1133,20 +1129,6 @@ class JobHostSummary(CreatedModifiedModel):
         self.failed = bool(self.dark or self.failures)
         update_fields.append('failed')
         super(JobHostSummary, self).save(*args, **kwargs)
-        self.update_host_last_job_summary()
-
-    def update_host_last_job_summary(self):
-        update_fields = []
-        if self.host is None:
-            return
-        if self.host.last_job_id != self.job_id:
-            self.host.last_job_id = self.job_id
-            update_fields.append('last_job_id')
-        if self.host.last_job_host_summary_id != self.id:
-            self.host.last_job_host_summary_id = self.id
-            update_fields.append('last_job_host_summary_id')
-        if update_fields:
-            self.host.save(update_fields=update_fields)
 
 
 class SystemJobOptions(BaseModel):

awx/main/models/notifications.py

@@ -23,7 +23,6 @@ from awx.main.notifications.email_backend import CustomEmailBackend
 from awx.main.notifications.slack_backend import SlackBackend
 from awx.main.notifications.twilio_backend import TwilioBackend
 from awx.main.notifications.pagerduty_backend import PagerDutyBackend
-from awx.main.notifications.hipchat_backend import HipChatBackend
 from awx.main.notifications.webhook_backend import WebhookBackend
 from awx.main.notifications.mattermost_backend import MattermostBackend
 from awx.main.notifications.grafana_backend import GrafanaBackend
@@ -44,7 +43,6 @@ class NotificationTemplate(CommonModelNameNotUnique):
                           ('twilio', _('Twilio'), TwilioBackend),
                           ('pagerduty', _('Pagerduty'), PagerDutyBackend),
                           ('grafana', _('Grafana'), GrafanaBackend),
-                          ('hipchat', _('HipChat'), HipChatBackend),
                           ('webhook', _('Webhook'), WebhookBackend),
                           ('mattermost', _('Mattermost'), MattermostBackend),
                           ('rocketchat', _('Rocket.Chat'), RocketChatBackend),

awx/main/models/projects.py

@@ -199,7 +199,7 @@ class ProjectOptions(models.Model):
         results = []
         project_path = self.get_project_path()
         if project_path:
-            for dirpath, dirnames, filenames in os.walk(smart_str(project_path)):
+            for dirpath, dirnames, filenames in os.walk(smart_str(project_path), followlinks=settings.AWX_SHOW_PLAYBOOK_LINKS):
                 if skip_directory(dirpath):
                     continue
                 for filename in filenames:

awx/main/models/schedules.py

@@ -191,7 +191,7 @@ class Schedule(PrimordialModel, LaunchTimeConfig):
         return rrule
 
     @classmethod
-    def rrulestr(cls, rrule, **kwargs):
+    def rrulestr(cls, rrule, fast_forward=True, **kwargs):
         """
         Apply our own custom rrule parsing requirements
         """
@@ -205,11 +205,17 @@ class Schedule(PrimordialModel, LaunchTimeConfig):
                     'A valid TZID must be provided (e.g., America/New_York)'
                 )
 
-        if 'MINUTELY' in rrule or 'HOURLY' in rrule:
+        if fast_forward and ('MINUTELY' in rrule or 'HOURLY' in rrule):
             try:
                 first_event = x[0]
-                if first_event < now() - datetime.timedelta(days=365 * 5):
-                    raise ValueError('RRULE values with more than 1000 events are not allowed.')
+                if first_event < now():
+                    # hourly/minutely rrules with far-past DTSTART values
+                    # are *really* slow to precompute
+                    # start *from* one week ago to speed things up drastically
+                    dtstart = x._rrule[0]._dtstart.strftime(':%Y%m%dT')
+                    new_start = (now() - datetime.timedelta(days=7)).strftime(':%Y%m%dT')
+                    new_rrule = rrule.replace(dtstart, new_start)
+                    return Schedule.rrulestr(new_rrule, fast_forward=False)
             except IndexError:
                 pass
         return x

awx/main/models/unified_jobs.py

@@ -150,7 +150,7 @@ class UnifiedJobTemplate(PolymorphicModel, CommonModelNameNotUnique, Notificatio
         default=None,
         editable=False,
         related_name='%(class)s_as_next_schedule+',
-        on_delete=models.SET_NULL,
+        on_delete=polymorphic.SET_NULL,
     )
     status = models.CharField(
         max_length=32,
@@ -413,9 +413,8 @@ class UnifiedJobTemplate(PolymorphicModel, CommonModelNameNotUnique, Notificatio
         if 'extra_vars' in validated_kwargs:
             unified_job.handle_extra_data(validated_kwargs['extra_vars'])
 
-        if not getattr(self, '_deprecated_credential_launch', False):
-            # Create record of provided prompts for relaunch and rescheduling
-            unified_job.create_config_from_prompts(kwargs, parent=self)
+        # Create record of provided prompts for relaunch and rescheduling
+        unified_job.create_config_from_prompts(kwargs, parent=self)
 
         # manually issue the create activity stream entry _after_ M2M relations
         # have been associated to the UJ
@@ -587,7 +586,7 @@ class UnifiedJob(PolymorphicModel, PasswordFieldsModel, CommonModelNameNotUnique
         null=True,
         default=None,
         editable=False,
-        on_delete=models.SET_NULL,
+        on_delete=polymorphic.SET_NULL,
     )
     dependent_jobs = models.ManyToManyField(
         'self',
@@ -707,7 +706,7 @@ class UnifiedJob(PolymorphicModel, PasswordFieldsModel, CommonModelNameNotUnique
         null=True,
         default=None,
         on_delete=polymorphic.SET_NULL,
-        help_text=_('The Rampart/Instance group the job was run under'),
+        help_text=_('The Instance group the job was run under'),
     )
     organization = models.ForeignKey(
         'Organization',

awx/main/notifications/grafana_backend.py

@@ -13,6 +13,19 @@ from django.utils.translation import ugettext_lazy as _
 from awx.main.notifications.base import AWXBaseEmailBackend
 from awx.main.notifications.custom_notification_base import CustomNotificationBase
 
+DEFAULT_MSG = CustomNotificationBase.DEFAULT_MSG
+
+DEFAULT_APPROVAL_RUNNING_MSG = CustomNotificationBase.DEFAULT_APPROVAL_RUNNING_MSG
+DEFAULT_APPROVAL_RUNNING_BODY = CustomNotificationBase.DEFAULT_APPROVAL_RUNNING_BODY
+
+DEFAULT_APPROVAL_APPROVED_MSG = CustomNotificationBase.DEFAULT_APPROVAL_APPROVED_MSG
+DEFAULT_APPROVAL_APPROVED_BODY = CustomNotificationBase.DEFAULT_APPROVAL_APPROVED_BODY
+
+DEFAULT_APPROVAL_TIMEOUT_MSG = CustomNotificationBase.DEFAULT_APPROVAL_TIMEOUT_MSG
+DEFAULT_APPROVAL_TIMEOUT_BODY = CustomNotificationBase.DEFAULT_APPROVAL_TIMEOUT_BODY
+
+DEFAULT_APPROVAL_DENIED_MSG = CustomNotificationBase.DEFAULT_APPROVAL_DENIED_MSG
+DEFAULT_APPROVAL_DENIED_BODY = CustomNotificationBase.DEFAULT_APPROVAL_DENIED_BODY
 
 logger = logging.getLogger('awx.main.notifications.grafana_backend')
 
@@ -25,31 +38,13 @@ class GrafanaBackend(AWXBaseEmailBackend, CustomNotificationBase):
     sender_parameter = None
 
     DEFAULT_BODY = "{{ job_metadata }}"
-    default_messages = {
-        "started": {
-            "body": DEFAULT_BODY, "message": CustomNotificationBase.DEFAULT_MSG
-        },
-        "success": {
-            "body": DEFAULT_BODY, "message": CustomNotificationBase.DEFAULT_MSG
-        },
-        "error": {
-            "body": DEFAULT_BODY, "message": CustomNotificationBase.DEFAULT_MSG
-        },
-        "workflow_approval": {
-            "running": {
-                "message": CustomNotificationBase.DEFAULT_APPROVAL_RUNNING_MSG, "body": None
-            },
-            "approved": {
-                "message": CustomNotificationBase.DEFAULT_APPROVAL_APPROVED_MSG, "body": None
-            },
-            "timed_out": {
-                "message": CustomNotificationBase.DEFAULT_APPROVAL_TIMEOUT_MSG, "body": None
-            },
-            "denied": {
-                "message": CustomNotificationBase.DEFAULT_APPROVAL_DENIED_MSG, "body": None
-            }
-        }
-    }
+    default_messages = {"started": {"body": DEFAULT_BODY, "message": DEFAULT_MSG},
+                        "success": {"body": DEFAULT_BODY, "message": DEFAULT_MSG},
+                        "error": {"body": DEFAULT_BODY, "message": DEFAULT_MSG},
+                        "workflow_approval": {"running": {"message": DEFAULT_APPROVAL_RUNNING_MSG, "body": DEFAULT_APPROVAL_RUNNING_BODY},
+                                              "approved": {"message": DEFAULT_APPROVAL_APPROVED_MSG,"body": DEFAULT_APPROVAL_APPROVED_BODY},
+                                              "timed_out": {"message": DEFAULT_APPROVAL_TIMEOUT_MSG, "body": DEFAULT_APPROVAL_TIMEOUT_BODY},
+                                              "denied": {"message": DEFAULT_APPROVAL_DENIED_MSG, "body": DEFAULT_APPROVAL_DENIED_BODY}}}
 
     def __init__(self, grafana_key,dashboardId=None, panelId=None, annotation_tags=None, grafana_no_verify_ssl=False, isRegion=True,
                  fail_silently=False, **kwargs):
@@ -89,7 +84,8 @@ class GrafanaBackend(AWXBaseEmailBackend, CustomNotificationBase):
             grafana_data['isRegion'] = self.isRegion
             grafana_data['dashboardId'] = self.dashboardId
             grafana_data['panelId'] = self.panelId
-            grafana_data['tags'] = self.annotation_tags
+            if self.annotation_tags:
+                grafana_data['tags'] = self.annotation_tags
             grafana_data['text'] = m.subject
             grafana_headers['Authorization'] = "Bearer {}".format(self.grafana_key)
             grafana_headers['Content-Type'] = "application/json"

awx/main/notifications/hipchat_backend.py

@@ -1,54 +0,0 @@
-# Copyright (c) 2016 Ansible, Inc.
-# All Rights Reserved.
-
-import logging
-
-import requests
-
-from django.utils.encoding import smart_text
-from django.utils.translation import ugettext_lazy as _
-
-from awx.main.notifications.base import AWXBaseEmailBackend
-from awx.main.notifications.custom_notification_base import CustomNotificationBase
-
-logger = logging.getLogger('awx.main.notifications.hipchat_backend')
-
-
-class HipChatBackend(AWXBaseEmailBackend, CustomNotificationBase):
-
-    init_parameters = {"token": {"label": "Token", "type": "password"},
-                       "rooms": {"label": "Destination Rooms", "type": "list"},
-                       "color": {"label": "Notification Color", "type": "string"},
-                       "api_url": {"label": "API Url (e.g: https://mycompany.hipchat.com)", "type": "string"},
-                       "notify": {"label": "Notify room", "type": "bool"},
-                       "message_from": {"label": "Label to be shown with notification", "type": "string"}}
-    recipient_parameter = "rooms"
-    sender_parameter = "message_from"
-
-    def __init__(self, token, color, api_url, notify, fail_silently=False, **kwargs):
-        super(HipChatBackend, self).__init__(fail_silently=fail_silently)
-        self.token = token
-        if color is not None:
-            self.color = color.lower()
-        self.api_url = api_url
-        self.notify = notify
-        
-    def send_messages(self, messages):
-        sent_messages = 0
-
-        for m in messages:
-            for rcp in m.recipients():
-                r = requests.post("{}/v2/room/{}/notification".format(self.api_url, rcp),
-                                  params={"auth_token": self.token},
-                                  verify=False,
-                                  json={"color": self.color,
-                                        "message": m.subject,
-                                        "notify": self.notify,
-                                        "from": m.from_email,
-                                        "message_format": "text"})
-                if r.status_code != 204:
-                    logger.error(smart_text(_("Error sending messages: {}").format(r.text)))
-                    if not self.fail_silently:
-                        raise Exception(smart_text(_("Error sending message to hipchat: {}").format(r.text)))
-                sent_messages += 1
-        return sent_messages

awx/main/notifications/mattermost_backend.py

@@ -3,7 +3,6 @@
 
 import logging
 import requests
-import json
 
 from django.utils.encoding import smart_text
 from django.utils.translation import ugettext_lazy as _
@@ -45,7 +44,7 @@ class MattermostBackend(AWXBaseEmailBackend, CustomNotificationBase):
             payload['text'] = m.subject
 
             r = requests.post("{}".format(m.recipients()[0]),
-                              data=json.dumps(payload), verify=(not self.mattermost_no_verify_ssl))
+                              json=payload, verify=(not self.mattermost_no_verify_ssl))
             if r.status_code >= 400:
                 logger.error(smart_text(_("Error sending notification mattermost: {}").format(r.text)))
                 if not self.fail_silently:

awx/main/notifications/pagerduty_backend.py

@@ -11,9 +11,20 @@ from django.utils.translation import ugettext_lazy as _
 from awx.main.notifications.base import AWXBaseEmailBackend
 from awx.main.notifications.custom_notification_base import CustomNotificationBase
 
-DEFAULT_BODY = CustomNotificationBase.DEFAULT_BODY
 DEFAULT_MSG = CustomNotificationBase.DEFAULT_MSG
 
+DEFAULT_APPROVAL_RUNNING_MSG = CustomNotificationBase.DEFAULT_APPROVAL_RUNNING_MSG
+DEFAULT_APPROVAL_RUNNING_BODY = CustomNotificationBase.DEFAULT_APPROVAL_RUNNING_BODY
+
+DEFAULT_APPROVAL_APPROVED_MSG = CustomNotificationBase.DEFAULT_APPROVAL_APPROVED_MSG
+DEFAULT_APPROVAL_APPROVED_BODY = CustomNotificationBase.DEFAULT_APPROVAL_APPROVED_BODY
+
+DEFAULT_APPROVAL_TIMEOUT_MSG = CustomNotificationBase.DEFAULT_APPROVAL_TIMEOUT_MSG
+DEFAULT_APPROVAL_TIMEOUT_BODY = CustomNotificationBase.DEFAULT_APPROVAL_TIMEOUT_BODY
+
+DEFAULT_APPROVAL_DENIED_MSG = CustomNotificationBase.DEFAULT_APPROVAL_DENIED_MSG
+DEFAULT_APPROVAL_DENIED_BODY = CustomNotificationBase.DEFAULT_APPROVAL_DENIED_BODY
+
 logger = logging.getLogger('awx.main.notifications.pagerduty_backend')
 
 
@@ -30,10 +41,10 @@ class PagerDutyBackend(AWXBaseEmailBackend, CustomNotificationBase):
     default_messages = {"started": {"message": DEFAULT_MSG, "body": DEFAULT_BODY},
                         "success": {"message": DEFAULT_MSG, "body": DEFAULT_BODY},
                         "error": {"message": DEFAULT_MSG, "body": DEFAULT_BODY},
-                        "workflow_approval": {"running": {"message": DEFAULT_MSG, "body": DEFAULT_BODY},
-                                              "approved": {"message": DEFAULT_MSG,"body": DEFAULT_BODY},
-                                              "timed_out": {"message": DEFAULT_MSG, "body": DEFAULT_BODY},
-                                              "denied": {"message": DEFAULT_MSG, "body": DEFAULT_BODY}}}
+                        "workflow_approval": {"running": {"message": DEFAULT_APPROVAL_RUNNING_MSG, "body": DEFAULT_APPROVAL_RUNNING_BODY},
+                                              "approved": {"message": DEFAULT_APPROVAL_APPROVED_MSG,"body": DEFAULT_APPROVAL_APPROVED_BODY},
+                                              "timed_out": {"message": DEFAULT_APPROVAL_TIMEOUT_MSG, "body": DEFAULT_APPROVAL_TIMEOUT_BODY},
+                                              "denied": {"message": DEFAULT_APPROVAL_DENIED_MSG, "body": DEFAULT_APPROVAL_DENIED_BODY}}}
 
     def __init__(self, subdomain, token, fail_silently=False, **kwargs):
         super(PagerDutyBackend, self).__init__(fail_silently=fail_silently)

awx/main/routing.py

@@ -1,14 +1,37 @@
+import redis
+import logging
+
 from django.conf.urls import url
+from django.conf import settings
+
 from channels.auth import AuthMiddlewareStack
 from channels.routing import ProtocolTypeRouter, URLRouter
+
 from . import consumers
 
+
+logger = logging.getLogger('awx.main.routing')
+
+
+class AWXProtocolTypeRouter(ProtocolTypeRouter):
+    def __init__(self, *args, **kwargs):
+        try:
+            r = redis.Redis.from_url(settings.BROKER_URL)
+            for k in r.scan_iter('asgi:*', 500):
+                logger.debug(f"cleaning up Redis key {k}")
+                r.delete(k)
+        except redis.exceptions.RedisError as e:
+            logger.warn("encountered an error communicating with redis.")
+            raise e
+        super().__init__(*args, **kwargs)
+
+
 websocket_urlpatterns = [
     url(r'websocket/$', consumers.EventConsumer),
     url(r'websocket/broadcast/$', consumers.BroadcastConsumer),
 ]
 
-application = ProtocolTypeRouter({
+application = AWXProtocolTypeRouter({
     'websocket': AuthMiddlewareStack(
         URLRouter(websocket_urlpatterns)
     ),

awx/main/scheduler/dag_simple.py

@@ -123,7 +123,7 @@ class SimpleDAG(object):
         self.root_nodes.discard(to_obj_ord)
 
         if from_obj_ord is None and to_obj_ord is None:
-            raise LookupError("From object {} and to object not found".format(from_obj, to_obj))
+            raise LookupError("From object {} and to object {} not found".format(from_obj, to_obj))
         elif from_obj_ord is None:
             raise LookupError("From object not found {}".format(from_obj))
         elif to_obj_ord is None:
@@ -152,8 +152,8 @@ class SimpleDAG(object):
             return self._get_children_by_label(this_ord, label)
         else:
             nodes = []
-            for l in self.node_from_edges_by_label.keys():
-                nodes.extend(self._get_children_by_label(this_ord, l))
+            for label_obj in self.node_from_edges_by_label.keys():
+                nodes.extend(self._get_children_by_label(this_ord, label_obj))
             return nodes
 
     def _get_parents_by_label(self, node_index, label):
@@ -168,8 +168,8 @@ class SimpleDAG(object):
             return self._get_parents_by_label(this_ord, label)
         else:
             nodes = []
-            for l in self.node_to_edges_by_label.keys():
-                nodes.extend(self._get_parents_by_label(this_ord, l))
+            for label_obj in self.node_to_edges_by_label.keys():
+                nodes.extend(self._get_parents_by_label(this_ord, label_obj))
             return nodes
 
     def get_root_nodes(self):

awx/main/scheduler/task_manager.py

@@ -10,7 +10,7 @@ import random
 
 # Django
 from django.db import transaction, connection
-from django.utils.translation import ugettext_lazy as _
+from django.utils.translation import ugettext_lazy as _, gettext_noop
 from django.utils.timezone import now as tz_now
 
 # AWX
@@ -114,7 +114,7 @@ class TaskManager():
                         logger.info('Refusing to start recursive workflow-in-workflow id={}, wfjt={}, ancestors={}'.format(
                             job.id, spawn_node.unified_job_template.pk, [wa.pk for wa in workflow_ancestors]))
                         display_list = [spawn_node.unified_job_template] + workflow_ancestors
-                        job.job_explanation = _(
+                        job.job_explanation = gettext_noop(
                             "Workflow Job spawned from workflow could not start because it "
                             "would result in recursion (spawn order, most recent first: {})"
                         ).format(', '.join(['<{}>'.format(tmp) for tmp in display_list]))
@@ -123,8 +123,8 @@ class TaskManager():
                             job.id, spawn_node.unified_job_template.pk, [wa.pk for wa in workflow_ancestors]))
                 if not job._resources_sufficient_for_launch():
                     can_start = False
-                    job.job_explanation = _("Job spawned from workflow could not start because it "
-                                            "was missing a related resource such as project or inventory")
+                    job.job_explanation = gettext_noop("Job spawned from workflow could not start because it "
+                                                       "was missing a related resource such as project or inventory")
                 if can_start:
                     if workflow_job.start_args:
                         start_args = json.loads(decrypt_field(workflow_job, 'start_args'))
@@ -132,8 +132,8 @@ class TaskManager():
                         start_args = {}
                     can_start = job.signal_start(**start_args)
                     if not can_start:
-                        job.job_explanation = _("Job spawned from workflow could not start because it "
-                                                "was not in the right state or required manual credentials")
+                        job.job_explanation = gettext_noop("Job spawned from workflow could not start because it "
+                                                           "was not in the right state or required manual credentials")
                 if not can_start:
                     job.status = 'failed'
                     job.save(update_fields=['status', 'job_explanation'])
@@ -173,7 +173,7 @@ class TaskManager():
                 workflow_job.status = new_status
                 if reason:
                     logger.info(reason)
-                    workflow_job.job_explanation = _("No error handling paths found, marking workflow as failed")
+                    workflow_job.job_explanation = gettext_noop("No error handling paths found, marking workflow as failed")
                     update_fields.append('job_explanation')
                 workflow_job.start_args = ''  # blank field to remove encrypted passwords
                 workflow_job.save(update_fields=update_fields)
@@ -226,7 +226,7 @@ class TaskManager():
                 # non-Ansible jobs on isolated instances run on controller
                 task.instance_group = rampart_group.controller
                 task.execution_node = random.choice(list(rampart_group.controller.instances.all().values_list('hostname', flat=True)))
-                logger.debug('Submitting isolated {} to queue {}.'.format(
+                logger.debug('Submitting isolated {} to queue {} on node {}.'.format(
                              task.log_format, task.instance_group.name, task.execution_node))
             elif controller_node:
                 task.instance_group = rampart_group
@@ -581,3 +581,4 @@ class TaskManager():
                 logger.debug("Starting Scheduler")
                 with task_manager_bulk_reschedule():
                     self._schedule()
+                logger.debug("Finishing Scheduler")

awx/main/signals.py

@@ -52,6 +52,7 @@ from awx.conf.utils import conf_to_dict
 __all__ = []
 
 logger = logging.getLogger('awx.main.signals')
+analytics_logger = logging.getLogger('awx.analytics.activity_stream')
 
 # Update has_active_failures for inventory/groups when a Host/Group is deleted,
 # when a Host-Group or Group-Group relationship is updated, or when a Job is deleted
@@ -149,9 +150,9 @@ def rbac_activity_stream(instance, sender, **kwargs):
 
 
 def cleanup_detached_labels_on_deleted_parent(sender, instance, **kwargs):
-    for l in instance.labels.all():
-        if l.is_candidate_for_detach():
-            l.delete()
+    for label in instance.labels.all():
+        if label.is_candidate_for_detach():
+            label.delete()
 
 
 def save_related_job_templates(sender, instance, **kwargs):
@@ -363,12 +364,24 @@ def model_serializer_mapping():
     }
 
 
+def emit_activity_stream_change(instance):
+    if 'migrate' in sys.argv:
+        # don't emit activity stream external logs during migrations, it
+        # could be really noisy
+        return
+    from awx.api.serializers import ActivityStreamSerializer
+    actor = None
+    if instance.actor:
+        actor = instance.actor.username
+    summary_fields = ActivityStreamSerializer(instance).get_summary_fields(instance)
+    analytics_logger.info('Activity Stream update entry for %s' % str(instance.object1),
+                          extra=dict(changes=instance.changes, relationship=instance.object_relationship_type,
+                          actor=actor, operation=instance.operation,
+                          object1=instance.object1, object2=instance.object2, summary_fields=summary_fields))
+
+
 def activity_stream_create(sender, instance, created, **kwargs):
     if created and activity_stream_enabled:
-        # TODO: remove deprecated_group conditional in 3.3
-        # Skip recording any inventory source directly associated with a group.
-        if isinstance(instance, InventorySource) and instance.deprecated_group:
-            return
         _type = type(instance)
         if getattr(_type, '_deferred', False):
             return
@@ -380,7 +393,7 @@ def activity_stream_create(sender, instance, created, **kwargs):
                 '{} ({})'.format(c.name, c.id)
                 for c in instance.credentials.iterator()
             ]
-            changes['labels'] = [l.name for l in instance.labels.iterator()]
+            changes['labels'] = [label.name for label in instance.labels.iterator()]
             if 'extra_vars' in changes:
                 changes['extra_vars'] = instance.display_extra_vars()
         if type(instance) == OAuth2AccessToken:
@@ -399,6 +412,9 @@ def activity_stream_create(sender, instance, created, **kwargs):
         else:
             activity_entry.setting = conf_to_dict(instance)
             activity_entry.save()
+        connection.on_commit(
+            lambda: emit_activity_stream_change(activity_entry)
+        )
 
 
 def activity_stream_update(sender, instance, **kwargs):
@@ -430,15 +446,14 @@ def activity_stream_update(sender, instance, **kwargs):
     else:
         activity_entry.setting = conf_to_dict(instance)
         activity_entry.save()
+    connection.on_commit(
+        lambda: emit_activity_stream_change(activity_entry)
+    )
 
 
 def activity_stream_delete(sender, instance, **kwargs):
     if not activity_stream_enabled:
         return
-    # TODO: remove deprecated_group conditional in 3.3
-    # Skip recording any inventory source directly associated with a group.
-    if isinstance(instance, InventorySource) and instance.deprecated_group:
-        return
     # Inventory delete happens in the task system rather than request-response-cycle.
     # If we trigger this handler there we may fall into db-integrity-related race conditions.
     # So we add flag verification to prevent normal signal handling. This funciton will be
@@ -467,6 +482,9 @@ def activity_stream_delete(sender, instance, **kwargs):
         object1=object1,
         actor=get_current_user_or_none())
     activity_entry.save()
+    connection.on_commit(
+        lambda: emit_activity_stream_change(activity_entry)
+    )
 
 
 def activity_stream_associate(sender, instance, **kwargs):
@@ -540,6 +558,9 @@ def activity_stream_associate(sender, instance, **kwargs):
                 activity_entry.role.add(role)
                 activity_entry.object_relationship_type = obj_rel
                 activity_entry.save()
+            connection.on_commit(
+                lambda: emit_activity_stream_change(activity_entry)
+            )
 
 
 @receiver(current_user_getter)

awx/main/tasks.py

@@ -31,7 +31,7 @@ from django.db.models.fields.related import ForeignKey
 from django.utils.timezone import now, timedelta
 from django.utils.encoding import smart_str
 from django.contrib.auth.models import User
-from django.utils.translation import ugettext_lazy as _
+from django.utils.translation import ugettext_lazy as _, gettext_noop
 from django.core.cache import cache
 from django.core.exceptions import ObjectDoesNotExist
 
@@ -67,12 +67,13 @@ from awx.main.queue import CallbackQueueDispatcher
 from awx.main.isolated import manager as isolated_manager
 from awx.main.dispatch.publish import task
 from awx.main.dispatch import get_local_queuename, reaper
-from awx.main.utils import (get_ssh_version, update_scm_url,
+from awx.main.utils import (update_scm_url,
                             ignore_inventory_computed_fields,
                             ignore_inventory_group_removal, extract_ansible_vars, schedule_task_manager,
                             get_awx_version)
 from awx.main.utils.ansible import read_ansible_config
 from awx.main.utils.common import _get_ansible_version, get_custom_venv_choices
+from awx.main.utils.external_logging import reconfigure_rsyslog
 from awx.main.utils.safe_yaml import safe_dump, sanitize_jinja
 from awx.main.utils.reload import stop_local_services
 from awx.main.utils.pglock import advisory_lock
@@ -141,6 +142,9 @@ def dispatch_startup():
     # everybody has moved to bigint, and remove this code entirely
     enforce_bigint_pk_migration()
 
+    # Update Tower's rsyslog.conf file based on loggins settings in the db
+    reconfigure_rsyslog()
+
 
 def inform_cluster_of_shutdown():
     try:
@@ -280,6 +284,12 @@ def handle_setting_changes(setting_keys):
     logger.debug('cache delete_many(%r)', cache_keys)
     cache.delete_many(cache_keys)
 
+    if any([
+        setting.startswith('LOG_AGGREGATOR')
+        for setting in setting_keys
+    ]):
+        reconfigure_rsyslog()
+
 
 @task(queue='tower_broadcast_all')
 def delete_project_files(project_path):
@@ -348,6 +358,9 @@ def gather_analytics():
     from rest_framework.fields import DateTimeField
     if not settings.INSIGHTS_TRACKING_STATE:
         return
+    if not (settings.AUTOMATION_ANALYTICS_URL and settings.REDHAT_USERNAME and settings.REDHAT_PASSWORD):
+        logger.debug('Not gathering analytics, configuration is invalid')
+        return
     last_gather = Setting.objects.filter(key='AUTOMATION_ANALYTICS_LAST_GATHER').first()
     if last_gather:
         last_time = DateTimeField().to_internal_value(last_gather.value)
@@ -548,7 +561,8 @@ def awx_periodic_scheduler():
                 continue
             if not can_start:
                 new_unified_job.status = 'failed'
-                new_unified_job.job_explanation = "Scheduled job could not start because it was not in the right state or required manual credentials"
+                new_unified_job.job_explanation = gettext_noop("Scheduled job could not start because it \
+                    was not in the right state or required manual credentials")
                 new_unified_job.save(update_fields=['status', 'job_explanation'])
                 new_unified_job.websocket_emit_status("failed")
             emit_channel_notification('schedules-changed', dict(id=schedule.id, group_name="schedules"))
@@ -887,21 +901,14 @@ class BaseTask(object):
         private_data = self.build_private_data(instance, private_data_dir)
         private_data_files = {'credentials': {}}
         if private_data is not None:
-            ssh_ver = get_ssh_version()
-            ssh_too_old = True if ssh_ver == "unknown" else Version(ssh_ver) < Version("6.0")
-            openssh_keys_supported = ssh_ver != "unknown" and Version(ssh_ver) >= Version("6.5")
             for credential, data in private_data.get('credentials', {}).items():
-                # Bail out now if a private key was provided in OpenSSH format
-                # and we're running an earlier version (<6.5).
-                if 'OPENSSH PRIVATE KEY' in data and not openssh_keys_supported:
-                    raise RuntimeError(OPENSSH_KEY_ERROR)
                 # OpenSSH formatted keys must have a trailing newline to be
                 # accepted by ssh-add.
                 if 'OPENSSH PRIVATE KEY' in data and not data.endswith('\n'):
                     data += '\n'
                 # For credentials used with ssh-add, write to a named pipe which
                 # will be read then closed, instead of leaving the SSH key on disk.
-                if credential and credential.credential_type.namespace in ('ssh', 'scm') and not ssh_too_old:
+                if credential and credential.credential_type.namespace in ('ssh', 'scm'):
                     try:
                         os.mkdir(os.path.join(private_data_dir, 'env'))
                     except OSError as e:
@@ -1212,6 +1219,8 @@ class BaseTask(object):
             else:
                 event_data['host_name'] = ''
                 event_data['host_id'] = ''
+            if event_data.get('event') == 'playbook_on_stats':
+                event_data['host_map'] = self.host_map
 
         if isinstance(self, RunProjectUpdate):
             # it's common for Ansible's SCM modules to print
@@ -1222,10 +1231,12 @@ class BaseTask(object):
             # this is a _little_ expensive to filter
             # with regex, but project updates don't have many events,
             # so it *should* have a negligible performance impact
+            task = event_data.get('event_data', {}).get('task_action')
             try:
-                event_data_json = json.dumps(event_data)
-                event_data_json = UriCleaner.remove_sensitive(event_data_json)
-                event_data = json.loads(event_data_json)
+                if task in ('git', 'hg', 'svn'):
+                    event_data_json = json.dumps(event_data)
+                    event_data_json = UriCleaner.remove_sensitive(event_data_json)
+                    event_data = json.loads(event_data_json)
             except json.JSONDecodeError:
                 pass
 
@@ -1411,7 +1422,6 @@ class BaseTask(object):
                 'status_handler': self.status_handler,
                 'settings': {
                     'job_timeout': self.get_instance_timeout(self.instance),
-                    'pexpect_timeout': getattr(settings, 'PEXPECT_TIMEOUT', 5),
                     'suppress_ansible_output': True,
                     **process_isolation_params,
                     **resource_profiling_params,
@@ -1466,7 +1476,7 @@ class BaseTask(object):
                                                            params.get('module'),
                                                            module_args,
                                                            ident=str(self.instance.pk))
-                self.event_ct = len(isolated_manager_instance.handled_events)
+                self.finished_callback(None)
             else:
                 res = ansible_runner.interface.run(**params)
                 status = res.status
@@ -2064,29 +2074,34 @@ class RunProjectUpdate(BaseTask):
         if settings.GALAXY_IGNORE_CERTS:
             env['ANSIBLE_GALAXY_IGNORE'] = True
         # Set up the public Galaxy server, if enabled
+        galaxy_configured = False
         if settings.PUBLIC_GALAXY_ENABLED:
-            galaxy_servers = [settings.PUBLIC_GALAXY_SERVER]
+            galaxy_servers = [settings.PUBLIC_GALAXY_SERVER]  # static setting
         else:
+            galaxy_configured = True
             galaxy_servers = []
         # Set up fallback Galaxy servers, if configured
         if settings.FALLBACK_GALAXY_SERVERS:
+            galaxy_configured = True
             galaxy_servers = settings.FALLBACK_GALAXY_SERVERS + galaxy_servers
         # Set up the primary Galaxy server, if configured
         if settings.PRIMARY_GALAXY_URL:
+            galaxy_configured = True
             galaxy_servers = [{'id': 'primary_galaxy'}] + galaxy_servers
             for key in GALAXY_SERVER_FIELDS:
                 value = getattr(settings, 'PRIMARY_GALAXY_{}'.format(key.upper()))
                 if value:
                     galaxy_servers[0][key] = value
-        for server in galaxy_servers:
-            for key in GALAXY_SERVER_FIELDS:
-                if not server.get(key):
-                    continue
-                env_key = ('ANSIBLE_GALAXY_SERVER_{}_{}'.format(server.get('id', 'unnamed'), key)).upper()
-                env[env_key] = server[key]
-        if galaxy_servers:
-            # now set the precedence of galaxy servers
-            env['ANSIBLE_GALAXY_SERVER_LIST'] = ','.join([server.get('id', 'unnamed') for server in galaxy_servers])
+        if galaxy_configured:
+            for server in galaxy_servers:
+                for key in GALAXY_SERVER_FIELDS:
+                    if not server.get(key):
+                        continue
+                    env_key = ('ANSIBLE_GALAXY_SERVER_{}_{}'.format(server.get('id', 'unnamed'), key)).upper()
+                    env[env_key] = server[key]
+            if galaxy_servers:
+                # now set the precedence of galaxy servers
+                env['ANSIBLE_GALAXY_SERVER_LIST'] = ','.join([server.get('id', 'unnamed') for server in galaxy_servers])
         return env
 
     def _build_scm_url_extra_vars(self, project_update):
@@ -2397,7 +2412,7 @@ class RunInventoryUpdate(BaseTask):
 
     @property
     def proot_show_paths(self):
-        return [self.get_path_to('..', 'plugins', 'inventory')]
+        return [self.get_path_to('..', 'plugins', 'inventory'), settings.AWX_ANSIBLE_COLLECTIONS_PATHS]
 
     def build_private_data(self, inventory_update, private_data_dir):
         """
@@ -2727,9 +2742,12 @@ class RunAdHocCommand(BaseTask):
         env['ANSIBLE_LOAD_CALLBACK_PLUGINS'] = '1'
         env['ANSIBLE_SFTP_BATCH_MODE'] = 'False'
 
-        # Specify empty SSH args (should disable ControlPersist entirely for
-        # ad hoc commands).
-        env.setdefault('ANSIBLE_SSH_ARGS', '')
+        # Create a directory for ControlPath sockets that is unique to each
+        # ad hoc command and visible inside the proot environment (when enabled).
+        cp_dir = os.path.join(private_data_dir, 'cp')
+        if not os.path.exists(cp_dir):
+            os.mkdir(cp_dir, 0o700)
+        env['ANSIBLE_SSH_CONTROL_PATH'] = cp_dir
 
         return env
 

awx/main/tests/conftest.py

@@ -107,11 +107,6 @@ def workflow_job_template_factory():
     return create_workflow_job_template
 
 
-@pytest.fixture
-def get_ssh_version(mocker):
-    return mocker.patch('awx.main.tasks.get_ssh_version', return_value='OpenSSH_6.9p1, LibreSSL 2.1.8')
-
-
 @pytest.fixture
 def job_template_with_survey_passwords_unit(job_template_with_survey_passwords_factory):
     return job_template_with_survey_passwords_factory(persisted=False)

awx/main/tests/data/inventory/plugins/azure_rm/files/azure_rm.yml

@@ -39,5 +39,5 @@ keyed_groups:
   prefix: ''
   separator: ''
 plain_host_names: true
-plugin: azure_rm
+plugin: azure.azcollection.azure_rm
 use_contrib_script_compatible_sanitization: true

awx/main/tests/data/inventory/plugins/ec2/files/aws_ec2.yml

@@ -1,6 +1,6 @@
 boto_profile: /tmp/my_boto_stuff
 compose:
-  ansible_host: public_ip_address
+  ansible_host: public_dns_name
   ec2_account_id: owner_id
   ec2_ami_launch_index: ami_launch_index | string
   ec2_architecture: architecture
@@ -50,9 +50,8 @@ filters:
 groups:
   ec2: true
 hostnames:
-- network-interface.addresses.association.public-ip
 - dns-name
-- private-dns-name
+iam_role_arn: arn:aws:iam::123456789012:role/test-role
 keyed_groups:
 - key: placement.availability_zone
   parent_group: zones
@@ -75,7 +74,7 @@ keyed_groups:
   parent_group: '{{ placement.region }}'
   prefix: ''
   separator: ''
-plugin: aws_ec2
+plugin: amazon.aws.aws_ec2
 regions:
 - us-east-2
 - ap-south-1

awx/main/tests/data/inventory/plugins/gce/files/gcp_compute.yml

@@ -40,7 +40,7 @@ keyed_groups:
 - key: image
   prefix: ''
   separator: ''
-plugin: gcp_compute
+plugin: google.cloud.gcp_compute
 projects:
 - fooo
 retrieve_image_info: true

awx/main/tests/data/inventory/plugins/openstack/files/file_reference

@@ -8,6 +8,7 @@ clouds:
       auth_url: https://foo.invalid
       domain_name: fooo
       password: fooo
+      project_domain_name: fooo
       project_name: fooo
       username: fooo
     private: false

awx/main/tests/data/inventory/plugins/openstack/files/openstack.yml

@@ -1,4 +1,4 @@
 expand_hostvars: true
 fail_on_errors: true
 inventory_hostname: uuid
-plugin: openstack
+plugin: openstack.cloud.openstack

awx/main/tests/data/inventory/plugins/rhv/env.json

@@ -0,0 +1,7 @@
+{
+    "ANSIBLE_TRANSFORM_INVALID_GROUP_CHARS": "never",
+    "OVIRT_INI_PATH": "{{ file_reference }}",
+    "OVIRT_PASSWORD": "fooo",
+    "OVIRT_URL": "https://foo.invalid",
+    "OVIRT_USERNAME": "fooo"
+}

awx/main/tests/data/inventory/plugins/rhv/files/file_reference

@@ -0,0 +1,5 @@
+[ovirt]
+ovirt_url=https://foo.invalid
+ovirt_username=fooo
+ovirt_password=fooo
+ovirt_ca_file=fooo

awx/main/tests/data/inventory/plugins/rhv/files/ovirt.yml

@@ -0,0 +1,20 @@
+base_source_var: value_of_var
+compose:
+  ansible_host: (devices.values() | list)[0][0] if devices else None
+groups:
+  dev: '"dev" in tags'
+keyed_groups:
+- key: cluster
+  prefix: cluster
+  separator: _
+- key: status
+  prefix: status
+  separator: _
+- key: tags
+  prefix: tag
+  separator: _
+ovirt_hostname_preference:
+- name
+- fqdn
+ovirt_insecure: false
+plugin: ovirt.ovirt.ovirt

awx/main/tests/data/inventory/plugins/satellite6/files/foreman.yml

@@ -1 +1,29 @@
-plugin: foreman
+base_source_var: value_of_var
+compose:
+  ansible_ssh_host: foreman['ip6'] | default(foreman['ip'], true)
+group_prefix: foo_group_prefix
+keyed_groups:
+- key: foreman['environment_name'] | lower | regex_replace(' ', '') | regex_replace('[^A-Za-z0-9_]', '_') | regex_replace('none', '')
+  prefix: foo_group_prefixenvironment_
+  separator: ''
+- key: foreman['location_name'] | lower | regex_replace(' ', '') | regex_replace('[^A-Za-z0-9_]', '_')
+  prefix: foo_group_prefixlocation_
+  separator: ''
+- key: foreman['organization_name'] | lower | regex_replace(' ', '') | regex_replace('[^A-Za-z0-9_]', '_')
+  prefix: foo_group_prefixorganization_
+  separator: ''
+- key: foreman['content_facet_attributes']['lifecycle_environment_name'] | lower | regex_replace(' ', '') | regex_replace('[^A-Za-z0-9_]', '_')
+  prefix: foo_group_prefixlifecycle_environment_
+  separator: ''
+- key: foreman['content_facet_attributes']['content_view_name'] | lower | regex_replace(' ', '') | regex_replace('[^A-Za-z0-9_]', '_')
+  prefix: foo_group_prefixcontent_view_
+  separator: ''
+- key: '"%s-%s-%s" | format(app, tier, color)'
+  separator: ''
+- key: '"%s-%s" | format(app, color)'
+  separator: ''
+legacy_hostvars: true
+plugin: theforeman.foreman.foreman
+want_facts: true
+want_hostcollections: true
+want_params: true

awx/main/tests/data/inventory/plugins/tower/files/tower.yml

@@ -1,3 +1,3 @@
 include_metadata: true
 inventory_id: 42
-plugin: tower
+plugin: awx.awx.tower

awx/main/tests/data/inventory/plugins/vmware/env.json

@@ -0,0 +1,7 @@
+{
+    "ANSIBLE_TRANSFORM_INVALID_GROUP_CHARS": "never",
+    "VMWARE_HOST": "https://foo.invalid",
+    "VMWARE_PASSWORD": "fooo",
+    "VMWARE_USER": "fooo",
+    "VMWARE_VALIDATE_CERTS": "False"
+}

awx/main/tests/data/inventory/plugins/vmware/files/vmware_vm_inventory.yml

@@ -0,0 +1,55 @@
+compose:
+  ansible_host: guest.ipAddress
+  ansible_ssh_host: guest.ipAddress
+  ansible_uuid: 99999999 | random | to_uuid
+  availablefield: availableField
+  configissue: configIssue
+  configstatus: configStatus
+  customvalue: customValue
+  effectiverole: effectiveRole
+  guestheartbeatstatus: guestHeartbeatStatus
+  layoutex: layoutEx
+  overallstatus: overallStatus
+  parentvapp: parentVApp
+  recenttask: recentTask
+  resourcepool: resourcePool
+  rootsnapshot: rootSnapshot
+  triggeredalarmstate: triggeredAlarmState
+filters:
+- config.zoo == "DC0_H0_VM0"
+hostnames:
+- config.foo
+keyed_groups:
+- key: config.asdf
+  prefix: ''
+  separator: ''
+plugin: community.vmware.vmware_vm_inventory
+properties:
+- availableField
+- configIssue
+- configStatus
+- customValue
+- datastore
+- effectiveRole
+- guestHeartbeatStatus
+- layout
+- layoutEx
+- name
+- network
+- overallStatus
+- parentVApp
+- permission
+- recentTask
+- resourcePool
+- rootSnapshot
+- snapshot
+- triggeredAlarmState
+- value
+- capability
+- config
+- guest
+- runtime
+- storage
+- summary
+strict: false
+with_nested_properties: true

awx/main/tests/data/inventory/scripts/ec2/files/file_reference

@@ -1,9 +1,11 @@
 [ec2]
 base_source_var = value_of_var
 boto_profile = /tmp/my_boto_stuff
+iam_role_arn = arn:aws:iam::123456789012:role/test-role
+hostname_variable = public_dns_name
+destination_variable = public_dns_name
 regions = us-east-2,ap-south-1
 regions_exclude = us-gov-west-1,cn-north-1
-destination_variable = public_dns_name
 vpc_destination_variable = ip_address
 route53 = False
 all_instances = True

awx/main/tests/data/inventory/scripts/openstack/files/file_reference

@@ -10,6 +10,7 @@ clouds:
       auth_url: https://foo.invalid
       domain_name: fooo
       password: fooo
+      project_domain_name: fooo
       project_name: fooo
       username: fooo
     private: false

awx/main/tests/data/inventory/scripts/satellite6/files/file_reference

@@ -6,12 +6,12 @@ user = fooo
 password = fooo
 
 [ansible]
-group_patterns = foo_group_patterns
+group_patterns = ["{app}-{tier}-{color}", "{app}-{color}"]
 want_facts = True
 want_hostcollections = True
 group_prefix = foo_group_prefix
 want_ansible_ssh_host = True
-rich_params = True
+rich_params = False
 
 [cache]
 path = /tmp

awx/main/tests/data/inventory/scripts/vmware/files/file_reference

@@ -5,6 +5,7 @@ username = fooo
 password = fooo
 server = https://foo.invalid
 base_source_var = value_of_var
-host_filters = foobaa
-groupby_patterns = fouo
+alias_pattern = {{ config.foo }}
+host_filters = {{ config.zoo == "DC0_H0_VM0" }}
+groupby_patterns = {{ config.asdf }}
 

awx/main/tests/docs/test_swagger_generation.py

@@ -105,9 +105,6 @@ class TestSwaggerGeneration():
             'get', 'put', 'patch', 'delete'
         ]
 
-        # Test deprecated paths
-        assert paths['/api/v2/jobs/{id}/extra_credentials/']['get']['deprecated'] is True
-
     @pytest.mark.parametrize('path', [
         '/api/',
         '/api/v2/',
@@ -177,7 +174,7 @@ class TestSwaggerGeneration():
                 data
             )
             data = re.sub(
-                r'"action_node": "awx-[^"]+"',
+                r'"action_node": "[^"]+"',
                 '"action_node": "awx"',
                 data
             )

awx/main/tests/factories/tower.py

@@ -220,7 +220,7 @@ def create_job_template(name, roles=None, persisted=True, webhook_service='', **
     if 'organization' in kwargs:
         org = kwargs['organization']
         if type(org) is not Organization:
-            org = mk_organization(org, '%s-desc'.format(org), persisted=persisted)
+            org = mk_organization(org, org, persisted=persisted)
 
     if 'credential' in kwargs:
         cred = kwargs['credential']
@@ -298,7 +298,7 @@ def create_organization(name, roles=None, persisted=True, **kwargs):
     labels = {}
     notification_templates = {}
 
-    org = mk_organization(name, '%s-desc'.format(name), persisted=persisted)
+    org = mk_organization(name, name, persisted=persisted)
 
     if 'inventories' in kwargs:
         for i in kwargs['inventories']:
@@ -319,11 +319,11 @@ def create_organization(name, roles=None, persisted=True, **kwargs):
     users = generate_users(org, teams, False, persisted, users=kwargs.get('users'))
 
     if 'labels' in kwargs:
-        for l in kwargs['labels']:
-            if type(l) is Label:
-                labels[l.name] = l
+        for label_obj in kwargs['labels']:
+            if type(label_obj) is Label:
+                labels[label_obj.name] = label_obj
             else:
-                labels[l] = mk_label(l, organization=org, persisted=persisted)
+                labels[label_obj] = mk_label(label_obj, organization=org, persisted=persisted)
 
     if 'notification_templates' in kwargs:
         for nt in kwargs['notification_templates']:

awx/main/tests/functional/analytics/test_collectors.py

@@ -0,0 +1,161 @@
+import pytest
+import tempfile
+import os
+import shutil
+import csv
+
+from django.utils.timezone import now
+from datetime import timedelta
+from django.db.backends.sqlite3.base import SQLiteCursorWrapper
+
+from awx.main.analytics import collectors
+
+from awx.main.models import (
+    ProjectUpdate,
+    InventorySource,
+    WorkflowJob,
+    WorkflowJobNode,
+    JobTemplate,
+)
+
+
+@pytest.fixture
+def sqlite_copy_expert(request):
+    # copy_expert is postgres-specific, and SQLite doesn't support it; mock its
+    # behavior to test that it writes a file that contains stdout from events
+    path = tempfile.mkdtemp(prefix="copied_tables")
+
+    def write_stdout(self, sql, fd):
+        # Would be cool if we instead properly disected the SQL query and verified
+        # it that way. But instead, we just take the nieve approach here.
+        assert sql.startswith("COPY (")
+        assert sql.endswith(") TO STDOUT WITH CSV HEADER")
+
+        sql = sql.replace("COPY (", "")
+        sql = sql.replace(") TO STDOUT WITH CSV HEADER", "")
+        # sqlite equivalent
+        sql = sql.replace("ARRAY_AGG", "GROUP_CONCAT")
+
+        # Remove JSON style queries
+        # TODO: could replace JSON style queries with sqlite kind of equivalents
+        sql_new = []
+        for line in sql.split("\n"):
+            if line.find("main_jobevent.event_data::") == -1:
+                sql_new.append(line)
+            elif not line.endswith(","):
+                sql_new[-1] = sql_new[-1].rstrip(",")
+        sql = "\n".join(sql_new)
+
+        self.execute(sql)
+        results = self.fetchall()
+        headers = [i[0] for i in self.description]
+
+        csv_handle = csv.writer(
+            fd,
+            delimiter=",",
+            quoting=csv.QUOTE_ALL,
+            escapechar="\\",
+            lineterminator="\n",
+        )
+        csv_handle.writerow(headers)
+        csv_handle.writerows(results)
+
+    setattr(SQLiteCursorWrapper, "copy_expert", write_stdout)
+    request.addfinalizer(lambda: shutil.rmtree(path))
+    request.addfinalizer(lambda: delattr(SQLiteCursorWrapper, "copy_expert"))
+    return path
+
+
+@pytest.mark.django_db
+def test_copy_tables_unified_job_query(
+    sqlite_copy_expert, project, inventory, job_template
+):
+    """
+    Ensure that various unified job types are in the output of the query.
+    """
+
+    time_start = now() - timedelta(hours=9)
+    inv_src = InventorySource.objects.create(
+        name="inventory_update1", inventory=inventory, source="gce"
+    )
+
+    project_update_name = ProjectUpdate.objects.create(
+        project=project, name="project_update1"
+    ).name
+    inventory_update_name = inv_src.create_unified_job().name
+    job_name = job_template.create_unified_job().name
+
+    with tempfile.TemporaryDirectory() as tmpdir:
+        collectors.copy_tables(time_start, tmpdir, subset="unified_jobs")
+        with open(os.path.join(tmpdir, "unified_jobs_table.csv")) as f:
+            lines = "".join([line for line in f])
+
+            assert project_update_name in lines
+            assert inventory_update_name in lines
+            assert job_name in lines
+
+
+@pytest.fixture
+def workflow_job(states=["new", "new", "new", "new", "new"]):
+    """
+    Workflow topology:
+           node[0]
+            /\
+          s/  \f
+          /    \
+    node[1,5] node[3]
+         /       \
+       s/         \f
+       /           \
+    node[2]       node[4]
+    """
+    wfj = WorkflowJob.objects.create()
+    jt = JobTemplate.objects.create(name="test-jt")
+    nodes = [
+        WorkflowJobNode.objects.create(workflow_job=wfj, unified_job_template=jt)
+        for i in range(0, 6)
+    ]
+    for node, state in zip(nodes, states):
+        if state:
+            node.job = jt.create_job()
+            node.job.status = state
+            node.job.save()
+            node.save()
+    nodes[0].success_nodes.add(nodes[1])
+    nodes[0].success_nodes.add(nodes[5])
+    nodes[1].success_nodes.add(nodes[2])
+    nodes[0].failure_nodes.add(nodes[3])
+    nodes[3].failure_nodes.add(nodes[4])
+    return wfj
+
+
+@pytest.mark.django_db
+def test_copy_tables_workflow_job_node_query(sqlite_copy_expert, workflow_job):
+    time_start = now() - timedelta(hours=9)
+
+    with tempfile.TemporaryDirectory() as tmpdir:
+        collectors.copy_tables(time_start, tmpdir, subset="workflow_job_node_query")
+        with open(os.path.join(tmpdir, "workflow_job_node_table.csv")) as f:
+            reader = csv.reader(f)
+            # Pop the headers
+            next(reader)
+            lines = [line for line in reader]
+
+            ids = [int(line[0]) for line in lines]
+
+            assert ids == list(
+                workflow_job.workflow_nodes.all().values_list("id", flat=True)
+            )
+
+            for index, relationship in zip(
+                [7, 8, 9], ["success_nodes", "failure_nodes", "always_nodes"]
+            ):
+                for i, l in enumerate(lines):
+                    related_nodes = (
+                        [int(e) for e in l[index].split(",")] if l[index] else []
+                    )
+                    assert related_nodes == list(
+                        getattr(workflow_job.workflow_nodes.all()[i], relationship)
+                        .all()
+                        .values_list("id", flat=True)
+                    ), f"(right side) workflow_nodes.all()[{i}].{relationship}.all()"

awx/main/tests/functional/analytics/test_counts.py

@@ -13,10 +13,10 @@ def test_empty():
         "active_host_count": 0,
         "credential": 0,
         "custom_inventory_script": 0,
-        "custom_virtualenvs": 0,   # dev env ansible3
+        "custom_virtualenvs": 0,  # dev env ansible3
         "host": 0,
         "inventory": 0,
-        "inventories": {'normal': 0, 'smart': 0},
+        "inventories": {"normal": 0, "smart": 0},
         "job_template": 0,
         "notification_template": 0,
         "organization": 0,
@@ -27,28 +27,97 @@ def test_empty():
         "user": 0,
         "workflow_job_template": 0,
         "unified_job": 0,
-        "pending_jobs": 0
+        "pending_jobs": 0,
     }
 
 
 @pytest.mark.django_db
-def test_database_counts(organization_factory, job_template_factory,
-                         workflow_job_template_factory):
-    objs = organization_factory('org', superusers=['admin'])
-    jt = job_template_factory('test', organization=objs.organization,
-                              inventory='test_inv', project='test_project',
-                              credential='test_cred')
-    workflow_job_template_factory('test')
+def test_database_counts(
+    organization_factory, job_template_factory, workflow_job_template_factory
+):
+    objs = organization_factory("org", superusers=["admin"])
+    jt = job_template_factory(
+        "test",
+        organization=objs.organization,
+        inventory="test_inv",
+        project="test_project",
+        credential="test_cred",
+    )
+    workflow_job_template_factory("test")
     models.Team(organization=objs.organization).save()
     models.Host(inventory=jt.inventory).save()
     models.Schedule(
-        rrule='DTSTART;TZID=America/New_York:20300504T150000',
-        unified_job_template=jt.job_template
+        rrule="DTSTART;TZID=America/New_York:20300504T150000",
+        unified_job_template=jt.job_template,
     ).save()
     models.CustomInventoryScript(organization=objs.organization).save()
 
     counts = collectors.counts(None)
-    for key in ('organization', 'team', 'user', 'inventory', 'credential',
-                'project', 'job_template', 'workflow_job_template', 'host',
-                'schedule', 'custom_inventory_script'):
+    for key in (
+        "organization",
+        "team",
+        "user",
+        "inventory",
+        "credential",
+        "project",
+        "job_template",
+        "workflow_job_template",
+        "host",
+        "schedule",
+        "custom_inventory_script",
+    ):
         assert counts[key] == 1
+
+
+@pytest.mark.django_db
+def test_inventory_counts(organization_factory, inventory_factory):
+    (inv1, inv2, inv3) = [inventory_factory(f"inv-{i}") for i in range(3)]
+
+    s1 = inv1.inventory_sources.create(name="src1", source="ec2")
+    s2 = inv1.inventory_sources.create(name="src2", source="file")
+    s3 = inv1.inventory_sources.create(name="src3", source="gce")
+
+    s1.hosts.create(name="host1", inventory=inv1)
+    s1.hosts.create(name="host2", inventory=inv1)
+    s1.hosts.create(name="host3", inventory=inv1)
+
+    s2.hosts.create(name="host4", inventory=inv1)
+    s2.hosts.create(name="host5", inventory=inv1)
+
+    s3.hosts.create(name="host6", inventory=inv1)
+
+    s1 = inv2.inventory_sources.create(name="src1", source="ec2")
+
+    s1.hosts.create(name="host1", inventory=inv2)
+    s1.hosts.create(name="host2", inventory=inv2)
+    s1.hosts.create(name="host3", inventory=inv2)
+
+    inv_counts = collectors.inventory_counts(None)
+
+    assert {
+        inv1.id: {
+            "name": "inv-0",
+            "kind": "",
+            "hosts": 6,
+            "sources": 3,
+            "source_list": [
+                {"name": "src1", "source": "ec2", "num_hosts": 3},
+                {"name": "src2", "source": "file", "num_hosts": 2},
+                {"name": "src3", "source": "gce", "num_hosts": 1},
+            ],
+        },
+        inv2.id: {
+            "name": "inv-1",
+            "kind": "",
+            "hosts": 3,
+            "sources": 1,
+            "source_list": [{"name": "src1", "source": "ec2", "num_hosts": 3}],
+        },
+        inv3.id: {
+            "name": "inv-2",
+            "kind": "",
+            "hosts": 0,
+            "sources": 0,
+            "source_list": [],
+        },
+    } == inv_counts

awx/main/tests/functional/api/test_activity_streams.py

@@ -1,7 +1,6 @@
 import pytest
 
 from awx.api.versioning import reverse
-from awx.main.middleware import ActivityStreamMiddleware
 from awx.main.models.activity_stream import ActivityStream
 from awx.main.access import ActivityStreamAccess
 from awx.conf.models import Setting
@@ -61,28 +60,6 @@ def test_ctint_activity_stream(monkeypatch, get, user, settings):
     assert response.data['summary_fields']['setting'][0]['name'] == 'FOO'
 
 
-@pytest.mark.django_db
-def test_middleware_actor_added(monkeypatch, post, get, user, settings):
-    settings.ACTIVITY_STREAM_ENABLED = True
-    u = user('admin-poster', True)
-
-    url = reverse('api:organization_list')
-    response = post(url,
-                    dict(name='test-org', description='test-desc'),
-                    u,
-                    middleware=ActivityStreamMiddleware())
-    assert response.status_code == 201
-
-    org_id = response.data['id']
-    activity_stream = ActivityStream.objects.filter(organization__pk=org_id).first()
-
-    url = reverse('api:activity_stream_detail', kwargs={'pk': activity_stream.pk})
-    response = get(url, u)
-
-    assert response.status_code == 200
-    assert response.data['summary_fields']['actor']['username'] == 'admin-poster'
-
-
 @pytest.mark.django_db
 def test_rbac_stream_resource_roles(activity_stream_entry, organization, org_admin, settings):
     settings.ACTIVITY_STREAM_ENABLED = True

awx/main/tests/functional/api/test_credential.py

@@ -972,7 +972,7 @@ def test_field_removal(put, organization, admin, credentialtype_ssh):
     ['insights_inventories', Inventory()],
     ['unifiedjobs', Job()],
     ['unifiedjobtemplates', JobTemplate()],
-    ['unifiedjobtemplates', InventorySource()],
+    ['unifiedjobtemplates', InventorySource(source='ec2')],
     ['projects', Project()],
     ['workflowjobnodes', WorkflowJobNode()],
 ])

awx/main/tests/functional/api/test_deprecated_credential_assignment.py

@@ -24,41 +24,6 @@ def job_template(job_template, project, inventory):
     return job_template
 
 
-@pytest.mark.django_db
-def test_extra_credentials_filtering(get, job_template, admin,
-                                     machine_credential, vault_credential, credential):
-    job_template.credentials.add(machine_credential)
-    job_template.credentials.add(vault_credential)
-    job_template.credentials.add(credential)
-    url = reverse(
-        'api:job_template_extra_credentials_list',
-        kwargs={'pk': job_template.pk}
-    )
-    resp = get(url, admin, expect=200)
-    assert resp.data['count'] == 1
-    assert resp.data['results'][0]['id'] == credential.pk
-
-
-@pytest.mark.django_db
-def test_extra_credentials_requires_cloud_or_net(get, post, job_template, admin,
-                                                 machine_credential, vault_credential, credential,
-                                                 net_credential):
-    url = reverse(
-        'api:job_template_extra_credentials_list',
-        kwargs={'pk': job_template.pk}
-    )
-
-    for cred in (machine_credential, vault_credential):
-        resp = post(url, {'associate': True, 'id': cred.pk}, admin, expect=400)
-        assert 'Extra credentials must be network or cloud.' in smart_str(resp.content)
-
-    post(url, {'associate': True, 'id': credential.pk}, admin, expect=204)
-    assert get(url, admin).data['count'] == 1
-
-    post(url, {'associate': True, 'id': net_credential.pk}, admin, expect=204)
-    assert get(url, admin).data['count'] == 2
-
-
 @pytest.mark.django_db
 def test_prevent_multiple_machine_creds(get, post, job_template, admin, machine_credential):
     url = reverse(
@@ -115,52 +80,6 @@ def test_prevent_multiple_machine_creds_at_launch(get, post, job_template, admin
     assert 'Cannot assign multiple Machine credentials.' in smart_str(resp.content)
 
 
-@pytest.mark.django_db
-def test_extra_credentials_unique_by_kind(get, post, job_template, admin,
-                                          credentialtype_aws):
-    url = reverse(
-        'api:job_template_extra_credentials_list',
-        kwargs={'pk': job_template.pk}
-    )
-
-    def _new_cred(name):
-        return {
-            'name': name,
-            'credential_type': credentialtype_aws.pk,
-            'inputs': {
-                'username': 'bob',
-                'password': 'secret',
-            }
-        }
-
-    post(url, _new_cred('First Cred'), admin, expect=201)
-    assert get(url, admin).data['count'] == 1
-
-    resp = post(url, _new_cred('Second Cred'), admin, expect=400)
-    assert 'Cannot assign multiple Amazon Web Services credentials.' in smart_str(resp.content)
-
-
-@pytest.mark.django_db
-def test_extra_credentials_at_launch(get, post, job_template, admin, credential):
-    url = reverse('api:job_template_launch', kwargs={'pk': job_template.pk})
-    pk = post(url, {'extra_credentials': [credential.pk]}, admin, expect=201).data['job']
-    summary_fields = get(reverse('api:job_detail', kwargs={'pk': pk}), admin).data['summary_fields']
-
-    assert len(summary_fields['credentials']) == 1
-
-
-@pytest.mark.django_db
-def test_modify_extra_credentials_at_launch(get, post, job_template, admin,
-                                            machine_credential, vault_credential, credential):
-    job_template.credentials.add(machine_credential)
-    job_template.credentials.add(vault_credential)
-    url = reverse('api:job_template_launch', kwargs={'pk': job_template.pk})
-    pk = post(url, {'extra_credentials': [credential.pk]}, admin, expect=201).data['job']
-
-    summary_fields = get(reverse('api:job_detail', kwargs={'pk': pk}), admin).data['summary_fields']
-    assert len(summary_fields['credentials']) == 3
-
-
 @pytest.mark.django_db
 def test_ssh_password_prompted_at_launch(get, post, job_template, admin, machine_credential):
     job_template.credentials.add(machine_credential)
@@ -229,25 +148,6 @@ def test_vault_credential_with_password_at_launch(get, post, job_template, admin
         signal_start.assert_called_with(vault_password='testing123')
 
 
-@pytest.mark.django_db
-def test_extra_creds_prompted_at_launch(get, post, job_template, admin, net_credential):
-    url = reverse('api:job_template_launch', kwargs={'pk': job_template.pk})
-    resp = post(url, {'extra_credentials': [net_credential.pk]}, admin, expect=201)
-
-    summary_fields = get(
-        reverse('api:job_detail', kwargs={'pk': resp.data['job']}),
-        admin
-    ).data['summary_fields']
-    assert len(summary_fields['credentials']) == 1
-
-
-@pytest.mark.django_db
-def test_invalid_mixed_credentials_specification(get, post, job_template, admin, net_credential):
-    url = reverse('api:job_template_launch', kwargs={'pk': job_template.pk})
-    post(url=url, data={'credentials': [net_credential.pk], 'extra_credentials': [net_credential.pk]},
-         user=admin, expect=400)
-
-
 @pytest.mark.django_db
 def test_deprecated_credential_activity_stream(patch, admin_user, machine_credential, job_template):
     job_template.credentials.add(machine_credential)

awx/main/tests/functional/api/test_job.py

@@ -22,20 +22,6 @@ from awx.main.models import (
 )
 
 
-@pytest.mark.django_db
-def test_extra_credentials(get, organization_factory, job_template_factory, credential):
-    objs = organization_factory("org", superusers=['admin'])
-    jt = job_template_factory("jt", organization=objs.organization,
-                              inventory='test_inv', project='test_proj').job_template
-    jt.credentials.add(credential)
-    jt.save()
-    job = jt.create_unified_job()
-
-    url = reverse('api:job_extra_credentials_list', kwargs={'pk': job.pk})
-    response = get(url, user=objs.superusers.admin)
-    assert response.data.get('count') == 1
-
-
 @pytest.mark.django_db
 def test_job_relaunch_permission_denied_response(
         post, get, inventory, project, credential, net_credential, machine_credential):
@@ -50,7 +36,7 @@ def test_job_relaunch_permission_denied_response(
     r = get(job.get_absolute_url(), jt_user, expect=200)
     assert r.data['summary_fields']['user_capabilities']['start']
 
-    # Job has prompted extra_credential, launch denied w/ message
+    # Job has prompted credential, launch denied w/ message
     job.launch_config.credentials.add(net_credential)
     r = post(reverse('api:job_relaunch', kwargs={'pk':job.pk}), {}, jt_user, expect=403)
     assert 'launched with prompted fields you do not have access to' in r.data['detail']
@@ -70,7 +56,7 @@ def test_job_relaunch_prompts_not_accepted_response(
     r = get(job.get_absolute_url(), jt_user, expect=200)
     assert r.data['summary_fields']['user_capabilities']['start']
 
-    # Job has prompted extra_credential, launch denied w/ message
+    # Job has prompted credential, launch denied w/ message
     job.launch_config.credentials.add(net_credential)
     r = post(reverse('api:job_relaunch', kwargs={'pk':job.pk}), {}, jt_user, expect=403)
 

awx/main/tests/functional/api/test_job_runtime_params.py

@@ -304,7 +304,7 @@ def test_job_launch_with_default_creds(machine_credential, vault_credential, dep
 @pytest.mark.django_db
 def test_job_launch_JT_enforces_unique_credentials_kinds(machine_credential, credentialtype_aws, deploy_jobtemplate):
     """
-    JT launching should require that extra_credentials have distinct CredentialTypes
+    JT launching should require that credentials have distinct CredentialTypes
     """
     creds = []
     for i in range(2):

awx/main/tests/functional/api/test_job_template.py

@@ -45,27 +45,6 @@ def test_create(post, project, machine_credential, inventory, alice, grant_proje
     )
 
 
-@pytest.mark.django_db
-def test_extra_credential_creation(get, post, organization_factory, job_template_factory, credentialtype_aws):
-    objs = organization_factory("org", superusers=['admin'])
-    jt = job_template_factory("jt", organization=objs.organization,
-                              inventory='test_inv', project='test_proj').job_template
-
-    url = reverse('api:job_template_extra_credentials_list', kwargs={'pk': jt.pk})
-    response = post(url, {
-        'name': 'My Cred',
-        'credential_type': credentialtype_aws.pk,
-        'inputs': {
-            'username': 'bob',
-            'password': 'secret',
-        }
-    }, objs.superusers.admin)
-    assert response.status_code == 201
-
-    response = get(url, user=objs.superusers.admin)
-    assert response.data.get('count') == 1
-
-
 @pytest.mark.django_db
 @pytest.mark.parametrize('kind', ['scm', 'insights'])
 def test_invalid_credential_kind_xfail(get, post, organization_factory, job_template_factory, kind):
@@ -87,42 +66,6 @@ def test_invalid_credential_kind_xfail(get, post, organization_factory, job_temp
     assert 'Cannot assign a Credential of kind `{}`.'.format(kind) in response.data.values()
 
 
-@pytest.mark.django_db
-def test_extra_credential_unique_type_xfail(get, post, organization_factory, job_template_factory, credentialtype_aws):
-    objs = organization_factory("org", superusers=['admin'])
-    jt = job_template_factory("jt", organization=objs.organization,
-                              inventory='test_inv', project='test_proj').job_template
-
-    url = reverse('api:job_template_extra_credentials_list', kwargs={'pk': jt.pk})
-    response = post(url, {
-        'name': 'My Cred',
-        'credential_type': credentialtype_aws.pk,
-        'inputs': {
-            'username': 'bob',
-            'password': 'secret',
-        }
-    }, objs.superusers.admin)
-    assert response.status_code == 201
-
-    response = get(url, user=objs.superusers.admin)
-    assert response.data.get('count') == 1
-
-    # this request should fail because you can't assign the same type (aws)
-    # twice
-    response = post(url, {
-        'name': 'My Cred',
-        'credential_type': credentialtype_aws.pk,
-        'inputs': {
-            'username': 'joe',
-            'password': 'another-secret',
-        }
-    }, objs.superusers.admin)
-    assert response.status_code == 400
-
-    response = get(url, user=objs.superusers.admin)
-    assert response.data.get('count') == 1
-
-
 @pytest.mark.django_db
 def test_create_with_forks_exceeding_maximum_xfail(alice, post, project, inventory, settings):
     project.use_role.members.add(alice)
@@ -143,60 +86,6 @@ def test_create_with_forks_exceeding_maximum_xfail(alice, post, project, invento
     assert 'Maximum number of forks (10) exceeded' in str(response.data)
 
 
-@pytest.mark.django_db
-def test_attach_extra_credential(get, post, organization_factory, job_template_factory, credential):
-    objs = organization_factory("org", superusers=['admin'])
-    jt = job_template_factory("jt", organization=objs.organization,
-                              inventory='test_inv', project='test_proj').job_template
-
-    url = reverse('api:job_template_extra_credentials_list', kwargs={'pk': jt.pk})
-    response = post(url, {
-        'associate': True,
-        'id': credential.id,
-    }, objs.superusers.admin)
-    assert response.status_code == 204
-
-    response = get(url, user=objs.superusers.admin)
-    assert response.data.get('count') == 1
-
-
-@pytest.mark.django_db
-def test_detach_extra_credential(get, post, organization_factory, job_template_factory, credential):
-    objs = organization_factory("org", superusers=['admin'])
-    jt = job_template_factory("jt", organization=objs.organization,
-                              inventory='test_inv', project='test_proj').job_template
-    jt.credentials.add(credential)
-    jt.save()
-
-    url = reverse('api:job_template_extra_credentials_list', kwargs={'pk': jt.pk})
-    response = post(url, {
-        'disassociate': True,
-        'id': credential.id,
-    }, objs.superusers.admin)
-    assert response.status_code == 204
-
-    response = get(url, user=objs.superusers.admin)
-    assert response.data.get('count') == 0
-
-
-@pytest.mark.django_db
-def test_attach_extra_credential_wrong_kind_xfail(get, post, organization_factory, job_template_factory, machine_credential):
-    """Extra credentials only allow net + cloud credentials"""
-    objs = organization_factory("org", superusers=['admin'])
-    jt = job_template_factory("jt", organization=objs.organization,
-                              inventory='test_inv', project='test_proj').job_template
-
-    url = reverse('api:job_template_extra_credentials_list', kwargs={'pk': jt.pk})
-    response = post(url, {
-        'associate': True,
-        'id': machine_credential.id,
-    }, objs.superusers.admin)
-    assert response.status_code == 400
-
-    response = get(url, user=objs.superusers.admin)
-    assert response.data.get('count') == 0
-
-
 @pytest.mark.django_db
 @pytest.mark.parametrize(
     "grant_project, grant_inventory, expect", [
@@ -368,57 +257,6 @@ def test_launch_with_pending_deletion_inventory_workflow(get, post, organization
     assert resp.data['inventory'] == ['The inventory associated with this Workflow is being deleted.']
 
 
-@pytest.mark.django_db
-def test_launch_with_extra_credentials(get, post, organization_factory,
-                                       job_template_factory, machine_credential,
-                                       credential, net_credential):
-    objs = organization_factory("org", superusers=['admin'])
-    jt = job_template_factory("jt", organization=objs.organization,
-                              inventory='test_inv', project='test_proj').job_template
-    jt.ask_credential_on_launch = True
-    jt.save()
-
-    resp = post(
-        reverse('api:job_template_launch', kwargs={'pk': jt.pk}),
-        dict(
-            credentials=[machine_credential.pk, credential.pk, net_credential.pk]
-        ),
-        objs.superusers.admin, expect=201
-    )
-    job_pk = resp.data.get('id')
-
-    resp = get(reverse('api:job_extra_credentials_list', kwargs={'pk': job_pk}), objs.superusers.admin)
-    assert resp.data.get('count') == 2
-
-    resp = get(reverse('api:job_template_extra_credentials_list', kwargs={'pk': jt.pk}), objs.superusers.admin)
-    assert resp.data.get('count') == 0
-
-
-@pytest.mark.django_db
-def test_launch_with_extra_credentials_not_allowed(get, post, organization_factory,
-                                                   job_template_factory, machine_credential,
-                                                   credential, net_credential):
-    objs = organization_factory("org", superusers=['admin'])
-    jt = job_template_factory("jt", organization=objs.organization,
-                              inventory='test_inv', project='test_proj').job_template
-    jt.credentials.add(machine_credential)
-    jt.ask_credential_on_launch = False
-    jt.save()
-
-    resp = post(
-        reverse('api:job_template_launch', kwargs={'pk': jt.pk}),
-        dict(
-            credentials=[machine_credential.pk, credential.pk, net_credential.pk]
-        ),
-        objs.superusers.admin
-    )
-    assert 'credentials' in resp.data['ignored_fields'].keys()
-    job_pk = resp.data.get('id')
-
-    resp = get(reverse('api:job_extra_credentials_list', kwargs={'pk': job_pk}), objs.superusers.admin)
-    assert resp.data.get('count') == 0
-
-
 @pytest.mark.django_db
 def test_jt_without_project(inventory):
     data = dict(name="Test", job_type="run",

awx/main/tests/functional/api/test_oauth.py

@@ -1,6 +1,8 @@
-import pytest
 import base64
 import json
+import time
+
+import pytest
 
 from django.db import connection
 from django.test.utils import override_settings
@@ -326,6 +328,38 @@ def test_refresh_accesstoken(oauth_application, post, get, delete, admin):
     assert original_refresh_token.revoked # is not None
 
 
+@pytest.mark.django_db
+def test_refresh_token_expiration_is_respected(oauth_application, post, get, delete, admin):
+    response = post(
+        reverse('api:o_auth2_application_token_list', kwargs={'pk': oauth_application.pk}),
+        {'scope': 'read'}, admin, expect=201
+    )
+    assert AccessToken.objects.count() == 1
+    assert RefreshToken.objects.count() == 1
+    refresh_token = RefreshToken.objects.get(token=response.data['refresh_token'])
+    refresh_url = drf_reverse('api:oauth_authorization_root_view') + 'token/'
+    short_lived = {
+        'ACCESS_TOKEN_EXPIRE_SECONDS': 1,
+        'AUTHORIZATION_CODE_EXPIRE_SECONDS': 1,
+        'REFRESH_TOKEN_EXPIRE_SECONDS': 1
+    }
+    time.sleep(1)
+    with override_settings(OAUTH2_PROVIDER=short_lived):
+        response = post(
+            refresh_url,
+            data='grant_type=refresh_token&refresh_token=' + refresh_token.token,
+            content_type='application/x-www-form-urlencoded',
+            HTTP_AUTHORIZATION='Basic ' + smart_str(base64.b64encode(smart_bytes(':'.join([
+                oauth_application.client_id, oauth_application.client_secret
+            ]))))
+        )
+    assert response.status_code == 403
+    assert b'The refresh token has expired.' in response.content
+    assert RefreshToken.objects.filter(token=refresh_token).exists()
+    assert AccessToken.objects.count() == 1
+    assert RefreshToken.objects.count() == 1
+
+
 
 @pytest.mark.django_db
 def test_revoke_access_then_refreshtoken(oauth_application, post, get, delete, admin):