Merge pull request #3045 from jakemcdermott/regenerate-package-lock

updating package-lock.json

Reviewed-by: Shane McDonald <me@shanemcd.com>
             https://github.com/shanemcd

.gitignore

@@ -1,3 +1,8 @@
+# Ignore generated schema
+swagger.json
+schema.json
+reference-schema.json
+
 # Tags
 .tags
 .tags1
@@ -52,10 +57,12 @@ __pycache__
 **/node_modules/**
 /tmp
 **/npm-debug.log*
+**/package-lock.json
 
 # UI build flag files
 awx/ui/.deps_built
 awx/ui/.release_built
+awx/ui/.release_deps_built
 
 # Testing
 .cache
@@ -112,7 +119,6 @@ local/
 *.mo
 requirements/vendor
 .i18n_built
-VERSION
 .idea/*
 
 # AWX python libs populated by requirements.txt

CONTRIBUTING.md

@@ -2,11 +2,11 @@
 
 Hi there! We're excited to have you as a contributor.
 
-Have questions about this document or anything not covered here? Come chat with us at `#ansible-awx` on irc.freenode.net, or submit your question to the [mailing list](https://groups.google.com/forum/#!forum/awx-project) .
+Have questions about this document or anything not covered here? Come chat with us at `#ansible-awx` on irc.freenode.net, or submit your question to the [mailing list](https://groups.google.com/forum/#!forum/awx-project).
 
 ## Table of contents
 
-* [Things to know prior to submitting code](#things-to-know-prior-to-contributing-code)
+* [Things to know prior to submitting code](#things-to-know-prior-to-submitting-code)
 * [Setting up your development environment](#setting-up-your-development-environment)
   * [Prerequisites](#prerequisites)
     * [Docker](#docker)
@@ -17,14 +17,14 @@ Have questions about this document or anything not covered here? Come chat with
     * [Create local settings](#create-local-settings)
     * [Build the base image](#build-the-base-image)
     * [Build the user interface](#build-the-user-interface)
-  # [Running the environment](#running-the-environment)
+  * [Running the environment](#running-the-environment)
     * [Start the containers](#start-the-containers)
     * [Start from the container shell](#start-from-the-container-shell)
   * [Post Build Steps](#post-build-steps)
-     * [Start a shell](#start-the-shell)
-     * [Create a superuser](#create-a-superuser)
-     * [Load the data](#load-the-data)
-     * [Building API Documentation](#build-documentation)
+    * [Start a shell](#start-a-shell)
+    * [Create a superuser](#create-a-superuser)
+    * [Load the data](#load-the-data)
+    * [Building API Documentation](#build-api-documentation)
   * [Accessing the AWX web interface](#accessing-the-awx-web-interface)
   * [Purging containers and images](#purging-containers-and-images)
 * [What should I work on?](#what-should-i-work-on)
@@ -34,10 +34,11 @@ Have questions about this document or anything not covered here? Come chat with
 ## Things to know prior to submitting code
 
 - All code submissions are done through pull requests against the `devel` branch.
-- You must use `git commit --signoff` for any commit to be merged, and agree that usage of --signoff constitutes agreement with the terms of [DCO 1.1](./DCO_1_1.md).  
+- You must use `git commit --signoff` for any commit to be merged, and agree that usage of --signoff constitutes agreement with the terms of [DCO 1.1](./DCO_1_1.md).
 - Take care to make sure no merge commits are in the submission, and use `git rebase` vs `git merge` for this reason.
+  - If collaborating with someone else on the same branch, consider using `--force-with-lease` instead of `--force`. This will prevent you from accidentally overwriting commits pushed by someone else. For more information, see https://git-scm.com/docs/git-push#git-push---force-with-leaseltrefnamegt
 - If submitting a large code change, it's a good idea to join the `#ansible-awx` channel on irc.freenode.net, and talk about what you would like to do or add first. This not only helps everyone know what's going on, it also helps save time and effort, if the community decides some changes are needed.
-- We ask all of our community members and contributors to adhere to the [Ansible code of conduct](http://docs.ansible.com/ansible/latest/community/code_of_conduct.html). If you have questions, or need assistance, please reach out to our community team at [codeofconduct@ansible.com](mailto:codeofconduct@ansible.com)   
+- We ask all of our community members and contributors to adhere to the [Ansible code of conduct](http://docs.ansible.com/ansible/latest/community/code_of_conduct.html). If you have questions, or need assistance, please reach out to our community team at [codeofconduct@ansible.com](mailto:codeofconduct@ansible.com)
 
 ## Setting up your development environment
 
@@ -49,7 +50,7 @@ The AWX development environment workflow and toolchain is based on Docker, and t
 
 Prior to starting the development services, you'll need `docker` and `docker-compose`. On Linux, you can generally find these in your distro's packaging, but you may find that Docker themselves maintain a separate repo that tracks more closely to the latest releases.
 
-For macOS and Windows, we recommend [Docker for Mac](https://www.docker.com/docker-mac) and [Docker for Windows](https://www.docker.com/docker-windows) 
+For macOS and Windows, we recommend [Docker for Mac](https://www.docker.com/docker-mac) and [Docker for Windows](https://www.docker.com/docker-windows)
 respectively.
 
 For Linux platforms, refer to the following from Docker:
@@ -86,8 +87,8 @@ If you're not using Docker for Mac, or Docker for Windows, you may need, or choo
 
 The AWX UI requires the following:
 
-- Node 6.x LTS version
-- NPM 3.x LTS
+- Node 8.x LTS
+- NPM 6.x LTS
 
 ### Build the environment
 
@@ -137,21 +138,21 @@ Run the following to build the AWX UI:
 ```
 ### Running the environment
 
-#### Start the containers 
+#### Start the containers
 
 Start the development containers by running the following:
 
 ```bash
 (host)$ make docker-compose
 ```
-    
-The above utilizes the image built in the previous step, and will automatically start all required services and dependent containers. Once the containers launch, your session will be attached to the *awx* container, and you'll be able to watch log messages and events in real time. You will see messages from Django, celery, and the front end build process.
+
+The above utilizes the image built in the previous step, and will automatically start all required services and dependent containers. Once the containers launch, your session will be attached to the *awx* container, and you'll be able to watch log messages and events in real time. You will see messages from Django and the front end build process.
 
 If you start a second terminal session, you can take a look at the running containers using the `docker ps` command. For example:
 
 ```bash
 # List running containers
-(host)$ docker ps 
+(host)$ docker ps
 
 $ docker ps
 CONTAINER ID        IMAGE                                              COMMAND                  CREATED             STATUS              PORTS                                                                                                                                      NAMES
@@ -174,7 +175,7 @@ The first time you start the environment, database migrations need to run in ord
 ```bash
 awx_1        | Operations to perform:
 awx_1        |   Synchronize unmigrated apps: solo, api, staticfiles, debug_toolbar, messages, channels, django_extensions, ui, rest_framework, polymorphic
-awx_1        |   Apply all migrations: sso, taggit, sessions, djcelery, sites, kombu_transport_django, social_auth, contenttypes, auth, conf, main
+awx_1        |   Apply all migrations: sso, taggit, sessions, sites, kombu_transport_django, social_auth, contenttypes, auth, conf, main
 awx_1        | Synchronizing apps without migrations:
 awx_1        |   Creating tables...
 awx_1        |     Running deferred SQL...
@@ -219,7 +220,7 @@ If you want to start and use the development environment, you'll first need to b
 ```
 
 The above will do all the setup tasks, including running database migrations, so it may take a couple minutes.
-    
+
 Now you can start each service individually, or start all services in a pre-configured tmux session like so:
 
 ```bash
@@ -248,9 +249,9 @@ Before you can log into AWX, you need to create an admin user. With this user yo
 (container)# awx-manage createsuperuser
 ```
 You will be prompted for a username, an email address, and a password, and you will be asked to confirm the password. The email address is not important, so just enter something that looks like an email address. Remember the username and password, as you will use them to log into the web interface for the first time.
-    
+
 ##### Load demo data
-  
+
 You can optionally load some demo data. This will create a demo project, inventory, and job template. From within the container shell, run the following to load the data:
 
 ```bash
@@ -276,7 +277,7 @@ in OpenAPI format.  A variety of online tools are available for translating
 this data into more consumable formats (such as HTML). http://editor.swagger.io
 is an example of one such service.
 
-### Accessing the AWX web interface 
+### Accessing the AWX web interface
 
 You can now log into the AWX web interface at [https://localhost:8043](https://localhost:8043), and access the API directly at [https://localhost:8043/api/](https://localhost:8043/api/).
 
@@ -289,7 +290,7 @@ When necessary, remove any AWX containers and images by running the following:
 ```bash
 (host)$ make docker-clean
 ```
-        
+
 ## What should I work on?
 
 For feature work, take a look at the current [Enhancements](https://github.com/ansible/awx/issues?q=is%3Aissue+is%3Aopen+label%3Atype%3Aenhancement).
@@ -329,7 +330,24 @@ We like to keep our commit history clean, and will require resubmission of pull
 
 Sometimes it might take us a while to fully review your PR. We try to keep the `devel` branch in good working order, and so we review requests carefully. Please be patient.
 
-All submitted PRs will have the linter and unit tests run against them, and the status reported in the PR.
+All submitted PRs will have the linter and unit tests run against them via Zuul, and the status reported in the PR.
+
+## PR Checks ran by Zuul
+Zuul jobs for awx are defined in the [zuul-jobs](https://github.com/ansible/zuul-jobs) repo.
+
+Zuul runs the following checks that must pass:
+1) `tox-awx-api-lint`
+2) `tox-awx-ui-lint`
+3) `tox-awx-api`
+4) `tox-awx-ui`
+5) `tox-awx-swagger`
+
+Zuul runs the following checks that are non-voting (can not pass but serve to inform PR reviewers):
+1) `tox-awx-detect-schema-change`
+    This check generates the schema and diffs it against a reference copy of the `devel` version of the schema.
+    Reviewers should inspect the `job-output.txt.gz` related to the check if their is a failure (grep for `diff -u -b` to find beginning of diff).
+    If the schema change is expected and makes sense in relation to the changes made by the PR, then you are good to go!
+    If not, the schema changes should be fixed, but this decision must be enforced by reviewers.
 
 ## Reporting Issues
 

DATA_MIGRATION.md

@@ -18,7 +18,7 @@ $ pip install --upgrade ansible-tower-cli
 
 The AWX host URL, user, and password must be set for the AWX instance to be exported:
 ```
-$ tower-cli config host <old-awx-host.example.com>
+$ tower-cli config host http://<old-awx-host.example.com>
 $ tower-cli config username <user>
 $ tower-cli config password <pass>
 ```
@@ -38,7 +38,7 @@ Export all objects
 
 Clean up remnants of the old AWX install:
 
-```docker rm -f $(ps -aq)```     # remove all old awx containers
+```docker rm -f $(docker ps -aq)```     # remove all old awx containers
 
 ```make clean-ui```              # clean up ui artifacts
 
@@ -62,7 +62,7 @@ For other install methods, refer to the [Install.md](https://github.com/ansible/
 Configure tower-cli for your new AWX host as shown earlier.  Import from a JSON file named assets.json
 
 ```
-$ tower-cli config host <new-awx-host.example.com>
+$ tower-cli config host http://<new-awx-host.example.com>
 $ tower-cli config username <user>
 $ tower-cli config password <pass>
 $ tower-cli send assets.json

INSTALL.md

@@ -62,8 +62,8 @@ Before you can run a deployment, you'll need the following installed in your loc
 - [docker-py](https://github.com/docker/docker-py) Python module
 - [GNU Make](https://www.gnu.org/software/make/)
 - [Git](https://git-scm.com/) Requires Version 1.8.4+
-- [Node 6.x LTS version](https://nodejs.org/en/download/)
-- [NPM 3.x LTS](https://docs.npmjs.com/)
+- [Node 8.x LTS version](https://nodejs.org/en/download/)
+- [NPM 6.x LTS](https://docs.npmjs.com/)
 
 ### System Requirements
 
@@ -119,12 +119,12 @@ To complete a deployment to OpenShift, you will obviously need access to an Open
 
 You will also need to have the `oc` command in your PATH. The `install.yml` playbook will call out to `oc` when logging into, and creating objects on the cluster.
 
-The default resource requests per-pod requires:
+The default resource requests per-deployment requires:
 
 > Memory: 6GB
 > CPU: 3 cores
 
-This can be tuned by overriding the variables found in [/installer/openshift/defaults/main.yml](/installer/openshift/defaults/main.yml). Special care should be taken when doing this as undersized instances will experience crashes and resource exhaustion.
+This can be tuned by overriding the variables found in [/installer/roles/kubernetes/defaults/main.yml](/installer/roles/kubernetes/defaults/main.yml). Special care should be taken when doing this as undersized instances will experience crashes and resource exhaustion.
 
 For more detail on how resource requests are formed see: [https://docs.openshift.com/container-platform/latest/dev_guide/compute_resources.html#dev-compute-resources](https://docs.openshift.com/container-platform/latest/dev_guide/compute_resources.html#dev-compute-resources)
 
@@ -236,7 +236,7 @@ Using /etc/ansible/ansible.cfg as config file
 }
 Operations to perform:
   Synchronize unmigrated apps: solo, api, staticfiles, messages, channels, django_extensions, ui, rest_framework, polymorphic
-  Apply all migrations: sso, taggit, sessions, djcelery, sites, kombu_transport_django, social_auth, contenttypes, auth, conf, main
+  Apply all migrations: sso, taggit, sessions, sites, kombu_transport_django, social_auth, contenttypes, auth, conf, main
 Synchronizing apps without migrations:
   Creating tables...
     Running deferred SQL...
@@ -426,7 +426,7 @@ If you choose to use the official images then the remote host will be the one to
 
 > As mentioned above, in [Prerequisites](#prerequisites-1), the prerequisites are required on the remote host.
 
-> When deploying to a remote host, the playook does not execute tasks with the `become` option. For this reason, make sure the user that connects to the remote host has privileges to run the `docker` command. This typically means that non-privileged users need to be part of the `docker` group.
+> When deploying to a remote host, the playbook does not execute tasks with the `become` option. For this reason, make sure the user that connects to the remote host has privileges to run the `docker` command. This typically means that non-privileged users need to be part of the `docker` group.
 
 
 #### Inventory variables
@@ -449,6 +449,10 @@ Before starting the build process, review the [inventory](./installer/inventory)
 
 When using docker-compose, the `docker-compose.yml` file will be created there (default `/var/lib/awx`).
 
+*ca_trust_dir*
+
+> If you're using a non trusted CA, provide a path where the untrusted Certs are stored on your Host.
+
 #### Docker registry
 
 If you wish to tag and push built images to a Docker registry, set the following variables in the inventory file:
@@ -548,7 +552,7 @@ Using /etc/ansible/ansible.cfg as config file
 }
 Operations to perform:
   Synchronize unmigrated apps: solo, api, staticfiles, messages, channels, django_extensions, ui, rest_framework, polymorphic
-  Apply all migrations: sso, taggit, sessions, djcelery, sites, kombu_transport_django, social_auth, contenttypes, auth, conf, main
+  Apply all migrations: sso, taggit, sessions, sites, kombu_transport_django, social_auth, contenttypes, auth, conf, main
 Synchronizing apps without migrations:
   Creating tables...
     Running deferred SQL...

Makefile

@@ -1,4 +1,4 @@
-PYTHON ?= python
+PYTHON ?= python3
 PYTHON_VERSION = $(shell $(PYTHON) -c "from distutils.sysconfig import get_python_version; print(get_python_version())")
 SITELIB=$(shell $(PYTHON) -c "from distutils.sysconfig import get_python_lib; print(get_python_lib())")
 OFFICIAL ?= no
@@ -11,11 +11,7 @@ GIT_BRANCH ?= $(shell git rev-parse --abbrev-ref HEAD)
 MANAGEMENT_COMMAND ?= awx-manage
 IMAGE_REPOSITORY_AUTH ?=
 IMAGE_REPOSITORY_BASE ?= https://gcr.io
-
-VERSION=$(shell git describe --long --first-parent)
-VERSION3=$(shell git describe --long --first-parent | sed 's/\-g.*//')
-VERSION3DOT=$(shell git describe --long --first-parent | sed 's/\-g.*//' | sed 's/\-/\./')
-RELEASE_VERSION=$(shell git describe --long --first-parent | sed 's@\([0-9.]\{1,\}\).*@\1@')
+VERSION := $(shell cat VERSION)
 
 # NOTE: This defaults the container image version to the branch that's active
 COMPOSE_TAG ?= $(GIT_BRANCH)
@@ -30,8 +26,6 @@ DEV_DOCKER_TAG_BASE ?= gcr.io/ansible-tower-engineering
 # Comma separated list
 SRC_ONLY_PKGS ?= cffi,pycparser,psycopg2,twilio
 
-CURWD = $(shell pwd)
-
 # Determine appropriate shasum command
 UNAME_S := $(shell uname -s)
 ifeq ($(UNAME_S),Linux)
@@ -48,20 +42,9 @@ DATE := $(shell date -u +%Y%m%d%H%M)
 NAME ?= awx
 GIT_REMOTE_URL = $(shell git config --get remote.origin.url)
 
-ifeq ($(OFFICIAL),yes)
-    VERSION_TARGET ?= $(RELEASE_VERSION)
-else
-    VERSION_TARGET ?= $(VERSION3DOT)
-endif
-
 # TAR build parameters
-ifeq ($(OFFICIAL),yes)
-    SDIST_TAR_NAME=$(NAME)-$(RELEASE_VERSION)
-    WHEEL_NAME=$(NAME)-$(RELEASE_VERSION)
-else
-    SDIST_TAR_NAME=$(NAME)-$(VERSION3DOT)
-    WHEEL_NAME=$(NAME)-$(VERSION3DOT)
-endif
+SDIST_TAR_NAME=$(NAME)-$(VERSION)
+WHEEL_NAME=$(NAME)-$(VERSION)
 
 SDIST_COMMAND ?= sdist
 WHEEL_COMMAND ?= bdist_wheel
@@ -70,12 +53,13 @@ WHEEL_FILE ?= $(WHEEL_NAME)-py2-none-any.whl
 
 # UI flag files
 UI_DEPS_FLAG_FILE = awx/ui/.deps_built
+UI_RELEASE_DEPS_FLAG_FILE = awx/ui/.release_deps_built
 UI_RELEASE_FLAG_FILE = awx/ui/.release_built
 
 I18N_FLAG_FILE = .i18n_built
 
 .PHONY: awx-link clean clean-tmp clean-venv requirements requirements_dev \
-	develop refresh adduser migrate dbchange dbshell runserver celeryd \
+	develop refresh adduser migrate dbchange dbshell runserver \
 	receiver test test_unit test_ansible test_coverage coverage_html \
 	dev_build release_build release_clean sdist \
 	ui-docker-machine ui-docker ui-release ui-devel \
@@ -90,6 +74,7 @@ clean-ui:
 	rm -rf awx/ui/test/e2e/reports/
 	rm -rf awx/ui/client/languages/
 	rm -f $(UI_DEPS_FLAG_FILE)
+	rm -f $(UI_RELEASE_DEPS_FLAG_FILE)
 	rm -f $(UI_RELEASE_FLAG_FILE)
 
 clean-tmp:
@@ -101,6 +86,11 @@ clean-venv:
 clean-dist:
 	rm -rf dist
 
+clean-schema:
+	rm -rf swagger.json
+	rm -rf schema.json
+	rm -rf reference-schema.json
+
 # Remove temporary build files, compiled Python files.
 clean: clean-ui clean-dist
 	rm -rf awx/public
@@ -112,7 +102,6 @@ clean: clean-ui clean-dist
 	rm -rf requirements/vendor
 	rm -rf tmp
 	rm -rf $(I18N_FLAG_FILE)
-	rm -f VERSION
 	mkdir tmp
 	rm -rf build $(NAME)-$(VERSION) *.egg-info
 	find . -type f -regex ".*\.py[co]$$" -delete
@@ -133,23 +122,30 @@ virtualenv_ansible:
 			mkdir $(VENV_BASE); \
 		fi; \
 		if [ ! -d "$(VENV_BASE)/ansible" ]; then \
-			virtualenv --system-site-packages $(VENV_BASE)/ansible && \
+			virtualenv -p python --system-site-packages $(VENV_BASE)/ansible && \
 			$(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) --ignore-installed six packaging appdirs && \
 			$(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) --ignore-installed setuptools==36.0.1 && \
 			$(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) --ignore-installed pip==9.0.1; \
 		fi; \
 	fi
 
+virtualenv_ansible_py3:
+	if [ "$(VENV_BASE)" ]; then \
+		if [ ! -d "$(VENV_BASE)" ]; then \
+			mkdir $(VENV_BASE); \
+		fi; \
+		if [ ! -d "$(VENV_BASE)/ansible3" ]; then \
+			python3 -m venv --system-site-packages $(VENV_BASE)/ansible3; \
+		fi; \
+	fi
+
 virtualenv_awx:
 	if [ "$(VENV_BASE)" ]; then \
 		if [ ! -d "$(VENV_BASE)" ]; then \
 			mkdir $(VENV_BASE); \
 		fi; \
 		if [ ! -d "$(VENV_BASE)/awx" ]; then \
-			virtualenv --system-site-packages $(VENV_BASE)/awx && \
-			$(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) --ignore-installed six packaging appdirs && \
-			$(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) --ignore-installed setuptools==36.0.1 && \
-			$(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) --ignore-installed pip==9.0.1; \
+			$(PYTHON) -m venv $(VENV_BASE)/awx; \
 		fi; \
 	fi
 
@@ -161,6 +157,11 @@ requirements_ansible: virtualenv_ansible
 	fi
 	$(VENV_BASE)/ansible/bin/pip uninstall --yes -r requirements/requirements_ansible_uninstall.txt
 
+requirements_ansible_py3: virtualenv_ansible_py3
+	cat requirements/requirements_ansible.txt requirements/requirements_ansible_git.txt | $(VENV_BASE)/ansible3/bin/pip3 install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) --ignore-installed -r /dev/stdin
+	$(VENV_BASE)/ansible3/bin/pip3 install ansible  # can't inherit from system ansible, it's py2
+	$(VENV_BASE)/ansible3/bin/pip3 uninstall --yes -r requirements/requirements_ansible_uninstall.txt
+
 requirements_ansible_dev:
 	if [ "$(VENV_BASE)" ]; then \
 		$(VENV_BASE)/ansible/bin/pip install pytest mock; \
@@ -168,11 +169,9 @@ requirements_ansible_dev:
 
 requirements_isolated:
 	if [ ! -d "$(VENV_BASE)/awx" ]; then \
-		virtualenv --system-site-packages $(VENV_BASE)/awx && \
-		$(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) --ignore-installed six packaging appdirs && \
-		$(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) --ignore-installed setuptools==35.0.2 && \
-		$(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) --ignore-installed pip==9.0.1; \
+		$(PYTHON) -m venv $(VENV_BASE)/awx; \
 	fi;
+	echo "include-system-site-packages = true" >> $(VENV_BASE)/awx/lib/python$(PYTHON_VERSION)/pyvenv.cfg
 	$(VENV_BASE)/awx/bin/pip install -r requirements/requirements_isolated.txt
 
 # Install third-party requirements needed for AWX's environment.
@@ -182,14 +181,15 @@ requirements_awx: virtualenv_awx
 	else \
 	    cat requirements/requirements.txt requirements/requirements_git.txt | $(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) --ignore-installed -r /dev/stdin ; \
 	fi
-	$(VENV_BASE)/awx/bin/pip uninstall --yes -r requirements/requirements_tower_uninstall.txt
+	echo "include-system-site-packages = true" >> $(VENV_BASE)/awx/lib/python$(PYTHON_VERSION)/pyvenv.cfg
+	#$(VENV_BASE)/awx/bin/pip uninstall --yes -r requirements/requirements_tower_uninstall.txt
 
 requirements_awx_dev:
 	$(VENV_BASE)/awx/bin/pip install -r requirements/requirements_dev.txt
 
 requirements: requirements_ansible requirements_awx
 
-requirements_dev: requirements requirements_awx_dev requirements_ansible_dev
+requirements_dev: requirements requirements_ansible_py3 requirements_awx_dev requirements_ansible_dev
 
 requirements_test: requirements
 
@@ -208,7 +208,7 @@ version_file:
 	if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
 	fi; \
-	python -c "import awx as awx; print awx.__version__" > /var/lib/awx/.awx_version; \
+	python -c "import awx; print(awx.__version__)" > /var/lib/awx/.awx_version; \
 
 # Do any one-time init tasks.
 comma := ,
@@ -221,7 +221,7 @@ init:
 	if [ "$(AWX_GROUP_QUEUES)" == "tower,thepentagon" ]; then \
 		$(MANAGEMENT_COMMAND) provision_instance --hostname=isolated; \
 		$(MANAGEMENT_COMMAND) register_queue --queuename='thepentagon' --hostnames=isolated --controller=tower; \
-		$(MANAGEMENT_COMMAND) generate_isolated_key | ssh -o "StrictHostKeyChecking no" root@isolated 'cat >> /root/.ssh/authorized_keys'; \
+		$(MANAGEMENT_COMMAND) generate_isolated_key > /awx_devel/awx/main/expect/authorized_keys; \
 	fi;
 
 # Refresh development environment after pulling new code.
@@ -250,7 +250,7 @@ server_noattach:
 	tmux new-session -d -s awx 'exec make uwsgi'
 	tmux rename-window 'AWX'
 	tmux select-window -t awx:0
-	tmux split-window -v 'exec make celeryd'
+	tmux split-window -v 'exec make dispatcher'
 	tmux new-window 'exec make daphne'
 	tmux select-window -t awx:1
 	tmux rename-window 'WebSockets'
@@ -272,7 +272,7 @@ supervisor:
 	@if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
 	fi; \
-	supervisord --configuration /supervisor.conf --pidfile=/tmp/supervisor_pid
+	supervisord --pidfile=/tmp/supervisor_pid
 
 # Alternate approach to tmux to run all development tasks specified in
 # Procfile.
@@ -282,12 +282,6 @@ honcho:
 	fi; \
 	honcho start -f tools/docker-compose/Procfile
 
-flower:
-	@if [ "$(VENV_BASE)" ]; then \
-		. $(VENV_BASE)/awx/bin/activate; \
-	fi; \
-	celery flower --address=0.0.0.0 --port=5555 --broker=amqp://guest:guest@$(RABBITMQ_HOST):5672//
-
 collectstatic:
 	@if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
@@ -298,7 +292,7 @@ uwsgi: collectstatic
 	@if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
 	fi; \
-    uwsgi -b 32768 --socket 127.0.0.1:8050 --module=awx.wsgi:application --home=/venv/awx --chdir=/awx_devel/ --vacuum --processes=5 --harakiri=120 --master --no-orphans --py-autoreload 1 --max-requests=1000 --stats /tmp/stats.socket --lazy-apps --logformat "%(addr) %(method) %(uri) - %(proto) %(status)" --hook-accepting1-once="exec:/bin/sh -c '[ -f /tmp/celery_pid ] && kill -1 `cat /tmp/celery_pid` || true'"
+    uwsgi -b 32768 --socket 127.0.0.1:8050 --module=awx.wsgi:application --home=/venv/awx --chdir=/awx_devel/ --vacuum --processes=5 --harakiri=120 --master --no-orphans --py-autoreload 1 --max-requests=1000 --stats /tmp/stats.socket --lazy-apps --logformat "%(addr) %(method) %(uri) - %(proto) %(status)" --hook-accepting1-once="exec:awx-manage run_dispatcher --reload"
 
 daphne:
 	@if [ "$(VENV_BASE)" ]; then \
@@ -319,13 +313,13 @@ runserver:
 	fi; \
 	$(PYTHON) manage.py runserver
 
-# Run to start the background celery worker for development.
-celeryd:
-	rm -f /tmp/celery_pid
+# Run to start the background task dispatcher for development.
+dispatcher:
 	@if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
 	fi; \
-	celery worker -A awx -l DEBUG -B -Ofair --autoscale=100,4 --schedule=$(CELERY_SCHEDULE_FILE) -n celery@$(COMPOSE_HOST) --pidfile /tmp/celery_pid
+	$(PYTHON) manage.py run_dispatcher
+
 
 # Run to start the zeromq callback receiver
 receiver:
@@ -361,18 +355,21 @@ pyflakes: reports
 pylint: reports
 	@(set -o pipefail && $@ | reports/$@.report)
 
+genschema: reports
+	$(MAKE) swagger PYTEST_ARGS="--genschema"
+
 swagger: reports
 	@if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
 	fi; \
-	(set -o pipefail && py.test awx/conf/tests/functional awx/main/tests/functional/api awx/main/tests/docs --release=$(VERSION_TARGET) | tee reports/$@.report)
+	(set -o pipefail && py.test $(PYTEST_ARGS) awx/conf/tests/functional awx/main/tests/functional/api awx/main/tests/docs --release=$(VERSION_TARGET) | tee reports/$@.report)
 
 check: flake8 pep8 # pyflakes pylint
 
 awx-link:
 	cp -R /tmp/awx.egg-info /awx_devel/ || true
 	sed -i "s/placeholder/$(shell git describe --long | sed 's/\./\\./g')/" /awx_devel/awx.egg-info/PKG-INFO
-	cp /tmp/awx.egg-link /venv/awx/lib/python2.7/site-packages/awx.egg-link
+	cp -f /tmp/awx.egg-link /venv/awx/lib/python$(PYTHON_VERSION)/site-packages/awx.egg-link
 
 TEST_DIRS ?= awx/main/tests/unit awx/main/tests/functional awx/conf/tests awx/sso/tests
 
@@ -381,7 +378,8 @@ test:
 	@if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
 	fi; \
-	py.test -n auto $(TEST_DIRS)
+	PYTHONDONTWRITEBYTECODE=1 py.test -p no:cacheprovider -n auto $(TEST_DIRS)
+	awx-manage check_migrations --dry-run --check  -n 'vNNN_missing_migration_file'
 
 test_combined: test_ansible test
 
@@ -468,7 +466,7 @@ messages:
 # generate l10n .json .mo
 languages: $(I18N_FLAG_FILE)
 
-$(I18N_FLAG_FILE): $(UI_DEPS_FLAG_FILE)
+$(I18N_FLAG_FILE): $(UI_RELEASE_DEPS_FLAG_FILE)
 	$(NPM_BIN) --prefix awx/ui run languages
 	$(PYTHON) tools/scripts/compilemessages.py
 	touch $(I18N_FLAG_FILE)
@@ -476,13 +474,31 @@ $(I18N_FLAG_FILE): $(UI_DEPS_FLAG_FILE)
 # End l10n TASKS
 # --------------------------------------
 
-# UI TASKS
+# UI RELEASE TASKS
+# --------------------------------------
+ui-release: $(UI_RELEASE_FLAG_FILE)
+
+$(UI_RELEASE_FLAG_FILE): $(I18N_FLAG_FILE) $(UI_RELEASE_DEPS_FLAG_FILE)
+	$(NPM_BIN) --prefix awx/ui run build-release
+	touch $(UI_RELEASE_FLAG_FILE)
+
+$(UI_RELEASE_DEPS_FLAG_FILE):
+	PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=1 $(NPM_BIN) --unsafe-perm --prefix awx/ui install --no-save awx/ui
+	touch $(UI_RELEASE_DEPS_FLAG_FILE)
+
+# END UI RELEASE TASKS
 # --------------------------------------
 
+# UI TASKS
+# --------------------------------------
 ui-deps: $(UI_DEPS_FLAG_FILE)
 
 $(UI_DEPS_FLAG_FILE):
-	$(NPM_BIN) --unsafe-perm --prefix awx/ui install awx/ui
+	@if [ -f ${UI_RELEASE_DEPS_FLAG_FILE} ]; then \
+		rm -rf awx/ui/node_modules; \
+		rm -f ${UI_RELEASE_DEPS_FLAG_FILE}; \
+	fi; \
+	$(NPM_BIN) --unsafe-perm --prefix awx/ui install --no-save awx/ui
 	touch $(UI_DEPS_FLAG_FILE)
 
 ui-docker-machine: $(UI_DEPS_FLAG_FILE)
@@ -496,12 +512,6 @@ ui-docker: $(UI_DEPS_FLAG_FILE)
 ui-devel: $(UI_DEPS_FLAG_FILE)
 	$(NPM_BIN) --prefix awx/ui run build-devel -- $(MAKEFLAGS)
 
-ui-release: $(UI_RELEASE_FLAG_FILE)
-
-$(UI_RELEASE_FLAG_FILE): $(I18N_FLAG_FILE) $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run build-release
-	touch $(UI_RELEASE_FLAG_FILE)
-
 ui-test: $(UI_DEPS_FLAG_FILE)
 	$(NPM_BIN) --prefix awx/ui run test
 
@@ -512,9 +522,6 @@ ui-test-ci: $(UI_DEPS_FLAG_FILE)
 	$(NPM_BIN) --prefix awx/ui run test:ci
 	$(NPM_BIN) --prefix awx/ui run unit
 
-testjs_ci:
-	echo "Update UI unittests later" #ui-test-ci
-
 jshint: $(UI_DEPS_FLAG_FILE)
 	$(NPM_BIN) run --prefix awx/ui jshint
 	$(NPM_BIN) run --prefix awx/ui lint
@@ -562,23 +569,38 @@ docker-isolated:
 	TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml -f tools/docker-isolated-override.yml create
 	docker start tools_awx_1
 	docker start tools_isolated_1
-	echo "__version__ = '`git describe --long | cut -d - -f 1-1`'" | docker exec -i tools_isolated_1 /bin/bash -c "cat > /venv/awx/lib/python2.7/site-packages/awx.py"
-	if [ "`docker exec -i -t tools_isolated_1 cat /root/.ssh/authorized_keys`" == "`docker exec -t tools_awx_1 cat /root/.ssh/id_rsa.pub`" ]; then \
-		echo "SSH keys already copied to isolated instance"; \
-	else \
-		docker exec "tools_isolated_1" bash -c "mkdir -p /root/.ssh && rm -f /root/.ssh/authorized_keys && echo $$(docker exec -t tools_awx_1 cat /root/.ssh/id_rsa.pub) >> /root/.ssh/authorized_keys"; \
-	fi
-	TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml -f tools/docker-isolated-override.yml up
+	echo "__version__ = '`git describe --long | cut -d - -f 1-1`'" | docker exec -i tools_isolated_1 /bin/bash -c "cat > /venv/awx/lib/python$(PYTHON_VERSION)/site-packages/awx.py"
+	CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml -f tools/docker-isolated-override.yml up
 
 # Docker Compose Development environment
 docker-compose: docker-auth
-	TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml up --no-recreate awx
+	CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml up --no-recreate awx
 
 docker-compose-cluster: docker-auth
-	TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose-cluster.yml up
+	CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose-cluster.yml up
 
 docker-compose-test: docker-auth
-	cd tools && TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm --service-ports awx /bin/bash
+	cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm --service-ports awx /bin/bash
+
+docker-compose-runtest:
+	cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm --service-ports awx /start_tests.sh
+
+docker-compose-build-swagger:
+	cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm --service-ports awx /start_tests.sh swagger
+
+docker-compose-genschema:
+	cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm --service-ports awx /start_tests.sh genschema
+	mv swagger.json schema.json
+
+docker-compose-detect-schema-change:
+	$(MAKE) docker-compose-genschema
+	curl https://s3.amazonaws.com/awx-public-ci-files/schema.json -o reference-schema.json
+	# Ignore differences in whitespace with -b
+	diff -u -b reference-schema.json schema.json
+
+docker-compose-clean:
+	cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm -w /awx_devel --service-ports awx make clean
+	cd tools && TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose rm -sf
 
 docker-compose-build: awx-devel-build
 
@@ -604,14 +626,13 @@ docker-refresh: docker-clean docker-compose
 
 # Docker Development Environment with Elastic Stack Connected
 docker-compose-elk: docker-auth
-	TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml -f tools/elastic/docker-compose.logstash-link.yml -f tools/elastic/docker-compose.elastic-override.yml up --no-recreate
+	CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml -f tools/elastic/docker-compose.logstash-link.yml -f tools/elastic/docker-compose.elastic-override.yml up --no-recreate
 
 docker-compose-cluster-elk: docker-auth
 	TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose-cluster.yml -f tools/elastic/docker-compose.logstash-link-cluster.yml -f tools/elastic/docker-compose.elastic-override.yml up --no-recreate
 
 minishift-dev:
-	ansible-playbook -i localhost, -e devtree_directory=$(CURWD) tools/clusterdevel/start_minishift_dev.yml
-
+	ansible-playbook -i localhost, -e devtree_directory=$(CURDIR) tools/clusterdevel/start_minishift_dev.yml
 
 clean-elk:
 	docker stop tools_kibana_1
@@ -625,5 +646,4 @@ psql-container:
 	docker run -it --net tools_default --rm postgres:9.6 sh -c 'exec psql -h "postgres" -p "5432" -U postgres'
 
 VERSION:
-	@echo $(VERSION_TARGET) > $@
-	@echo "awx: $(VERSION_TARGET)"
+	@echo "awx: $(VERSION)"

README.md

@@ -1,7 +1,6 @@
-[![Run Status](https://api.shippable.com/projects/591c82a22f895107009e8b35/badge?branch=devel)](https://app.shippable.com/github/ansible/awx)
+[![Gated by Zuul](https://zuul-ci.org/gated.svg)](https://ansible.softwarefactory-project.io/zuul/status)
 
-AWX
-===
+<img src="https://raw.githubusercontent.com/ansible/awx-logos/master/awx/ui/client/assets/logo-login.svg?sanitize=true" width=200 alt="AWX" />
 
 AWX provides a web-based user interface, REST API, and task engine built on top of [Ansible](https://github.com/ansible/ansible). It is the upstream project for [Tower](https://www.ansible.com/tower), a commercial derivative of AWX.  
 
@@ -11,6 +10,8 @@ To learn more about using AWX, and Tower, view the [Tower docs site](http://docs
 
 The AWX Project Frequently Asked Questions can be found [here](https://www.ansible.com/awx-project-faq).
 
+The AWX logos and branding assets are covered by [our trademark guidelines](https://github.com/ansible/awx-logos/blob/master/TRADEMARKS.md).
+
 Contributing
 ------------
 

VERSION

@@ -0,0 +1 @@
+3.0.0

awx/__init__.py

@@ -12,14 +12,6 @@ __version__ = get_distribution('awx').version
 __all__ = ['__version__']
 
 
-# Isolated nodes do not have celery installed
-try:
-    from .celery import app as celery_app # noqa
-    __all__.append('celery_app')
-except ImportError:
-    pass
-
-
 # Check for the presence/absence of "devonly" module to determine if running
 # from a source code checkout or release packaage.
 try:
@@ -29,6 +21,48 @@ except ImportError: # pragma: no cover
     MODE = 'production'
 
 
+import hashlib
+
+try:
+    import django
+    from django.utils.encoding import force_bytes
+    from django.db.backends.base.schema import BaseDatabaseSchemaEditor
+    from django.db.backends.base import schema
+    HAS_DJANGO = True
+except ImportError:
+    HAS_DJANGO = False
+
+
+if HAS_DJANGO is True:
+    # This line exists to make sure we don't regress on FIPS support if we
+    # upgrade Django; if you're upgrading Django and see this error,
+    # update the version check below, and confirm that FIPS still works.
+    if django.__version__ != '1.11.16':
+        raise RuntimeError("Django version other than 1.11.16 detected {}. \
+                Subclassing BaseDatabaseSchemaEditor is known to work for Django 1.11.16 \
+                and may not work in newer Django versions.".format(django.__version__))
+
+
+    class FipsBaseDatabaseSchemaEditor(BaseDatabaseSchemaEditor):
+
+        @classmethod
+        def _digest(cls, *args):
+            """
+            Generates a 32-bit digest of a set of arguments that can be used to
+            shorten identifying names.
+            """
+            try:
+                h = hashlib.md5()
+            except ValueError:
+                h = hashlib.md5(usedforsecurity=False)
+            for arg in args:
+                h.update(force_bytes(arg))
+            return h.hexdigest()[:8]
+
+
+    schema.BaseDatabaseSchemaEditor = FipsBaseDatabaseSchemaEditor
+
+
 def find_commands(management_dir):
     # Modified version of function from django/core/management/__init__.py.
     command_dir = os.path.join(management_dir, 'commands')

awx/api/conf.py

@@ -43,7 +43,7 @@ register(
     help_text=_('Dictionary for customizing OAuth 2 timeouts, available items are '
                 '`ACCESS_TOKEN_EXPIRE_SECONDS`, the duration of access tokens in the number '
                 'of seconds, and `AUTHORIZATION_CODE_EXPIRE_SECONDS`, the duration of '
-                'authorization grants in the number of seconds.'),
+                'authorization codes in the number of seconds.'),
     category=_('Authentication'),
     category_slug='authentication',
 )

awx/api/fields.py

@@ -97,7 +97,7 @@ class DeprecatedCredentialField(serializers.IntegerField):
         kwargs['allow_null'] = True
         kwargs['default'] = None
         kwargs['min_value'] = 1
-        kwargs['help_text'] = 'This resource has been deprecated and will be removed in a future release'
+        kwargs.setdefault('help_text', 'This resource has been deprecated and will be removed in a future release')
         super(DeprecatedCredentialField, self).__init__(**kwargs)
 
     def to_internal_value(self, pk):

awx/api/filters.py

@@ -25,7 +25,6 @@ from rest_framework.filters import BaseFilterBackend
 from awx.main.utils import get_type_for_model, to_python_boolean
 from awx.main.utils.db import get_all_field_names
 from awx.main.models.credential import CredentialType
-from awx.main.models.rbac import RoleAncestorEntry
 
 
 class V1CredentialFilterBackend(BaseFilterBackend):
@@ -66,7 +65,7 @@ class TypeFilterBackend(BaseFilterBackend):
                 model = queryset.model
                 model_type = get_type_for_model(model)
                 if 'polymorphic_ctype' in get_all_field_names(model):
-                    types_pks = set([v for k,v in types_map.items() if k in types])
+                    types_pks = set([v for k, v in types_map.items() if k in types])
                     queryset = queryset.filter(polymorphic_ctype_id__in=types_pks)
                 elif model_type in types:
                     queryset = queryset
@@ -193,7 +192,7 @@ class FieldLookupBackend(BaseFilterBackend):
 
     def value_to_python(self, model, lookup, value):
         try:
-            lookup = lookup.encode("ascii")
+            lookup.encode("ascii")
         except UnicodeEncodeError:
             raise ValueError("%r is not an allowed field name. Must be ascii encodable." % lookup)
 
@@ -347,12 +346,12 @@ class FieldLookupBackend(BaseFilterBackend):
                     else:
                         args.append(Q(**{k:v}))
                 for role_name in role_filters:
+                    if not hasattr(queryset.model, 'accessible_pk_qs'):
+                        raise ParseError(_(
+                            'Cannot apply role_level filter to this list because its model '
+                            'does not use roles for access control.'))
                     args.append(
-                        Q(pk__in=RoleAncestorEntry.objects.filter(
-                            ancestor__in=request.user.roles.all(),
-                            content_type_id=ContentType.objects.get_for_model(queryset.model).id,
-                            role_field=role_name
-                        ).values_list('object_id').distinct())
+                        Q(pk__in=queryset.model.accessible_pk_qs(request.user, role_name))
                     )
                 if or_filters:
                     q = Q()
@@ -364,12 +363,12 @@ class FieldLookupBackend(BaseFilterBackend):
                     args.append(q)
                 if search_filters and search_filter_relation == 'OR':
                     q = Q()
-                    for term, constrains in search_filters.iteritems():
+                    for term, constrains in search_filters.items():
                         for constrain in constrains:
                             q |= Q(**{constrain: term})
                     args.append(q)
                 elif search_filters and search_filter_relation == 'AND':
-                    for term, constrains in search_filters.iteritems():
+                    for term, constrains in search_filters.items():
                         q_chain = Q()
                         for constrain in constrains:
                             q_chain |= Q(**{constrain: term})

awx/api/generics.py

@@ -6,7 +6,7 @@ import inspect
 import logging
 import time
 import six
-import urllib    
+import urllib.parse
 
 # Django
 from django.conf import settings
@@ -56,7 +56,7 @@ __all__ = ['APIView', 'GenericAPIView', 'ListAPIView', 'SimpleListAPIView',
            'ParentMixin',
            'DeleteLastUnattachLabelMixin',
            'SubListAttachDetachAPIView',
-           'CopyAPIView']
+           'CopyAPIView', 'BaseUsersList',]
 
 logger = logging.getLogger('awx.api.generics')
 analytics_logger = logging.getLogger('awx.analytics.performance')
@@ -90,9 +90,9 @@ class LoggedLoginView(auth_views.LoginView):
             logger.info(smart_text(u"User {} logged in.".format(self.request.user.username)))
             ret.set_cookie('userLoggedIn', 'true')
             current_user = UserSerializer(self.request.user)
-            current_user = JSONRenderer().render(current_user.data)
-            current_user = urllib.quote('%s' % current_user, '')
-            ret.set_cookie('current_user', current_user)
+            current_user = smart_text(JSONRenderer().render(current_user.data))
+            current_user = urllib.parse.quote('%s' % current_user, '')
+            ret.set_cookie('current_user', current_user, secure=settings.SESSION_COOKIE_SECURE or None)
 
             return ret
         else:
@@ -305,7 +305,7 @@ class APIView(views.APIView):
         # submitted data was rejected.
         request_method = getattr(self, '_raw_data_request_method', None)
         response_status = getattr(self, '_raw_data_response_status', 0)
-        if request_method in ('POST', 'PUT', 'PATCH') and response_status in xrange(400, 500):
+        if request_method in ('POST', 'PUT', 'PATCH') and response_status in range(400, 500):
             return self.request.data.copy()
 
         return data
@@ -348,7 +348,7 @@ class GenericAPIView(generics.GenericAPIView, APIView):
         # form.
         if hasattr(self, '_raw_data_form_marker'):
             # Always remove read only fields from serializer.
-            for name, field in serializer.fields.items():
+            for name, field in list(serializer.fields.items()):
                 if getattr(field, 'read_only', None):
                     del serializer.fields[name]
             serializer._data = self.update_raw_data(serializer.data)
@@ -390,7 +390,6 @@ class GenericAPIView(generics.GenericAPIView, APIView):
             ]:
                 d[key] = self.metadata_class().get_serializer_info(serializer, method=method)
         d['settings'] = settings
-        d['has_named_url'] = self.model in settings.NAMED_URL_GRAPH
         return d
 
 
@@ -553,9 +552,8 @@ class SubListDestroyAPIView(DestroyAPIView, SubListAPIView):
 
     def perform_list_destroy(self, instance_list):
         if self.check_sub_obj_permission:
-            # Check permissions for all before deleting, avoiding half-deleted lists
             for instance in instance_list:
-                if self.has_delete_permission(instance):
+                if not self.has_delete_permission(instance):
                     raise PermissionDenied()
         for instance in instance_list:
             self.perform_destroy(instance, check_permission=False)
@@ -750,7 +748,7 @@ class SubListAttachDetachAPIView(SubListCreateAttachDetachAPIView):
     def update_raw_data(self, data):
         request_method = getattr(self, '_raw_data_request_method', None)
         response_status = getattr(self, '_raw_data_response_status', 0)
-        if request_method == 'POST' and response_status in xrange(400, 500):
+        if request_method == 'POST' and response_status in range(400, 500):
             return super(SubListAttachDetachAPIView, self).update_raw_data(data)
         return {'id': None}
 
@@ -991,3 +989,22 @@ class CopyAPIView(GenericAPIView):
         serializer = self._get_copy_return_serializer(new_obj)
         headers = {'Location': new_obj.get_absolute_url(request=request)}
         return Response(serializer.data, status=status.HTTP_201_CREATED, headers=headers)
+
+
+class BaseUsersList(SubListCreateAttachDetachAPIView):
+    def post(self, request, *args, **kwargs):
+        ret = super(BaseUsersList, self).post( request, *args, **kwargs)
+        if ret.status_code != 201:
+            return ret
+        try:
+            if ret.data is not None and request.data.get('is_system_auditor', False):
+                # This is a faux-field that just maps to checking the system
+                # auditor role member list.. unfortunately this means we can't
+                # set it on creation, and thus needs to be set here.
+                user = User.objects.get(id=ret.data['id'])
+                user.is_system_auditor = request.data['is_system_auditor']
+                ret.data['is_system_auditor'] = request.data['is_system_auditor']
+        except AttributeError as exc:
+            print(exc)
+            pass
+        return ret

awx/api/metadata.py

@@ -63,12 +63,15 @@ class Metadata(metadata.SimpleMetadata):
                 verbose_name = smart_text(opts.verbose_name)
                 field_info['help_text'] = field_help_text[field.field_name].format(verbose_name)
 
-            for model_field in serializer.Meta.model._meta.fields:
-                if field.field_name == model_field.name:
-                    field_info['filterable'] = True
-                    break
+            if field.field_name == 'type':
+                field_info['filterable'] = True
             else:
-                field_info['filterable'] = False
+                for model_field in serializer.Meta.model._meta.fields:
+                    if field.field_name == model_field.name:
+                        field_info['filterable'] = True
+                        break
+                else:
+                    field_info['filterable'] = False
 
         # Indicate if a field has a default value.
         # FIXME: Still isn't showing all default values?
@@ -154,7 +157,7 @@ class Metadata(metadata.SimpleMetadata):
             finally:
                 view.request = request
 
-            for field, meta in actions[method].items():
+            for field, meta in list(actions[method].items()):
                 if not isinstance(meta, dict):
                     continue
 

awx/api/parsers.py

@@ -5,6 +5,7 @@ import json
 # Django
 from django.conf import settings
 from django.utils import six
+from django.utils.encoding import smart_str
 from django.utils.translation import ugettext_lazy as _
 
 # Django REST Framework
@@ -25,7 +26,7 @@ class JSONParser(parsers.JSONParser):
         encoding = parser_context.get('encoding', settings.DEFAULT_CHARSET)
 
         try:
-            data = stream.read().decode(encoding)
+            data = smart_str(stream.read(), encoding=encoding)
             if not data:
                 return {}
             obj = json.loads(data, object_pairs_hook=OrderedDict)

awx/api/serializers.py

@@ -8,7 +8,7 @@ import logging
 import operator
 import re
 import six
-import urllib
+import urllib.parse
 from collections import OrderedDict
 from datetime import timedelta
 
@@ -40,6 +40,7 @@ from rest_framework.utils.serializer_helpers import ReturnList
 from polymorphic.models import PolymorphicModel
 
 # AWX
+from awx.main.access import get_user_capabilities
 from awx.main.constants import (
     SCHEDULEABLE_PROVIDERS,
     ANSI_SGR_PATTERN,
@@ -49,7 +50,6 @@ from awx.main.constants import (
 )
 from awx.main.models import * # noqa
 from awx.main.models.base import NEW_JOB_TYPE_CHOICES
-from awx.main.access import get_user_capabilities
 from awx.main.fields import ImplicitRoleField
 from awx.main.utils import (
     get_type_for_model, get_model_for_type, timestamp_apiformat,
@@ -61,7 +61,7 @@ from awx.main.redact import UriCleaner, REPLACE_STR
 
 from awx.main.validators import vars_validate_or_raise
 
-from awx.conf.license import feature_enabled
+from awx.conf.license import feature_enabled, LicenseForbids
 from awx.api.versioning import reverse, get_request_version
 from awx.api.fields import (BooleanNullField, CharNullField, ChoiceNullField,
                             VerbatimField, DeprecatedCredentialField)
@@ -104,7 +104,7 @@ SUMMARIZABLE_FK_FIELDS = {
     'project_update': DEFAULT_SUMMARY_FIELDS + ('status', 'failed',),
     'credential': DEFAULT_SUMMARY_FIELDS + ('kind', 'cloud', 'credential_type_id'),
     'vault_credential': DEFAULT_SUMMARY_FIELDS + ('kind', 'cloud', 'credential_type_id'),
-    'job': DEFAULT_SUMMARY_FIELDS + ('status', 'failed', 'elapsed'),
+    'job': DEFAULT_SUMMARY_FIELDS + ('status', 'failed', 'elapsed', 'type'),
     'job_template': DEFAULT_SUMMARY_FIELDS,
     'workflow_job_template': DEFAULT_SUMMARY_FIELDS,
     'workflow_job': DEFAULT_SUMMARY_FIELDS,
@@ -203,11 +203,11 @@ class BaseSerializerMetaclass(serializers.SerializerMetaclass):
 
     @staticmethod
     def _is_list_of_strings(x):
-        return isinstance(x, (list, tuple)) and all([isinstance(y, basestring) for y in x])
+        return isinstance(x, (list, tuple)) and all([isinstance(y, str) for y in x])
 
     @staticmethod
     def _is_extra_kwargs(x):
-        return isinstance(x, dict) and all([isinstance(k, basestring) and isinstance(v, dict) for k,v in x.items()])
+        return isinstance(x, dict) and all([isinstance(k, str) and isinstance(v, dict) for k,v in x.items()])
 
     @classmethod
     def _update_meta(cls, base, meta, other=None):
@@ -259,9 +259,7 @@ class BaseSerializerMetaclass(serializers.SerializerMetaclass):
         return super(BaseSerializerMetaclass, cls).__new__(cls, name, bases, attrs)
 
 
-class BaseSerializer(serializers.ModelSerializer):
-
-    __metaclass__ = BaseSerializerMetaclass
+class BaseSerializer(serializers.ModelSerializer, metaclass=BaseSerializerMetaclass):
 
     class Meta:
         fields = ('id', 'type', 'url', 'related', 'summary_fields', 'created',
@@ -284,7 +282,7 @@ class BaseSerializer(serializers.ModelSerializer):
         # The following lines fix the problem of being able to pass JSON dict into PrimaryKeyRelatedField.
         data = kwargs.get('data', False)
         if data:
-            for field_name, field_instance in six.iteritems(self.fields):
+            for field_name, field_instance in self.fields.items():
                 if isinstance(field_instance, ManyRelatedField) and not field_instance.read_only:
                     if isinstance(data.get(field_name, False), dict):
                         raise serializers.ValidationError(_('Cannot use dictionary for %s' % field_name))
@@ -294,7 +292,7 @@ class BaseSerializer(serializers.ModelSerializer):
         """
         The request version component of the URL as an integer i.e., 1 or 2
         """
-        return get_request_version(self.context.get('request'))
+        return get_request_version(self.context.get('request')) or 1
 
     def get_type(self, obj):
         return get_type_for_model(self.Meta.model)
@@ -612,7 +610,7 @@ class BaseSerializer(serializers.ModelSerializer):
                         v2.extend(e)
                     else:
                         v2.append(e)
-                d[k] = map(force_text, v2)
+                d[k] = list(map(force_text, v2))
             raise ValidationError(d)
         return attrs
 
@@ -632,9 +630,7 @@ class EmptySerializer(serializers.Serializer):
     pass
 
 
-class BaseFactSerializer(BaseSerializer):
-
-    __metaclass__ = BaseSerializerMetaclass
+class BaseFactSerializer(BaseSerializer, metaclass=BaseSerializerMetaclass):
 
     def get_fields(self):
         ret = super(BaseFactSerializer, self).get_fields()
@@ -1311,16 +1307,12 @@ class ProjectSerializer(UnifiedJobTemplateSerializer, ProjectOptionsSerializer):
         'admin', 'update',
         {'copy': 'organization.project_admin'}
     ]
-    scm_delete_on_next_update = serializers.BooleanField(
-        read_only=True,
-        help_text=_('This field has been deprecated and will be removed in a future release'))
 
     class Meta:
         model = Project
-        fields = ('*', 'organization', 'scm_delete_on_next_update', 'scm_update_on_launch',
+        fields = ('*', 'organization', 'scm_update_on_launch',
                   'scm_update_cache_timeout', 'scm_revision', 'custom_virtualenv',) + \
                  ('last_update_failed', 'last_updated')  # Backwards compatibility
-        read_only_fields = ('scm_delete_on_next_update',)
 
     def get_related(self, obj):
         res = super(ProjectSerializer, self).get_related(obj)
@@ -1503,6 +1495,12 @@ class InventorySerializer(BaseSerializerWithVariables):
         'admin', 'adhoc',
         {'copy': 'organization.inventory_admin'}
     ]
+    groups_with_active_failures = serializers.IntegerField(
+        read_only=True,
+        min_value=0,
+        help_text=_('This field has been deprecated and will be removed in a future release')
+    )
+
 
     class Meta:
         model = Inventory
@@ -1724,6 +1722,11 @@ class AnsibleFactsSerializer(BaseSerializer):
 
 class GroupSerializer(BaseSerializerWithVariables):
     capabilities_prefetch = ['inventory.admin', 'inventory.adhoc']
+    groups_with_active_failures = serializers.IntegerField(
+        read_only=True,
+        min_value=0,
+        help_text=_('This field has been deprecated and will be removed in a future release')
+    )
 
     class Meta:
         model = Group
@@ -1903,7 +1906,9 @@ class CustomInventoryScriptSerializer(BaseSerializer):
 
 
 class InventorySourceOptionsSerializer(BaseSerializer):
-    credential = DeprecatedCredentialField()
+    credential = DeprecatedCredentialField(
+        help_text=_('Cloud credential to use for inventory updates.')
+    )
 
     class Meta:
         fields = ('*', 'source', 'source_path', 'source_script', 'source_vars', 'credential',
@@ -2130,10 +2135,10 @@ class InventorySourceSerializer(UnifiedJobTemplateSerializer, InventorySourceOpt
             return attrs.get(fd, self.instance and getattr(self.instance, fd) or None)
 
         if get_field_from_model_or_attrs('source') != 'scm':
-            redundant_scm_fields = filter(
+            redundant_scm_fields = list(filter(
                 lambda x: attrs.get(x, None),
                 ['source_project', 'source_path', 'update_on_project_update']
-            )
+            ))
             if redundant_scm_fields:
                 raise serializers.ValidationError(
                     {"detail": _("Cannot set %s if not SCM type." % ' '.join(redundant_scm_fields))}
@@ -2456,17 +2461,17 @@ class CredentialTypeSerializer(BaseSerializer):
                     field['help_text'] = _(field['help_text'])
                 if field['type'] == 'become_method':
                     field.pop('type')
-                    field['choices'] = map(operator.itemgetter(0), CHOICES_PRIVILEGE_ESCALATION_METHODS)
+                    field['choices'] = list(map(operator.itemgetter(0), CHOICES_PRIVILEGE_ESCALATION_METHODS))
         return value
 
     def filter_field_metadata(self, fields, method):
         # API-created/modified CredentialType kinds are limited to
         # `cloud` and `net`
         if method in ('PUT', 'POST'):
-            fields['kind']['choices'] = filter(
+            fields['kind']['choices'] = list(filter(
                 lambda choice: choice[0] in ('cloud', 'net'),
                 fields['kind']['choices']
-            )
+            ))
         return fields
 
 
@@ -2617,8 +2622,8 @@ class CredentialSerializer(BaseSerializer):
                 raise serializers.ValidationError({"kind": _('"%s" is not a valid choice' % kind)})
             data['credential_type'] = credential_type.pk
             value = OrderedDict(
-                {'credential_type': credential_type}.items() +
-                super(CredentialSerializer, self).to_internal_value(data).items()
+                list({'credential_type': credential_type}.items()) +
+                list(super(CredentialSerializer, self).to_internal_value(data).items())
             )
 
             # Make a set of the keys in the POST/PUT payload
@@ -2974,12 +2979,16 @@ class JobTemplateMixin(object):
     '''
 
     def _recent_jobs(self, obj):
-        if hasattr(obj, 'workflow_jobs'):
-            job_mgr = obj.workflow_jobs
-        else:
-            job_mgr = obj.jobs
-        return [{'id': x.id, 'status': x.status, 'finished': x.finished}
-                for x in job_mgr.all().order_by('-created')[:10]]
+        # Exclude "joblets", jobs that ran as part of a sliced workflow job
+        uj_qs = obj.unifiedjob_unified_jobs.exclude(job__job_slice_count__gt=1).order_by('-created')
+        # Would like to apply an .only, but does not play well with non_polymorphic
+        # .only('id', 'status', 'finished', 'polymorphic_ctype_id')
+        optimized_qs = uj_qs.non_polymorphic()
+        return [{
+            'id': x.id, 'status': x.status, 'finished': x.finished,
+            # Make type consistent with API top-level key, for instance workflow_job
+            'type': x.get_real_instance_class()._meta.verbose_name.replace(' ', '_')
+        } for x in optimized_qs[:10]]
 
     def get_summary_fields(self, obj):
         d = super(JobTemplateMixin, self).get_summary_fields(obj)
@@ -3009,7 +3018,7 @@ class JobTemplateSerializer(JobTemplateMixin, UnifiedJobTemplateSerializer, JobO
         fields = ('*', 'host_config_key', 'ask_diff_mode_on_launch', 'ask_variables_on_launch', 'ask_limit_on_launch', 'ask_tags_on_launch',
                   'ask_skip_tags_on_launch', 'ask_job_type_on_launch', 'ask_verbosity_on_launch', 'ask_inventory_on_launch',
                   'ask_credential_on_launch', 'survey_enabled', 'become_enabled', 'diff_mode',
-                  'allow_simultaneous', 'custom_virtualenv')
+                  'allow_simultaneous', 'custom_virtualenv', 'job_slice_count')
 
     def get_related(self, obj):
         res = super(JobTemplateSerializer, self).get_related(obj)
@@ -3026,6 +3035,7 @@ class JobTemplateSerializer(JobTemplateMixin, UnifiedJobTemplateSerializer, JobO
             labels = self.reverse('api:job_template_label_list', kwargs={'pk': obj.pk}),
             object_roles = self.reverse('api:job_template_object_roles_list', kwargs={'pk': obj.pk}),
             instance_groups = self.reverse('api:job_template_instance_groups_list', kwargs={'pk': obj.pk}),
+            slice_workflow_jobs = self.reverse('api:job_template_slice_workflow_jobs_list', kwargs={'pk': obj.pk}),
         ))
         if self.version > 1:
             res['copy'] = self.reverse('api:job_template_copy', kwargs={'pk': obj.pk})
@@ -3047,7 +3057,7 @@ class JobTemplateSerializer(JobTemplateMixin, UnifiedJobTemplateSerializer, JobO
 
         prompting_error_message = _("Must either set a default value or ask to prompt on launch.")
         if project is None:
-            raise serializers.ValidationError({'project': _("Job types 'run' and 'check' must have assigned a project.")})
+            raise serializers.ValidationError({'project': _("Job Templates must have a project assigned.")})
         elif inventory is None and not get_field_from_model_or_attrs('ask_inventory_on_launch'):
             raise serializers.ValidationError({'inventory': prompting_error_message})
 
@@ -3056,6 +3066,13 @@ class JobTemplateSerializer(JobTemplateMixin, UnifiedJobTemplateSerializer, JobO
     def validate_extra_vars(self, value):
         return vars_validate_or_raise(value)
 
+    def validate_job_slice_count(self, value):
+        if value > 1 and not feature_enabled('workflows'):
+            raise LicenseForbids({'job_slice_count': [_(
+                "Job slicing is a workflows-based feature and your license does not allow use of workflows."
+            )]})
+        return value
+
     def get_summary_fields(self, obj):
         summary_fields = super(JobTemplateSerializer, self).get_summary_fields(obj)
         all_creds = []
@@ -3100,19 +3117,46 @@ class JobTemplateSerializer(JobTemplateMixin, UnifiedJobTemplateSerializer, JobO
         return summary_fields
 
 
+class JobTemplateWithSpecSerializer(JobTemplateSerializer):
+    '''
+    Used for activity stream entries.
+    '''
+
+    class Meta:
+        model = JobTemplate
+        fields = ('*', 'survey_spec')
+
 
 class JobSerializer(UnifiedJobSerializer, JobOptionsSerializer):
 
     passwords_needed_to_start = serializers.ReadOnlyField()
-    ask_diff_mode_on_launch = serializers.ReadOnlyField()
-    ask_variables_on_launch = serializers.ReadOnlyField()
-    ask_limit_on_launch = serializers.ReadOnlyField()
-    ask_skip_tags_on_launch = serializers.ReadOnlyField()
-    ask_tags_on_launch = serializers.ReadOnlyField()
-    ask_job_type_on_launch = serializers.ReadOnlyField()
-    ask_verbosity_on_launch = serializers.ReadOnlyField()
-    ask_inventory_on_launch = serializers.ReadOnlyField()
-    ask_credential_on_launch = serializers.ReadOnlyField()
+    ask_diff_mode_on_launch = serializers.BooleanField(
+        read_only=True,
+        help_text=_('This field has been deprecated and will be removed in a future release'))
+    ask_variables_on_launch = serializers.BooleanField(
+        read_only=True,
+        help_text=_('This field has been deprecated and will be removed in a future release'))
+    ask_limit_on_launch = serializers.BooleanField(
+        read_only=True,
+        help_text=_('This field has been deprecated and will be removed in a future release'))
+    ask_skip_tags_on_launch = serializers.BooleanField(
+        read_only=True,
+        help_text=_('This field has been deprecated and will be removed in a future release'))
+    ask_tags_on_launch = serializers.BooleanField(
+        read_only=True,
+        help_text=_('This field has been deprecated and will be removed in a future release'))
+    ask_job_type_on_launch = serializers.BooleanField(
+        read_only=True,
+        help_text=_('This field has been deprecated and will be removed in a future release'))
+    ask_verbosity_on_launch = serializers.BooleanField(
+        read_only=True,
+        help_text=_('This field has been deprecated and will be removed in a future release'))
+    ask_inventory_on_launch = serializers.BooleanField(
+        read_only=True,
+        help_text=_('This field has been deprecated and will be removed in a future release'))
+    ask_credential_on_launch = serializers.BooleanField(
+        read_only=True,
+        help_text=_('This field has been deprecated and will be removed in a future release'))
     artifacts = serializers.SerializerMethodField()
 
     class Meta:
@@ -3121,7 +3165,7 @@ class JobSerializer(UnifiedJobSerializer, JobOptionsSerializer):
                   'ask_variables_on_launch', 'ask_limit_on_launch', 'ask_tags_on_launch', 'ask_skip_tags_on_launch',
                   'ask_job_type_on_launch', 'ask_verbosity_on_launch', 'ask_inventory_on_launch',
                   'ask_credential_on_launch', 'allow_simultaneous', 'artifacts', 'scm_revision',
-                  'instance_group', 'diff_mode')
+                  'instance_group', 'diff_mode', 'job_slice_number', 'job_slice_count')
 
     def get_related(self, obj):
         res = super(JobSerializer, self).get_related(obj)
@@ -3439,12 +3483,16 @@ class AdHocCommandSerializer(UnifiedJobSerializer):
             ret['name'] = obj.module_name
         return ret
 
+    def validate(self, attrs):
+        ret = super(AdHocCommandSerializer, self).validate(attrs)
+        return ret
+
     def validate_extra_vars(self, value):
         redacted_extra_vars, removed_vars = extract_ansible_vars(value)
         if removed_vars:
             raise serializers.ValidationError(_(
                 "{} are prohibited from use in ad hoc commands."
-            ).format(", ".join(removed_vars)))
+            ).format(", ".join(sorted(removed_vars, reverse=True))))
         return vars_validate_or_raise(value)
 
 
@@ -3555,7 +3603,7 @@ class WorkflowJobTemplateSerializer(JobTemplateMixin, LabelsListMixin, UnifiedJo
     class Meta:
         model = WorkflowJobTemplate
         fields = ('*', 'extra_vars', 'organization', 'survey_enabled', 'allow_simultaneous',
-                  'ask_variables_on_launch',)
+                  'ask_variables_on_launch', 'inventory', 'ask_inventory_on_launch',)
 
     def get_related(self, obj):
         res = super(WorkflowJobTemplateSerializer, self).get_related(obj)
@@ -3583,12 +3631,24 @@ class WorkflowJobTemplateSerializer(JobTemplateMixin, LabelsListMixin, UnifiedJo
         return vars_validate_or_raise(value)
 
 
+class WorkflowJobTemplateWithSpecSerializer(WorkflowJobTemplateSerializer):
+    '''
+    Used for activity stream entries.
+    '''
+
+    class Meta:
+        model = WorkflowJobTemplate
+        fields = ('*', 'survey_spec')
+
+
 class WorkflowJobSerializer(LabelsListMixin, UnifiedJobSerializer):
 
     class Meta:
         model = WorkflowJob
         fields = ('*', 'workflow_job_template', 'extra_vars', 'allow_simultaneous',
-                  '-execution_node', '-event_processing_finished', '-controller_node',)
+                  'job_template', 'is_sliced_job',
+                  '-execution_node', '-event_processing_finished', '-controller_node',
+                  'inventory',)
 
     def get_related(self, obj):
         res = super(WorkflowJobSerializer, self).get_related(obj)
@@ -3596,6 +3656,8 @@ class WorkflowJobSerializer(LabelsListMixin, UnifiedJobSerializer):
             res['workflow_job_template'] = self.reverse('api:workflow_job_template_detail',
                                                         kwargs={'pk': obj.workflow_job_template.pk})
             res['notifications'] = self.reverse('api:workflow_job_notifications_list', kwargs={'pk': obj.pk})
+        if obj.job_template_id:
+            res['job_template'] = self.reverse('api:job_template_detail', kwargs={'pk': obj.job_template_id})
         res['workflow_nodes'] = self.reverse('api:workflow_job_workflow_nodes_list', kwargs={'pk': obj.pk})
         res['labels'] = self.reverse('api:workflow_job_label_list', kwargs={'pk': obj.pk})
         res['activity_stream'] = self.reverse('api:workflow_job_activity_stream_list', kwargs={'pk': obj.pk})
@@ -3658,7 +3720,7 @@ class LaunchConfigurationBaseSerializer(BaseSerializer):
             for field in self.instance._meta.fields:
                 setattr(mock_obj, field.name, getattr(self.instance, field.name))
         field_names = set(field.name for field in self.Meta.model._meta.fields)
-        for field_name, value in attrs.items():
+        for field_name, value in list(attrs.items()):
             setattr(mock_obj, field_name, value)
             if field_name not in field_names:
                 attrs.pop(field_name)
@@ -3669,7 +3731,7 @@ class LaunchConfigurationBaseSerializer(BaseSerializer):
         if obj is None:
             return ret
         if 'extra_data' in ret and obj.survey_passwords:
-            ret['extra_data'] = obj.display_extra_data()
+            ret['extra_data'] = obj.display_extra_vars()
         return ret
 
     def get_summary_fields(self, obj):
@@ -3810,9 +3872,6 @@ class WorkflowJobTemplateNodeSerializer(LaunchConfigurationBaseSerializer):
             ujt_obj = attrs['unified_job_template']
         elif self.instance:
             ujt_obj = self.instance.unified_job_template
-        if isinstance(ujt_obj, (WorkflowJobTemplate)):
-            raise serializers.ValidationError({
-                "unified_job_template": _("Cannot nest a %s inside a WorkflowJobTemplate") % ujt_obj.__class__.__name__})
         if 'credential' in deprecated_fields:  # TODO: remove when v2 API is deprecated
             cred = deprecated_fields['credential']
             attrs['credential'] = cred
@@ -3861,7 +3920,8 @@ class WorkflowJobNodeSerializer(LaunchConfigurationBaseSerializer):
     class Meta:
         model = WorkflowJobNode
         fields = ('*', 'credential', 'job', 'workflow_job', '-name', '-description', 'id', 'url', 'related',
-                  'unified_job_template', 'success_nodes', 'failure_nodes', 'always_nodes',)
+                  'unified_job_template', 'success_nodes', 'failure_nodes', 'always_nodes',
+                  'do_not_run',)
 
     def get_related(self, obj):
         res = super(WorkflowJobNodeSerializer, self).get_related(obj)
@@ -4274,7 +4334,7 @@ class JobLaunchSerializer(BaseSerializer):
                             passwords_needed=cred.passwords_needed
                         )
                         if cred.credential_type.managed_by_tower and 'vault_id' in cred.credential_type.defined_fields:
-                            cred_dict['vault_id'] = cred.inputs.get('vault_id') or None
+                            cred_dict['vault_id'] = cred.get_input('vault_id', default=None)
                         defaults_dict.setdefault(field_name, []).append(cred_dict)
             else:
                 defaults_dict[field_name] = getattr(obj, field_name)
@@ -4363,37 +4423,63 @@ class JobLaunchSerializer(BaseSerializer):
 class WorkflowJobLaunchSerializer(BaseSerializer):
 
     can_start_without_user_input = serializers.BooleanField(read_only=True)
+    defaults = serializers.SerializerMethodField()
     variables_needed_to_start = serializers.ReadOnlyField()
     survey_enabled = serializers.SerializerMethodField()
     extra_vars = VerbatimField(required=False, write_only=True)
+    inventory = serializers.PrimaryKeyRelatedField(
+        queryset=Inventory.objects.all(),
+        required=False, write_only=True
+    )
     workflow_job_template_data = serializers.SerializerMethodField()
 
     class Meta:
         model = WorkflowJobTemplate
-        fields = ('can_start_without_user_input', 'extra_vars',
-                  'survey_enabled', 'variables_needed_to_start',
+        fields = ('ask_inventory_on_launch', 'can_start_without_user_input', 'defaults', 'extra_vars',
+                  'inventory', 'survey_enabled', 'variables_needed_to_start',
                   'node_templates_missing', 'node_prompts_rejected',
-                  'workflow_job_template_data')
+                  'workflow_job_template_data', 'survey_enabled')
+        read_only_fields = ('ask_inventory_on_launch',)
 
     def get_survey_enabled(self, obj):
         if obj:
             return obj.survey_enabled and 'spec' in obj.survey_spec
         return False
 
+    def get_defaults(self, obj):
+        defaults_dict = {}
+        for field_name in WorkflowJobTemplate.get_ask_mapping().keys():
+            if field_name == 'inventory':
+                defaults_dict[field_name] = dict(
+                    name=getattrd(obj, '%s.name' % field_name, None),
+                    id=getattrd(obj, '%s.pk' % field_name, None))
+            else:
+                defaults_dict[field_name] = getattr(obj, field_name)
+        return defaults_dict
+
     def get_workflow_job_template_data(self, obj):
         return dict(name=obj.name, id=obj.id, description=obj.description)
 
     def validate(self, attrs):
-        obj = self.instance
+        template = self.instance
 
-        accepted, rejected, errors = obj._accept_or_ignore_job_kwargs(
-            _exclude_errors=['required'],
-            **attrs)
+        accepted, rejected, errors = template._accept_or_ignore_job_kwargs(**attrs)
+        self._ignored_fields = rejected
 
-        WFJT_extra_vars = obj.extra_vars
-        attrs = super(WorkflowJobLaunchSerializer, self).validate(attrs)
-        obj.extra_vars = WFJT_extra_vars
-        return attrs
+        if template.inventory and template.inventory.pending_deletion is True:
+            errors['inventory'] = _("The inventory associated with this Workflow is being deleted.")
+        elif 'inventory' in accepted and accepted['inventory'].pending_deletion:
+            errors['inventory'] = _("The provided inventory is being deleted.")
+
+        if errors:
+            raise serializers.ValidationError(errors)
+
+        WFJT_extra_vars = template.extra_vars
+        WFJT_inventory = template.inventory
+        super(WorkflowJobLaunchSerializer, self).validate(attrs)
+        template.extra_vars = WFJT_extra_vars
+        template.inventory = WFJT_inventory
+        return accepted
 
 
 class NotificationTemplateSerializer(BaseSerializer):
@@ -4404,11 +4490,11 @@ class NotificationTemplateSerializer(BaseSerializer):
         model = NotificationTemplate
         fields = ('*', 'organization', 'notification_type', 'notification_configuration')
 
-    type_map = {"string": (str, unicode),
+    type_map = {"string": (str,),
                 "int": (int,),
                 "bool": (bool,),
                 "list": (list,),
-                "password": (str, unicode),
+                "password": (str,),
                 "object": (dict, OrderedDict)}
 
     def to_representation(self, obj):
@@ -4533,13 +4619,13 @@ class SchedulePreviewSerializer(BaseSerializer):
     # - COUNT > 999
     def validate_rrule(self, value):
         rrule_value = value
-        multi_by_month_day = ".*?BYMONTHDAY[\:\=][0-9]+,-*[0-9]+"
-        multi_by_month = ".*?BYMONTH[\:\=][0-9]+,[0-9]+"
-        by_day_with_numeric_prefix = ".*?BYDAY[\:\=][0-9]+[a-zA-Z]{2}"
-        match_count = re.match(".*?(COUNT\=[0-9]+)", rrule_value)
-        match_multiple_dtstart = re.findall(".*?(DTSTART(;[^:]+)?\:[0-9]+T[0-9]+Z?)", rrule_value)
-        match_native_dtstart = re.findall(".*?(DTSTART:[0-9]+T[0-9]+) ", rrule_value)
-        match_multiple_rrule = re.findall(".*?(RRULE\:)", rrule_value)
+        multi_by_month_day = r".*?BYMONTHDAY[\:\=][0-9]+,-*[0-9]+"
+        multi_by_month = r".*?BYMONTH[\:\=][0-9]+,[0-9]+"
+        by_day_with_numeric_prefix = r".*?BYDAY[\:\=][0-9]+[a-zA-Z]{2}"
+        match_count = re.match(r".*?(COUNT\=[0-9]+)", rrule_value)
+        match_multiple_dtstart = re.findall(r".*?(DTSTART(;[^:]+)?\:[0-9]+T[0-9]+Z?)", rrule_value)
+        match_native_dtstart = re.findall(r".*?(DTSTART:[0-9]+T[0-9]+) ", rrule_value)
+        match_multiple_rrule = re.findall(r".*?(RRULE\:)", rrule_value)
         if not len(match_multiple_dtstart):
             raise serializers.ValidationError(_('Valid DTSTART required in rrule. Value should start with: DTSTART:YYYYMMDDTHHMMSSZ'))
         if len(match_native_dtstart):
@@ -4612,6 +4698,23 @@ class ScheduleSerializer(LaunchConfigurationBaseSerializer, SchedulePreviewSeria
             res['inventory'] = obj.unified_job_template.inventory.get_absolute_url(self.context.get('request'))
         return res
 
+    def get_summary_fields(self, obj):
+        summary_fields = super(ScheduleSerializer, self).get_summary_fields(obj)
+        if 'inventory' in summary_fields:
+            return summary_fields
+
+        inventory = None
+        if obj.unified_job_template and getattr(obj.unified_job_template, 'inventory', None):
+            inventory = obj.unified_job_template.inventory
+        else:
+            return summary_fields
+
+        summary_fields['inventory'] = dict()
+        for field in SUMMARIZABLE_FK_FIELDS['inventory']:
+            summary_fields['inventory'][field] = getattr(inventory, field, None)
+
+        return summary_fields
+
     def validate_unified_job_template(self, value):
         if type(value) == InventorySource and value.source not in SCHEDULEABLE_PROVIDERS:
             raise serializers.ValidationError(_('Inventory Source must be a cloud resource.'))
@@ -4774,7 +4877,7 @@ class ActivityStreamSerializer(BaseSerializer):
         for key in summary_dict.keys():
             if 'id' not in summary_dict[key]:
                 summary_dict[key] = summary_dict[key] + ('id',)
-        field_list = summary_dict.items()
+        field_list = list(summary_dict.items())
         # Needed related fields that are not in the default summary fields
         field_list += [
             ('workflow_job_template_node', ('id', 'unified_job_template_id')),
@@ -4794,7 +4897,7 @@ class ActivityStreamSerializer(BaseSerializer):
 
     def get_fields(self):
         ret = super(ActivityStreamSerializer, self).get_fields()
-        for key, field in ret.items():
+        for key, field in list(ret.items()):
             if key == 'changes':
                 field.help_text = _('A summary of the new and changed values when an object is created, updated, or deleted')
             if key == 'object1':
@@ -4835,10 +4938,6 @@ class ActivityStreamSerializer(BaseSerializer):
 
     def get_related(self, obj):
         rel = {}
-        VIEW_NAME_EXCEPTIONS = {
-            'custom_inventory_script': 'inventory_script_detail',
-            'o_auth2_access_token': 'o_auth2_token_detail'
-        }
         if obj.actor is not None:
             rel['actor'] = self.reverse('api:user_detail', kwargs={'pk': obj.actor.pk})
         for fk, __ in self._local_summarizable_fk_fields:
@@ -4852,11 +4951,12 @@ class ActivityStreamSerializer(BaseSerializer):
                     if getattr(thisItem, 'id', None) in id_list:
                         continue
                     id_list.append(getattr(thisItem, 'id', None))
-                    if fk in VIEW_NAME_EXCEPTIONS:
-                        view_name = VIEW_NAME_EXCEPTIONS[fk]
+                    if hasattr(thisItem, 'get_absolute_url'):
+                        rel_url = thisItem.get_absolute_url(self.context.get('request'))
                     else:
                         view_name = fk + '_detail'
-                    rel[fk].append(self.reverse('api:' + view_name, kwargs={'pk': thisItem.id}))
+                        rel_url = self.reverse('api:' + view_name, kwargs={'pk': thisItem.id})
+                    rel[fk].append(rel_url)
 
                     if fk == 'schedule':
                         rel['unified_job_template'] = thisItem.unified_job_template.get_absolute_url(self.context.get('request'))
@@ -4939,7 +5039,7 @@ class FactVersionSerializer(BaseFactSerializer):
         }
         res['fact_view'] = '%s?%s' % (
             reverse('api:host_fact_compare_view', kwargs={'pk': obj.host.pk}, request=self.context.get('request')),
-            urllib.urlencode(params)
+            urllib.parse.urlencode(params)
         )
         return res
 

awx/api/templates/api/_list_common.md

@@ -54,8 +54,6 @@ within all designated text fields of a model.
 
     ?search=findme
 
-_Added in AWX 1.4_
-
 (_Added in Ansible Tower 3.1.0_) Search across related fields:
 
     ?related__search=findme
@@ -84,7 +82,7 @@ To exclude results matching certain criteria, prefix the field parameter with
 
     ?not__field=value
 
-(_Added in AWX 1.4_) By default, all query string filters are AND'ed together, so
+By default, all query string filters are AND'ed together, so
 only the results matching *all* filters will be returned.  To combine results
 matching *any* one of multiple criteria, prefix each query string parameter
 with `or__`:

awx/api/templates/api/inventory_script_view.md

@@ -10,7 +10,7 @@ object containing groups, including the hosts, children and variables for each
 group.  The response data is equivalent to that returned by passing the
 `--list` argument to an inventory script.
 
-_(Added in AWX 1.3)_ Specify a query string of `?hostvars=1` to retrieve the JSON
+Specify a query string of `?hostvars=1` to retrieve the JSON
 object above including all host variables.  The `['_meta']['hostvars']` object
 in the response contains an entry for each host with its variables.  This
 response format can be used with Ansible 1.3 and later to avoid making a
@@ -18,11 +18,19 @@ separate API request for each host.  Refer to
 [Tuning the External Inventory Script](http://docs.ansible.com/developing_inventory.html#tuning-the-external-inventory-script)
 for more information on this feature.
 
-_(Added in AWX 1.4)_ By default, the inventory script will only return hosts that
+By default, the inventory script will only return hosts that
 are enabled in the inventory.  This feature allows disabled hosts to be skipped
 when running jobs without removing them from the inventory.  Specify a query
 string of `?all=1` to return all hosts, including disabled ones.
 
+Specify a query string of `?towervars=1` to add variables
+to the hostvars of each host that specifies its enabled state and database ID.
+
+Specify a query string of `?subset=slice2of5` to produce an inventory that
+has a restricted number of hosts according to the rules of job slicing.
+
+To apply multiple query strings, join them with the `&` character, like `?hostvars=1&all=1`.
+
 ## Host Response
 
 Make a GET request to this resource with a query string similar to

awx/api/templates/api/retrieve_api_view.md

@@ -1,7 +1,3 @@
-{% if has_named_url %}
-### Note: starting from api v2, this resource object can be accessed via its named URL.
-{% endif %}
-
 # Retrieve {{ model_verbose_name|title|anora }}:
 
 Make GET request to this resource to retrieve a single {{ model_verbose_name }}

awx/api/templates/api/retrieve_destroy_api_view.md

@@ -1,7 +1,3 @@
-{% if has_named_url %}
-### Note: starting from api v2, this resource object can be accessed via its named URL.
-{% endif %}
-
 {% ifmeth GET %}
 # Retrieve {{ model_verbose_name|title|anora }}:
 

awx/api/templates/api/retrieve_update_api_view.md

@@ -1,7 +1,3 @@
-{% if has_named_url %}
-### Note: starting from api v2, this resource object can be accessed via its named URL.
-{% endif %}
-
 {% ifmeth GET %}
 # Retrieve {{ model_verbose_name|title|anora }}:
 

awx/api/templates/api/retrieve_update_destroy_api_view.md

@@ -1,7 +1,3 @@
-{% if has_named_url %}
-### Note: starting from api v2, this resource object can be accessed via its named URL.
-{% endif %}
-
 {% ifmeth GET %}
 # Retrieve {{ model_verbose_name|title|anora }}:
 

awx/api/urls/job_template.py

@@ -8,6 +8,7 @@ from awx.api.views import (
     JobTemplateDetail,
     JobTemplateLaunch,
     JobTemplateJobsList,
+    JobTemplateSliceWorkflowJobsList,
     JobTemplateCallback,
     JobTemplateSchedulesList,
     JobTemplateSurveySpec,
@@ -28,6 +29,7 @@ urls = [
     url(r'^(?P<pk>[0-9]+)/$', JobTemplateDetail.as_view(), name='job_template_detail'),
     url(r'^(?P<pk>[0-9]+)/launch/$', JobTemplateLaunch.as_view(), name='job_template_launch'),
     url(r'^(?P<pk>[0-9]+)/jobs/$', JobTemplateJobsList.as_view(), name='job_template_jobs_list'),
+    url(r'^(?P<pk>[0-9]+)/slice_workflow_jobs/$', JobTemplateSliceWorkflowJobsList.as_view(), name='job_template_slice_workflow_jobs_list'),
     url(r'^(?P<pk>[0-9]+)/callback/$', JobTemplateCallback.as_view(), name='job_template_callback'),
     url(r'^(?P<pk>[0-9]+)/schedules/$', JobTemplateSchedulesList.as_view(), name='job_template_schedules_list'),
     url(r'^(?P<pk>[0-9]+)/survey_spec/$', JobTemplateSurveySpec.as_view(), name='job_template_survey_spec'),

awx/api/views/inventory.py

@@ -0,0 +1,211 @@
+# Copyright (c) 2018 Red Hat, Inc.
+# All Rights Reserved.
+
+# Python
+import logging
+
+# Django
+from django.conf import settings
+from django.db.models import Q
+from django.contrib.contenttypes.models import ContentType
+from django.utils.translation import ugettext_lazy as _
+
+# Django REST Framework
+from rest_framework.exceptions import PermissionDenied
+from rest_framework.response import Response
+from rest_framework import status
+
+# AWX
+from awx.main.models import (
+    ActivityStream,
+    Inventory,
+    JobTemplate,
+    Role,
+    User,
+    InstanceGroup,
+    InventoryUpdateEvent,
+    InventoryUpdate,
+    InventorySource,
+    CustomInventoryScript,
+)
+from awx.api.generics import (
+    ListCreateAPIView,
+    RetrieveUpdateDestroyAPIView,
+    SubListAPIView,
+    SubListAttachDetachAPIView,
+    ResourceAccessList,
+    CopyAPIView,
+)
+
+from awx.api.serializers import (
+    InventorySerializer,
+    ActivityStreamSerializer,
+    RoleSerializer,
+    InstanceGroupSerializer,
+    InventoryUpdateEventSerializer,
+    CustomInventoryScriptSerializer,
+    InventoryDetailSerializer,
+    JobTemplateSerializer,
+)
+from awx.api.views.mixin import (
+    ActivityStreamEnforcementMixin,
+    RelatedJobsPreventDeleteMixin,
+    ControlledByScmMixin,
+)
+
+logger = logging.getLogger('awx.api.views.organization')
+
+
+class InventoryUpdateEventsList(SubListAPIView):
+
+    model = InventoryUpdateEvent
+    serializer_class = InventoryUpdateEventSerializer
+    parent_model = InventoryUpdate
+    relationship = 'inventory_update_events'
+    view_name = _('Inventory Update Events List')
+    search_fields = ('stdout',)
+
+    def finalize_response(self, request, response, *args, **kwargs):
+        response['X-UI-Max-Events'] = settings.MAX_UI_JOB_EVENTS
+        return super(InventoryUpdateEventsList, self).finalize_response(request, response, *args, **kwargs)
+
+
+class InventoryScriptList(ListCreateAPIView):
+
+    model = CustomInventoryScript
+    serializer_class = CustomInventoryScriptSerializer
+
+
+class InventoryScriptDetail(RetrieveUpdateDestroyAPIView):
+
+    model = CustomInventoryScript
+    serializer_class = CustomInventoryScriptSerializer
+
+    def destroy(self, request, *args, **kwargs):
+        instance = self.get_object()
+        can_delete = request.user.can_access(self.model, 'delete', instance)
+        if not can_delete:
+            raise PermissionDenied(_("Cannot delete inventory script."))
+        for inv_src in InventorySource.objects.filter(source_script=instance):
+            inv_src.source_script = None
+            inv_src.save()
+        return super(InventoryScriptDetail, self).destroy(request, *args, **kwargs)
+
+
+class InventoryScriptObjectRolesList(SubListAPIView):
+
+    model = Role
+    serializer_class = RoleSerializer
+    parent_model = CustomInventoryScript
+    search_fields = ('role_field', 'content_type__model',)
+
+    def get_queryset(self):
+        po = self.get_parent_object()
+        content_type = ContentType.objects.get_for_model(self.parent_model)
+        return Role.objects.filter(content_type=content_type, object_id=po.pk)
+
+
+class InventoryScriptCopy(CopyAPIView):
+
+    model = CustomInventoryScript
+    copy_return_serializer_class = CustomInventoryScriptSerializer
+
+
+class InventoryList(ListCreateAPIView):
+
+    model = Inventory
+    serializer_class = InventorySerializer
+
+    def get_queryset(self):
+        qs = Inventory.accessible_objects(self.request.user, 'read_role')
+        qs = qs.select_related('admin_role', 'read_role', 'update_role', 'use_role', 'adhoc_role')
+        qs = qs.prefetch_related('created_by', 'modified_by', 'organization')
+        return qs
+
+
+class InventoryDetail(RelatedJobsPreventDeleteMixin, ControlledByScmMixin, RetrieveUpdateDestroyAPIView):
+
+    model = Inventory
+    serializer_class = InventoryDetailSerializer
+
+    def update(self, request, *args, **kwargs):
+        obj = self.get_object()
+        kind = self.request.data.get('kind') or kwargs.get('kind')
+
+        # Do not allow changes to an Inventory kind.
+        if kind is not None and obj.kind != kind:
+            return self.http_method_not_allowed(request, *args, **kwargs)
+        return super(InventoryDetail, self).update(request, *args, **kwargs)
+
+    def destroy(self, request, *args, **kwargs):
+        obj = self.get_object()
+        if not request.user.can_access(self.model, 'delete', obj):
+            raise PermissionDenied()
+        self.check_related_active_jobs(obj)  # related jobs mixin
+        try:
+            obj.schedule_deletion(getattr(request.user, 'id', None))
+            return Response(status=status.HTTP_202_ACCEPTED)
+        except RuntimeError as e:
+            return Response(dict(error=_("{0}".format(e))), status=status.HTTP_400_BAD_REQUEST)
+
+
+class InventoryActivityStreamList(ActivityStreamEnforcementMixin, SubListAPIView):
+
+    model = ActivityStream
+    serializer_class = ActivityStreamSerializer
+    parent_model = Inventory
+    relationship = 'activitystream_set'
+    search_fields = ('changes',)
+
+    def get_queryset(self):
+        parent = self.get_parent_object()
+        self.check_parent_access(parent)
+        qs = self.request.user.get_queryset(self.model)
+        return qs.filter(Q(inventory=parent) | Q(host__in=parent.hosts.all()) | Q(group__in=parent.groups.all()))
+
+
+class InventoryInstanceGroupsList(SubListAttachDetachAPIView):
+
+    model = InstanceGroup
+    serializer_class = InstanceGroupSerializer
+    parent_model = Inventory
+    relationship = 'instance_groups'
+
+
+class InventoryAccessList(ResourceAccessList):
+
+    model = User # needs to be User for AccessLists's
+    parent_model = Inventory
+
+
+class InventoryObjectRolesList(SubListAPIView):
+
+    model = Role
+    serializer_class = RoleSerializer
+    parent_model = Inventory
+    search_fields = ('role_field', 'content_type__model',)
+
+    def get_queryset(self):
+        po = self.get_parent_object()
+        content_type = ContentType.objects.get_for_model(self.parent_model)
+        return Role.objects.filter(content_type=content_type, object_id=po.pk)
+
+
+class InventoryJobTemplateList(SubListAPIView):
+
+    model = JobTemplate
+    serializer_class = JobTemplateSerializer
+    parent_model = Inventory
+    relationship = 'jobtemplates'
+
+    def get_queryset(self):
+        parent = self.get_parent_object()
+        self.check_parent_access(parent)
+        qs = self.request.user.get_queryset(self.model)
+        return qs.filter(inventory=parent)
+
+
+class InventoryCopy(CopyAPIView):
+
+    model = Inventory
+    copy_return_serializer_class = InventorySerializer

awx/api/views/mixin.py

@@ -0,0 +1,309 @@
+# Copyright (c) 2018 Red Hat, Inc.
+# All Rights Reserved.
+
+import dateutil
+import logging
+
+from django.db.models import (
+    Count,
+    F,
+)
+from django.db import transaction
+from django.shortcuts import get_object_or_404
+from django.utils.timezone import now
+from django.utils.translation import ugettext_lazy as _
+
+from rest_framework.permissions import SAFE_METHODS
+from rest_framework.exceptions import PermissionDenied
+from rest_framework.response import Response
+from rest_framework import status
+
+from awx.main.constants import ACTIVE_STATES
+from awx.main.utils import (
+    get_object_or_400,
+    parse_yaml_or_json,
+)
+from awx.main.models.ha import (
+    Instance,
+    InstanceGroup,
+)
+from awx.main.models.organization import Team
+from awx.main.models.projects import Project
+from awx.main.models.inventory import Inventory
+from awx.main.models.jobs import JobTemplate
+from awx.conf.license import (
+    feature_enabled,
+    LicenseForbids,
+)
+from awx.api.exceptions import ActiveJobConflict
+
+logger = logging.getLogger('awx.api.views.mixin')
+
+
+class ActivityStreamEnforcementMixin(object):
+    '''
+    Mixin to check that license supports activity streams.
+    '''
+    def check_permissions(self, request):
+        ret = super(ActivityStreamEnforcementMixin, self).check_permissions(request)
+        if not feature_enabled('activity_streams'):
+            raise LicenseForbids(_('Your license does not allow use of the activity stream.'))
+        return ret
+
+
+class SystemTrackingEnforcementMixin(object):
+    '''
+    Mixin to check that license supports system tracking.
+    '''
+    def check_permissions(self, request):
+        ret = super(SystemTrackingEnforcementMixin, self).check_permissions(request)
+        if not feature_enabled('system_tracking'):
+            raise LicenseForbids(_('Your license does not permit use of system tracking.'))
+        return ret
+
+
+class WorkflowsEnforcementMixin(object):
+    '''
+    Mixin to check that license supports workflows.
+    '''
+    def check_permissions(self, request):
+        ret = super(WorkflowsEnforcementMixin, self).check_permissions(request)
+        if not feature_enabled('workflows') and request.method not in ('GET', 'OPTIONS', 'DELETE'):
+            raise LicenseForbids(_('Your license does not allow use of workflows.'))
+        return ret
+
+
+class UnifiedJobDeletionMixin(object):
+    '''
+    Special handling when deleting a running unified job object.
+    '''
+    def destroy(self, request, *args, **kwargs):
+        obj = self.get_object()
+        if not request.user.can_access(self.model, 'delete', obj):
+            raise PermissionDenied()
+        try:
+            if obj.unified_job_node.workflow_job.status in ACTIVE_STATES:
+                raise PermissionDenied(detail=_('Cannot delete job resource when associated workflow job is running.'))
+        except self.model.unified_job_node.RelatedObjectDoesNotExist:
+            pass
+        # Still allow deletion of new status, because these can be manually created
+        if obj.status in ACTIVE_STATES and obj.status != 'new':
+            raise PermissionDenied(detail=_("Cannot delete running job resource."))
+        elif not obj.event_processing_finished:
+            # Prohibit deletion if job events are still coming in
+            if obj.finished and now() < obj.finished + dateutil.relativedelta.relativedelta(minutes=1):
+                # less than 1 minute has passed since job finished and events are not in
+                return Response({"error": _("Job has not finished processing events.")},
+                                status=status.HTTP_400_BAD_REQUEST)
+            else:
+                # if it has been > 1 minute, events are probably lost
+                logger.warning('Allowing deletion of {} through the API without all events '
+                               'processed.'.format(obj.log_format))
+        obj.delete()
+        return Response(status=status.HTTP_204_NO_CONTENT)
+
+
+class InstanceGroupMembershipMixin(object):
+    '''
+    This mixin overloads attach/detach so that it calls InstanceGroup.save(),
+    triggering a background recalculation of policy-based instance group
+    membership.
+    '''
+    def attach(self, request, *args, **kwargs):
+        response = super(InstanceGroupMembershipMixin, self).attach(request, *args, **kwargs)
+        sub_id, res = self.attach_validate(request)
+        if status.is_success(response.status_code):
+            if self.parent_model is Instance:
+                ig_obj = get_object_or_400(self.model, pk=sub_id)
+                inst_name = ig_obj.hostname
+            else:
+                inst_name = get_object_or_400(self.model, pk=sub_id).hostname
+            with transaction.atomic():
+                ig_qs = InstanceGroup.objects.select_for_update()
+                if self.parent_model is Instance:
+                    ig_obj = get_object_or_400(ig_qs, pk=sub_id)
+                else:
+                    # similar to get_parent_object, but selected for update
+                    parent_filter = {
+                        self.lookup_field: self.kwargs.get(self.lookup_field, None),
+                    }
+                    ig_obj = get_object_or_404(ig_qs, **parent_filter)
+                if inst_name not in ig_obj.policy_instance_list:
+                    ig_obj.policy_instance_list.append(inst_name)
+                    ig_obj.save(update_fields=['policy_instance_list'])
+        return response
+
+    def is_valid_relation(self, parent, sub, created=False):
+        if sub.is_isolated():
+            return {'error': _('Isolated instances may not be added or removed from instances groups via the API.')}
+        if self.parent_model is InstanceGroup:
+            ig_obj = self.get_parent_object()
+            if ig_obj.controller_id is not None:
+                return {'error': _('Isolated instance group membership may not be managed via the API.')}
+        return None
+
+    def unattach_validate(self, request):
+        (sub_id, res) = super(InstanceGroupMembershipMixin, self).unattach_validate(request)
+        if res:
+            return (sub_id, res)
+        sub = get_object_or_400(self.model, pk=sub_id)
+        attach_errors = self.is_valid_relation(None, sub)
+        if attach_errors:
+            return (sub_id, Response(attach_errors, status=status.HTTP_400_BAD_REQUEST))
+        return (sub_id, res)
+
+    def unattach(self, request, *args, **kwargs):
+        response = super(InstanceGroupMembershipMixin, self).unattach(request, *args, **kwargs)
+        if status.is_success(response.status_code):
+            sub_id = request.data.get('id', None)
+            if self.parent_model is Instance:
+                inst_name = self.get_parent_object().hostname
+            else:
+                inst_name = get_object_or_400(self.model, pk=sub_id).hostname
+            with transaction.atomic():
+                ig_qs = InstanceGroup.objects.select_for_update()
+                if self.parent_model is Instance:
+                    ig_obj = get_object_or_400(ig_qs, pk=sub_id)
+                else:
+                    # similar to get_parent_object, but selected for update
+                    parent_filter = {
+                        self.lookup_field: self.kwargs.get(self.lookup_field, None),
+                    }
+                    ig_obj = get_object_or_404(ig_qs, **parent_filter)
+                if inst_name in ig_obj.policy_instance_list:
+                    ig_obj.policy_instance_list.pop(ig_obj.policy_instance_list.index(inst_name))
+                    ig_obj.save(update_fields=['policy_instance_list'])
+        return response
+
+
+class RelatedJobsPreventDeleteMixin(object):
+    def perform_destroy(self, obj):
+        self.check_related_active_jobs(obj)
+        return super(RelatedJobsPreventDeleteMixin, self).perform_destroy(obj)
+
+    def check_related_active_jobs(self, obj):
+        active_jobs = obj.get_active_jobs()
+        if len(active_jobs) > 0:
+            raise ActiveJobConflict(active_jobs)
+        time_cutoff = now() - dateutil.relativedelta.relativedelta(minutes=1)
+        recent_jobs = obj._get_related_jobs().filter(finished__gte = time_cutoff)
+        for unified_job in recent_jobs.get_real_instances():
+            if not unified_job.event_processing_finished:
+                raise PermissionDenied(_(
+                    'Related job {} is still processing events.'
+                ).format(unified_job.log_format))
+
+
+class OrganizationCountsMixin(object):
+
+    def get_serializer_context(self, *args, **kwargs):
+        full_context = super(OrganizationCountsMixin, self).get_serializer_context(*args, **kwargs)
+
+        if self.request is None:
+            return full_context
+
+        db_results = {}
+        org_qs = self.model.accessible_objects(self.request.user, 'read_role')
+        org_id_list = org_qs.values('id')
+        if len(org_id_list) == 0:
+            if self.request.method == 'POST':
+                full_context['related_field_counts'] = {}
+            return full_context
+
+        inv_qs = Inventory.accessible_objects(self.request.user, 'read_role')
+        project_qs = Project.accessible_objects(self.request.user, 'read_role')
+
+        # Produce counts of Foreign Key relationships
+        db_results['inventories'] = inv_qs\
+            .values('organization').annotate(Count('organization')).order_by('organization')
+
+        db_results['teams'] = Team.accessible_objects(
+            self.request.user, 'read_role').values('organization').annotate(
+            Count('organization')).order_by('organization')
+
+        JT_project_reference = 'project__organization'
+        JT_inventory_reference = 'inventory__organization'
+        db_results['job_templates_project'] = JobTemplate.accessible_objects(
+            self.request.user, 'read_role').exclude(
+            project__organization=F(JT_inventory_reference)).values(JT_project_reference).annotate(
+            Count(JT_project_reference)).order_by(JT_project_reference)
+
+        db_results['job_templates_inventory'] = JobTemplate.accessible_objects(
+            self.request.user, 'read_role').values(JT_inventory_reference).annotate(
+            Count(JT_inventory_reference)).order_by(JT_inventory_reference)
+
+        db_results['projects'] = project_qs\
+            .values('organization').annotate(Count('organization')).order_by('organization')
+
+        # Other members and admins of organization are always viewable
+        db_results['users'] = org_qs.annotate(
+            users=Count('member_role__members', distinct=True),
+            admins=Count('admin_role__members', distinct=True)
+        ).values('id', 'users', 'admins')
+
+        count_context = {}
+        for org in org_id_list:
+            org_id = org['id']
+            count_context[org_id] = {
+                'inventories': 0, 'teams': 0, 'users': 0, 'job_templates': 0,
+                'admins': 0, 'projects': 0}
+
+        for res, count_qs in db_results.items():
+            if res == 'job_templates_project':
+                org_reference = JT_project_reference
+            elif res == 'job_templates_inventory':
+                org_reference = JT_inventory_reference
+            elif res == 'users':
+                org_reference = 'id'
+            else:
+                org_reference = 'organization'
+            for entry in count_qs:
+                org_id = entry[org_reference]
+                if org_id in count_context:
+                    if res == 'users':
+                        count_context[org_id]['admins'] = entry['admins']
+                        count_context[org_id]['users'] = entry['users']
+                        continue
+                    count_context[org_id][res] = entry['%s__count' % org_reference]
+
+        # Combine the counts for job templates by project and inventory
+        for org in org_id_list:
+            org_id = org['id']
+            count_context[org_id]['job_templates'] = 0
+            for related_path in ['job_templates_project', 'job_templates_inventory']:
+                if related_path in count_context[org_id]:
+                    count_context[org_id]['job_templates'] += count_context[org_id].pop(related_path)
+
+        full_context['related_field_counts'] = count_context
+
+        return full_context
+
+
+class ControlledByScmMixin(object):
+    '''
+    Special method to reset SCM inventory commit hash
+    if anything that it manages changes.
+    '''
+
+    def _reset_inv_src_rev(self, obj):
+        if self.request.method in SAFE_METHODS or not obj:
+            return
+        project_following_sources = obj.inventory_sources.filter(
+            update_on_project_update=True, source='scm')
+        if project_following_sources:
+            # Allow inventory changes unrelated to variables
+            if self.model == Inventory and (
+                    not self.request or not self.request.data or
+                    parse_yaml_or_json(self.request.data.get('variables', '')) == parse_yaml_or_json(obj.variables)):
+                return
+            project_following_sources.update(scm_last_revision='')
+
+    def get_object(self):
+        obj = super(ControlledByScmMixin, self).get_object()
+        self._reset_inv_src_rev(obj)
+        return obj
+
+    def get_parent_object(self):
+        obj = super(ControlledByScmMixin, self).get_parent_object()
+        self._reset_inv_src_rev(obj)
+        return obj

awx/api/views/organization.py

@@ -0,0 +1,247 @@
+# Copyright (c) 2018 Red Hat, Inc.
+# All Rights Reserved.
+
+# Python
+import logging
+
+# Django
+from django.db.models import Count
+from django.contrib.contenttypes.models import ContentType
+from django.utils.translation import ugettext_lazy as _
+
+# AWX
+from awx.conf.license import (
+    feature_enabled,
+    LicenseForbids,
+)
+from awx.main.models import (
+    ActivityStream,
+    Inventory,
+    Project,
+    JobTemplate,
+    WorkflowJobTemplate,
+    Organization,
+    NotificationTemplate,
+    Role,
+    User,
+    Team,
+    InstanceGroup,
+)
+from awx.api.generics import (
+    ListCreateAPIView,
+    RetrieveUpdateDestroyAPIView,
+    SubListAPIView,
+    SubListCreateAttachDetachAPIView,
+    SubListAttachDetachAPIView,
+    ResourceAccessList,
+    BaseUsersList,
+)
+
+from awx.api.serializers import (
+    OrganizationSerializer,
+    InventorySerializer,
+    ProjectSerializer,
+    UserSerializer,
+    TeamSerializer,
+    ActivityStreamSerializer,
+    RoleSerializer,
+    NotificationTemplateSerializer,
+    WorkflowJobTemplateSerializer,
+    InstanceGroupSerializer,
+)
+from awx.api.views.mixin import (
+    ActivityStreamEnforcementMixin,
+    RelatedJobsPreventDeleteMixin,
+    OrganizationCountsMixin,
+)
+
+logger = logging.getLogger('awx.api.views.organization')
+
+
+class OrganizationList(OrganizationCountsMixin, ListCreateAPIView):
+
+    model = Organization
+    serializer_class = OrganizationSerializer
+
+    def get_queryset(self):
+        qs = Organization.accessible_objects(self.request.user, 'read_role')
+        qs = qs.select_related('admin_role', 'auditor_role', 'member_role', 'read_role')
+        qs = qs.prefetch_related('created_by', 'modified_by')
+        return qs
+
+    def create(self, request, *args, **kwargs):
+        """Create a new organzation.
+
+        If there is already an organization and the license of this
+        instance does not permit multiple organizations, then raise
+        LicenseForbids.
+        """
+        # Sanity check: If the multiple organizations feature is disallowed
+        # by the license, then we are only willing to create this organization
+        # if no organizations exist in the system.
+        if (not feature_enabled('multiple_organizations') and
+                self.model.objects.exists()):
+            raise LicenseForbids(_('Your license only permits a single '
+                                   'organization to exist.'))
+
+        # Okay, create the organization as usual.
+        return super(OrganizationList, self).create(request, *args, **kwargs)
+
+
+class OrganizationDetail(RelatedJobsPreventDeleteMixin, RetrieveUpdateDestroyAPIView):
+
+    model = Organization
+    serializer_class = OrganizationSerializer
+
+    def get_serializer_context(self, *args, **kwargs):
+        full_context = super(OrganizationDetail, self).get_serializer_context(*args, **kwargs)
+
+        if not hasattr(self, 'kwargs') or 'pk' not in self.kwargs:
+            return full_context
+        org_id = int(self.kwargs['pk'])
+
+        org_counts = {}
+        access_kwargs = {'accessor': self.request.user, 'role_field': 'read_role'}
+        direct_counts = Organization.objects.filter(id=org_id).annotate(
+            users=Count('member_role__members', distinct=True),
+            admins=Count('admin_role__members', distinct=True)
+        ).values('users', 'admins')
+
+        if not direct_counts:
+            return full_context
+
+        org_counts = direct_counts[0]
+        org_counts['inventories'] = Inventory.accessible_objects(**access_kwargs).filter(
+            organization__id=org_id).count()
+        org_counts['teams'] = Team.accessible_objects(**access_kwargs).filter(
+            organization__id=org_id).count()
+        org_counts['projects'] = Project.accessible_objects(**access_kwargs).filter(
+            organization__id=org_id).count()
+        org_counts['job_templates'] = JobTemplate.accessible_objects(**access_kwargs).filter(
+            project__organization__id=org_id).count()
+
+        full_context['related_field_counts'] = {}
+        full_context['related_field_counts'][org_id] = org_counts
+
+        return full_context
+
+
+class OrganizationInventoriesList(SubListAPIView):
+
+    model = Inventory
+    serializer_class = InventorySerializer
+    parent_model = Organization
+    relationship = 'inventories'
+
+
+class OrganizationUsersList(BaseUsersList):
+
+    model = User
+    serializer_class = UserSerializer
+    parent_model = Organization
+    relationship = 'member_role.members'
+
+
+class OrganizationAdminsList(BaseUsersList):
+
+    model = User
+    serializer_class = UserSerializer
+    parent_model = Organization
+    relationship = 'admin_role.members'
+
+
+class OrganizationProjectsList(SubListCreateAttachDetachAPIView):
+
+    model = Project
+    serializer_class = ProjectSerializer
+    parent_model = Organization
+    relationship = 'projects'
+    parent_key = 'organization'
+
+
+class OrganizationWorkflowJobTemplatesList(SubListCreateAttachDetachAPIView):
+
+    model = WorkflowJobTemplate
+    serializer_class = WorkflowJobTemplateSerializer
+    parent_model = Organization
+    relationship = 'workflows'
+    parent_key = 'organization'
+
+
+class OrganizationTeamsList(SubListCreateAttachDetachAPIView):
+
+    model = Team
+    serializer_class = TeamSerializer
+    parent_model = Organization
+    relationship = 'teams'
+    parent_key = 'organization'
+
+
+class OrganizationActivityStreamList(ActivityStreamEnforcementMixin, SubListAPIView):
+
+    model = ActivityStream
+    serializer_class = ActivityStreamSerializer
+    parent_model = Organization
+    relationship = 'activitystream_set'
+    search_fields = ('changes',)
+
+
+class OrganizationNotificationTemplatesList(SubListCreateAttachDetachAPIView):
+
+    model = NotificationTemplate
+    serializer_class = NotificationTemplateSerializer
+    parent_model = Organization
+    relationship = 'notification_templates'
+    parent_key = 'organization'
+
+
+class OrganizationNotificationTemplatesAnyList(SubListCreateAttachDetachAPIView):
+
+    model = NotificationTemplate
+    serializer_class = NotificationTemplateSerializer
+    parent_model = Organization
+    relationship = 'notification_templates_any'
+
+
+class OrganizationNotificationTemplatesErrorList(SubListCreateAttachDetachAPIView):
+
+    model = NotificationTemplate
+    serializer_class = NotificationTemplateSerializer
+    parent_model = Organization
+    relationship = 'notification_templates_error'
+
+
+class OrganizationNotificationTemplatesSuccessList(SubListCreateAttachDetachAPIView):
+
+    model = NotificationTemplate
+    serializer_class = NotificationTemplateSerializer
+    parent_model = Organization
+    relationship = 'notification_templates_success'
+
+
+class OrganizationInstanceGroupsList(SubListAttachDetachAPIView):
+
+    model = InstanceGroup
+    serializer_class = InstanceGroupSerializer
+    parent_model = Organization
+    relationship = 'instance_groups'
+
+
+class OrganizationAccessList(ResourceAccessList):
+
+    model = User # needs to be User for AccessLists's
+    parent_model = Organization
+
+
+class OrganizationObjectRolesList(SubListAPIView):
+
+    model = Role
+    serializer_class = RoleSerializer
+    parent_model = Organization
+    search_fields = ('role_field', 'content_type__model',)
+
+    def get_queryset(self):
+        po = self.get_parent_object()
+        content_type = ContentType.objects.get_for_model(self.parent_model)
+        return Role.objects.filter(content_type=content_type, object_id=po.pk)
+

awx/api/views/root.py

@@ -0,0 +1,279 @@
+# Copyright (c) 2018 Ansible, Inc.
+# All Rights Reserved.
+
+import logging
+import operator
+import json
+from collections import OrderedDict
+
+from django.conf import settings
+from django.utils.encoding import smart_text
+from django.utils.decorators import method_decorator
+from django.views.decorators.csrf import ensure_csrf_cookie
+from django.template.loader import render_to_string
+from django.utils.translation import ugettext_lazy as _
+
+from rest_framework.permissions import AllowAny, IsAuthenticated
+from rest_framework.response import Response
+from rest_framework import status
+
+from awx.api.generics import APIView
+from awx.main.ha import is_ha_environment
+from awx.main.utils import (
+    get_awx_version,
+    get_ansible_version,
+    get_custom_venv_choices,
+    to_python_boolean,
+)
+from awx.api.versioning import reverse, get_request_version, drf_reverse
+from awx.conf.license import get_license, feature_enabled
+from awx.main.models import (
+    Project,
+    Organization,
+    Instance,
+    InstanceGroup,
+    JobTemplate,
+)
+
+logger = logging.getLogger('awx.api.views.root')
+
+
+class ApiRootView(APIView):
+
+    permission_classes = (AllowAny,)
+    view_name = _('REST API')
+    versioning_class = None
+    swagger_topic = 'Versioning'
+
+    @method_decorator(ensure_csrf_cookie)
+    def get(self, request, format=None):
+        ''' List supported API versions '''
+
+        v1 = reverse('api:api_v1_root_view', kwargs={'version': 'v1'})
+        v2 = reverse('api:api_v2_root_view', kwargs={'version': 'v2'})
+        data = OrderedDict()
+        data['description'] = _('AWX REST API')
+        data['current_version'] = v2
+        data['available_versions'] = dict(v1 = v1, v2 = v2)
+        data['oauth2'] = drf_reverse('api:oauth_authorization_root_view')
+        if feature_enabled('rebranding'):
+            data['custom_logo'] = settings.CUSTOM_LOGO
+            data['custom_login_info'] = settings.CUSTOM_LOGIN_INFO
+        return Response(data)
+
+
+class ApiOAuthAuthorizationRootView(APIView):
+
+    permission_classes = (AllowAny,)
+    view_name = _("API OAuth 2 Authorization Root")
+    versioning_class = None
+    swagger_topic = 'Authentication'
+
+    def get(self, request, format=None):
+        data = OrderedDict()
+        data['authorize'] = drf_reverse('api:authorize')
+        data['token'] = drf_reverse('api:token')
+        data['revoke_token'] = drf_reverse('api:revoke-token')
+        return Response(data)
+
+
+class ApiVersionRootView(APIView):
+
+    permission_classes = (AllowAny,)
+    swagger_topic = 'Versioning'
+
+    def get(self, request, format=None):
+        ''' List top level resources '''
+        data = OrderedDict()
+        data['ping'] = reverse('api:api_v1_ping_view', request=request)
+        data['instances'] = reverse('api:instance_list', request=request)
+        data['instance_groups'] = reverse('api:instance_group_list', request=request)
+        data['config'] = reverse('api:api_v1_config_view', request=request)
+        data['settings'] = reverse('api:setting_category_list', request=request)
+        data['me'] = reverse('api:user_me_list', request=request)
+        data['dashboard'] = reverse('api:dashboard_view', request=request)
+        data['organizations'] = reverse('api:organization_list', request=request)
+        data['users'] = reverse('api:user_list', request=request)
+        data['projects'] = reverse('api:project_list', request=request)
+        data['project_updates'] = reverse('api:project_update_list', request=request)
+        data['teams'] = reverse('api:team_list', request=request)
+        data['credentials'] = reverse('api:credential_list', request=request)
+        if get_request_version(request) > 1:
+            data['credential_types'] = reverse('api:credential_type_list', request=request)
+            data['applications'] = reverse('api:o_auth2_application_list', request=request)
+            data['tokens'] = reverse('api:o_auth2_token_list', request=request)
+        data['inventory'] = reverse('api:inventory_list', request=request)
+        data['inventory_scripts'] = reverse('api:inventory_script_list', request=request)
+        data['inventory_sources'] = reverse('api:inventory_source_list', request=request)
+        data['inventory_updates'] = reverse('api:inventory_update_list', request=request)
+        data['groups'] = reverse('api:group_list', request=request)
+        data['hosts'] = reverse('api:host_list', request=request)
+        data['job_templates'] = reverse('api:job_template_list', request=request)
+        data['jobs'] = reverse('api:job_list', request=request)
+        data['job_events'] = reverse('api:job_event_list', request=request)
+        data['ad_hoc_commands'] = reverse('api:ad_hoc_command_list', request=request)
+        data['system_job_templates'] = reverse('api:system_job_template_list', request=request)
+        data['system_jobs'] = reverse('api:system_job_list', request=request)
+        data['schedules'] = reverse('api:schedule_list', request=request)
+        data['roles'] = reverse('api:role_list', request=request)
+        data['notification_templates'] = reverse('api:notification_template_list', request=request)
+        data['notifications'] = reverse('api:notification_list', request=request)
+        data['labels'] = reverse('api:label_list', request=request)
+        data['unified_job_templates'] = reverse('api:unified_job_template_list', request=request)
+        data['unified_jobs'] = reverse('api:unified_job_list', request=request)
+        data['activity_stream'] = reverse('api:activity_stream_list', request=request)
+        data['workflow_job_templates'] = reverse('api:workflow_job_template_list', request=request)
+        data['workflow_jobs'] = reverse('api:workflow_job_list', request=request)
+        data['workflow_job_template_nodes'] = reverse('api:workflow_job_template_node_list', request=request)
+        data['workflow_job_nodes'] = reverse('api:workflow_job_node_list', request=request)
+        return Response(data)
+
+
+class ApiV1RootView(ApiVersionRootView):
+    view_name = _('Version 1')
+
+
+class ApiV2RootView(ApiVersionRootView):
+    view_name = _('Version 2')
+
+
+class ApiV1PingView(APIView):
+    """A simple view that reports very basic information about this
+    instance, which is acceptable to be public information.
+    """
+    permission_classes = (AllowAny,)
+    authentication_classes = ()
+    view_name = _('Ping')
+    swagger_topic = 'System Configuration'
+
+    def get(self, request, format=None):
+        """Return some basic information about this instance
+
+        Everything returned here should be considered public / insecure, as
+        this requires no auth and is intended for use by the installer process.
+        """
+        response = {
+            'ha': is_ha_environment(),
+            'version': get_awx_version(),
+            'active_node': settings.CLUSTER_HOST_ID,
+        }
+
+        response['instances'] = []
+        for instance in Instance.objects.all():
+            response['instances'].append(dict(node=instance.hostname, heartbeat=instance.modified,
+                                              capacity=instance.capacity, version=instance.version))
+            sorted(response['instances'], key=operator.itemgetter('node'))
+        response['instance_groups'] = []
+        for instance_group in InstanceGroup.objects.all():
+            response['instance_groups'].append(dict(name=instance_group.name,
+                                                    capacity=instance_group.capacity,
+                                                    instances=[x.hostname for x in instance_group.instances.all()]))
+        return Response(response)
+
+
+class ApiV1ConfigView(APIView):
+
+    permission_classes = (IsAuthenticated,)
+    view_name = _('Configuration')
+    swagger_topic = 'System Configuration'
+
+    def check_permissions(self, request):
+        super(ApiV1ConfigView, self).check_permissions(request)
+        if not request.user.is_superuser and request.method.lower() not in {'options', 'head', 'get'}:
+            self.permission_denied(request)  # Raises PermissionDenied exception.
+
+    def get(self, request, format=None):
+        '''Return various sitewide configuration settings'''
+
+        if request.user.is_superuser or request.user.is_system_auditor:
+            license_data = get_license(show_key=True)
+        else:
+            license_data = get_license(show_key=False)
+        if not license_data.get('valid_key', False):
+            license_data = {}
+        if license_data and 'features' in license_data and 'activity_streams' in license_data['features']:
+            # FIXME: Make the final setting value dependent on the feature?
+            license_data['features']['activity_streams'] &= settings.ACTIVITY_STREAM_ENABLED
+
+        pendo_state = settings.PENDO_TRACKING_STATE if settings.PENDO_TRACKING_STATE in ('off', 'anonymous', 'detailed') else 'off'
+
+        data = dict(
+            time_zone=settings.TIME_ZONE,
+            license_info=license_data,
+            version=get_awx_version(),
+            ansible_version=get_ansible_version(),
+            eula=render_to_string("eula.md") if license_data.get('license_type', 'UNLICENSED') != 'open' else '',
+            analytics_status=pendo_state
+        )
+
+        # If LDAP is enabled, user_ldap_fields will return a list of field
+        # names that are managed by LDAP and should be read-only for users with
+        # a non-empty ldap_dn attribute.
+        if getattr(settings, 'AUTH_LDAP_SERVER_URI', None) and feature_enabled('ldap'):
+            user_ldap_fields = ['username', 'password']
+            user_ldap_fields.extend(getattr(settings, 'AUTH_LDAP_USER_ATTR_MAP', {}).keys())
+            user_ldap_fields.extend(getattr(settings, 'AUTH_LDAP_USER_FLAGS_BY_GROUP', {}).keys())
+            data['user_ldap_fields'] = user_ldap_fields
+
+        if request.user.is_superuser \
+                or request.user.is_system_auditor \
+                or Organization.accessible_objects(request.user, 'admin_role').exists() \
+                or Organization.accessible_objects(request.user, 'auditor_role').exists():
+            data.update(dict(
+                project_base_dir = settings.PROJECTS_ROOT,
+                project_local_paths = Project.get_local_path_choices(),
+                custom_virtualenvs = get_custom_venv_choices()
+            ))
+        elif JobTemplate.accessible_objects(request.user, 'admin_role').exists():
+            data['custom_virtualenvs'] = get_custom_venv_choices()
+
+        return Response(data)
+
+    def post(self, request):
+        if not isinstance(request.data, dict):
+            return Response({"error": _("Invalid license data")}, status=status.HTTP_400_BAD_REQUEST)
+        if "eula_accepted" not in request.data:
+            return Response({"error": _("Missing 'eula_accepted' property")}, status=status.HTTP_400_BAD_REQUEST)
+        try:
+            eula_accepted = to_python_boolean(request.data["eula_accepted"])
+        except ValueError:
+            return Response({"error": _("'eula_accepted' value is invalid")}, status=status.HTTP_400_BAD_REQUEST)
+
+        if not eula_accepted:
+            return Response({"error": _("'eula_accepted' must be True")}, status=status.HTTP_400_BAD_REQUEST)
+        request.data.pop("eula_accepted")
+        try:
+            data_actual = json.dumps(request.data)
+        except Exception:
+            logger.info(smart_text(u"Invalid JSON submitted for license."),
+                        extra=dict(actor=request.user.username))
+            return Response({"error": _("Invalid JSON")}, status=status.HTTP_400_BAD_REQUEST)
+        try:
+            from awx.main.utils.common import get_licenser
+            license_data = json.loads(data_actual)
+            license_data_validated = get_licenser(**license_data).validate()
+        except Exception:
+            logger.warning(smart_text(u"Invalid license submitted."),
+                           extra=dict(actor=request.user.username))
+            return Response({"error": _("Invalid License")}, status=status.HTTP_400_BAD_REQUEST)
+
+        # If the license is valid, write it to the database.
+        if license_data_validated['valid_key']:
+            settings.LICENSE = license_data
+            settings.TOWER_URL_BASE = "{}://{}".format(request.scheme, request.get_host())
+            return Response(license_data_validated)
+
+        logger.warning(smart_text(u"Invalid license submitted."),
+                       extra=dict(actor=request.user.username))
+        return Response({"error": _("Invalid license")}, status=status.HTTP_400_BAD_REQUEST)
+
+    def delete(self, request):
+        try:
+            settings.LICENSE = {}
+            return Response(status=status.HTTP_204_NO_CONTENT)
+        except Exception:
+            # FIX: Log
+            return Response({"error": _("Failed to remove license.")}, status=status.HTTP_400_BAD_REQUEST)
+
+
+

awx/celery.py

@@ -1,25 +0,0 @@
-
-# Copyright (c) 2017 Ansible, Inc.
-# All Rights Reserved.
-
-from __future__ import absolute_import, unicode_literals
-
-import os
-from celery import Celery
-from django.conf import settings # noqa
-
-
-try:
-    import awx.devonly # noqa
-    MODE = 'development'
-except ImportError: # pragma: no cover
-    MODE = 'production'
-
-os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'awx.settings.%s' % MODE)
-
-app = Celery('awx')
-app.config_from_object('django.conf:settings')
-app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
-
-if __name__ == '__main__':
-    app.start()

awx/conf/fields.py

@@ -1,6 +1,6 @@
 # Python
 import logging
-import urlparse
+import urllib.parse as urlparse
 from collections import OrderedDict
 
 # Django
@@ -71,7 +71,7 @@ class StringListBooleanField(ListField):
                 return False
             elif value in NullBooleanField.NULL_VALUES:
                 return None
-            elif isinstance(value, basestring):
+            elif isinstance(value, str):
                 return self.child.to_representation(value)
         except TypeError:
             pass
@@ -88,7 +88,7 @@ class StringListBooleanField(ListField):
                 return False
             elif data in NullBooleanField.NULL_VALUES:
                 return None
-            elif isinstance(data, basestring):
+            elif isinstance(data, str):
                 return self.child.run_validation(data)
         except TypeError:
             pass

awx/conf/management/commands/migrate_to_database_settings.py

@@ -460,10 +460,10 @@ class Command(BaseCommand):
                     elif file_to_comment not in to_comment_patterns:
                         to_comment_patterns.append(file_to_comment)
             # Run once in dry-run mode to catch any errors from updating the files.
-            diffs = comment_assignments(to_comment_patterns, to_comment.keys(), dry_run=True, backup_suffix=self.backup_suffix)
+            diffs = comment_assignments(to_comment_patterns, list(to_comment.keys()), dry_run=True, backup_suffix=self.backup_suffix)
             # Then, if really updating, run again.
             if not self.dry_run and not self.no_comment:
-                diffs = comment_assignments(to_comment_patterns, to_comment.keys(), dry_run=False, backup_suffix=self.backup_suffix)
+                diffs = comment_assignments(to_comment_patterns, list(to_comment.keys()), dry_run=False, backup_suffix=self.backup_suffix)
                 if license_file_to_comment:
                     diffs.extend(self._comment_license_file(dry_run=False))
                 if local_settings_file_to_comment:

awx/conf/migrations/0006_v331_ldap_group_type.py

@@ -0,0 +1,18 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+
+# AWX
+from awx.conf.migrations._ldap_group_type import fill_ldap_group_type_params
+
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('conf', '0005_v330_rename_two_session_settings'),
+    ]
+
+    operations = [
+        migrations.RunPython(fill_ldap_group_type_params),
+    ]

awx/conf/migrations/_ldap_group_type.py

@@ -0,0 +1,30 @@
+
+import inspect
+
+from django.conf import settings
+from django.utils.timezone import now
+
+
+def fill_ldap_group_type_params(apps, schema_editor):
+    group_type = settings.AUTH_LDAP_GROUP_TYPE
+    Setting = apps.get_model('conf', 'Setting')
+
+    group_type_params = {'name_attr': 'cn', 'member_attr': 'member'}
+    qs = Setting.objects.filter(key='AUTH_LDAP_GROUP_TYPE_PARAMS')
+    entry = None
+    if qs.exists():
+        entry = qs[0]
+        group_type_params = entry.value
+    else:
+        entry = Setting(key='AUTH_LDAP_GROUP_TYPE_PARAMS',
+                        value=group_type_params,
+                        created=now(),
+                        modified=now())
+
+    init_attrs = set(inspect.getargspec(group_type.__init__).args[1:])
+    for k in group_type_params.keys():
+        if k not in init_attrs:
+            del group_type_params[k]
+
+    entry.value = group_type_params
+    entry.save()

awx/conf/models.py

@@ -33,7 +33,7 @@ class Setting(CreatedModifiedModel):
         on_delete=models.CASCADE,
     ))
 
-    def __unicode__(self):
+    def __str__(self):
         try:
             json_value = json.dumps(self.value)
         except ValueError:

awx/conf/settings.py

@@ -2,20 +2,21 @@
 from collections import namedtuple
 import contextlib
 import logging
+import re
 import sys
 import threading
 import time
-import StringIO
 import traceback
-
-import six
+import urllib.parse
+from io import StringIO
 
 # Django
 from django.conf import LazySettings
 from django.conf import settings, UserSettingsHolder
 from django.core.cache import cache as django_cache
 from django.core.exceptions import ImproperlyConfigured
-from django.db import ProgrammingError, OperationalError, transaction, connection
+from django.db import transaction, connection
+from django.db.utils import Error as DBError
 from django.utils.functional import cached_property
 
 # Django REST Framework
@@ -60,6 +61,15 @@ SETTING_CACHE_DEFAULTS = True
 __all__ = ['SettingsWrapper', 'get_settings_to_cache', 'SETTING_CACHE_NOTSET']
 
 
+def normalize_broker_url(value):
+    parts = value.rsplit('@', 1)
+    match = re.search('(amqp://[^:]+:)(.*)', parts[0])
+    if match:
+        prefix, password = match.group(1), match.group(2)
+        parts[0] = prefix + urllib.parse.quote(password)
+    return '@'.join(parts)
+
+
 @contextlib.contextmanager
 def _ctit_db_wrapper(trans_safe=False):
     '''
@@ -79,21 +89,21 @@ def _ctit_db_wrapper(trans_safe=False):
                     logger.debug('Obtaining database settings in spite of broken transaction.')
                     transaction.set_rollback(False)
         yield
-    except (ProgrammingError, OperationalError):
+    except DBError:
         if 'migrate' in sys.argv and get_tower_migration_version() < '310':
             logger.info('Using default settings until version 3.1 migration.')
         else:
             # We want the _full_ traceback with the context
             # First we get the current call stack, which constitutes the "top",
             # it has the context up to the point where the context manager is used
-            top_stack = StringIO.StringIO()
+            top_stack = StringIO()
             traceback.print_stack(file=top_stack)
             top_lines = top_stack.getvalue().strip('\n').split('\n')
             top_stack.close()
             # Get "bottom" stack from the local error that happened
             # inside of the "with" block this wraps
             exc_type, exc_value, exc_traceback = sys.exc_info()
-            bottom_stack = StringIO.StringIO()
+            bottom_stack = StringIO()
             traceback.print_tb(exc_traceback, file=bottom_stack)
             bottom_lines = bottom_stack.getvalue().strip('\n').split('\n')
             # Glue together top and bottom where overlap is found
@@ -115,7 +125,8 @@ def _ctit_db_wrapper(trans_safe=False):
             bottom_stack.close()
             # Log the combined stack
             if trans_safe:
-                logger.warning('Database settings are not available, using defaults, error:\n{}'.format(tb_string))
+                if 'check_migrations' not in sys.argv:
+                    logger.warning('Database settings are not available, using defaults, error:\n{}'.format(tb_string))
             else:
                 logger.error('Error modifying something related to database settings.\n{}'.format(tb_string))
     finally:
@@ -156,15 +167,6 @@ class EncryptedCacheProxy(object):
     def get(self, key, **kwargs):
         value = self.cache.get(key, **kwargs)
         value = self._handle_encryption(self.decrypter, key, value)
-
-        # python-memcached auto-encodes unicode on cache set in python2
-        # https://github.com/linsomniac/python-memcached/issues/79
-        # https://github.com/linsomniac/python-memcached/blob/288c159720eebcdf667727a859ef341f1e908308/memcache.py#L961
-        if six.PY2 and isinstance(value, six.binary_type):
-            try:
-                six.text_type(value)
-            except UnicodeDecodeError:
-                value = value.decode('utf-8')
         logger.debug('cache get(%r, %r) -> %r', key, empty, filter_sensitive(self.registry, key, value))
         return value
 
@@ -444,7 +446,16 @@ class SettingsWrapper(UserSettingsHolder):
                 value = self._get_local(name)
         if value is not empty:
             return value
-        return self._get_default(name)
+        value = self._get_default(name)
+        # sometimes users specify RabbitMQ passwords that contain
+        # unescaped : and @ characters that confused urlparse, e.g.,
+        # amqp://guest:a@ns:ibl3#@localhost:5672//
+        #
+        # detect these scenarios, and automatically escape the user's
+        # password so it just works
+        if name == 'BROKER_URL':
+            value = normalize_broker_url(value)
+        return value
 
     def _set_local(self, name, value):
         field = self.registry.get_setting_field(name)

awx/conf/signals.py

@@ -9,15 +9,11 @@ from django.core.cache import cache
 from django.dispatch import receiver
 
 # Tower
-import awx.main.signals
 from awx.conf import settings_registry
 from awx.conf.models import Setting
-from awx.conf.serializers import SettingSerializer
 
 logger = logging.getLogger('awx.conf.signals')
 
-awx.main.signals.model_serializer_mapping[Setting] = SettingSerializer
-
 __all__ = []
 
 

awx/conf/tests/functional/test_api.py

@@ -1,5 +1,5 @@
 import pytest
-import mock
+from unittest import mock
 
 from rest_framework import serializers
 

awx/conf/tests/functional/test_reencrypt_migration.py

@@ -1,38 +0,0 @@
-# -*- coding: utf-8 -*-
-
-# Copyright (c) 2017 Ansible, Inc.
-# All Rights Reserved.
-import pytest
-import mock
-
-from django.apps import apps
-from awx.conf.migrations._reencrypt import (
-    replace_aesecb_fernet,
-    encrypt_field,
-    decrypt_field,
-)
-from awx.conf.settings import Setting
-from awx.main.utils import decrypt_field as new_decrypt_field
-
-
-@pytest.mark.django_db
-@pytest.mark.parametrize("old_enc, new_enc, value", [
-    ('$encrypted$UTF8$AES', '$encrypted$UTF8$AESCBC$', u'Iñtërnâtiônàlizætiøn'),
-    ('$encrypted$AES$', '$encrypted$AESCBC$', 'test'),
-])
-def test_settings(old_enc, new_enc, value):
-    with mock.patch('awx.conf.models.encrypt_field', encrypt_field):
-        with mock.patch('awx.conf.settings.decrypt_field', decrypt_field):
-            setting = Setting.objects.create(key='SOCIAL_AUTH_GITHUB_SECRET', value=value)
-    assert setting.value.startswith(old_enc)
-
-    replace_aesecb_fernet(apps, None)
-    setting.refresh_from_db()
-
-    assert setting.value.startswith(new_enc)
-    assert new_decrypt_field(setting, 'value') == value
-
-    # This is here for a side-effect.
-    # Exception if the encryption type of AESCBC is not properly skipped, ensures
-    # our `startswith` calls don't have typos
-    replace_aesecb_fernet(apps, None)

awx/conf/tests/unit/test_settings.py

@@ -4,6 +4,7 @@
 # All Rights Reserved.
 
 from contextlib import contextmanager
+import codecs
 from uuid import uuid4
 import time
 
@@ -67,7 +68,7 @@ def test_cached_settings_unicode_is_auto_decoded(settings):
     # https://github.com/linsomniac/python-memcached/issues/79
     # https://github.com/linsomniac/python-memcached/blob/288c159720eebcdf667727a859ef341f1e908308/memcache.py#L961
 
-    value = six.u('Iñtërnâtiônàlizætiøn').encode('utf-8')  # this simulates what python-memcached does on cache.set()
+    value = 'Iñtërnâtiônàlizætiøn'  # this simulates what python-memcached does on cache.set()
     settings.cache.set('DEBUG', value)
     assert settings.cache.get('DEBUG') == six.u('Iñtërnâtiônàlizætiøn')
 
@@ -262,7 +263,7 @@ def test_setting_from_db_with_unicode(settings, mocker, encrypted):
         encrypted=encrypted
     )
     # this simulates a bug in python-memcached; see https://github.com/linsomniac/python-memcached/issues/79
-    value = six.u('Iñtërnâtiônàlizætiøn').encode('utf-8')
+    value = 'Iñtërnâtiônàlizætiøn'
 
     setting_from_db = mocker.Mock(id=1, key='AWX_SOME_SETTING', value=value)
     mocks = mocker.Mock(**{
@@ -272,8 +273,8 @@ def test_setting_from_db_with_unicode(settings, mocker, encrypted):
         }),
     })
     with mocker.patch('awx.conf.models.Setting.objects.filter', return_value=mocks):
-        assert settings.AWX_SOME_SETTING == six.u('Iñtërnâtiônàlizætiøn')
-        assert settings.cache.get('AWX_SOME_SETTING') == six.u('Iñtërnâtiônàlizætiøn')
+        assert settings.AWX_SOME_SETTING == 'Iñtërnâtiônàlizætiøn'
+        assert settings.cache.get('AWX_SOME_SETTING') == 'Iñtërnâtiônàlizætiøn'
 
 
 @pytest.mark.defined_in_file(AWX_SOME_SETTING='DEFAULT')
@@ -434,7 +435,7 @@ def test_sensitive_cache_data_is_encrypted(settings, mocker):
 
     def rot13(obj, attribute):
         assert obj.pk == 123
-        return getattr(obj, attribute).encode('rot13')
+        return codecs.encode(getattr(obj, attribute), 'rot_13')
 
     native_cache = LocMemCache(str(uuid4()), {})
     cache = EncryptedCacheProxy(
@@ -471,7 +472,7 @@ def test_readonly_sensitive_cache_data_is_encrypted(settings):
 
     def rot13(obj, attribute):
         assert obj.pk is None
-        return getattr(obj, attribute).encode('rot13')
+        return codecs.encode(getattr(obj, attribute), 'rot_13')
 
     native_cache = LocMemCache(str(uuid4()), {})
     cache = EncryptedCacheProxy(

awx/conf/utils.py

@@ -102,7 +102,7 @@ def comment_assignments_in_file(filename, assignment_names, dry_run=True, backup
         if not dry_run:
             if backup_filename:
                 shutil.copy2(filename, backup_filename)
-            with open(filename, 'wb') as fileobj:
+            with open(filename, 'w') as fileobj:
                 fileobj.write(new_file_data)
     return '\n'.join(diff_lines)
 

awx/conf/views.py

@@ -72,7 +72,7 @@ class SettingSingletonDetail(RetrieveUpdateDestroyAPIView):
 
     def get_queryset(self):
         self.category_slug = self.kwargs.get('category_slug', 'all')
-        all_category_slugs = settings_registry.get_registered_categories(features_enabled=get_licensed_features()).keys()
+        all_category_slugs = list(settings_registry.get_registered_categories(features_enabled=get_licensed_features()).keys())
         for slug_to_delete in VERSION_SPECIFIC_CATEGORIES_TO_EXCLUDE[get_request_version(self.request)]:
             all_category_slugs.remove(slug_to_delete)
         if self.request.user.is_superuser or getattr(self.request.user, 'is_system_auditor', False):
@@ -123,7 +123,7 @@ class SettingSingletonDetail(RetrieveUpdateDestroyAPIView):
             if key == 'LICENSE' or settings_registry.is_setting_read_only(key):
                 continue
             if settings_registry.is_setting_encrypted(key) and \
-                    isinstance(value, basestring) and \
+                    isinstance(value, str) and \
                     value.startswith('$encrypted$'):
                 continue
             setattr(serializer.instance, key, value)
@@ -210,7 +210,7 @@ class SettingLoggingTest(GenericAPIView):
 # in URL patterns and reverse URL lookups, converting CamelCase names to
 # lowercase_with_underscore (e.g. MyView.as_view() becomes my_view).
 this_module = sys.modules[__name__]
-for attr, value in locals().items():
+for attr, value in list(locals().items()):
     if isinstance(value, type) and issubclass(value, APIView):
         name = camelcase_to_underscore(attr)
         view = value.as_view()

awx/lib/awx_display_callback/cleanup.py

@@ -24,7 +24,12 @@ import os
 import pwd
 
 # PSUtil
-import psutil
+try:
+    import psutil
+except ImportError:
+    raise ImportError('psutil is missing; {}bin/pip install psutil'.format(
+        os.environ['VIRTUAL_ENV']
+    ))
 
 __all__ = []
 

awx/lib/awx_display_callback/events.py

@@ -27,9 +27,13 @@ import os
 import stat
 import threading
 import uuid
-import memcache
 
-from six.moves import xrange
+try:
+    import memcache
+except ImportError:
+    raise ImportError('python-memcached is missing; {}bin/pip install python-memcached'.format(
+        os.environ['VIRTUAL_ENV']
+    ))
 
 __all__ = ['event_context']
 
@@ -50,9 +54,8 @@ class IsolatedFileWrite:
         filename = '{}-partial.json'.format(event_uuid)
         dropoff_location = os.path.join(self.private_data_dir, 'artifacts', 'job_events', filename)
         write_location = '.'.join([dropoff_location, 'tmp'])
-        partial_data = json.dumps(value)
         with os.fdopen(os.open(write_location, os.O_WRONLY | os.O_CREAT, stat.S_IRUSR | stat.S_IWUSR), 'w') as f:
-            f.write(partial_data)
+            f.write(value)
         os.rename(write_location, dropoff_location)
 
 
@@ -148,7 +151,7 @@ class EventContext(object):
         if event not in ('playbook_on_stats',) and "res" in event_data and len(str(event_data['res'])) > max_res:
             event_data['res'] = {}
         event_dict = dict(event=event, event_data=event_data)
-        for key in event_data.keys():
+        for key in list(event_data.keys()):
             if key in ('job_id', 'ad_hoc_command_id', 'project_update_id', 'uuid', 'parent_uuid', 'created',):
                 event_dict[key] = event_data.pop(key)
             elif key in ('verbosity', 'pid'):
@@ -159,11 +162,11 @@ class EventContext(object):
         return {}
 
     def dump(self, fileobj, data, max_width=78, flush=False):
-        b64data = base64.b64encode(json.dumps(data))
+        b64data = base64.b64encode(json.dumps(data).encode('utf-8')).decode()
         with self.display_lock:
             # pattern corresponding to OutputEventFilter expectation
             fileobj.write(u'\x1b[K')
-            for offset in xrange(0, len(b64data), max_width):
+            for offset in range(0, len(b64data), max_width):
                 chunk = b64data[offset:offset + max_width]
                 escaped_chunk = u'{}\x1b[{}D'.format(chunk, len(chunk))
                 fileobj.write(escaped_chunk)
@@ -173,7 +176,7 @@ class EventContext(object):
 
     def dump_begin(self, fileobj):
         begin_dict = self.get_begin_dict()
-        self.cache.set(":1:ev-{}".format(begin_dict['uuid']), begin_dict)
+        self.cache.set(":1:ev-{}".format(begin_dict['uuid']), json.dumps(begin_dict))
         self.dump(fileobj, {'uuid': begin_dict['uuid']})
 
     def dump_end(self, fileobj):

awx/lib/awx_display_callback/module.py

@@ -475,6 +475,15 @@ class BaseCallbackModule(CallbackBase):
         with self.capture_event_data('runner_retry', **event_data):
             super(BaseCallbackModule, self).v2_runner_retry(result)
 
+    def v2_runner_on_start(self, host, task):
+        event_data = dict(
+            host=host.get_name(),
+            task=task
+        )
+        with self.capture_event_data('runner_on_start', **event_data):
+            super(BaseCallbackModule, self).v2_runner_on_start(host, task)
+            
+
 
 class AWXDefaultCallbackModule(BaseCallbackModule, DefaultCallbackModule):
 

awx/lib/tests/test_display_callback.py

@@ -5,11 +5,11 @@ from __future__ import absolute_import
 
 from collections import OrderedDict
 import json
-import mock
 import os
 import shutil
 import sys
 import tempfile
+from unittest import mock
 
 import pytest
 

awx/main/access.py

@@ -524,7 +524,7 @@ class UserAccess(BaseAccess):
         # A user can be changed if they are themselves, or by org admins or
         # superusers.  Change permission implies changing only certain fields
         # that a user should be able to edit for themselves.
-        if not settings.MANAGE_ORGANIZATION_AUTH:
+        if not settings.MANAGE_ORGANIZATION_AUTH and not self.user.is_superuser:
             return False
         return bool(self.user == obj or self.can_admin(obj, data))
 
@@ -577,7 +577,7 @@ class UserAccess(BaseAccess):
         return False
 
     def can_attach(self, obj, sub_obj, relationship, *args, **kwargs):
-        if not settings.MANAGE_ORGANIZATION_AUTH:
+        if not settings.MANAGE_ORGANIZATION_AUTH and not self.user.is_superuser:
             return False
 
         # Reverse obj and sub_obj, defer to RoleAccess if this is a role assignment.
@@ -587,7 +587,7 @@ class UserAccess(BaseAccess):
         return super(UserAccess, self).can_attach(obj, sub_obj, relationship, *args, **kwargs)
 
     def can_unattach(self, obj, sub_obj, relationship, *args, **kwargs):
-        if not settings.MANAGE_ORGANIZATION_AUTH:
+        if not settings.MANAGE_ORGANIZATION_AUTH and not self.user.is_superuser:
             return False
 
         if relationship == 'roles':
@@ -611,7 +611,8 @@ class OAuth2ApplicationAccess(BaseAccess):
     select_related = ('user',)
 
     def filtered_queryset(self):
-        return self.model.objects.filter(organization__in=self.user.organizations)
+        org_access_qs = Organization.accessible_objects(self.user, 'member_role')
+        return self.model.objects.filter(organization__in=org_access_qs)
 
     def can_change(self, obj, data):
         return self.user.is_superuser or self.check_related('organization', Organization, data, obj=obj, 
@@ -1156,13 +1157,10 @@ class TeamAccess(BaseAccess):
     def can_attach(self, obj, sub_obj, relationship, *args, **kwargs):
         """Reverse obj and sub_obj, defer to RoleAccess if this is an assignment
         of a resource role to the team."""
-        if not settings.MANAGE_ORGANIZATION_AUTH:
-            return False
+        # MANAGE_ORGANIZATION_AUTH setting checked in RoleAccess
         if isinstance(sub_obj, Role):
             if sub_obj.content_object is None:
                 raise PermissionDenied(_("The {} role cannot be assigned to a team").format(sub_obj.name))
-            elif isinstance(sub_obj.content_object, User):
-                raise PermissionDenied(_("The admin_role for a User cannot be assigned to a team"))
 
             if isinstance(sub_obj.content_object, ResourceMixin):
                 role_access = RoleAccess(self.user)
@@ -1174,9 +1172,7 @@ class TeamAccess(BaseAccess):
                                                   *args, **kwargs)
 
     def can_unattach(self, obj, sub_obj, relationship, *args, **kwargs):
-        if not settings.MANAGE_ORGANIZATION_AUTH:
-            return False
-
+        # MANAGE_ORGANIZATION_AUTH setting checked in RoleAccess
         if isinstance(sub_obj, Role):
             if isinstance(sub_obj.content_object, ResourceMixin):
                 role_access = RoleAccess(self.user)
@@ -1212,7 +1208,7 @@ class ProjectAccess(BaseAccess):
     @check_superuser
     def can_add(self, data):
         if not data:  # So the browseable API will work
-            return Organization.accessible_objects(self.user, 'admin_role').exists()
+            return Organization.accessible_objects(self.user, 'project_admin_role').exists()
         return (self.check_related('organization', Organization, data, role_field='project_admin_role', mandatory=True) and
                 self.check_related('credential', Credential, data, role_field='use_role'))
 
@@ -1280,6 +1276,7 @@ class JobTemplateAccess(BaseAccess):
         'instance_groups',
         'credentials__credential_type',
         Prefetch('labels', queryset=Label.objects.all().order_by('name')),
+        Prefetch('last_job', queryset=UnifiedJob.objects.non_polymorphic()),
     )
 
     def filtered_queryset(self):
@@ -1393,13 +1390,15 @@ class JobTemplateAccess(BaseAccess):
             'job_tags', 'force_handlers', 'skip_tags', 'ask_variables_on_launch',
             'ask_tags_on_launch', 'ask_job_type_on_launch', 'ask_skip_tags_on_launch',
             'ask_inventory_on_launch', 'ask_credential_on_launch', 'survey_enabled',
-            'custom_virtualenv', 'diff_mode',
+            'custom_virtualenv', 'diff_mode', 'timeout', 'job_slice_count',
 
             # These fields are ignored, but it is convenient for QA to allow clients to post them
             'last_job_run', 'created', 'modified',
         ]
 
         for k, v in data.items():
+            if k not in [x.name for x in obj._meta.concrete_fields]:
+                continue
             if hasattr(obj, k) and getattr(obj, k) != v:
                 if k not in field_whitelist and v != getattr(obj, '%s_id' % k, None) \
                         and not (hasattr(obj, '%s_id' % k) and getattr(obj, '%s_id' % k) is None and v == ''): # Equate '' to None in the case of foreign keys
@@ -1788,7 +1787,7 @@ class WorkflowJobNodeAccess(BaseAccess):
 
     def filtered_queryset(self):
         return self.model.objects.filter(
-            workflow_job__workflow_job_template__in=WorkflowJobTemplate.accessible_objects(
+            workflow_job__unified_job_template__in=UnifiedJobTemplate.accessible_pk_qs(
                 self.user, 'read_role'))
 
     @check_superuser
@@ -1839,8 +1838,10 @@ class WorkflowJobTemplateAccess(BaseAccess):
         if 'survey_enabled' in data and data['survey_enabled']:
             self.check_license(feature='surveys')
 
-        return self.check_related('organization', Organization, data, role_field='workflow_admin_role',
-                                  mandatory=True)
+        return (
+            self.check_related('organization', Organization, data, role_field='workflow_admin_role', mandatory=True) and
+            self.check_related('inventory', Inventory, data, role_field='use_role')
+        )
 
     def can_copy(self, obj):
         if self.save_messages:
@@ -1894,8 +1895,11 @@ class WorkflowJobTemplateAccess(BaseAccess):
         if self.user.is_superuser:
             return True
 
-        return (self.check_related('organization', Organization, data, role_field='workflow_admin_role', obj=obj) and
-                self.user in obj.admin_role)
+        return (
+            self.check_related('organization', Organization, data, role_field='workflow_admin_role', obj=obj) and
+            self.check_related('inventory', Inventory, data, role_field='use_role', obj=obj) and
+            self.user in obj.admin_role
+        )
 
     def can_delete(self, obj):
         return self.user.is_superuser or self.user in obj.admin_role
@@ -1914,7 +1918,7 @@ class WorkflowJobAccess(BaseAccess):
 
     def filtered_queryset(self):
         return WorkflowJob.objects.filter(
-            workflow_job_template__in=WorkflowJobTemplate.accessible_objects(
+            unified_job_template__in=UnifiedJobTemplate.accessible_pk_qs(
                 self.user, 'read_role'))
 
     def can_add(self, data):
@@ -1946,27 +1950,39 @@ class WorkflowJobAccess(BaseAccess):
         if self.user.is_superuser:
             return True
 
-        wfjt = obj.workflow_job_template
+        template = obj.workflow_job_template
+        if not template and obj.job_template_id:
+            template = obj.job_template
         # only superusers can relaunch orphans
-        if not wfjt:
+        if not template:
             return False
 
-        # If job was launched by another user, it could have survey passwords
-        if obj.created_by_id != self.user.pk:
-            # Obtain prompts used to start original job
-            JobLaunchConfig = obj._meta.get_field('launch_config').related_model
-            try:
-                config = JobLaunchConfig.objects.get(job=obj)
-            except JobLaunchConfig.DoesNotExist:
-                config = None
+        # Obtain prompts used to start original job
+        JobLaunchConfig = obj._meta.get_field('launch_config').related_model
+        try:
+            config = JobLaunchConfig.objects.get(job=obj)
+        except JobLaunchConfig.DoesNotExist:
+            if self.save_messages:
+                self.messages['detail'] = _('Workflow Job was launched with unknown prompts.')
+            return False
 
-            if config is None or config.prompts_dict():
+        # Check if access to prompts to prevent relaunch
+        if config.prompts_dict():
+            if obj.created_by_id != self.user.pk:
                 if self.save_messages:
                     self.messages['detail'] = _('Job was launched with prompts provided by another user.')
                 return False
+            if not JobLaunchConfigAccess(self.user).can_add({'reference_obj': config}):
+                if self.save_messages:
+                    self.messages['detail'] = _('Job was launched with prompts you lack access to.')
+                return False
+            if config.has_unprompted(template):
+                if self.save_messages:
+                    self.messages['detail'] = _('Job was launched with prompts no longer accepted.')
+                return False
 
         # execute permission to WFJT is mandatory for any relaunch
-        return (self.user in wfjt.execute_role)
+        return (self.user in template.execute_role)
 
     def can_recreate(self, obj):
         node_qs = obj.workflow_job_nodes.all().prefetch_related('inventory', 'credentials', 'unified_job_template')
@@ -2549,14 +2565,13 @@ class RoleAccess(BaseAccess):
         # Unsupported for now
         return False
 
-    def can_attach(self, obj, sub_obj, relationship, data,
-                   skip_sub_obj_read_check=False):
-        return self.can_unattach(obj, sub_obj, relationship, data, skip_sub_obj_read_check)
+    def can_attach(self, obj, sub_obj, relationship, *args, **kwargs):
+        return self.can_unattach(obj, sub_obj, relationship, *args, **kwargs)
 
     @check_superuser
     def can_unattach(self, obj, sub_obj, relationship, data=None, skip_sub_obj_read_check=False):
         if isinstance(obj.content_object, Team):
-            if not settings.MANAGE_ORGANIZATION_AUTH:
+            if not settings.MANAGE_ORGANIZATION_AUTH and not self.user.is_superuser:
                 return False
 
         if not skip_sub_obj_read_check and relationship in ['members', 'member_role.parents', 'parents']:

awx/main/conf.py

@@ -197,6 +197,18 @@ register(
     category_slug='jobs',
 )
 
+register(
+    'AWX_ISOLATED_VERBOSITY',
+    field_class=fields.IntegerField,
+    min_value=0,
+    max_value=5,
+    label=_('Verbosity level for isolated node management tasks'),
+    help_text=_('This can be raised to aid in debugging connection issues for isolated task execution'),
+    category=_('Jobs'),
+    category_slug='jobs',
+    default=0
+)
+
 register(
     'AWX_ISOLATED_CHECK_INTERVAL',
     field_class=fields.IntegerField,

awx/main/constants.py

@@ -29,3 +29,11 @@ STANDARD_INVENTORY_UPDATE_ENV = {
 CAN_CANCEL = ('new', 'pending', 'waiting', 'running')
 ACTIVE_STATES = CAN_CANCEL
 CENSOR_VALUE = '************'
+ENV_BLACKLIST = frozenset((
+    'VIRTUAL_ENV', 'PATH', 'PYTHONPATH', 'PROOT_TMP_DIR', 'JOB_ID',
+    'INVENTORY_ID', 'INVENTORY_SOURCE_ID', 'INVENTORY_UPDATE_ID',
+    'AD_HOC_COMMAND_ID', 'REST_API_URL', 'REST_API_TOKEN', 'MAX_EVENT_RES',
+    'CALLBACK_QUEUE', 'CALLBACK_CONNECTION', 'CACHE',
+    'JOB_CALLBACK_DEBUG', 'INVENTORY_HOSTVARS', 'FACT_QUEUE',
+    'AWX_HOST', 'PROJECT_REVISION'
+))

awx/main/consumers.py

@@ -4,6 +4,7 @@ import logging
 from channels import Group
 from channels.auth import channel_session_user_from_http, channel_session_user
 
+from django.utils.encoding import smart_str
 from django.http.cookie import parse_cookie
 from django.core.serializers.json import DjangoJSONEncoder
 
@@ -30,7 +31,7 @@ def ws_connect(message):
         # store the valid CSRF token from the cookie so we can compare it later
         # on ws_receive
         cookie_token = parse_cookie(
-            headers.get('cookie')
+            smart_str(headers.get(b'cookie'))
         ).get('csrftoken')
         if cookie_token:
             message.channel_session[XRF_KEY] = cookie_token

awx/main/dispatch/__init__.py

@@ -0,0 +1,5 @@
+from django.conf import settings
+
+
+def get_local_queuename():
+    return settings.CLUSTER_HOST_ID

awx/main/dispatch/control.py

@@ -0,0 +1,58 @@
+import logging
+import socket
+
+from django.conf import settings
+
+from awx.main.dispatch import get_local_queuename
+from kombu import Connection, Queue, Exchange, Producer, Consumer
+
+logger = logging.getLogger('awx.main.dispatch')
+
+
+class Control(object):
+
+    services = ('dispatcher', 'callback_receiver')
+    result = None
+
+    def __init__(self, service, host=None):
+        if service not in self.services:
+            raise RuntimeError('{} must be in {}'.format(service, self.services))
+        self.service = service
+        self.queuename = host or get_local_queuename()
+        self.queue = Queue(self.queuename, Exchange(self.queuename), routing_key=self.queuename)
+
+    def publish(self, msg, conn, **kwargs):
+        producer = Producer(
+            exchange=self.queue.exchange,
+            channel=conn,
+            routing_key=self.queuename
+        )
+        producer.publish(msg, expiration=5, **kwargs)
+
+    def status(self, *args, **kwargs):
+        return self.control_with_reply('status', *args, **kwargs)
+
+    def running(self, *args, **kwargs):
+        return self.control_with_reply('running', *args, **kwargs)
+
+    def control_with_reply(self, command, timeout=5):
+        logger.warn('checking {} {} for {}'.format(self.service, command, self.queuename))
+        reply_queue = Queue(name="amq.rabbitmq.reply-to")
+        self.result = None
+        with Connection(settings.BROKER_URL) as conn:
+            with Consumer(conn, reply_queue, callbacks=[self.process_message], no_ack=True):
+                self.publish({'control': command}, conn, reply_to='amq.rabbitmq.reply-to')
+                try:
+                    conn.drain_events(timeout=timeout)
+                except socket.timeout:
+                    logger.error('{} did not reply within {}s'.format(self.service, timeout))
+                    raise
+        return self.result
+
+    def control(self, msg, **kwargs):
+        with Connection(settings.BROKER_URL) as conn:
+            self.publish(msg, conn)
+
+    def process_message(self, body, message):
+        self.result = body
+        message.ack()

awx/main/dispatch/pool.py

@@ -0,0 +1,412 @@
+import logging
+import os
+import sys
+import random
+import traceback
+from uuid import uuid4
+
+import collections
+from multiprocessing import Process
+from multiprocessing import Queue as MPQueue
+from queue import Full as QueueFull, Empty as QueueEmpty
+
+from django.conf import settings
+from django.db import connection as django_connection, connections
+from django.core.cache import cache as django_cache
+from jinja2 import Template
+import psutil
+
+from awx.main.models import UnifiedJob
+from awx.main.dispatch import reaper
+
+logger = logging.getLogger('awx.main.dispatch')
+
+
+class PoolWorker(object):
+    '''
+    Used to track a worker child process and its pending and finished messages.
+
+    This class makes use of two distinct multiprocessing.Queues to track state:
+
+    - self.queue: this is a queue which represents pending messages that should
+                  be handled by this worker process; as new AMQP messages come
+                  in, a pool will put() them into this queue; the child
+                  process that is forked will get() from this queue and handle
+                  received messages in an endless loop
+    - self.finished: this is a queue which the worker process uses to signal
+                     that it has finished processing a message
+
+    When a message is put() onto this worker, it is tracked in
+    self.managed_tasks.
+
+    Periodically, the worker will call .calculate_managed_tasks(), which will
+    cause messages in self.finished to be removed from self.managed_tasks.
+
+    In this way, self.managed_tasks represents a view of the messages assigned
+    to a specific process.  The message at [0] is the least-recently inserted
+    message, and it represents what the worker is running _right now_
+    (self.current_task).
+
+    A worker is "busy" when it has at least one message in self.managed_tasks.
+    It is "idle" when self.managed_tasks is empty.
+    '''
+
+    def __init__(self, queue_size, target, args):
+        self.messages_sent = 0
+        self.messages_finished = 0
+        self.managed_tasks = collections.OrderedDict()
+        self.finished = MPQueue(queue_size)
+        self.queue = MPQueue(queue_size)
+        self.process = Process(target=target, args=(self.queue, self.finished) + args)
+        self.process.daemon = True
+
+    def start(self):
+        self.process.start()
+
+    def put(self, body):
+        uuid = '?'
+        if isinstance(body, dict):
+            if not body.get('uuid'):
+                body['uuid'] = str(uuid4())
+            uuid = body['uuid']
+        logger.debug('delivered {} to worker[{}] qsize {}'.format(
+            uuid, self.pid, self.qsize
+        ))
+        self.managed_tasks[uuid] = body
+        self.queue.put(body, block=True, timeout=5)
+        self.messages_sent += 1
+        self.calculate_managed_tasks()
+
+    def quit(self):
+        '''
+        Send a special control message to the worker that tells it to exit
+        gracefully.
+        '''
+        self.queue.put('QUIT')
+
+    @property
+    def pid(self):
+        return self.process.pid
+
+    @property
+    def qsize(self):
+        return self.queue.qsize()
+
+    @property
+    def alive(self):
+        return self.process.is_alive()
+
+    @property
+    def mb(self):
+        if self.alive:
+            return '{:0.3f}'.format(
+                psutil.Process(self.pid).memory_info().rss / 1024.0 / 1024.0
+            )
+        return '0'
+
+    @property
+    def exitcode(self):
+        return str(self.process.exitcode)
+
+    def calculate_managed_tasks(self):
+        # look to see if any tasks were finished
+        finished = []
+        for _ in range(self.finished.qsize()):
+            try:
+                finished.append(self.finished.get(block=False))
+            except QueueEmpty:
+                break  # qsize is not always _totally_ up to date
+
+        # if any tasks were finished, removed them from the managed tasks for
+        # this worker
+        for uuid in finished:
+            self.messages_finished += 1
+            del self.managed_tasks[uuid]
+
+    @property
+    def current_task(self):
+        self.calculate_managed_tasks()
+        # the task at [0] is the one that's running right now (or is about to
+        # be running)
+        if len(self.managed_tasks):
+            return self.managed_tasks[list(self.managed_tasks.keys())[0]]
+
+        return None
+
+    @property
+    def orphaned_tasks(self):
+        orphaned = []
+        if not self.alive:
+            # if this process had a running task that never finished,
+            # requeue its error callbacks
+            current_task = self.current_task
+            if isinstance(current_task, dict):
+                orphaned.extend(current_task.get('errbacks', []))
+
+            # if this process has any pending messages requeue them
+            for _ in range(self.qsize):
+                try:
+                    message = self.queue.get(block=False)
+                    if message != 'QUIT':
+                        orphaned.append(message)
+                except QueueEmpty:
+                    break  # qsize is not always _totally_ up to date
+            if len(orphaned):
+                logger.error(
+                    'requeuing {} messages from gone worker pid:{}'.format(
+                        len(orphaned), self.pid
+                    )
+                )
+        return orphaned
+
+    @property
+    def busy(self):
+        self.calculate_managed_tasks()
+        return len(self.managed_tasks) > 0
+
+    @property
+    def idle(self):
+        return not self.busy
+
+
+class WorkerPool(object):
+    '''
+    Creates a pool of forked PoolWorkers.
+
+    As WorkerPool.write(...) is called (generally, by a kombu consumer
+    implementation when it receives an AMQP message), messages are passed to
+    one of the multiprocessing Queues where some work can be done on them.
+
+    class MessagePrinter(awx.main.dispatch.worker.BaseWorker):
+
+        def perform_work(self, body):
+            print(body)
+
+    pool = WorkerPool(min_workers=4)  # spawn four worker processes
+    pool.init_workers(MessagePrint().work_loop)
+    pool.write(
+        0,  # preferred worker 0
+        'Hello, World!'
+    )
+    '''
+
+    debug_meta = ''
+
+    def __init__(self, min_workers=None, queue_size=None):
+        self.name = settings.CLUSTER_HOST_ID
+        self.pid = os.getpid()
+        self.min_workers = min_workers or settings.JOB_EVENT_WORKERS
+        self.queue_size = queue_size or settings.JOB_EVENT_MAX_QUEUE_SIZE
+        self.workers = []
+
+    def __len__(self):
+        return len(self.workers)
+
+    def init_workers(self, target, *target_args):
+        self.target = target
+        self.target_args = target_args
+        for idx in range(self.min_workers):
+            self.up()
+
+    def up(self):
+        idx = len(self.workers)
+        # It's important to close these because we're _about_ to fork, and we
+        # don't want the forked processes to inherit the open sockets
+        # for the DB and memcached connections (that way lies race conditions)
+        django_connection.close()
+        django_cache.close()
+        worker = PoolWorker(self.queue_size, self.target, (idx,) + self.target_args)
+        self.workers.append(worker)
+        try:
+            worker.start()
+        except Exception:
+            logger.exception('could not fork')
+        else:
+            logger.warn('scaling up worker pid:{}'.format(worker.pid))
+        return idx, worker
+
+    def debug(self, *args, **kwargs):
+        self.cleanup()
+        tmpl = Template(
+            '{{ pool.name }}[pid:{{ pool.pid }}] workers total={{ workers|length }} {{ meta }} \n'
+            '{% for w in workers %}'
+            '.  worker[pid:{{ w.pid }}]{% if not w.alive %} GONE exit={{ w.exitcode }}{% endif %}'
+            ' sent={{ w.messages_sent }}'
+            ' finished={{ w.messages_finished }}'
+            ' qsize={{ w.managed_tasks|length }}'
+            ' rss={{ w.mb }}MB'
+            '{% for task in w.managed_tasks.values() %}'
+            '\n     - {% if loop.index0 == 0 %}running {% else %}queued {% endif %}'
+            '{{ task["uuid"] }} '
+            '{% if "task" in task %}'
+            '{{ task["task"].rsplit(".", 1)[-1] }}'
+            # don't print kwargs, they often contain launch-time secrets
+            '(*{{ task.get("args", []) }})'
+            '{% endif %}'
+            '{% endfor %}'
+            '{% if not w.managed_tasks|length %}'
+            ' [IDLE]'
+            '{% endif %}'
+            '\n'
+            '{% endfor %}'
+        )
+        return tmpl.render(pool=self, workers=self.workers, meta=self.debug_meta)
+
+    def write(self, preferred_queue, body):
+        queue_order = sorted(range(len(self.workers)), key=lambda x: -1 if x==preferred_queue else x)
+        write_attempt_order = []
+        for queue_actual in queue_order:
+            try:
+                self.workers[queue_actual].put(body)
+                return queue_actual
+            except QueueFull:
+                pass
+            except Exception:
+                tb = traceback.format_exc()
+                logger.warn("could not write to queue %s" % preferred_queue)
+                logger.warn("detail: {}".format(tb))
+            write_attempt_order.append(preferred_queue)
+        logger.warn("could not write payload to any queue, attempted order: {}".format(write_attempt_order))
+        return None
+
+    def stop(self, signum):
+        try:
+            for worker in self.workers:
+                os.kill(worker.pid, signum)
+        except Exception:
+            logger.exception('could not kill {}'.format(worker.pid))
+
+
+class AutoscalePool(WorkerPool):
+    '''
+    An extended pool implementation that automatically scales workers up and
+    down based on demand
+    '''
+
+    def __init__(self, *args, **kwargs):
+        self.max_workers = kwargs.pop('max_workers', None)
+        super(AutoscalePool, self).__init__(*args, **kwargs)
+
+        if self.max_workers is None:
+            settings_absmem = getattr(settings, 'SYSTEM_TASK_ABS_MEM', None)
+            if settings_absmem is not None:
+                total_memory_gb = int(settings_absmem)
+            else:
+                total_memory_gb = (psutil.virtual_memory().total >> 30) + 1  # noqa: round up
+            # 5 workers per GB of total memory
+            self.max_workers = (total_memory_gb * 5)
+
+        # max workers can't be less than min_workers
+        self.max_workers = max(self.min_workers, self.max_workers)
+
+    @property
+    def should_grow(self):
+        if len(self.workers) < self.min_workers:
+            # If we don't have at least min_workers, add more
+            return True
+        # If every worker is busy doing something, add more
+        return all([w.busy for w in self.workers])
+
+    @property
+    def full(self):
+        return len(self.workers) == self.max_workers
+
+    @property
+    def debug_meta(self):
+        return 'min={} max={}'.format(self.min_workers, self.max_workers)
+
+    def cleanup(self):
+        """
+        Perform some internal account and cleanup.  This is run on
+        every cluster node heartbeat:
+
+        1.  Discover worker processes that exited, and recover messages they
+            were handling.
+        2.  Clean up unnecessary, idle workers.
+        3.  Check to see if the database says this node is running any tasks
+            that aren't actually running.  If so, reap them.
+        """
+        orphaned = []
+        for w in self.workers[::]:
+            if not w.alive:
+                # the worker process has exited
+                # 1. take the task it was running and enqueue the error
+                #    callbacks
+                # 2. take any pending tasks delivered to its queue and
+                #    send them to another worker
+                logger.error('worker pid:{} is gone (exit={})'.format(w.pid, w.exitcode))
+                if w.current_task:
+                    if w.current_task != 'QUIT':
+                        try:
+                            for j in UnifiedJob.objects.filter(celery_task_id=w.current_task['uuid']):
+                                reaper.reap_job(j, 'failed')
+                        except Exception:
+                            logger.exception('failed to reap job UUID {}'.format(w.current_task['uuid']))
+                orphaned.extend(w.orphaned_tasks)
+                self.workers.remove(w)
+            elif w.idle and len(self.workers) > self.min_workers:
+                # the process has an empty queue (it's idle) and we have
+                # more processes in the pool than we need (> min)
+                # send this process a message so it will exit gracefully
+                # at the next opportunity
+                logger.warn('scaling down worker pid:{}'.format(w.pid))
+                w.quit()
+                self.workers.remove(w)
+
+        for m in orphaned:
+            # if all the workers are dead, spawn at least one
+            if not len(self.workers):
+                self.up()
+            idx = random.choice(range(len(self.workers)))
+            self.write(idx, m)
+
+        # if the database says a job is running on this node, but it's *not*,
+        # then reap it
+        running_uuids = []
+        for worker in self.workers:
+            worker.calculate_managed_tasks()
+            running_uuids.extend(list(worker.managed_tasks.keys()))
+        try:
+            reaper.reap(excluded_uuids=running_uuids)
+        except Exception:
+            # we _probably_ failed here due to DB connectivity issues, so
+            # don't use our logger (it accesses the database for configuration)
+            _, _, tb = sys.exc_info()
+            traceback.print_tb(tb)
+            for conn in connections.all():
+                # If the database connection has a hiccup, re-establish a new
+                # connection
+                conn.close_if_unusable_or_obsolete()
+
+    def up(self):
+        if self.full:
+            # if we can't spawn more workers, just toss this message into a
+            # random worker's backlog
+            idx = random.choice(range(len(self.workers)))
+            return idx, self.workers[idx]
+        else:
+            return super(AutoscalePool, self).up()
+
+    def write(self, preferred_queue, body):
+        try:
+            # when the cluster heartbeat occurs, clean up internally
+            if isinstance(body, dict) and 'cluster_node_heartbeat' in body['task']:
+                self.cleanup()
+            if self.should_grow:
+                self.up()
+            # we don't care about "preferred queue" round robin distribution, just
+            # find the first non-busy worker and claim it
+            workers = self.workers[:]
+            random.shuffle(workers)
+            for w in workers:
+                if not w.busy:
+                    w.put(body)
+                    break
+            else:
+                return super(AutoscalePool, self).write(preferred_queue, body)
+        except Exception:
+            for conn in connections.all():
+                # If the database connection has a hiccup, re-establish a new
+                # connection
+                conn.close_if_unusable_or_obsolete()
+            logger.exception('failed to write inbound message')

awx/main/dispatch/publish.py

@@ -0,0 +1,128 @@
+import inspect
+import logging
+import sys
+from uuid import uuid4
+
+from django.conf import settings
+from kombu import Connection, Exchange, Producer
+
+logger = logging.getLogger('awx.main.dispatch')
+
+
+def serialize_task(f):
+    return '.'.join([f.__module__, f.__name__])
+
+
+class task:
+    """
+    Used to decorate a function or class so that it can be run asynchronously
+    via the task dispatcher.  Tasks can be simple functions:
+
+    @task()
+    def add(a, b):
+        return a + b
+
+    ...or classes that define a `run` method:
+
+    @task()
+    class Adder:
+        def run(self, a, b):
+            return a + b
+
+    # Tasks can be run synchronously...
+    assert add(1, 1) == 2
+    assert Adder().run(1, 1) == 2
+
+    # ...or published to a queue:
+    add.apply_async([1, 1])
+    Adder.apply_async([1, 1])
+
+    # Tasks can also define a specific target queue or exchange type:
+
+    @task(queue='slow-tasks')
+    def snooze():
+        time.sleep(10)
+
+    @task(queue='tower_broadcast', exchange_type='fanout')
+    def announce():
+        print("Run this everywhere!")
+    """
+
+    def __init__(self, queue=None, exchange_type=None):
+        self.queue = queue
+        self.exchange_type = exchange_type
+
+    def __call__(self, fn=None):
+        queue = self.queue
+        exchange_type = self.exchange_type
+
+        class PublisherMixin(object):
+
+            queue = None
+
+            @classmethod
+            def delay(cls, *args, **kwargs):
+                return cls.apply_async(args, kwargs)
+
+            @classmethod
+            def apply_async(cls, args=None, kwargs=None, queue=None, uuid=None, **kw):
+                task_id = uuid or str(uuid4())
+                args = args or []
+                kwargs = kwargs or {}
+                queue = (
+                    queue or
+                    getattr(cls.queue, 'im_func', cls.queue) or
+                    settings.CELERY_DEFAULT_QUEUE
+                )
+                obj = {
+                    'uuid': task_id,
+                    'args': args,
+                    'kwargs': kwargs,
+                    'task': cls.name
+                }
+                obj.update(**kw)
+                if callable(queue):
+                    queue = queue()
+                if not settings.IS_TESTING(sys.argv):
+                    with Connection(settings.BROKER_URL) as conn:
+                        exchange = Exchange(queue, type=exchange_type or 'direct')
+                        producer = Producer(conn)
+                        logger.debug('publish {}({}, queue={})'.format(
+                            cls.name,
+                            task_id,
+                            queue
+                        ))
+                        producer.publish(obj,
+                                         serializer='json',
+                                         compression='bzip2',
+                                         exchange=exchange,
+                                         declare=[exchange],
+                                         delivery_mode="persistent",
+                                         routing_key=queue)
+                return (obj, queue)
+
+        # If the object we're wrapping *is* a class (e.g., RunJob), return
+        # a *new* class that inherits from the wrapped class *and* BaseTask
+        # In this way, the new class returned by our decorator is the class
+        # being decorated *plus* PublisherMixin so cls.apply_async() and
+        # cls.delay() work
+        bases = []
+        ns = {'name': serialize_task(fn), 'queue': queue}
+        if inspect.isclass(fn):
+            bases = list(fn.__bases__)
+            ns.update(fn.__dict__)
+        cls = type(
+            fn.__name__,
+            tuple(bases + [PublisherMixin]),
+            ns
+        )
+        if inspect.isclass(fn):
+            return cls
+
+        # if the object being decorated is *not* a class (it's a Python
+        # function), make fn.apply_async and fn.delay proxy through to the
+        # PublisherMixin we dynamically created above
+        setattr(fn, 'name', cls.name)
+        setattr(fn, 'apply_async', cls.apply_async)
+        setattr(fn, 'delay', cls.delay)
+        return fn

awx/main/dispatch/reaper.py

@@ -0,0 +1,49 @@
+from datetime import timedelta
+import logging
+
+from django.db.models import Q
+from django.utils.timezone import now as tz_now
+from django.contrib.contenttypes.models import ContentType
+
+from awx.main.models import Instance, UnifiedJob, WorkflowJob
+
+logger = logging.getLogger('awx.main.dispatch')
+
+
+def reap_job(j, status):
+    if UnifiedJob.objects.get(id=j.id).status not in ('running', 'waiting'):
+        # just in case, don't reap jobs that aren't running
+        return
+    j.status = status
+    j.start_args = ''  # blank field to remove encrypted passwords
+    j.job_explanation += ' '.join((
+        'Task was marked as running in Tower but was not present in',
+        'the job queue, so it has been marked as failed.',
+    ))
+    j.save(update_fields=['status', 'start_args', 'job_explanation'])
+    if hasattr(j, 'send_notification_templates'):
+        j.send_notification_templates('failed')
+    j.websocket_emit_status(status)
+    logger.error(
+        '{} is no longer running; reaping'.format(j.log_format)
+    )
+
+
+def reap(instance=None, status='failed', excluded_uuids=[]):
+    '''
+    Reap all jobs in waiting|running for this instance.
+    '''
+    me = instance or Instance.objects.me()
+    now = tz_now()
+    workflow_ctype_id = ContentType.objects.get_for_model(WorkflowJob).id
+    jobs = UnifiedJob.objects.filter(
+        (
+            Q(status='running') |
+            Q(status='waiting', modified__lte=now - timedelta(seconds=60))
+        ) & (
+            Q(execution_node=me.hostname) |
+            Q(controller_node=me.hostname)
+        ) & ~Q(polymorphic_ctype_id=workflow_ctype_id)
+    ).exclude(celery_task_id__in=excluded_uuids)
+    for j in jobs:
+        reap_job(j, status)

awx/main/dispatch/worker/__init__.py

@@ -0,0 +1,3 @@
+from .base import AWXConsumer, BaseWorker  # noqa
+from .callback import CallbackBrokerWorker  # noqa
+from .task import TaskWorker  # noqa

awx/main/dispatch/worker/base.py

@@ -0,0 +1,151 @@
+# Copyright (c) 2018 Ansible by Red Hat
+# All Rights Reserved.
+
+import os
+import logging
+import signal
+from uuid import UUID
+from queue import Empty as QueueEmpty
+
+from django import db
+from kombu import Producer
+from kombu.mixins import ConsumerMixin
+
+from awx.main.dispatch.pool import WorkerPool
+
+logger = logging.getLogger('awx.main.dispatch')
+
+
+def signame(sig):
+    return dict(
+        (k, v) for v, k in signal.__dict__.items()
+        if v.startswith('SIG') and not v.startswith('SIG_')
+    )[sig]
+
+
+class WorkerSignalHandler:
+
+    def __init__(self):
+        self.kill_now = False
+        signal.signal(signal.SIGINT, self.exit_gracefully)
+
+    def exit_gracefully(self, *args, **kwargs):
+        self.kill_now = True
+
+
+class AWXConsumer(ConsumerMixin):
+
+    def __init__(self, name, connection, worker, queues=[], pool=None):
+        self.connection = connection
+        self.total_messages = 0
+        self.queues = queues
+        self.worker = worker
+        self.pool = pool
+        if pool is None:
+            self.pool = WorkerPool()
+        self.pool.init_workers(self.worker.work_loop)
+
+    def get_consumers(self, Consumer, channel):
+        logger.debug(self.listening_on)
+        return [Consumer(queues=self.queues, accept=['json'],
+                         callbacks=[self.process_task])]
+
+    @property
+    def listening_on(self):
+        return 'listening on {}'.format([
+            '{} [{}]'.format(q.name, q.exchange.type) for q in self.queues
+        ])
+
+    def control(self, body, message):
+        logger.warn(body)
+        control = body.get('control')
+        if control in ('status', 'running'):
+            producer = Producer(
+                channel=self.connection,
+                routing_key=message.properties['reply_to']
+            )
+            if control == 'status':
+                msg = '\n'.join([self.listening_on, self.pool.debug()])
+            elif control == 'running':
+                msg = []
+                for worker in self.pool.workers:
+                    worker.calculate_managed_tasks()
+                    msg.extend(worker.managed_tasks.keys())
+            producer.publish(msg)
+        elif control == 'reload':
+            for worker in self.pool.workers:
+                worker.quit()
+        else:
+            logger.error('unrecognized control message: {}'.format(control))
+        message.ack()
+
+    def process_task(self, body, message):
+        if 'control' in body:
+            return self.control(body, message)
+        if len(self.pool):
+            if "uuid" in body and body['uuid']:
+                try:
+                    queue = UUID(body['uuid']).int % len(self.pool)
+                except Exception:
+                    queue = self.total_messages % len(self.pool)
+            else:
+                queue = self.total_messages % len(self.pool)
+        else:
+            queue = 0
+        self.pool.write(queue, body)
+        self.total_messages += 1
+        message.ack()
+
+    def run(self, *args, **kwargs):
+        signal.signal(signal.SIGINT, self.stop)
+        signal.signal(signal.SIGTERM, self.stop)
+        self.worker.on_start()
+        super(AWXConsumer, self).run(*args, **kwargs)
+
+    def stop(self, signum, frame):
+        self.should_stop = True  # this makes the kombu mixin stop consuming
+        logger.debug('received {}, stopping'.format(signame(signum)))
+        self.worker.on_stop()
+        raise SystemExit()
+
+
+class BaseWorker(object):
+
+    def work_loop(self, queue, finished, idx, *args):
+        ppid = os.getppid()
+        signal_handler = WorkerSignalHandler()
+        while not signal_handler.kill_now:
+            # if the parent PID changes, this process has been orphaned
+            # via e.g., segfault or sigkill, we should exit too
+            if os.getppid() != ppid:
+                break
+            try:
+                body = queue.get(block=True, timeout=1)
+                if body == 'QUIT':
+                    break
+            except QueueEmpty:
+                continue
+            except Exception as e:
+                logger.error("Exception on worker {}, restarting: ".format(idx) + str(e))
+                continue
+            try:
+                for conn in db.connections.all():
+                    # If the database connection has a hiccup during the prior message, close it
+                    # so we can establish a new connection
+                    conn.close_if_unusable_or_obsolete()
+                self.perform_work(body, *args)
+            finally:
+                if 'uuid' in body:
+                    uuid = body['uuid']
+                    logger.debug('task {} is finished'.format(uuid))
+                    finished.put(uuid)
+        logger.warn('worker exiting gracefully pid:{}'.format(os.getpid()))
+
+    def perform_work(self, body):
+        raise NotImplementedError()
+
+    def on_start(self):
+        pass
+
+    def on_stop(self):
+        pass

awx/main/dispatch/worker/callback.py

@@ -0,0 +1,127 @@
+import logging
+import time
+import traceback
+
+from django.conf import settings
+from django.db import DatabaseError, OperationalError, connection as django_connection
+from django.db.utils import InterfaceError, InternalError
+
+from awx.main.consumers import emit_channel_notification
+from awx.main.models import (JobEvent, AdHocCommandEvent, ProjectUpdateEvent,
+                             InventoryUpdateEvent, SystemJobEvent, UnifiedJob)
+
+from .base import BaseWorker
+
+logger = logging.getLogger('awx.main.commands.run_callback_receiver')
+
+
+class CallbackBrokerWorker(BaseWorker):
+    '''
+    A worker implementation that deserializes callback event data and persists
+    it into the database.
+
+    The code that *builds* these types of messages is found in the AWX display
+    callback (`awx.lib.awx_display_callback`).
+    '''
+
+    MAX_RETRIES = 2
+
+    def perform_work(self, body):
+        try:
+            event_map = {
+                'job_id': JobEvent,
+                'ad_hoc_command_id': AdHocCommandEvent,
+                'project_update_id': ProjectUpdateEvent,
+                'inventory_update_id': InventoryUpdateEvent,
+                'system_job_id': SystemJobEvent,
+            }
+
+            if not any([key in body for key in event_map]):
+                raise Exception('Payload does not have a job identifier')
+
+            def _save_event_data():
+                for key, cls in event_map.items():
+                    if key in body:
+                        cls.create_from_data(**body)
+
+            job_identifier = 'unknown job'
+            job_key = 'unknown'
+            for key in event_map.keys():
+                if key in body:
+                    job_identifier = body[key]
+                    job_key = key
+                    break
+
+            if settings.DEBUG:
+                from pygments import highlight
+                from pygments.lexers import PythonLexer
+                from pygments.formatters import Terminal256Formatter
+                from pprint import pformat
+                if body.get('event') == 'EOF':
+                    event_thing = 'EOF event'
+                else:
+                    event_thing = 'event {}'.format(body.get('counter', 'unknown'))
+                logger.info('Callback worker received {} for {} {}'.format(
+                    event_thing, job_key[:-len('_id')], job_identifier
+                ))
+                logger.debug('Body: {}'.format(
+                    highlight(pformat(body, width=160), PythonLexer(), Terminal256Formatter(style='friendly'))
+                )[:1024 * 4])
+
+            if body.get('event') == 'EOF':
+                try:
+                    final_counter = body.get('final_counter', 0)
+                    logger.info('Event processing is finished for Job {}, sending notifications'.format(job_identifier))
+                    # EOF events are sent when stdout for the running task is
+                    # closed. don't actually persist them to the database; we
+                    # just use them to report `summary` websocket events as an
+                    # approximation for when a job is "done"
+                    emit_channel_notification(
+                        'jobs-summary',
+                        dict(group_name='jobs', unified_job_id=job_identifier, final_counter=final_counter)
+                    )
+                    # Additionally, when we've processed all events, we should
+                    # have all the data we need to send out success/failure
+                    # notification templates
+                    uj = UnifiedJob.objects.get(pk=job_identifier)
+                    if hasattr(uj, 'send_notification_templates'):
+                        retries = 0
+                        while retries < 5:
+                            if uj.finished:
+                                uj.send_notification_templates('succeeded' if uj.status == 'successful' else 'failed')
+                                break
+                            else:
+                                # wait a few seconds to avoid a race where the
+                                # events are persisted _before_ the UJ.status
+                                # changes from running -> successful
+                                retries += 1
+                                time.sleep(1)
+                                uj = UnifiedJob.objects.get(pk=job_identifier)
+                except Exception:
+                    logger.exception('Worker failed to emit notifications: Job {}'.format(job_identifier))
+                return
+
+            retries = 0
+            while retries <= self.MAX_RETRIES:
+                try:
+                    _save_event_data()
+                    break
+                except (OperationalError, InterfaceError, InternalError):
+                    if retries >= self.MAX_RETRIES:
+                        logger.exception('Worker could not re-establish database connectivity, giving up on event for Job {}'.format(job_identifier))
+                        return
+                    delay = 60 * retries
+                    logger.exception('Database Error Saving Job Event, retry #{i} in {delay} seconds:'.format(
+                        i=retries + 1,
+                        delay=delay
+                    ))
+                    django_connection.close()
+                    time.sleep(delay)
+                    retries += 1
+                except DatabaseError:
+                    logger.exception('Database Error Saving Job Event for Job {}'.format(job_identifier))
+                    break
+        except Exception as exc:
+            tb = traceback.format_exc()
+            logger.error('Callback Task Processor Raised Exception: %r', exc)
+            logger.error('Detail: {}'.format(tb))

awx/main/dispatch/worker/task.py

@@ -0,0 +1,121 @@
+import inspect
+import logging
+import importlib
+import sys
+import traceback
+
+import six
+
+from awx.main.tasks import dispatch_startup, inform_cluster_of_shutdown
+
+from .base import BaseWorker
+
+logger = logging.getLogger('awx.main.dispatch')
+
+
+class TaskWorker(BaseWorker):
+    '''
+    A worker implementation that deserializes task messages and runs native
+    Python code.
+
+    The code that *builds* these types of messages is found in
+    `awx.main.dispatch.publish`.
+    '''
+
+    @classmethod
+    def resolve_callable(cls, task):
+        '''
+        Transform a dotted notation task into an imported, callable function, e.g.,
+
+        awx.main.tasks.delete_inventory
+        awx.main.tasks.RunProjectUpdate
+        '''
+        if not task.startswith('awx.'):
+            raise ValueError('{} is not a valid awx task'.format(task))
+        module, target = task.rsplit('.', 1)
+        module = importlib.import_module(module)
+        _call = None
+        if hasattr(module, target):
+            _call = getattr(module, target, None)
+        if not (
+            hasattr(_call, 'apply_async') and hasattr(_call, 'delay')
+        ):
+            raise ValueError('{} is not decorated with @task()'.format(task))
+
+        return _call
+
+    def run_callable(self, body):
+        '''
+        Given some AMQP message, import the correct Python code and run it.
+        '''
+        task = body['task']
+        uuid = body.get('uuid', '<unknown>')
+        args = body.get('args', [])
+        kwargs = body.get('kwargs', {})
+        _call = TaskWorker.resolve_callable(task)
+        if inspect.isclass(_call):
+            # the callable is a class, e.g., RunJob; instantiate and
+            # return its `run()` method
+            _call = _call().run
+        # don't print kwargs, they often contain launch-time secrets
+        logger.debug('task {} starting {}(*{})'.format(uuid, task, args))
+        return _call(*args, **kwargs)
+
+    def perform_work(self, body):
+        '''
+        Import and run code for a task e.g.,
+
+        body = {
+            'args': [8],
+            'callbacks': [{
+                'args': [],
+                'kwargs': {}
+                'task': u'awx.main.tasks.handle_work_success'
+            }],
+            'errbacks': [{
+                'args': [],
+                'kwargs': {},
+                'task': 'awx.main.tasks.handle_work_error'
+            }],
+            'kwargs': {},
+            'task': u'awx.main.tasks.RunProjectUpdate'
+        }
+        '''
+        result = None
+        try:
+            result = self.run_callable(body)
+        except Exception as exc:
+            result = exc
+
+            try:
+                if getattr(exc, 'is_awx_task_error', False):
+                    # Error caused by user / tracked in job output
+                    logger.warning(six.text_type("{}").format(exc))
+                else:
+                    task = body['task']
+                    args = body.get('args', [])
+                    kwargs = body.get('kwargs', {})
+                    logger.exception('Worker failed to run task {}(*{}, **{}'.format(
+                        task, args, kwargs
+                    ))
+            except Exception:
+                # It's fairly critical that this code _not_ raise exceptions on logging
+                # If you configure external logging in a way that _it_ fails, there's
+                # not a lot we can do here; sys.stderr.write is a final hail mary
+                _, _, tb = sys.exc_info()
+                traceback.print_tb(tb)
+
+            for callback in body.get('errbacks', []) or []:
+                callback['uuid'] = body['uuid']
+                self.perform_work(callback)
+
+        for callback in body.get('callbacks', []) or []:
+            callback['uuid'] = body['uuid']
+            self.perform_work(callback)
+        return result
+
+    def on_start(self):
+        dispatch_startup()
+
+    def on_stop(self):
+        inform_cluster_of_shutdown()

awx/main/exceptions.py

@@ -4,11 +4,6 @@
 import six
 
 
-# Celery does not respect exception type when using a serializer different than pickle;
-# and awx uses the json serializer
-# https://github.com/celery/celery/issues/3586
-
-
 class _AwxTaskError():
     def build_exception(self, task, message=None):
         if message is None:
@@ -36,5 +31,3 @@ class _AwxTaskError():
 
 
 AwxTaskError = _AwxTaskError()
-
-

awx/main/expect/.gitignore

@@ -0,0 +1 @@
+authorized_keys

awx/main/expect/isolated_manager.py

@@ -1,6 +1,5 @@
 import base64
 import codecs
-import StringIO
 import json
 import os
 import shutil
@@ -9,8 +8,10 @@ import tempfile
 import time
 import logging
 from distutils.version import LooseVersion as Version
+from io import StringIO
 
 from django.conf import settings
+from django.utils.encoding import smart_bytes, smart_str
 
 import awx
 from awx.main.expect import run
@@ -38,7 +39,7 @@ class IsolatedManager(object):
         :param stdout_handle:       a file-like object for capturing stdout
         :param ssh_key_path:        a filepath where SSH key data can be read
         :param expect_passwords:    a dict of regular expression password prompts
-                                    to input values, i.e., {r'Password:\s*?$':
+                                    to input values, i.e., {r'Password:*?$':
                                     'some_password'}
         :param cancelled_callback:  a callable - which returns `True` or `False`
                                     - signifying if the job has been prematurely
@@ -101,6 +102,8 @@ class IsolatedManager(object):
         ]
         if extra_vars:
             args.extend(['-e', json.dumps(extra_vars)])
+        if settings.AWX_ISOLATED_VERBOSITY:
+            args.append('-%s' % ('v' * min(5, settings.AWX_ISOLATED_VERBOSITY)))
         return args
 
     @staticmethod
@@ -117,10 +120,10 @@ class IsolatedManager(object):
 
     @classmethod
     def awx_playbook_path(cls):
-        return os.path.join(
+        return os.path.abspath(os.path.join(
             os.path.dirname(awx.__file__),
             'playbooks'
-        )
+        ))
 
     def path_to(self, *args):
         return os.path.join(self.private_data_dir, *args)
@@ -142,7 +145,7 @@ class IsolatedManager(object):
 
         # if an ssh private key fifo exists, read its contents and delete it
         if self.ssh_key_path:
-            buff = StringIO.StringIO()
+            buff = StringIO()
             with open(self.ssh_key_path, 'r') as fifo:
                 for line in fifo:
                     buff.write(line)
@@ -154,7 +157,10 @@ class IsolatedManager(object):
         # into a variable, and will replicate the data into a named pipe on the
         # isolated instance
         secrets_path = os.path.join(self.private_data_dir, 'env')
-        run.open_fifo_write(secrets_path, base64.b64encode(json.dumps(secrets)))
+        run.open_fifo_write(
+            secrets_path,
+            smart_str(base64.b64encode(smart_bytes(json.dumps(secrets))))
+        )
 
         self.build_isolated_job_data()
 
@@ -174,7 +180,7 @@ class IsolatedManager(object):
         args = self._build_args('run_isolated.yml', '%s,' % self.host, extra_vars)
         if self.instance.verbosity:
             args.append('-%s' % ('v' * min(5, self.instance.verbosity)))
-        buff = StringIO.StringIO()
+        buff = StringIO()
         logger.debug('Starting job {} on isolated host with `run_isolated.yml` playbook.'.format(self.instance.id))
         status, rc = IsolatedManager.run_pexpect(
             args, self.awx_playbook_path(), self.management_env, buff,
@@ -244,7 +250,7 @@ class IsolatedManager(object):
         os.makedirs(self.path_to('artifacts', 'job_events'), mode=stat.S_IXUSR + stat.S_IWUSR + stat.S_IRUSR)
 
     def _missing_artifacts(self, path_list, output):
-        missing_artifacts = filter(lambda path: not os.path.exists(path), path_list)
+        missing_artifacts = list(filter(lambda path: not os.path.exists(path), path_list))
         for path in missing_artifacts:
             self.stdout_handle.write('ansible did not exit cleanly, missing `{}`.\n'.format(path))
         if missing_artifacts:
@@ -282,7 +288,7 @@ class IsolatedManager(object):
         status = 'failed'
         output = ''
         rc = None
-        buff = StringIO.StringIO()
+        buff = StringIO()
         last_check = time.time()
         seek = 0
         job_timeout = remaining = self.job_timeout
@@ -303,7 +309,7 @@ class IsolatedManager(object):
                 time.sleep(1)
                 continue
 
-            buff = StringIO.StringIO()
+            buff = StringIO()
             logger.debug('Checking on isolated job {} with `check_isolated.yml`.'.format(self.instance.id))
             status, rc = IsolatedManager.run_pexpect(
                 args, self.awx_playbook_path(), self.management_env, buff,
@@ -318,7 +324,7 @@ class IsolatedManager(object):
 
             path = self.path_to('artifacts', 'stdout')
             if os.path.exists(path):
-                with open(path, 'r') as f:
+                with codecs.open(path, 'r', encoding='utf-8') as f:
                     f.seek(seek)
                     for line in f:
                         self.stdout_handle.write(line)
@@ -340,7 +346,7 @@ class IsolatedManager(object):
         elif status == 'failed':
             # if we were unable to retrieve job reults from the isolated host,
             # print stdout of the `check_isolated.yml` playbook for clues
-            self.stdout_handle.write(output)
+            self.stdout_handle.write(smart_str(output))
 
         return status, rc
 
@@ -355,7 +361,7 @@ class IsolatedManager(object):
         }
         args = self._build_args('clean_isolated.yml', '%s,' % self.host, extra_vars)
         logger.debug('Cleaning up job {} on isolated host with `clean_isolated.yml` playbook.'.format(self.instance.id))
-        buff = StringIO.StringIO()
+        buff = StringIO()
         timeout = max(60, 2 * settings.AWX_ISOLATED_CONNECTION_TIMEOUT)
         status, rc = IsolatedManager.run_pexpect(
             args, self.awx_playbook_path(), self.management_env, buff,
@@ -407,46 +413,52 @@ class IsolatedManager(object):
         args = cls._build_args('heartbeat_isolated.yml', hostname_string)
         args.extend(['--forks', str(len(instance_qs))])
         env = cls._base_management_env()
-        env['ANSIBLE_STDOUT_CALLBACK'] = 'json'
-
-        buff = StringIO.StringIO()
-        timeout = max(60, 2 * settings.AWX_ISOLATED_CONNECTION_TIMEOUT)
-        status, rc = IsolatedManager.run_pexpect(
-            args, cls.awx_playbook_path(), env, buff,
-            idle_timeout=timeout, job_timeout=timeout,
-            pexpect_timeout=5
-        )
-        output = buff.getvalue().encode('utf-8')
-        buff.close()
 
         try:
-            result = json.loads(output)
-            if not isinstance(result, dict):
-                raise TypeError('Expected a dict but received {}.'.format(str(type(result))))
-        except (ValueError, AssertionError, TypeError):
-            logger.exception('Failed to read status from isolated instances, output:\n {}'.format(output))
-            return
+            facts_path = tempfile.mkdtemp()
+            env['ANSIBLE_CACHE_PLUGIN'] = 'jsonfile'
+            env['ANSIBLE_CACHE_PLUGIN_CONNECTION'] = facts_path
 
-        for instance in instance_qs:
-            try:
-                task_result = result['plays'][0]['tasks'][0]['hosts'][instance.hostname]
-            except (KeyError, IndexError):
+            buff = StringIO()
+            timeout = max(60, 2 * settings.AWX_ISOLATED_CONNECTION_TIMEOUT)
+            status, rc = IsolatedManager.run_pexpect(
+                args, cls.awx_playbook_path(), env, buff,
+                idle_timeout=timeout, job_timeout=timeout,
+                pexpect_timeout=5
+            )
+            heartbeat_stdout = buff.getvalue().encode('utf-8')
+            buff.close()
+
+            for instance in instance_qs:
+                output = heartbeat_stdout
                 task_result = {}
-            if 'capacity_cpu' in task_result and 'capacity_mem' in task_result:
-                cls.update_capacity(instance, task_result, awx_application_version)
-                logger.debug('Isolated instance {} successful heartbeat'.format(instance.hostname))
-            elif instance.capacity == 0:
-                logger.debug('Isolated instance {} previously marked as lost, could not re-join.'.format(
-                    instance.hostname))
-            else:
-                logger.warning('Could not update status of isolated instance {}, msg={}'.format(
-                    instance.hostname, task_result.get('msg', 'unknown failure')
-                ))
-                if instance.is_lost(isolated=True):
-                    instance.capacity = 0
-                    instance.save(update_fields=['capacity'])
-                    logger.error('Isolated instance {} last checked in at {}, marked as lost.'.format(
-                        instance.hostname, instance.modified))
+                try:
+                    with open(os.path.join(facts_path, instance.hostname), 'r') as facts_data:
+                        output = facts_data.read()
+                    task_result = json.loads(output)
+                except Exception:
+                    logger.exception('Failed to read status from isolated instances, output:\n {}'.format(output))
+                if 'awx_capacity_cpu' in task_result and 'awx_capacity_mem' in task_result:
+                    task_result = {
+                        'capacity_cpu': task_result['awx_capacity_cpu'],
+                        'capacity_mem': task_result['awx_capacity_mem'],
+                        'version': task_result['awx_capacity_version']
+                    }
+                    cls.update_capacity(instance, task_result, awx_application_version)
+                    logger.debug('Isolated instance {} successful heartbeat'.format(instance.hostname))
+                elif instance.capacity == 0:
+                    logger.debug('Isolated instance {} previously marked as lost, could not re-join.'.format(
+                        instance.hostname))
+                else:
+                    logger.warning('Could not update status of isolated instance {}'.format(instance.hostname))
+                    if instance.is_lost(isolated=True):
+                        instance.capacity = 0
+                        instance.save(update_fields=['capacity'])
+                        logger.error('Isolated instance {} last checked in at {}, marked as lost.'.format(
+                            instance.hostname, instance.modified))
+        finally:
+            if os.path.exists(facts_path):
+                shutil.rmtree(facts_path)
 
     @staticmethod
     def get_stdout_handle(instance, private_data_dir, event_data_key='job_id'):

awx/main/expect/run.py

@@ -4,7 +4,6 @@ import argparse
 import base64
 import codecs
 import collections
-import StringIO
 import logging
 import json
 import os
@@ -13,12 +12,15 @@ import pipes
 import re
 import signal
 import sys
-import thread
+import threading
 import time
+try:
+    from io import StringIO
+except ImportError:
+    from StringIO import StringIO
 
 import pexpect
 import psutil
-import six
 
 
 logger = logging.getLogger('awx.main.utils.expect')
@@ -49,7 +51,10 @@ def open_fifo_write(path, data):
     reads data from the pipe.
     '''
     os.mkfifo(path, 0o600)
-    thread.start_new_thread(lambda p, d: open(p, 'w').write(d), (path, data))
+    threading.Thread(
+        target=lambda p, d: open(p, 'w').write(d),
+        args=(path, data)
+    ).start()
 
 
 def run_pexpect(args, cwd, env, logfile,
@@ -71,7 +76,7 @@ def run_pexpect(args, cwd, env, logfile,
                                 - signifying if the job has been prematurely
                                   cancelled
     :param expect_passwords:    a dict of regular expression password prompts
-                                to input values, i.e., {r'Password:\s*?$':
+                                to input values, i.e., {r'Password:*?$':
                                 'some_password'}
     :param extra_update_fields: a dict used to specify DB fields which should
                                 be updated on the underlying model
@@ -97,14 +102,8 @@ def run_pexpect(args, cwd, env, logfile,
         # enforce usage of an OrderedDict so that the ordering of elements in
         # `keys()` matches `values()`.
         expect_passwords = collections.OrderedDict(expect_passwords)
-    password_patterns = expect_passwords.keys()
-    password_values = expect_passwords.values()
-
-    # pexpect needs all env vars to be utf-8 encoded strings
-    # https://github.com/pexpect/pexpect/issues/512
-    for k, v in env.items():
-        if isinstance(v, six.text_type):
-            env[k] = v.encode('utf-8')
+    password_patterns = list(expect_passwords.keys())
+    password_values = list(expect_passwords.values())
 
     child = pexpect.spawn(
         args[0], args[1:], cwd=cwd, env=env, ignore_sighup=True,
@@ -208,6 +207,12 @@ def run_isolated_job(private_data_dir, secrets, logfile=sys.stdout):
     env['AWX_ISOLATED_DATA_DIR'] = private_data_dir
     env['PYTHONPATH'] = env.get('PYTHONPATH', '') + callback_dir + ':'
 
+    venv_path = env.get('VIRTUAL_ENV')
+    if venv_path and not os.path.exists(venv_path):
+        raise RuntimeError(
+            'a valid Python virtualenv does not exist at {}'.format(venv_path)
+        )
+
     return run_pexpect(args, cwd, env, logfile,
                        expect_passwords=expect_passwords,
                        idle_timeout=idle_timeout,
@@ -226,7 +231,11 @@ def handle_termination(pid, args, proot_cmd, is_cancel=True):
                       instance's cancel_flag.
     '''
     try:
-        if proot_cmd in ' '.join(args):
+        if sys.version_info > (3, 0):
+            used_proot = proot_cmd.encode('utf-8') in args
+        else:
+            used_proot = proot_cmd in ' '.join(args)
+        if used_proot:
             if not psutil:
                 os.kill(pid, signal.SIGKILL)
             else:
@@ -247,8 +256,8 @@ def handle_termination(pid, args, proot_cmd, is_cancel=True):
 
 
 def __run__(private_data_dir):
-    buff = StringIO.StringIO()
-    with open(os.path.join(private_data_dir, 'env'), 'r') as f:
+    buff = StringIO()
+    with codecs.open(os.path.join(private_data_dir, 'env'), 'r', encoding='utf-8') as f:
         for line in f:
             buff.write(line)
 

awx/main/fields.py

@@ -7,7 +7,7 @@ import json
 import operator
 import re
 import six
-import urllib
+import urllib.parse
 
 from jinja2 import Environment, StrictUndefined
 from jinja2.exceptions import UndefinedError, TemplateSyntaxError
@@ -46,7 +46,7 @@ from awx.main.utils.filters import SmartFilter
 from awx.main.utils.encryption import encrypt_value, decrypt_value, get_encryption_key
 from awx.main.validators import validate_ssh_private_key
 from awx.main.models.rbac import batch_role_ancestor_rebuilding, Role
-from awx.main.constants import CHOICES_PRIVILEGE_ESCALATION_METHODS
+from awx.main.constants import CHOICES_PRIVILEGE_ESCALATION_METHODS, ENV_BLACKLIST
 from awx.main import utils
 
 
@@ -251,6 +251,9 @@ class ImplicitRoleField(models.ForeignKey):
             if type(field_name) == tuple:
                 continue
 
+            if type(field_name) == bytes:
+                field_name = field_name.decode('utf-8')
+
             if field_name.startswith('singleton:'):
                 continue
 
@@ -373,7 +376,7 @@ class SmartFilterField(models.TextField):
         # https://docs.python.org/2/library/stdtypes.html#truth-value-testing
         if not value:
             return None
-        value = urllib.unquote(value)
+        value = urllib.parse.unquote(value)
         try:
             SmartFilter().query_from_string(value)
         except RuntimeError as e:
@@ -407,9 +410,6 @@ class JSONSchemaField(JSONBField):
             self.schema(model_instance),
             format_checker=self.format_checker
         ).iter_errors(value):
-            # strip Python unicode markers from jsonschema validation errors
-            error.message = re.sub(r'\bu(\'|")', r'\1', error.message)
-
             if error.validator == 'pattern' and 'error' in error.schema:
                 error.message = six.text_type(error.schema['error']).format(instance=error.instance)
             elif error.validator == 'type':
@@ -514,10 +514,10 @@ class CredentialInputField(JSONSchemaField):
             field = field.copy()
             if field['type'] == 'become_method':
                 field.pop('type')
-                field['choices'] = map(operator.itemgetter(0), CHOICES_PRIVILEGE_ESCALATION_METHODS)
+                field['choices'] = list(map(operator.itemgetter(0), CHOICES_PRIVILEGE_ESCALATION_METHODS))
             properties[field['id']] = field
             if field.get('choices', []):
-                field['enum'] = field['choices'][:]
+                field['enum'] = list(field['choices'])[:]
         return {
             'type': 'object',
             'properties': properties,
@@ -573,10 +573,10 @@ class CredentialInputField(JSONSchemaField):
                 # string)
                 match = re.search(
                     # 'foo' is a dependency of 'bar'
-                    "'"         # apostrophe
-                    "([^']+)"   # one or more non-apostrophes (first group)
-                    "'[\w ]+'"  # one or more words/spaces
-                    "([^']+)",  # second group
+                    r"'"         # apostrophe
+                    r"([^']+)"   # one or more non-apostrophes (first group)
+                    r"'[\w ]+'"  # one or more words/spaces
+                    r"([^']+)",  # second group
                     error.message,
                 )
                 if match:
@@ -755,7 +755,7 @@ class CredentialTypeInjectorField(JSONSchemaField):
                 'file': {
                     'type': 'object',
                     'patternProperties': {
-                        '^template(\.[a-zA-Z_]+[a-zA-Z0-9_]*)?$': {'type': 'string'},
+                        r'^template(\.[a-zA-Z_]+[a-zA-Z0-9_]*)?$': {'type': 'string'},
                     },
                     'additionalProperties': False,
                 },
@@ -767,7 +767,12 @@ class CredentialTypeInjectorField(JSONSchemaField):
                         # of underscores, digits, and alphabetics from the portable
                         # character set. The first character of a name is not
                         # a digit.
-                        '^[a-zA-Z_]+[a-zA-Z0-9_]*$': {'type': 'string'},
+                        '^[a-zA-Z_]+[a-zA-Z0-9_]*$': {
+                            'type': 'string',
+                            # The environment variable _value_ can be any ascii,
+                            # but pexpect will choke on any unicode
+                            'pattern': '^[\x00-\x7F]*$'
+                        },
                     },
                     'additionalProperties': False,
                 },
@@ -783,6 +788,19 @@ class CredentialTypeInjectorField(JSONSchemaField):
             'additionalProperties': False
         }
 
+    def validate_env_var_allowed(self, env_var):
+        if env_var.startswith('ANSIBLE_'):
+            raise django_exceptions.ValidationError(
+                _('Environment variable {} may affect Ansible configuration so its '
+                  'use is not allowed in credentials.').format(env_var),
+                code='invalid', params={'value': env_var},
+            )
+        if env_var in ENV_BLACKLIST:
+            raise django_exceptions.ValidationError(
+                _('Environment variable {} is blacklisted from use in credentials.').format(env_var),
+                code='invalid', params={'value': env_var},
+            )
+
     def validate(self, value, model_instance):
         super(CredentialTypeInjectorField, self).validate(
             value, model_instance
@@ -806,14 +824,14 @@ class CredentialTypeInjectorField(JSONSchemaField):
         )
 
         class ExplodingNamespace:
-            def __unicode__(self):
+            def __str__(self):
                 raise UndefinedError(_('Must define unnamed file injector in order to reference `tower.filename`.'))
 
         class TowerNamespace:
             def __init__(self):
                 self.filename = ExplodingNamespace()
 
-            def __unicode__(self):
+            def __str__(self):
                 raise UndefinedError(_('Cannot directly reference reserved `tower` namespace container.'))
 
         valid_namespace['tower'] = TowerNamespace()
@@ -834,6 +852,9 @@ class CredentialTypeInjectorField(JSONSchemaField):
                 setattr(valid_namespace['tower'].filename, template_name, 'EXAMPLE_FILENAME')
 
         for type_, injector in value.items():
+            if type_ == 'env':
+                for key in injector.keys():
+                    self.validate_env_var_allowed(key)
             for key, tmpl in injector.items():
                 try:
                     Environment(

awx/main/management/commands/check_migrations.py

@@ -0,0 +1,12 @@
+from django.db import connections
+from django.db.backends.sqlite3.base import DatabaseWrapper
+from django.core.management.commands.makemigrations import Command as MakeMigrations
+
+
+class Command(MakeMigrations):
+
+    def execute(self, *args, **options):
+        settings = connections['default'].settings_dict.copy()
+        settings['ENGINE'] = 'sqlite3'
+        connections['default'] = DatabaseWrapper(settings)
+        return MakeMigrations().execute(*args, **options)

awx/main/management/commands/cleanup_facts.py

@@ -3,6 +3,7 @@
 
 # Python
 import re
+import sys
 from dateutil.relativedelta import relativedelta
 
 # Django
@@ -129,6 +130,7 @@ class Command(BaseCommand):
 
     @transaction.atomic
     def handle(self, *args, **options):
+        sys.stderr.write("This command has been deprecated and will be removed in a future release.\n")
         if not feature_enabled('system_tracking'):
             raise CommandError("The System Tracking feature is not enabled for your instance")
         cleanup_facts = CleanupFacts()

awx/main/management/commands/create_preload_data.py

@@ -15,6 +15,8 @@ class Command(BaseCommand):
     def handle(self, *args, **kwargs):
         # Sanity check: Is there already an organization in the system?
         if Organization.objects.count():
+            print('An organization is already in the system, exiting.')
+            print('(changed: False)')
             return
 
         # Create a default organization as the first superuser found.
@@ -54,3 +56,4 @@ class Command(BaseCommand):
                 jt.credentials.add(c)
         print('Default organization added.')
         print('Demo Credential, Inventory, and Job Template added.')
+        print('(changed: True)')

awx/main/management/commands/deprovision_instance.py

@@ -2,7 +2,6 @@
 # All Rights Reserved
 
 import subprocess
-import warnings
 
 from django.db import transaction
 from django.core.management.base import BaseCommand, CommandError
@@ -24,31 +23,28 @@ class Command(BaseCommand):
     def add_arguments(self, parser):
         parser.add_argument('--hostname', dest='hostname', type=str,
                             help='Hostname used during provisioning')
-        parser.add_argument('--name', dest='name', type=str,
-                            help='(PENDING DEPRECIATION) Hostname used during provisioning')
 
     @transaction.atomic
     def handle(self, *args, **options):
         # TODO: remove in 3.3
-        if options.get('name'):
-            warnings.warn("`--name` is depreciated in favor of `--hostname`, and will be removed in release 3.3.")
-            if options.get('hostname'):
-                raise CommandError("Cannot accept both --name and --hostname.")
-            options['hostname'] = options['name']
         hostname = options.get('hostname')
         if not hostname:
             raise CommandError("--hostname is a required argument")
         with advisory_lock('instance_registration_%s' % hostname):
             instance = Instance.objects.filter(hostname=hostname)
             if instance.exists():
+                isolated = instance.first().is_isolated()
                 instance.delete()
                 print("Instance Removed")
-                result = subprocess.Popen("rabbitmqctl forget_cluster_node rabbitmq@{}".format(hostname), shell=True).wait()
-                if result != 0:
-                    print("Node deprovisioning may have failed when attempting to "
-                          "remove the RabbitMQ instance {} from the cluster".format(hostname))
-                else:
+                if isolated:
                     print('Successfully deprovisioned {}'.format(hostname))
+                else:
+                    result = subprocess.Popen("rabbitmqctl forget_cluster_node rabbitmq@{}".format(hostname), shell=True).wait()
+                    if result != 0:
+                        print("Node deprovisioning may have failed when attempting to "
+                              "remove the RabbitMQ instance {} from the cluster".format(hostname))
+                    else:
+                        print('Successfully deprovisioned {}'.format(hostname))
                 print('(changed: True)')
             else:
                 print('No instance found matching name {}'.format(hostname))

awx/main/management/commands/deprovision_node.py

@@ -1,17 +0,0 @@
-# Copyright (c) 2017 Ansible by Red Hat
-# All Rights Reserved
-
-# Borrow from another AWX command
-from awx.main.management.commands.deprovision_instance import Command as OtherCommand
-
-# Python
-import warnings
-
-
-class Command(OtherCommand):
-
-    def handle(self, *args, **options):
-        # TODO: delete this entire file in 3.3
-        warnings.warn('This command is replaced with `deprovision_instance` and will '
-                      'be removed in release 3.3.')
-        return super(Command, self).handle(*args, **options)

awx/main/management/commands/generate_isolated_key.py

@@ -1,6 +1,7 @@
 # Copyright (c) 2015 Ansible, Inc.
 # All Rights Reserved
 import datetime
+from django.utils.encoding import smart_str
 
 from cryptography.hazmat.backends import default_backend
 from cryptography.hazmat.primitives import serialization
@@ -35,10 +36,10 @@ class Command(BaseCommand):
         ).save()
         pemfile = Setting.objects.create(
             key='AWX_ISOLATED_PUBLIC_KEY',
-            value=key.public_key().public_bytes(
+            value=smart_str(key.public_key().public_bytes(
                 encoding=serialization.Encoding.OpenSSH,
                 format=serialization.PublicFormat.OpenSSH
-            ) + " generated-by-awx@%s" % datetime.datetime.utcnow().isoformat()
+            )) + " generated-by-awx@%s" % datetime.datetime.utcnow().isoformat()
         )
         pemfile.save()
         print(pemfile.value)

awx/main/management/commands/inventory_import.py

@@ -4,6 +4,7 @@
 # Python
 import json
 import logging
+import fnmatch
 import os
 import re
 import subprocess
@@ -15,12 +16,20 @@ import shutil
 # Django
 from django.conf import settings
 from django.core.management.base import BaseCommand, CommandError
-from django.core.exceptions import ImproperlyConfigured
 from django.db import connection, transaction
 from django.utils.encoding import smart_text
 
-# AWX
-from awx.main.models import * # noqa
+# AWX inventory imports
+from awx.main.models.inventory import (
+    Inventory,
+    InventorySource,
+    InventoryUpdate,
+    Host
+)
+from awx.main.utils.mem_inventory import MemInventory, dict_to_mem_data
+
+# other AWX imports
+from awx.main.models.rbac import batch_role_ancestor_rebuilding
 from awx.main.utils import (
     ignore_inventory_computed_fields,
     check_proot_installed,
@@ -28,7 +37,6 @@ from awx.main.utils import (
     build_proot_temp_dir,
     get_licenser
 )
-from awx.main.utils.mem_inventory import MemInventory, dict_to_mem_data
 from awx.main.signals import disable_activity_stream
 from awx.main.constants import STANDARD_INVENTORY_UPDATE_ENV
 
@@ -63,60 +71,46 @@ class AnsibleInventoryLoader(object):
     use the ansible-inventory CLI utility to convert it into in-memory
     representational objects. Example:
         /usr/bin/ansible/ansible-inventory -i hosts --list
-    If it fails to find this, it uses the backported script instead
     '''
 
-    def __init__(self, source, group_filter_re=None, host_filter_re=None, is_custom=False):
+    def __init__(self, source, is_custom=False, venv_path=None):
         self.source = source
         self.source_dir = functioning_dir(self.source)
         self.is_custom = is_custom
         self.tmp_private_dir = None
         self.method = 'ansible-inventory'
-        self.group_filter_re = group_filter_re
-        self.host_filter_re = host_filter_re
-
-        self.is_vendored_source = False
-        if self.source_dir == os.path.join(settings.BASE_DIR, 'plugins', 'inventory'):
-            self.is_vendored_source = True
+        if venv_path:
+            self.venv_path = venv_path
+        else:
+            self.venv_path = settings.ANSIBLE_VENV_PATH
 
     def build_env(self):
         env = dict(os.environ.items())
-        env['VIRTUAL_ENV'] = settings.ANSIBLE_VENV_PATH
-        env['PATH'] = os.path.join(settings.ANSIBLE_VENV_PATH, "bin") + ":" + env['PATH']
+        env['VIRTUAL_ENV'] = self.venv_path
+        env['PATH'] = os.path.join(self.venv_path, "bin") + ":" + env['PATH']
         # Set configuration items that should always be used for updates
         for key, value in STANDARD_INVENTORY_UPDATE_ENV.items():
             if key not in env:
                 env[key] = value
-        venv_libdir = os.path.join(settings.ANSIBLE_VENV_PATH, "lib")
+        venv_libdir = os.path.join(self.venv_path, "lib")
         env.pop('PYTHONPATH', None)  # default to none if no python_ver matches
-        if os.path.isdir(os.path.join(venv_libdir, "python2.7")):
-            env['PYTHONPATH'] = os.path.join(venv_libdir, "python2.7", "site-packages") + ":"
+        for version in os.listdir(venv_libdir):
+            if fnmatch.fnmatch(version, 'python[23].*'):
+                if os.path.isdir(os.path.join(venv_libdir, version)):
+                    env['PYTHONPATH'] = os.path.join(venv_libdir, version, "site-packages") + ":"
+                    break
+        # For internal inventory updates, these are not reported in the job_env API
+        logger.info('Using VIRTUAL_ENV: {}'.format(env['VIRTUAL_ENV']))
+        logger.info('Using PATH: {}'.format(env['PATH']))
+        logger.info('Using PYTHONPATH: {}'.format(env.get('PYTHONPATH', None)))
         return env
 
     def get_base_args(self):
         # get ansible-inventory absolute path for running in bubblewrap/proot, in Popen
-        for path in os.environ["PATH"].split(os.pathsep):
-            potential_path = os.path.join(path.strip('"'), 'ansible-inventory')
-            if os.path.isfile(potential_path) and os.access(potential_path, os.X_OK):
-                logger.debug('Using system install of ansible-inventory CLI: {}'.format(potential_path))
-                return [potential_path, '-i', self.source]
-
-        # Stopgap solution for group_vars, do not use backported module for official
-        # vendored cloud modules or custom scripts TODO: remove after Ansible 2.3 deprecation
-        if self.is_vendored_source or self.is_custom:
-            self.method = 'inventory script invocation'
-            return [self.source]
-
-        # ansible-inventory was not found, look for backported module TODO: remove after Ansible 2.3 deprecation
-        abs_module_path = os.path.abspath(os.path.join(
-            os.path.dirname(__file__), '..', '..', '..', 'plugins',
-            'ansible_inventory', 'backport.py'))
-        self.method = 'ansible-inventory backport'
-
-        if not os.path.exists(abs_module_path):
-            raise ImproperlyConfigured('Cannot find inventory module')
-        logger.debug('Using backported ansible-inventory module: {}'.format(abs_module_path))
-        return [abs_module_path, '-i', self.source]
+        abs_ansible_inventory = shutil.which('ansible-inventory')
+        bargs= [abs_ansible_inventory, '-i', self.source]
+        logger.debug('Using base command: {}'.format(' '.join(bargs)))
+        return bargs
 
     def get_proot_args(self, cmd, env):
         cwd = os.getcwd()
@@ -155,6 +149,8 @@ class AnsibleInventoryLoader(object):
 
         proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, env=env)
         stdout, stderr = proc.communicate()
+        stdout = smart_text(stdout)
+        stderr = smart_text(stderr)
 
         if self.tmp_private_dir:
             shutil.rmtree(self.tmp_private_dir, True)
@@ -177,80 +173,7 @@ class AnsibleInventoryLoader(object):
         base_args = self.get_base_args()
         logger.info('Reading Ansible inventory source: %s', self.source)
 
-        data = self.command_to_json(base_args + ['--list'])
-
-        # TODO: remove after we run custom scripts through ansible-inventory
-        if self.is_custom and '_meta' not in data or 'hostvars' not in data['_meta']:
-            # Invoke the executable once for each host name we've built up
-            # to set their variables
-            data.setdefault('_meta', {})
-            data['_meta'].setdefault('hostvars', {})
-            logger.warning('Re-calling script for hostvars individually.')
-            for group_name, group_data in data.iteritems():
-                if group_name == '_meta':
-                    continue
-
-                if isinstance(group_data, dict):
-                    group_host_list = group_data.get('hosts', [])
-                elif isinstance(group_data, list):
-                    group_host_list = group_data
-                else:
-                    logger.warning('Group data for "%s" is not a dict or list',
-                                   group_name)
-                    group_host_list = []
-
-                for hostname in group_host_list:
-                    logger.debug('Obtaining hostvars for %s' % hostname.encode('utf-8'))
-                    hostdata = self.command_to_json(
-                        base_args + ['--host', hostname.encode("utf-8")]
-                    )
-                    if isinstance(hostdata, dict):
-                        data['_meta']['hostvars'][hostname] = hostdata
-                    else:
-                        logger.warning(
-                            'Expected dict of vars for host "%s" when '
-                            'calling with `--host`, got %s instead',
-                            k, str(type(data))
-                        )
-
-        logger.info('Processing JSON output...')
-        inventory = MemInventory(
-            group_filter_re=self.group_filter_re, host_filter_re=self.host_filter_re)
-        inventory = dict_to_mem_data(data, inventory=inventory)
-
-        return inventory
-
-
-def load_inventory_source(source, group_filter_re=None,
-                          host_filter_re=None, exclude_empty_groups=False,
-                          is_custom=False):
-    '''
-    Load inventory from given source directory or file.
-    '''
-    # Sanity check: We sanitize these module names for our API but Ansible proper doesn't follow
-    # good naming conventions
-    source = source.replace('rhv.py', 'ovirt4.py')
-    source = source.replace('satellite6.py', 'foreman.py')
-    source = source.replace('vmware.py', 'vmware_inventory.py')
-    if not os.path.exists(source):
-        raise IOError('Source does not exist: %s' % source)
-    source = os.path.join(os.getcwd(), os.path.dirname(source),
-                          os.path.basename(source))
-    source = os.path.normpath(os.path.abspath(source))
-
-    inventory = AnsibleInventoryLoader(
-        source=source,
-        group_filter_re=group_filter_re,
-        host_filter_re=host_filter_re,
-        is_custom=is_custom).load()
-
-    logger.debug('Finished loading from source: %s', source)
-    # Exclude groups that are completely empty.
-    if exclude_empty_groups:
-        inventory.delete_empty_groups()
-    logger.info('Loaded %d groups, %d hosts', len(inventory.all_group.all_groups),
-                len(inventory.all_group.all_hosts))
-    return inventory.all_group
+        return self.command_to_json(base_args + ['--list'])
 
 
 class Command(BaseCommand):
@@ -268,6 +191,8 @@ class Command(BaseCommand):
         parser.add_argument('--inventory-id', dest='inventory_id', type=int,
                             default=None, metavar='i',
                             help='id of inventory to sync')
+        parser.add_argument('--venv', dest='venv', type=str, default=None,
+                            help='absolute path to the AWX custom virtualenv to use')
         parser.add_argument('--overwrite', dest='overwrite', action='store_true', default=False,
                             help='overwrite the destination hosts and groups')
         parser.add_argument('--overwrite-vars', dest='overwrite_vars',
@@ -347,7 +272,7 @@ class Command(BaseCommand):
             if enabled is not default:
                 enabled_value = getattr(self, 'enabled_value', None)
                 if enabled_value is not None:
-                    enabled = bool(unicode(enabled_value) == unicode(enabled))
+                    enabled = bool(str(enabled_value) == str(enabled))
                 else:
                     enabled = bool(enabled)
         if enabled is default:
@@ -357,6 +282,19 @@ class Command(BaseCommand):
         else:
             raise NotImplementedError('Value of enabled {} not understood.'.format(enabled))
 
+    def get_source_absolute_path(self, source):
+        # Sanity check: We sanitize these module names for our API but Ansible proper doesn't follow
+        # good naming conventions
+        source = source.replace('rhv.py', 'ovirt4.py')
+        source = source.replace('satellite6.py', 'foreman.py')
+        source = source.replace('vmware.py', 'vmware_inventory.py')
+        if not os.path.exists(source):
+            raise IOError('Source does not exist: %s' % source)
+        source = os.path.join(os.getcwd(), os.path.dirname(source),
+                              os.path.basename(source))
+        source = os.path.normpath(os.path.abspath(source))
+        return source
+
     def load_inventory_from_database(self):
         '''
         Load inventory and related objects from the database.
@@ -369,9 +307,9 @@ class Command(BaseCommand):
         try:
             self.inventory = Inventory.objects.get(**q)
         except Inventory.DoesNotExist:
-            raise CommandError('Inventory with %s = %s cannot be found' % q.items()[0])
+            raise CommandError('Inventory with %s = %s cannot be found' % list(q.items())[0])
         except Inventory.MultipleObjectsReturned:
-            raise CommandError('Inventory with %s = %s returned multiple results' % q.items()[0])
+            raise CommandError('Inventory with %s = %s returned multiple results' % list(q.items())[0])
         logger.info('Updating inventory %d: %s' % (self.inventory.pk,
                                                    self.inventory.name))
 
@@ -471,7 +409,7 @@ class Command(BaseCommand):
         if self.instance_id_var:
             all_instance_ids = self.mem_instance_id_map.keys()
             instance_ids = []
-            for offset in xrange(0, len(all_instance_ids), self._batch_size):
+            for offset in range(0, len(all_instance_ids), self._batch_size):
                 instance_ids = all_instance_ids[offset:(offset + self._batch_size)]
                 for host_pk in hosts_qs.filter(instance_id__in=instance_ids).values_list('pk', flat=True):
                     del_host_pks.discard(host_pk)
@@ -479,19 +417,19 @@ class Command(BaseCommand):
                 del_host_pks.discard(host_pk)
             all_host_names = list(set(self.mem_instance_id_map.values()) - set(self.all_group.all_hosts.keys()))
         else:
-            all_host_names = self.all_group.all_hosts.keys()
-        for offset in xrange(0, len(all_host_names), self._batch_size):
+            all_host_names = list(self.all_group.all_hosts.keys())
+        for offset in range(0, len(all_host_names), self._batch_size):
             host_names = all_host_names[offset:(offset + self._batch_size)]
             for host_pk in hosts_qs.filter(name__in=host_names).values_list('pk', flat=True):
                 del_host_pks.discard(host_pk)
         # Now delete all remaining hosts in batches.
         all_del_pks = sorted(list(del_host_pks))
-        for offset in xrange(0, len(all_del_pks), self._batch_size):
+        for offset in range(0, len(all_del_pks), self._batch_size):
             del_pks = all_del_pks[offset:(offset + self._batch_size)]
             for host in hosts_qs.filter(pk__in=del_pks):
                 host_name = host.name
                 host.delete()
-                logger.info('Deleted host "%s"', host_name)
+                logger.debug('Deleted host "%s"', host_name)
         if settings.SQL_DEBUG:
             logger.warning('host deletions took %d queries for %d hosts',
                            len(connection.queries) - queries_before,
@@ -509,8 +447,8 @@ class Command(BaseCommand):
         groups_qs = self.inventory_source.groups.all()
         # Build list of all group pks, remove those that should not be deleted.
         del_group_pks = set(groups_qs.values_list('pk', flat=True))
-        all_group_names = self.all_group.all_groups.keys()
-        for offset in xrange(0, len(all_group_names), self._batch_size):
+        all_group_names = list(self.all_group.all_groups.keys())
+        for offset in range(0, len(all_group_names), self._batch_size):
             group_names = all_group_names[offset:(offset + self._batch_size)]
             for group_pk in groups_qs.filter(name__in=group_names).values_list('pk', flat=True):
                 del_group_pks.discard(group_pk)
@@ -522,13 +460,13 @@ class Command(BaseCommand):
             del_group_pks.discard(self.inventory_source.deprecated_group_id)
         # Now delete all remaining groups in batches.
         all_del_pks = sorted(list(del_group_pks))
-        for offset in xrange(0, len(all_del_pks), self._batch_size):
+        for offset in range(0, len(all_del_pks), self._batch_size):
             del_pks = all_del_pks[offset:(offset + self._batch_size)]
             for group in groups_qs.filter(pk__in=del_pks):
                 group_name = group.name
                 with ignore_inventory_computed_fields():
                     group.delete()
-                logger.info('Group "%s" deleted', group_name)
+                logger.debug('Group "%s" deleted', group_name)
         if settings.SQL_DEBUG:
             logger.warning('group deletions took %d queries for %d groups',
                            len(connection.queries) - queries_before,
@@ -549,7 +487,7 @@ class Command(BaseCommand):
         db_groups = self.inventory_source.groups
         for db_group in db_groups.all():
             if self.inventory_source.deprecated_group_id == db_group.id:  # TODO: remove in 3.3
-                logger.info(
+                logger.debug(
                     'Group "%s" from v1 API child group/host connections preserved',
                     db_group.name
                 )
@@ -561,25 +499,25 @@ class Command(BaseCommand):
             for mem_group in mem_children:
                 db_children_name_pk_map.pop(mem_group.name, None)
             del_child_group_pks = list(set(db_children_name_pk_map.values()))
-            for offset in xrange(0, len(del_child_group_pks), self._batch_size):
+            for offset in range(0, len(del_child_group_pks), self._batch_size):
                 child_group_pks = del_child_group_pks[offset:(offset + self._batch_size)]
                 for db_child in db_children.filter(pk__in=child_group_pks):
                     group_group_count += 1
                     db_group.children.remove(db_child)
-                    logger.info('Group "%s" removed from group "%s"',
-                                db_child.name, db_group.name)
+                    logger.debug('Group "%s" removed from group "%s"',
+                                 db_child.name, db_group.name)
             # FIXME: Inventory source group relationships
             # Delete group/host relationships not present in imported data.
             db_hosts = db_group.hosts
             del_host_pks = set(db_hosts.values_list('pk', flat=True))
             mem_hosts = self.all_group.all_groups[db_group.name].hosts
             all_mem_host_names = [h.name for h in mem_hosts if not h.instance_id]
-            for offset in xrange(0, len(all_mem_host_names), self._batch_size):
+            for offset in range(0, len(all_mem_host_names), self._batch_size):
                 mem_host_names = all_mem_host_names[offset:(offset + self._batch_size)]
                 for db_host_pk in db_hosts.filter(name__in=mem_host_names).values_list('pk', flat=True):
                     del_host_pks.discard(db_host_pk)
             all_mem_instance_ids = [h.instance_id for h in mem_hosts if h.instance_id]
-            for offset in xrange(0, len(all_mem_instance_ids), self._batch_size):
+            for offset in range(0, len(all_mem_instance_ids), self._batch_size):
                 mem_instance_ids = all_mem_instance_ids[offset:(offset + self._batch_size)]
                 for db_host_pk in db_hosts.filter(instance_id__in=mem_instance_ids).values_list('pk', flat=True):
                     del_host_pks.discard(db_host_pk)
@@ -587,15 +525,15 @@ class Command(BaseCommand):
             for db_host_pk in all_db_host_pks:
                 del_host_pks.discard(db_host_pk)
             del_host_pks = list(del_host_pks)
-            for offset in xrange(0, len(del_host_pks), self._batch_size):
+            for offset in range(0, len(del_host_pks), self._batch_size):
                 del_pks = del_host_pks[offset:(offset + self._batch_size)]
                 for db_host in db_hosts.filter(pk__in=del_pks):
                     group_host_count += 1
                     if db_host not in db_group.hosts.all():
                         continue
                     db_group.hosts.remove(db_host)
-                    logger.info('Host "%s" removed from group "%s"',
-                                db_host.name, db_group.name)
+                    logger.debug('Host "%s" removed from group "%s"',
+                                 db_host.name, db_group.name)
         if settings.SQL_DEBUG:
             logger.warning('group-group and group-host deletions took %d queries for %d relationships',
                            len(connection.queries) - queries_before,
@@ -614,9 +552,9 @@ class Command(BaseCommand):
         if db_variables != all_obj.variables_dict:
             all_obj.variables = json.dumps(db_variables)
             all_obj.save(update_fields=['variables'])
-            logger.info('Inventory variables updated from "all" group')
+            logger.debug('Inventory variables updated from "all" group')
         else:
-            logger.info('Inventory variables unmodified')
+            logger.debug('Inventory variables unmodified')
 
     def _create_update_groups(self):
         '''
@@ -635,7 +573,7 @@ class Command(BaseCommand):
             if len(v.parents) == 1 and v.parents[0].name == 'all':
                 root_group_names.add(k)
         existing_group_names = set()
-        for offset in xrange(0, len(all_group_names), self._batch_size):
+        for offset in range(0, len(all_group_names), self._batch_size):
             group_names = all_group_names[offset:(offset + self._batch_size)]
             for group in self.inventory.groups.filter(name__in=group_names):
                 mem_group = self.all_group.all_groups[group.name]
@@ -648,11 +586,11 @@ class Command(BaseCommand):
                     group.variables = json.dumps(db_variables)
                     group.save(update_fields=['variables'])
                     if self.overwrite_vars:
-                        logger.info('Group "%s" variables replaced', group.name)
+                        logger.debug('Group "%s" variables replaced', group.name)
                     else:
-                        logger.info('Group "%s" variables updated', group.name)
+                        logger.debug('Group "%s" variables updated', group.name)
                 else:
-                    logger.info('Group "%s" variables unmodified', group.name)
+                    logger.debug('Group "%s" variables unmodified', group.name)
                 existing_group_names.add(group.name)
                 self._batch_add_m2m(self.inventory_source.groups, group)
         for group_name in all_group_names:
@@ -666,7 +604,7 @@ class Command(BaseCommand):
                     'description':'imported'
                 }
             )[0]
-            logger.info('Group "%s" added', group.name)
+            logger.debug('Group "%s" added', group.name)
             self._batch_add_m2m(self.inventory_source.groups, group)
         self._batch_add_m2m(self.inventory_source.groups, flush=True)
         if settings.SQL_DEBUG:
@@ -705,24 +643,24 @@ class Command(BaseCommand):
         if update_fields:
             db_host.save(update_fields=update_fields)
         if 'name' in update_fields:
-            logger.info('Host renamed from "%s" to "%s"', old_name, mem_host.name)
+            logger.debug('Host renamed from "%s" to "%s"', old_name, mem_host.name)
         if 'instance_id' in update_fields:
             if old_instance_id:
-                logger.info('Host "%s" instance_id updated', mem_host.name)
+                logger.debug('Host "%s" instance_id updated', mem_host.name)
             else:
-                logger.info('Host "%s" instance_id added', mem_host.name)
+                logger.debug('Host "%s" instance_id added', mem_host.name)
         if 'variables' in update_fields:
             if self.overwrite_vars:
-                logger.info('Host "%s" variables replaced', mem_host.name)
+                logger.debug('Host "%s" variables replaced', mem_host.name)
             else:
-                logger.info('Host "%s" variables updated', mem_host.name)
+                logger.debug('Host "%s" variables updated', mem_host.name)
         else:
-            logger.info('Host "%s" variables unmodified', mem_host.name)
+            logger.debug('Host "%s" variables unmodified', mem_host.name)
         if 'enabled' in update_fields:
             if enabled:
-                logger.info('Host "%s" is now enabled', mem_host.name)
+                logger.debug('Host "%s" is now enabled', mem_host.name)
             else:
-                logger.info('Host "%s" is now disabled', mem_host.name)
+                logger.debug('Host "%s" is now disabled', mem_host.name)
         self._batch_add_m2m(self.inventory_source.hosts, db_host)
 
     def _create_update_hosts(self):
@@ -739,7 +677,7 @@ class Command(BaseCommand):
         mem_host_instance_id_map = {}
         mem_host_name_map = {}
         mem_host_names_to_update = set(self.all_group.all_hosts.keys())
-        for k,v in self.all_group.all_hosts.iteritems():
+        for k,v in self.all_group.all_hosts.items():
             mem_host_name_map[k] = v
             instance_id = self._get_instance_id(v.variables)
             if instance_id in self.db_instance_id_map:
@@ -749,7 +687,7 @@ class Command(BaseCommand):
 
         # Update all existing hosts where we know the PK based on instance_id.
         all_host_pks = sorted(mem_host_pk_map.keys())
-        for offset in xrange(0, len(all_host_pks), self._batch_size):
+        for offset in range(0, len(all_host_pks), self._batch_size):
             host_pks = all_host_pks[offset:(offset + self._batch_size)]
             for db_host in self.inventory.hosts.filter( pk__in=host_pks):
                 if db_host.pk in host_pks_updated:
@@ -761,7 +699,7 @@ class Command(BaseCommand):
 
         # Update all existing hosts where we know the instance_id.
         all_instance_ids = sorted(mem_host_instance_id_map.keys())
-        for offset in xrange(0, len(all_instance_ids), self._batch_size):
+        for offset in range(0, len(all_instance_ids), self._batch_size):
             instance_ids = all_instance_ids[offset:(offset + self._batch_size)]
             for db_host in self.inventory.hosts.filter( instance_id__in=instance_ids):
                 if db_host.pk in host_pks_updated:
@@ -773,7 +711,7 @@ class Command(BaseCommand):
 
         # Update all existing hosts by name.
         all_host_names = sorted(mem_host_name_map.keys())
-        for offset in xrange(0, len(all_host_names), self._batch_size):
+        for offset in range(0, len(all_host_names), self._batch_size):
             host_names = all_host_names[offset:(offset + self._batch_size)]
             for db_host in self.inventory.hosts.filter( name__in=host_names):
                 if db_host.pk in host_pks_updated:
@@ -796,9 +734,9 @@ class Command(BaseCommand):
                 host_attrs['instance_id'] = instance_id
             db_host = self.inventory.hosts.update_or_create(name=mem_host_name, defaults=host_attrs)[0]
             if enabled is False:
-                logger.info('Host "%s" added (disabled)', mem_host_name)
+                logger.debug('Host "%s" added (disabled)', mem_host_name)
             else:
-                logger.info('Host "%s" added', mem_host_name)
+                logger.debug('Host "%s" added', mem_host_name)
             self._batch_add_m2m(self.inventory_source.hosts, db_host)
 
         self._batch_add_m2m(self.inventory_source.hosts, flush=True)
@@ -815,22 +753,22 @@ class Command(BaseCommand):
         '''
         if settings.SQL_DEBUG:
             queries_before = len(connection.queries)
-        all_group_names = sorted([k for k,v in self.all_group.all_groups.iteritems() if v.children])
+        all_group_names = sorted([k for k,v in self.all_group.all_groups.items() if v.children])
         group_group_count = 0
-        for offset in xrange(0, len(all_group_names), self._batch_size):
+        for offset in range(0, len(all_group_names), self._batch_size):
             group_names = all_group_names[offset:(offset + self._batch_size)]
             for db_group in self.inventory.groups.filter(name__in=group_names):
                 mem_group = self.all_group.all_groups[db_group.name]
                 group_group_count += len(mem_group.children)
                 all_child_names = sorted([g.name for g in mem_group.children])
-                for offset2 in xrange(0, len(all_child_names), self._batch_size):
+                for offset2 in range(0, len(all_child_names), self._batch_size):
                     child_names = all_child_names[offset2:(offset2 + self._batch_size)]
                     db_children_qs = self.inventory.groups.filter(name__in=child_names)
                     for db_child in db_children_qs.filter(children__id=db_group.id):
-                        logger.info('Group "%s" already child of group "%s"', db_child.name, db_group.name)
+                        logger.debug('Group "%s" already child of group "%s"', db_child.name, db_group.name)
                     for db_child in db_children_qs.exclude(children__id=db_group.id):
                         self._batch_add_m2m(db_group.children, db_child)
-                    logger.info('Group "%s" added as child of "%s"', db_child.name, db_group.name)
+                    logger.debug('Group "%s" added as child of "%s"', db_child.name, db_group.name)
                 self._batch_add_m2m(db_group.children, flush=True)
         if settings.SQL_DEBUG:
             logger.warning('Group-group updates took %d queries for %d group-group relationships',
@@ -842,31 +780,31 @@ class Command(BaseCommand):
         # belongs.
         if settings.SQL_DEBUG:
             queries_before = len(connection.queries)
-        all_group_names = sorted([k for k,v in self.all_group.all_groups.iteritems() if v.hosts])
+        all_group_names = sorted([k for k,v in self.all_group.all_groups.items() if v.hosts])
         group_host_count = 0
-        for offset in xrange(0, len(all_group_names), self._batch_size):
+        for offset in range(0, len(all_group_names), self._batch_size):
             group_names = all_group_names[offset:(offset + self._batch_size)]
             for db_group in self.inventory.groups.filter(name__in=group_names):
                 mem_group = self.all_group.all_groups[db_group.name]
                 group_host_count += len(mem_group.hosts)
                 all_host_names = sorted([h.name for h in mem_group.hosts if not h.instance_id])
-                for offset2 in xrange(0, len(all_host_names), self._batch_size):
+                for offset2 in range(0, len(all_host_names), self._batch_size):
                     host_names = all_host_names[offset2:(offset2 + self._batch_size)]
                     db_hosts_qs = self.inventory.hosts.filter(name__in=host_names)
                     for db_host in db_hosts_qs.filter(groups__id=db_group.id):
-                        logger.info('Host "%s" already in group "%s"', db_host.name, db_group.name)
+                        logger.debug('Host "%s" already in group "%s"', db_host.name, db_group.name)
                     for db_host in db_hosts_qs.exclude(groups__id=db_group.id):
                         self._batch_add_m2m(db_group.hosts, db_host)
-                        logger.info('Host "%s" added to group "%s"', db_host.name, db_group.name)
+                        logger.debug('Host "%s" added to group "%s"', db_host.name, db_group.name)
                 all_instance_ids = sorted([h.instance_id for h in mem_group.hosts if h.instance_id])
-                for offset2 in xrange(0, len(all_instance_ids), self._batch_size):
+                for offset2 in range(0, len(all_instance_ids), self._batch_size):
                     instance_ids = all_instance_ids[offset2:(offset2 + self._batch_size)]
                     db_hosts_qs = self.inventory.hosts.filter(instance_id__in=instance_ids)
                     for db_host in db_hosts_qs.filter(groups__id=db_group.id):
-                        logger.info('Host "%s" already in group "%s"', db_host.name, db_group.name)
+                        logger.debug('Host "%s" already in group "%s"', db_host.name, db_group.name)
                     for db_host in db_hosts_qs.exclude(groups__id=db_group.id):
                         self._batch_add_m2m(db_group.hosts, db_host)
-                        logger.info('Host "%s" added to group "%s"', db_host.name, db_group.name)
+                        logger.debug('Host "%s" added to group "%s"', db_host.name, db_group.name)
                 self._batch_add_m2m(db_group.hosts, flush=True)
         if settings.SQL_DEBUG:
             logger.warning('Group-host updates took %d queries for %d group-host relationships',
@@ -926,6 +864,7 @@ class Command(BaseCommand):
         self.set_logging_level()
         self.inventory_name = options.get('inventory_name', None)
         self.inventory_id = options.get('inventory_id', None)
+        venv_path = options.get('venv', None)
         self.overwrite = bool(options.get('overwrite', False))
         self.overwrite_vars = bool(options.get('overwrite_vars', False))
         self.keep_vars = bool(options.get('keep_vars', False))
@@ -938,7 +877,7 @@ class Command(BaseCommand):
         self.exclude_empty_groups = bool(options.get('exclude_empty_groups', False))
         self.instance_id_var = options.get('instance_id_var', None)
 
-        self.celery_invoked = False if os.getenv('INVENTORY_SOURCE_ID', None) is None else True
+        self.invoked_from_dispatcher = False if os.getenv('INVENTORY_SOURCE_ID', None) is None else True
 
         # Load inventory and related objects from database.
         if self.inventory_name and self.inventory_id:
@@ -986,12 +925,26 @@ class Command(BaseCommand):
                         self.inventory_update.status = 'running'
                         self.inventory_update.save()
 
-            # Load inventory from source.
-            self.all_group = load_inventory_source(self.source,
-                                                   self.group_filter_re,
-                                                   self.host_filter_re,
-                                                   self.exclude_empty_groups,
-                                                   self.is_custom)
+            source = self.get_source_absolute_path(self.source)
+
+            data = AnsibleInventoryLoader(source=source, is_custom=self.is_custom, venv_path=venv_path).load()
+
+            logger.debug('Finished loading from source: %s', source)
+            logger.info('Processing JSON output...')
+            inventory = MemInventory(
+                group_filter_re=self.group_filter_re, host_filter_re=self.host_filter_re)
+            inventory = dict_to_mem_data(data, inventory=inventory)
+
+            del data  # forget dict from import, could be large
+
+            logger.info('Loaded %d groups, %d hosts', len(inventory.all_group.all_groups),
+                        len(inventory.all_group.all_hosts))
+
+            if self.exclude_empty_groups:
+                inventory.delete_empty_groups()
+
+            self.all_group = inventory.all_group
+
             if settings.DEBUG:
                 # depending on inventory source, this output can be
                 # *exceedingly* verbose - crawling a deeply nested
@@ -1062,7 +1015,7 @@ class Command(BaseCommand):
                 exc = e
             transaction.rollback()
 
-        if self.celery_invoked is False:
+        if self.invoked_from_dispatcher is False:
             with ignore_inventory_computed_fields():
                 self.inventory_update = InventoryUpdate.objects.get(pk=self.inventory_update.pk)
                 self.inventory_update.result_traceback = tb
@@ -1074,4 +1027,4 @@ class Command(BaseCommand):
         if exc and isinstance(exc, CommandError):
             sys.exit(1)
         elif exc:
-            raise
+            raise exc

awx/main/management/commands/list_instances.py

@@ -6,6 +6,22 @@ from django.core.management.base import BaseCommand
 import six
 
 
+class Ungrouped(object):
+
+    name = 'ungrouped'
+    policy_instance_percentage = None
+    policy_instance_minimum = None
+    controller = None
+
+    @property
+    def instances(self):
+        return Instance.objects.filter(rampart_groups__isnull=True)
+
+    @property
+    def capacity(self):
+        return sum([x.capacity for x in self.instances])
+
+
 class Command(BaseCommand):
     """List instances from the Tower database
     """
@@ -13,12 +29,28 @@ class Command(BaseCommand):
     def handle(self, *args, **options):
         super(Command, self).__init__()
 
-        for instance in Instance.objects.all():
-            print(six.text_type(
-                "hostname: {0.hostname}; created: {0.created}; "
-                "heartbeat: {0.modified}; capacity: {0.capacity}").format(instance))
-        for instance_group in InstanceGroup.objects.all():
-            print(six.text_type(
-                "Instance Group: {0.name}; created: {0.created}; "
-                "capacity: {0.capacity}; members: {1}").format(instance_group,
-                                                               [x.hostname for x in instance_group.instances.all()]))
+        groups = list(InstanceGroup.objects.all())
+        ungrouped = Ungrouped()
+        if len(ungrouped.instances):
+            groups.append(ungrouped)
+
+        for instance_group in groups:
+            fmt = '[{0.name} capacity={0.capacity}'
+            if instance_group.policy_instance_percentage:
+                fmt += ' policy={0.policy_instance_percentage}%'
+            if instance_group.policy_instance_minimum:
+                fmt += ' policy>={0.policy_instance_minimum}'
+            if instance_group.controller:
+                fmt += ' controller={0.controller.name}'
+            print(six.text_type(fmt + ']').format(instance_group))
+            for x in instance_group.instances.all():
+                color = '\033[92m'
+                if x.capacity == 0 or x.enabled is False:
+                    color = '\033[91m'
+                fmt = '\t' + color + '{0.hostname} capacity={0.capacity} version={1}'
+                if x.last_isolated_check:
+                    fmt += ' last_isolated_check="{0.last_isolated_check:%Y-%m-%d %H:%M:%S}"'
+                if x.capacity:
+                    fmt += ' heartbeat="{0.modified:%Y-%m-%d %H:%M:%S}"'
+                print(six.text_type(fmt + '\033[0m').format(x, x.version or '?'))
+            print('')

awx/main/management/commands/register_instance.py

@@ -1,17 +0,0 @@
-# Copyright (c) 2017 Ansible by Red Hat
-# All Rights Reserved
-
-# Borrow from another AWX command
-from awx.main.management.commands.provision_instance import Command as OtherCommand
-
-# Python
-import warnings
-
-
-class Command(OtherCommand):
-
-    def handle(self, *args, **options):
-        # TODO: delete this entire file in 3.3
-        warnings.warn('This command is replaced with `provision_instance` and will '
-                      'be removed in release 3.3.')
-        return super(Command, self).handle(*args, **options)

awx/main/management/commands/register_queue.py

@@ -19,11 +19,11 @@ class InstanceNotFound(Exception):
 class Command(BaseCommand):
 
     def add_arguments(self, parser):
-        parser.add_argument('--queuename', dest='queuename', type=lambda s: six.text_type(s, 'utf8'),
+        parser.add_argument('--queuename', dest='queuename', type=str,
                             help='Queue to create/update')
-        parser.add_argument('--hostnames', dest='hostnames', type=lambda s: six.text_type(s, 'utf8'),
+        parser.add_argument('--hostnames', dest='hostnames', type=str,
                             help='Comma-Delimited Hosts to add to the Queue (will not remove already assigned instances)')
-        parser.add_argument('--controller', dest='controller', type=lambda s: six.text_type(s, 'utf8'),
+        parser.add_argument('--controller', dest='controller', type=str,
                             default='', help='The controlling group (makes this an isolated group)')
         parser.add_argument('--instance_percent', dest='instance_percent', type=int, default=0,
                             help='The percentage of active instances that will be assigned to this group'),

awx/main/management/commands/replay_job_events.py

@@ -4,6 +4,7 @@
 import sys
 import time
 import json
+import random
 
 from django.utils import timezone
 from django.core.management.base import BaseCommand
@@ -26,7 +27,21 @@ from awx.api.serializers import (
 )
 
 
-class ReplayJobEvents():
+class JobStatusLifeCycle():
+    def emit_job_status(self, job, status):
+        # {"status": "successful", "project_id": 13, "unified_job_id": 659, "group_name": "jobs"}
+        job.websocket_emit_status(status)
+
+    def determine_job_event_finish_status_index(self, job_event_count, random_seed):
+        if random_seed == 0:
+            return job_event_count - 1
+
+        random.seed(random_seed)
+        job_event_index = random.randint(0, job_event_count - 1)
+        return job_event_index
+
+
+class ReplayJobEvents(JobStatusLifeCycle):
 
     recording_start = None
     replay_start = None
@@ -76,9 +91,10 @@ class ReplayJobEvents():
             job_events = job.inventory_update_events.order_by('created')
         elif type(job) is SystemJob:
             job_events = job.system_job_events.order_by('created')
-        if job_events.count() == 0:
+        count = job_events.count()
+        if count == 0:
             raise RuntimeError("No events for job id {}".format(job.id))
-        return job_events
+        return job_events, count
 
     def get_serializer(self, job):
         if type(job) is Job:
@@ -95,7 +111,7 @@ class ReplayJobEvents():
             raise RuntimeError("Job is of type {} and replay is not yet supported.".format(type(job)))
             sys.exit(1)
 
-    def run(self, job_id, speed=1.0, verbosity=0, skip=0):
+    def run(self, job_id, speed=1.0, verbosity=0, skip_range=[], random_seed=0, final_status_delay=0, debug=False):
         stats = {
             'events_ontime': {
                 'total': 0,
@@ -119,17 +135,27 @@ class ReplayJobEvents():
         }
         try:
             job = self.get_job(job_id)
-            job_events = self.get_job_events(job)
+            job_events, job_event_count = self.get_job_events(job)
             serializer = self.get_serializer(job)
         except RuntimeError as e:
             print("{}".format(e.message))
             sys.exit(1)
 
         je_previous = None
+
+        self.emit_job_status(job, 'pending')
+        self.emit_job_status(job, 'waiting')
+        self.emit_job_status(job, 'running')
+
+        finish_status_index = self.determine_job_event_finish_status_index(job_event_count, random_seed)
+
         for n, je_current in enumerate(job_events):
-            if n < skip:
+            if je_current.counter in skip_range:
                 continue
 
+            if debug:
+                input("{} of {}:".format(n, job_event_count))
+
             if not je_previous:
                 stats['recording_start'] = je_current.created
                 self.start(je_current.created)
@@ -146,7 +172,7 @@ class ReplayJobEvents():
                 print("recording: next job in {} seconds".format(recording_diff))
             if replay_offset >= 0:
                 replay_diff = recording_diff - replay_offset
-                
+
                 if replay_diff > 0:
                     stats['events_ontime']['total'] += 1
                     if verbosity >= 3:
@@ -167,6 +193,11 @@ class ReplayJobEvents():
             stats['events_total'] += 1
             je_previous = je_current
 
+            if n == finish_status_index:
+                if final_status_delay != 0:
+                    self.sleep(final_status_delay)
+                self.emit_job_status(job, job.status)
+
         if stats['events_total'] > 2:
             stats['replay_end'] = self.now()
             stats['replay_duration'] = (stats['replay_end'] - stats['replay_start']).total_seconds()
@@ -193,19 +224,39 @@ class Command(BaseCommand):
 
     help = 'Replay job events over websockets ordered by created on date.'
 
+    def _parse_slice_range(self, slice_arg):
+        slice_arg = tuple([int(n) for n in slice_arg.split(':')])
+        slice_obj = slice(*slice_arg)
+
+        start = slice_obj.start or 0
+        stop = slice_obj.stop or -1
+        step = slice_obj.step or 1
+
+        return range(start, stop, step)
+
     def add_arguments(self, parser):
         parser.add_argument('--job_id', dest='job_id', type=int, metavar='j',
                             help='Id of the job to replay (job or adhoc)')
-        parser.add_argument('--speed', dest='speed', type=int, metavar='s',
+        parser.add_argument('--speed', dest='speed', type=float, metavar='s',
                             help='Speedup factor.')
-        parser.add_argument('--skip', dest='skip', type=int, metavar='k',
-                            help='Number of events to skip.')
+        parser.add_argument('--skip-range', dest='skip_range', type=str, metavar='k',
+                            default='0:-1:1', help='Range of events to skip')
+        parser.add_argument('--random-seed', dest='random_seed', type=int, metavar='r',
+                            default=0, help='Random number generator seed to use when determining job_event index to emit final job status')
+        parser.add_argument('--final-status-delay', dest='final_status_delay', type=float, metavar='f',
+                            default=0, help='Delay between event and final status emit')
+        parser.add_argument('--debug', dest='debug', type=bool, metavar='d',
+                            default=False, help='Enable step mode to control emission of job events one at a time.')
 
     def handle(self, *args, **options):
         job_id = options.get('job_id')
         speed = options.get('speed') or 1
         verbosity = options.get('verbosity') or 0
-        skip = options.get('skip') or 0
+        random_seed = options.get('random_seed')
+        final_status_delay = options.get('final_status_delay')
+        debug = options.get('debug')
+        skip = self._parse_slice_range(options.get('skip_range'))
 
         replayer = ReplayJobEvents()
-        replayer.run(job_id, speed, verbosity, skip)
+        replayer.run(job_id, speed=speed, verbosity=verbosity, skip_range=skip, random_seed=random_seed,
+                     final_status_delay=final_status_delay, debug=debug)

awx/main/management/commands/revoke_oauth2_tokens.py

@@ -0,0 +1,37 @@
+# Django
+from django.core.management.base import BaseCommand, CommandError
+from django.contrib.auth.models import User
+from django.core.exceptions import ObjectDoesNotExist
+
+# AWX
+from awx.main.models.oauth import OAuth2AccessToken
+from oauth2_provider.models import RefreshToken
+
+
+def revoke_tokens(token_list):
+    for token in token_list:
+        token.revoke()
+        print('revoked {} {}'.format(token.__class__.__name__, token.token))
+
+
+class Command(BaseCommand):
+    """Command that revokes OAuth2 access tokens."""
+    help='Revokes OAuth2 access tokens.  Use --all to revoke access and refresh tokens.'
+
+    def add_arguments(self, parser):
+        parser.add_argument('--user', dest='user', type=str, help='revoke OAuth2 tokens for a specific username')
+        parser.add_argument('--all', dest='all', action='store_true', help='revoke OAuth2 access tokens and refresh tokens')
+
+    def handle(self, *args, **options):
+        if not options['user']:
+            if options['all']:
+                revoke_tokens(RefreshToken.objects.filter(revoked=None))
+            revoke_tokens(OAuth2AccessToken.objects.all())
+        else:
+            try:
+                user = User.objects.get(username=options['user'])
+            except ObjectDoesNotExist:
+                raise CommandError('A user with that username does not exist.')
+            if options['all']:
+                revoke_tokens(RefreshToken.objects.filter(revoked=None).filter(user=user))
+            revoke_tokens(user.main_oauth2accesstoken.filter(user=user))

awx/main/management/commands/run_callback_receiver.py

@@ -1,224 +1,11 @@
 # Copyright (c) 2015 Ansible, Inc.
 # All Rights Reserved.
 
-# Python
-import logging
-import os
-import signal
-import time
-from uuid import UUID
-from multiprocessing import Process
-from multiprocessing import Queue as MPQueue
-from Queue import Empty as QueueEmpty
-from Queue import Full as QueueFull
-
-from kombu import Connection, Exchange, Queue
-from kombu.mixins import ConsumerMixin
-
-# Django
 from django.conf import settings
 from django.core.management.base import BaseCommand
-from django.db import connection as django_connection
-from django.db import DatabaseError, OperationalError
-from django.db.utils import InterfaceError, InternalError
-from django.core.cache import cache as django_cache
+from kombu import Connection, Exchange, Queue
 
-# AWX
-from awx.main.models import * # noqa
-from awx.main.consumers import emit_channel_notification
-
-logger = logging.getLogger('awx.main.commands.run_callback_receiver')
-
-
-class WorkerSignalHandler:
-
-    def __init__(self):
-        self.kill_now = False
-        signal.signal(signal.SIGINT, self.exit_gracefully)
-        signal.signal(signal.SIGTERM, self.exit_gracefully)
-
-    def exit_gracefully(self, *args, **kwargs):
-        self.kill_now = True
-
-
-class CallbackBrokerWorker(ConsumerMixin):
-
-    MAX_RETRIES = 2
-
-    def __init__(self, connection, use_workers=True):
-        self.connection = connection
-        self.worker_queues = []
-        self.total_messages = 0
-        self.init_workers(use_workers)
-
-    def init_workers(self, use_workers=True):
-        def shutdown_handler(active_workers):
-            def _handler(signum, frame):
-                try:
-                    for active_worker in active_workers:
-                        active_worker.terminate()
-                    signal.signal(signum, signal.SIG_DFL)
-                    os.kill(os.getpid(), signum) # Rethrow signal, this time without catching it
-                except Exception:
-                    logger.exception('Error in shutdown_handler')
-            return _handler
-
-        if use_workers:
-            django_connection.close()
-            django_cache.close()
-            for idx in range(settings.JOB_EVENT_WORKERS):
-                queue_actual = MPQueue(settings.JOB_EVENT_MAX_QUEUE_SIZE)
-                w = Process(target=self.callback_worker, args=(queue_actual, idx,))
-                w.start()
-                if settings.DEBUG:
-                    logger.info('Started worker %s' % str(idx))
-                self.worker_queues.append([0, queue_actual, w])
-        elif settings.DEBUG:
-            logger.warn('Started callback receiver (no workers)')
-
-        signal.signal(signal.SIGINT, shutdown_handler([p[2] for p in self.worker_queues]))
-        signal.signal(signal.SIGTERM, shutdown_handler([p[2] for p in self.worker_queues]))
-
-    def get_consumers(self, Consumer, channel):
-        return [Consumer(queues=[Queue(settings.CALLBACK_QUEUE,
-                                       Exchange(settings.CALLBACK_QUEUE, type='direct'),
-                                       routing_key=settings.CALLBACK_QUEUE)],
-                         accept=['json'],
-                         callbacks=[self.process_task])]
-
-    def process_task(self, body, message):
-        if "uuid" in body and body['uuid']:
-            try:
-                queue = UUID(body['uuid']).int % settings.JOB_EVENT_WORKERS
-            except Exception:
-                queue = self.total_messages % settings.JOB_EVENT_WORKERS
-        else:
-            queue = self.total_messages % settings.JOB_EVENT_WORKERS
-        self.write_queue_worker(queue, body)
-        self.total_messages += 1
-        message.ack()
-
-    def write_queue_worker(self, preferred_queue, body):
-        queue_order = sorted(range(settings.JOB_EVENT_WORKERS), cmp=lambda x, y: -1 if x==preferred_queue else 0)
-        write_attempt_order = []
-        for queue_actual in queue_order:
-            try:
-                worker_actual = self.worker_queues[queue_actual]
-                worker_actual[1].put(body, block=True, timeout=5)
-                worker_actual[0] += 1
-                return queue_actual
-            except QueueFull:
-                pass
-            except Exception:
-                import traceback
-                tb = traceback.format_exc()
-                logger.warn("Could not write to queue %s" % preferred_queue)
-                logger.warn("Detail: {}".format(tb))
-            write_attempt_order.append(preferred_queue)
-        logger.warn("Could not write payload to any queue, attempted order: {}".format(write_attempt_order))
-        return None
-
-    def callback_worker(self, queue_actual, idx):
-        signal_handler = WorkerSignalHandler()
-        while not signal_handler.kill_now:
-            try:
-                body = queue_actual.get(block=True, timeout=1)
-            except QueueEmpty:
-                continue
-            except Exception as e:
-                logger.error("Exception on worker thread, restarting: " + str(e))
-                continue
-            try:
-
-                event_map = {
-                    'job_id': JobEvent,
-                    'ad_hoc_command_id': AdHocCommandEvent,
-                    'project_update_id': ProjectUpdateEvent,
-                    'inventory_update_id': InventoryUpdateEvent,
-                    'system_job_id': SystemJobEvent,
-                }
-
-                if not any([key in body for key in event_map]):
-                    raise Exception('Payload does not have a job identifier')
-                if settings.DEBUG:
-                    from pygments import highlight
-                    from pygments.lexers import PythonLexer
-                    from pygments.formatters import Terminal256Formatter
-                    from pprint import pformat
-                    logger.info('Body: {}'.format(
-                        highlight(pformat(body, width=160), PythonLexer(), Terminal256Formatter(style='friendly'))
-                    )[:1024 * 4])
-
-                def _save_event_data():
-                    for key, cls in event_map.items():
-                        if key in body:
-                            cls.create_from_data(**body)
-
-                job_identifier = 'unknown job'
-                for key in event_map.keys():
-                    if key in body:
-                        job_identifier = body[key]
-                        break
-
-                if body.get('event') == 'EOF':
-                    try:
-                        final_counter = body.get('final_counter', 0)
-                        logger.info('Event processing is finished for Job {}, sending notifications'.format(job_identifier))
-                        # EOF events are sent when stdout for the running task is
-                        # closed. don't actually persist them to the database; we
-                        # just use them to report `summary` websocket events as an
-                        # approximation for when a job is "done"
-                        emit_channel_notification(
-                            'jobs-summary',
-                            dict(group_name='jobs', unified_job_id=job_identifier, final_counter=final_counter)
-                        )
-                        # Additionally, when we've processed all events, we should
-                        # have all the data we need to send out success/failure
-                        # notification templates
-                        uj = UnifiedJob.objects.get(pk=job_identifier)
-                        if hasattr(uj, 'send_notification_templates'):
-                            retries = 0
-                            while retries < 5:
-                                if uj.finished:
-                                    uj.send_notification_templates('succeeded' if uj.status == 'successful' else 'failed')
-                                    break
-                                else:
-                                    # wait a few seconds to avoid a race where the
-                                    # events are persisted _before_ the UJ.status
-                                    # changes from running -> successful
-                                    retries += 1
-                                    time.sleep(1)
-                                    uj = UnifiedJob.objects.get(pk=job_identifier)
-                    except Exception:
-                        logger.exception('Worker failed to emit notifications: Job {}'.format(job_identifier))
-                    continue
-
-                retries = 0
-                while retries <= self.MAX_RETRIES:
-                    try:
-                        _save_event_data()
-                        break
-                    except (OperationalError, InterfaceError, InternalError) as e:
-                        if retries >= self.MAX_RETRIES:
-                            logger.exception('Worker could not re-establish database connectivity, shutting down gracefully: Job {}'.format(job_identifier))
-                            os.kill(os.getppid(), signal.SIGINT)
-                            return
-                        delay = 60 * retries
-                        logger.exception('Database Error Saving Job Event, retry #{i} in {delay} seconds:'.format(
-                            i=retries + 1,
-                            delay=delay
-                        ))
-                        django_connection.close()
-                        time.sleep(delay)
-                        retries += 1
-                    except DatabaseError as e:
-                        logger.exception('Database Error Saving Job Event for Job {}'.format(job_identifier))
-                        break
-            except Exception as exc:
-                import traceback
-                tb = traceback.format_exc()
-                logger.error('Callback Task Processor Raised Exception: %r', exc)
-                logger.error('Detail: {}'.format(tb))
+from awx.main.dispatch.worker import AWXConsumer, CallbackBrokerWorker
 
 
 class Command(BaseCommand):
@@ -231,8 +18,22 @@ class Command(BaseCommand):
 
     def handle(self, *arg, **options):
         with Connection(settings.BROKER_URL) as conn:
+            consumer = None
             try:
-                worker = CallbackBrokerWorker(conn)
-                worker.run()
+                consumer = AWXConsumer(
+                    'callback_receiver',
+                    conn,
+                    CallbackBrokerWorker(),
+                    [
+                        Queue(
+                            settings.CALLBACK_QUEUE,
+                            Exchange(settings.CALLBACK_QUEUE, type='direct'),
+                            routing_key=settings.CALLBACK_QUEUE
+                        )
+                    ]
+                )
+                consumer.run()
             except KeyboardInterrupt:
                 print('Terminating Callback Receiver')
+                if consumer:
+                    consumer.stop()

awx/main/management/commands/run_dispatcher.py

@@ -0,0 +1,149 @@
+# Copyright (c) 2015 Ansible, Inc.
+# All Rights Reserved.
+import os
+import logging
+from multiprocessing import Process
+
+from django.conf import settings
+from django.core.cache import cache as django_cache
+from django.core.management.base import BaseCommand
+from django.db import connection as django_connection, connections
+from kombu import Connection, Exchange, Queue
+
+from awx.main.dispatch import get_local_queuename, reaper
+from awx.main.dispatch.control import Control
+from awx.main.dispatch.pool import AutoscalePool
+from awx.main.dispatch.worker import AWXConsumer, TaskWorker
+
+logger = logging.getLogger('awx.main.dispatch')
+
+
+def construct_bcast_queue_name(common_name):
+    return common_name + '_' + settings.CLUSTER_HOST_ID
+
+
+class Command(BaseCommand):
+    help = 'Launch the task dispatcher'
+
+    def add_arguments(self, parser):
+        parser.add_argument('--status', dest='status', action='store_true',
+                            help='print the internal state of any running dispatchers')
+        parser.add_argument('--running', dest='running', action='store_true',
+                            help='print the UUIDs of any tasked managed by this dispatcher')
+        parser.add_argument('--reload', dest='reload', action='store_true',
+                            help=('cause the dispatcher to recycle all of its worker processes;'
+                                  'running jobs will run to completion first'))
+
+    def beat(self):
+        from celery import Celery
+        from celery.beat import PersistentScheduler
+        from celery.apps import beat
+
+        class AWXScheduler(PersistentScheduler):
+
+            def __init__(self, *args, **kwargs):
+                self.ppid = os.getppid()
+                super(AWXScheduler, self).__init__(*args, **kwargs)
+
+            def setup_schedule(self):
+                super(AWXScheduler, self).setup_schedule()
+                self.update_from_dict(settings.CELERYBEAT_SCHEDULE)
+
+            def tick(self, *args, **kwargs):
+                if os.getppid() != self.ppid:
+                    # if the parent PID changes, this process has been orphaned
+                    # via e.g., segfault or sigkill, we should exit too
+                    raise SystemExit()
+                return super(AWXScheduler, self).tick(*args, **kwargs)
+
+            def apply_async(self, entry, producer=None, advance=True, **kwargs):
+                for conn in connections.all():
+                    # If the database connection has a hiccup, re-establish a new
+                    # connection
+                    conn.close_if_unusable_or_obsolete()
+                task = TaskWorker.resolve_callable(entry.task)
+                result, queue = task.apply_async()
+
+                class TaskResult(object):
+                    id = result['uuid']
+
+                return TaskResult()
+
+        sched_file = '/var/lib/awx/beat.db'
+        app = Celery()
+        app.conf.BROKER_URL = settings.BROKER_URL
+        app.conf.CELERY_TASK_RESULT_EXPIRES = False
+
+        # celery in py3 seems to have a bug where the celerybeat schedule
+        # shelve can become corrupted; we've _only_ seen this in Ubuntu and py36
+        # it can be avoided by detecting and removing the corrupted file
+        # at some point, we'll just stop using celerybeat, because it's clearly
+        # buggy, too -_-
+        #
+        # https://github.com/celery/celery/issues/4777
+        sched = AWXScheduler(schedule_filename=sched_file, app=app)
+        try:
+            sched.setup_schedule()
+        except Exception:
+            logger.exception('{} is corrupted, removing.'.format(sched_file))
+            sched._remove_db()
+        finally:
+            try:
+                sched.close()
+            except Exception:
+                logger.exception('{} failed to sync/close'.format(sched_file))
+
+        beat.Beat(
+            30,
+            app,
+            schedule=sched_file, scheduler_cls=AWXScheduler
+        ).run()
+
+    def handle(self, *arg, **options):
+        if options.get('status'):
+            print(Control('dispatcher').status())
+            return
+        if options.get('running'):
+            print(Control('dispatcher').running())
+            return
+        if options.get('reload'):
+            return Control('dispatcher').control({'control': 'reload'})
+
+        # It's important to close these because we're _about_ to fork, and we
+        # don't want the forked processes to inherit the open sockets
+        # for the DB and memcached connections (that way lies race conditions)
+        django_connection.close()
+        django_cache.close()
+        beat = Process(target=self.beat)
+        beat.daemon = True
+        beat.start()
+
+        reaper.reap()
+        consumer = None
+        with Connection(settings.BROKER_URL) as conn:
+            try:
+                bcast = 'tower_broadcast_all'
+                queues = [
+                    Queue(q, Exchange(q), routing_key=q)
+                    for q in (settings.AWX_CELERY_QUEUES_STATIC + [get_local_queuename()])
+                ]
+                queues.append(
+                    Queue(
+                        construct_bcast_queue_name(bcast),
+                        exchange=Exchange(bcast, type='fanout'),
+                        routing_key=bcast,
+                        reply=True
+                    )
+                )
+                consumer = AWXConsumer(
+                    'dispatcher',
+                    conn,
+                    TaskWorker(),
+                    queues,
+                    AutoscalePool(min_workers=4)
+                )
+                consumer.run()
+            except KeyboardInterrupt:
+                logger.debug('Terminating Task Dispatcher')
+                if consumer:
+                    consumer.stop()

awx/main/management/commands/watch_celery.py

@@ -1,66 +0,0 @@
-import datetime
-import os
-import signal
-import subprocess
-import sys
-import time
-
-from celery import Celery
-from django.core.management.base import BaseCommand
-from django.conf import settings
-
-
-class Command(BaseCommand):
-    """Watch local celery workers"""
-    help=("Sends a periodic ping to the local celery process over AMQP to ensure "
-          "it's responsive; this command is only intended to run in an environment "
-          "where celeryd is running")
-
-    #
-    # Just because celery is _running_ doesn't mean it's _working_; it's
-    # imperative that celery workers are _actually_ handling AMQP messages on
-    # their appropriate queues for awx to function.  Unfortunately, we've been
-    # plagued by a variety of bugs in celery that cause it to hang and become
-    # an unresponsive zombie, such as:
-    #
-    # https://github.com/celery/celery/issues/4185
-    # https://github.com/celery/celery/issues/4457
-    #
-    # The goal of this code is periodically send a broadcast AMQP message to
-    # the celery process on the local host via celery.app.control.ping;
-    # If that _fails_, we attempt to determine the pid of the celery process
-    # and send SIGHUP (which tends to resolve these sorts of issues for us).
-    #
-
-    INTERVAL = 60
-
-    def _log(self, msg):
-        sys.stderr.write(datetime.datetime.utcnow().isoformat())
-        sys.stderr.write(' ')
-        sys.stderr.write(msg)
-        sys.stderr.write('\n')
-
-    def handle(self, **options):
-        app = Celery('awx')
-        app.config_from_object('django.conf:settings')
-        while True:
-            try:
-                pongs = app.control.ping(['celery@{}'.format(settings.CLUSTER_HOST_ID)], timeout=30)
-            except Exception:
-                pongs = []
-            if not pongs:
-                self._log('celery is not responsive to ping over local AMQP')
-                pid = self.getpid()
-                if pid:
-                    self._log('sending SIGHUP to {}'.format(pid))
-                    os.kill(pid, signal.SIGHUP)
-            time.sleep(self.INTERVAL)
-
-    def getpid(self):
-        cmd = 'supervisorctl pid tower-processes:awx-celeryd'
-        if os.path.exists('/supervisor_task.conf'):
-            cmd = 'supervisorctl -c /supervisor_task.conf pid tower-processes:celery'
-        try:
-            return int(subprocess.check_output(cmd, shell=True))
-        except Exception:
-            self._log('could not detect celery pid')

awx/main/middleware.py

@@ -1,9 +1,9 @@
 # Copyright (c) 2015 Ansible, Inc.
 # All Rights Reserved.
 
+import uuid
 import logging
 import threading
-import uuid
 import six
 import time
 import cProfile
@@ -126,10 +126,12 @@ class SessionTimeoutMiddleware(object):
     """
 
     def process_response(self, request, response):
+        should_skip = 'HTTP_X_WS_SESSION_QUIET' in request.META
         req_session = getattr(request, 'session', None)
-        if req_session and not req_session.is_empty():
-                request.session.set_expiry(request.session.get_expiry_age())
-                response['Session-Timeout'] = int(settings.SESSION_COOKIE_AGE)
+        if req_session and not req_session.is_empty() and should_skip is False:
+            expiry = int(settings.SESSION_COOKIE_AGE)
+            request.session.set_expiry(expiry)
+            response['Session-Timeout'] = expiry
         return response
 
 

awx/main/migrations/0001_initial.py

@@ -27,7 +27,7 @@ class Migration(migrations.Migration):
             name='ActivityStream',
             fields=[
                 ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
-                ('operation', models.CharField(max_length=13, choices=[(b'create', 'Entity Created'), (b'update', 'Entity Updated'), (b'delete', 'Entity Deleted'), (b'associate', 'Entity Associated with another Entity'), (b'disassociate', 'Entity was Disassociated with another Entity')])),
+                ('operation', models.CharField(max_length=13, choices=[('create', 'Entity Created'), ('update', 'Entity Updated'), ('delete', 'Entity Deleted'), ('associate', 'Entity Associated with another Entity'), ('disassociate', 'Entity was Disassociated with another Entity')])),
                 ('timestamp', models.DateTimeField(auto_now_add=True)),
                 ('changes', models.TextField(blank=True)),
                 ('object_relationship_type', models.TextField(blank=True)),
@@ -42,8 +42,8 @@ class Migration(migrations.Migration):
                 ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                 ('created', models.DateTimeField(default=None, editable=False)),
                 ('modified', models.DateTimeField(default=None, editable=False)),
-                ('host_name', models.CharField(default=b'', max_length=1024, editable=False)),
-                ('event', models.CharField(max_length=100, choices=[(b'runner_on_failed', 'Host Failed'), (b'runner_on_ok', 'Host OK'), (b'runner_on_unreachable', 'Host Unreachable'), (b'runner_on_skipped', 'Host Skipped')])),
+                ('host_name', models.CharField(default='', max_length=1024, editable=False)),
+                ('event', models.CharField(max_length=100, choices=[('runner_on_failed', 'Host Failed'), ('runner_on_ok', 'Host OK'), ('runner_on_unreachable', 'Host Unreachable'), ('runner_on_skipped', 'Host Skipped')])),
                 ('event_data', jsonfield.fields.JSONField(default={}, blank=True)),
                 ('failed', models.BooleanField(default=False, editable=False)),
                 ('changed', models.BooleanField(default=False, editable=False)),
@@ -60,8 +60,8 @@ class Migration(migrations.Migration):
                 ('created', models.DateTimeField(auto_now_add=True)),
                 ('modified', models.DateTimeField(auto_now=True)),
                 ('expires', models.DateTimeField(default=django.utils.timezone.now)),
-                ('request_hash', models.CharField(default=b'', max_length=40, blank=True)),
-                ('reason', models.CharField(default=b'', help_text='Reason the auth token was invalidated.', max_length=1024, blank=True)),
+                ('request_hash', models.CharField(default='', max_length=40, blank=True)),
+                ('reason', models.CharField(default='', help_text='Reason the auth token was invalidated.', max_length=1024, blank=True)),
                 ('user', models.ForeignKey(related_name='auth_tokens', to=settings.AUTH_USER_MODEL)),
             ],
         ),
@@ -71,22 +71,22 @@ class Migration(migrations.Migration):
                 ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                 ('created', models.DateTimeField(default=None, editable=False)),
                 ('modified', models.DateTimeField(default=None, editable=False)),
-                ('description', models.TextField(default=b'', blank=True)),
+                ('description', models.TextField(default='', blank=True)),
                 ('active', models.BooleanField(default=True, editable=False)),
                 ('name', models.CharField(max_length=512)),
-                ('kind', models.CharField(default=b'ssh', max_length=32, choices=[(b'ssh', 'Machine'), (b'scm', 'Source Control'), (b'aws', 'Amazon Web Services'), (b'rax', 'Rackspace'), (b'vmware', 'VMware vCenter'), (b'gce', 'Google Compute Engine'), (b'azure', 'Microsoft Azure'), (b'openstack', 'OpenStack')])),
+                ('kind', models.CharField(default='ssh', max_length=32, choices=[('ssh', 'Machine'), ('scm', 'Source Control'), ('aws', 'Amazon Web Services'), ('rax', 'Rackspace'), ('vmware', 'VMware vCenter'), ('gce', 'Google Compute Engine'), ('azure', 'Microsoft Azure'), ('openstack', 'OpenStack')])),
                 ('cloud', models.BooleanField(default=False, editable=False)),
-                ('host', models.CharField(default=b'', help_text='The hostname or IP address to use.', max_length=1024, verbose_name='Host', blank=True)),
-                ('username', models.CharField(default=b'', help_text='Username for this credential.', max_length=1024, verbose_name='Username', blank=True)),
-                ('password', models.CharField(default=b'', help_text='Password for this credential (or "ASK" to prompt the user for machine credentials).', max_length=1024, verbose_name='Password', blank=True)),
-                ('security_token', models.CharField(default=b'', help_text='Security Token for this credential', max_length=1024, verbose_name='Security Token', blank=True)),
-                ('project', models.CharField(default=b'', help_text='The identifier for the project.', max_length=100, verbose_name='Project', blank=True)),
-                ('ssh_key_data', models.TextField(default=b'', help_text='RSA or DSA private key to be used instead of password.', verbose_name='SSH private key', blank=True)),
-                ('ssh_key_unlock', models.CharField(default=b'', help_text='Passphrase to unlock SSH private key if encrypted (or "ASK" to prompt the user for machine credentials).', max_length=1024, verbose_name='SSH key unlock', blank=True)),
-                ('become_method', models.CharField(default=b'', help_text='Privilege escalation method.', max_length=32, blank=True, choices=[(b'', 'None'), (b'sudo', 'Sudo'), (b'su', 'Su'), (b'pbrun', 'Pbrun'), (b'pfexec', 'Pfexec')])),
-                ('become_username', models.CharField(default=b'', help_text='Privilege escalation username.', max_length=1024, blank=True)),
-                ('become_password', models.CharField(default=b'', help_text='Password for privilege escalation method.', max_length=1024, blank=True)),
-                ('vault_password', models.CharField(default=b'', help_text='Vault password (or "ASK" to prompt the user).', max_length=1024, blank=True)),
+                ('host', models.CharField(default='', help_text='The hostname or IP address to use.', max_length=1024, verbose_name='Host', blank=True)),
+                ('username', models.CharField(default='', help_text='Username for this credential.', max_length=1024, verbose_name='Username', blank=True)),
+                ('password', models.CharField(default='', help_text='Password for this credential (or "ASK" to prompt the user for machine credentials).', max_length=1024, verbose_name='Password', blank=True)),
+                ('security_token', models.CharField(default='', help_text='Security Token for this credential', max_length=1024, verbose_name='Security Token', blank=True)),
+                ('project', models.CharField(default='', help_text='The identifier for the project.', max_length=100, verbose_name='Project', blank=True)),
+                ('ssh_key_data', models.TextField(default='', help_text='RSA or DSA private key to be used instead of password.', verbose_name='SSH private key', blank=True)),
+                ('ssh_key_unlock', models.CharField(default='', help_text='Passphrase to unlock SSH private key if encrypted (or "ASK" to prompt the user for machine credentials).', max_length=1024, verbose_name='SSH key unlock', blank=True)),
+                ('become_method', models.CharField(default='', help_text='Privilege escalation method.', max_length=32, blank=True, choices=[('', 'None'), ('sudo', 'Sudo'), ('su', 'Su'), ('pbrun', 'Pbrun'), ('pfexec', 'Pfexec')])),
+                ('become_username', models.CharField(default='', help_text='Privilege escalation username.', max_length=1024, blank=True)),
+                ('become_password', models.CharField(default='', help_text='Password for privilege escalation method.', max_length=1024, blank=True)),
+                ('vault_password', models.CharField(default='', help_text='Vault password (or "ASK" to prompt the user).', max_length=1024, blank=True)),
                 ('created_by', models.ForeignKey(related_name="{u'class': 'credential', u'app_label': 'main'}(class)s_created+", on_delete=django.db.models.deletion.SET_NULL, default=None, editable=False, to=settings.AUTH_USER_MODEL, null=True)),
                 ('modified_by', models.ForeignKey(related_name="{u'class': 'credential', u'app_label': 'main'}(class)s_modified+", on_delete=django.db.models.deletion.SET_NULL, default=None, editable=False, to=settings.AUTH_USER_MODEL, null=True)),
                 ('tags', taggit.managers.TaggableManager(to='taggit.Tag', through='taggit.TaggedItem', blank=True, help_text='A comma-separated list of tags.', verbose_name='Tags')),
@@ -101,10 +101,10 @@ class Migration(migrations.Migration):
                 ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                 ('created', models.DateTimeField(default=None, editable=False)),
                 ('modified', models.DateTimeField(default=None, editable=False)),
-                ('description', models.TextField(default=b'', blank=True)),
+                ('description', models.TextField(default='', blank=True)),
                 ('active', models.BooleanField(default=True, editable=False)),
                 ('name', models.CharField(max_length=512)),
-                ('script', models.TextField(default=b'', help_text='Inventory script contents', blank=True)),
+                ('script', models.TextField(default='', help_text='Inventory script contents', blank=True)),
                 ('created_by', models.ForeignKey(related_name="{u'class': 'custominventoryscript', u'app_label': 'main'}(class)s_created+", on_delete=django.db.models.deletion.SET_NULL, default=None, editable=False, to=settings.AUTH_USER_MODEL, null=True)),
                 ('modified_by', models.ForeignKey(related_name="{u'class': 'custominventoryscript', u'app_label': 'main'}(class)s_modified+", on_delete=django.db.models.deletion.SET_NULL, default=None, editable=False, to=settings.AUTH_USER_MODEL, null=True)),
             ],
@@ -118,10 +118,10 @@ class Migration(migrations.Migration):
                 ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                 ('created', models.DateTimeField(default=None, editable=False)),
                 ('modified', models.DateTimeField(default=None, editable=False)),
-                ('description', models.TextField(default=b'', blank=True)),
+                ('description', models.TextField(default='', blank=True)),
                 ('active', models.BooleanField(default=True, editable=False)),
                 ('name', models.CharField(max_length=512)),
-                ('variables', models.TextField(default=b'', help_text='Group variables in JSON or YAML format.', blank=True)),
+                ('variables', models.TextField(default='', help_text='Group variables in JSON or YAML format.', blank=True)),
                 ('total_hosts', models.PositiveIntegerField(default=0, help_text='Total number of hosts directly or indirectly in this group.', editable=False)),
                 ('has_active_failures', models.BooleanField(default=False, help_text='Flag indicating whether this group has any hosts with active failures.', editable=False)),
                 ('hosts_with_active_failures', models.PositiveIntegerField(default=0, help_text='Number of hosts in this group with active failures.', editable=False)),
@@ -140,12 +140,12 @@ class Migration(migrations.Migration):
                 ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                 ('created', models.DateTimeField(default=None, editable=False)),
                 ('modified', models.DateTimeField(default=None, editable=False)),
-                ('description', models.TextField(default=b'', blank=True)),
+                ('description', models.TextField(default='', blank=True)),
                 ('active', models.BooleanField(default=True, editable=False)),
                 ('name', models.CharField(max_length=512)),
                 ('enabled', models.BooleanField(default=True, help_text='Is this host online and available for running jobs?')),
-                ('instance_id', models.CharField(default=b'', max_length=100, blank=True)),
-                ('variables', models.TextField(default=b'', help_text='Host variables in JSON or YAML format.', blank=True)),
+                ('instance_id', models.CharField(default='', max_length=100, blank=True)),
+                ('variables', models.TextField(default='', help_text='Host variables in JSON or YAML format.', blank=True)),
                 ('has_active_failures', models.BooleanField(default=False, help_text='Flag indicating whether the last job failed for this host.', editable=False)),
                 ('has_inventory_sources', models.BooleanField(default=False, help_text='Flag indicating whether this host was created/updated from any external inventory sources.', editable=False)),
                 ('created_by', models.ForeignKey(related_name="{u'class': 'host', u'app_label': 'main'}(class)s_created+", on_delete=django.db.models.deletion.SET_NULL, default=None, editable=False, to=settings.AUTH_USER_MODEL, null=True)),
@@ -171,10 +171,10 @@ class Migration(migrations.Migration):
                 ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                 ('created', models.DateTimeField(default=None, editable=False)),
                 ('modified', models.DateTimeField(default=None, editable=False)),
-                ('description', models.TextField(default=b'', blank=True)),
+                ('description', models.TextField(default='', blank=True)),
                 ('active', models.BooleanField(default=True, editable=False)),
                 ('name', models.CharField(unique=True, max_length=512)),
-                ('variables', models.TextField(default=b'', help_text='Inventory variables in JSON or YAML format.', blank=True)),
+                ('variables', models.TextField(default='', help_text='Inventory variables in JSON or YAML format.', blank=True)),
                 ('has_active_failures', models.BooleanField(default=False, help_text='Flag indicating whether any hosts in this inventory have failed.', editable=False)),
                 ('total_hosts', models.PositiveIntegerField(default=0, help_text='Total number of hosts in this inventory.', editable=False)),
                 ('hosts_with_active_failures', models.PositiveIntegerField(default=0, help_text='Number of hosts in this inventory with active failures.', editable=False)),
@@ -197,14 +197,14 @@ class Migration(migrations.Migration):
                 ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                 ('created', models.DateTimeField(default=None, editable=False)),
                 ('modified', models.DateTimeField(default=None, editable=False)),
-                ('event', models.CharField(max_length=100, choices=[(b'runner_on_failed', 'Host Failed'), (b'runner_on_ok', 'Host OK'), (b'runner_on_error', 'Host Failure'), (b'runner_on_skipped', 'Host Skipped'), (b'runner_on_unreachable', 'Host Unreachable'), (b'runner_on_no_hosts', 'No Hosts Remaining'), (b'runner_on_async_poll', 'Host Polling'), (b'runner_on_async_ok', 'Host Async OK'), (b'runner_on_async_failed', 'Host Async Failure'), (b'runner_on_file_diff', 'File Difference'), (b'playbook_on_start', 'Playbook Started'), (b'playbook_on_notify', 'Running Handlers'), (b'playbook_on_no_hosts_matched', 'No Hosts Matched'), (b'playbook_on_no_hosts_remaining', 'No Hosts Remaining'), (b'playbook_on_task_start', 'Task Started'), (b'playbook_on_vars_prompt', 'Variables Prompted'), (b'playbook_on_setup', 'Gathering Facts'), (b'playbook_on_import_for_host', 'internal: on Import for Host'), (b'playbook_on_not_import_for_host', 'internal: on Not Import for Host'), (b'playbook_on_play_start', 'Play Started'), (b'playbook_on_stats', 'Playbook Complete')])),
+                ('event', models.CharField(max_length=100, choices=[('runner_on_failed', 'Host Failed'), ('runner_on_ok', 'Host OK'), ('runner_on_error', 'Host Failure'), ('runner_on_skipped', 'Host Skipped'), ('runner_on_unreachable', 'Host Unreachable'), ('runner_on_no_hosts', 'No Hosts Remaining'), ('runner_on_async_poll', 'Host Polling'), ('runner_on_async_ok', 'Host Async OK'), ('runner_on_async_failed', 'Host Async Failure'), ('runner_on_file_diff', 'File Difference'), ('playbook_on_start', 'Playbook Started'), ('playbook_on_notify', 'Running Handlers'), ('playbook_on_no_hosts_matched', 'No Hosts Matched'), ('playbook_on_no_hosts_remaining', 'No Hosts Remaining'), ('playbook_on_task_start', 'Task Started'), ('playbook_on_vars_prompt', 'Variables Prompted'), ('playbook_on_setup', 'Gathering Facts'), ('playbook_on_import_for_host', 'internal: on Import for Host'), ('playbook_on_not_import_for_host', 'internal: on Not Import for Host'), ('playbook_on_play_start', 'Play Started'), ('playbook_on_stats', 'Playbook Complete')])),
                 ('event_data', jsonfield.fields.JSONField(default={}, blank=True)),
                 ('failed', models.BooleanField(default=False, editable=False)),
                 ('changed', models.BooleanField(default=False, editable=False)),
-                ('host_name', models.CharField(default=b'', max_length=1024, editable=False)),
-                ('play', models.CharField(default=b'', max_length=1024, editable=False)),
-                ('role', models.CharField(default=b'', max_length=1024, editable=False)),
-                ('task', models.CharField(default=b'', max_length=1024, editable=False)),
+                ('host_name', models.CharField(default='', max_length=1024, editable=False)),
+                ('play', models.CharField(default='', max_length=1024, editable=False)),
+                ('role', models.CharField(default='', max_length=1024, editable=False)),
+                ('task', models.CharField(default='', max_length=1024, editable=False)),
                 ('counter', models.PositiveIntegerField(default=0)),
                 ('host', models.ForeignKey(related_name='job_events_as_primary_host', on_delete=django.db.models.deletion.SET_NULL, default=None, editable=False, to='main.Host', null=True)),
                 ('hosts', models.ManyToManyField(related_name='job_events', editable=False, to='main.Host')),
@@ -220,7 +220,7 @@ class Migration(migrations.Migration):
                 ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                 ('created', models.DateTimeField(default=None, editable=False)),
                 ('modified', models.DateTimeField(default=None, editable=False)),
-                ('host_name', models.CharField(default=b'', max_length=1024, editable=False)),
+                ('host_name', models.CharField(default='', max_length=1024, editable=False)),
                 ('changed', models.PositiveIntegerField(default=0, editable=False)),
                 ('dark', models.PositiveIntegerField(default=0, editable=False)),
                 ('failures', models.PositiveIntegerField(default=0, editable=False)),
@@ -250,7 +250,7 @@ class Migration(migrations.Migration):
                 ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                 ('created', models.DateTimeField(default=None, editable=False)),
                 ('modified', models.DateTimeField(default=None, editable=False)),
-                ('description', models.TextField(default=b'', blank=True)),
+                ('description', models.TextField(default='', blank=True)),
                 ('active', models.BooleanField(default=True, editable=False)),
                 ('name', models.CharField(unique=True, max_length=512)),
                 ('admins', models.ManyToManyField(related_name='admin_of_organizations', to=settings.AUTH_USER_MODEL, blank=True)),
@@ -269,10 +269,10 @@ class Migration(migrations.Migration):
                 ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                 ('created', models.DateTimeField(default=None, editable=False)),
                 ('modified', models.DateTimeField(default=None, editable=False)),
-                ('description', models.TextField(default=b'', blank=True)),
+                ('description', models.TextField(default='', blank=True)),
                 ('active', models.BooleanField(default=True, editable=False)),
                 ('name', models.CharField(max_length=512)),
-                ('permission_type', models.CharField(max_length=64, choices=[(b'read', 'Read Inventory'), (b'write', 'Edit Inventory'), (b'admin', 'Administrate Inventory'), (b'run', 'Deploy To Inventory'), (b'check', 'Deploy To Inventory (Dry Run)'), (b'scan', 'Scan an Inventory'), (b'create', 'Create a Job Template')])),
+                ('permission_type', models.CharField(max_length=64, choices=[('read', 'Read Inventory'), ('write', 'Edit Inventory'), ('admin', 'Administrate Inventory'), ('run', 'Deploy To Inventory'), ('check', 'Deploy To Inventory (Dry Run)'), ('scan', 'Scan an Inventory'), ('create', 'Create a Job Template')])),
                 ('run_ad_hoc_commands', models.BooleanField(default=False, help_text='Execute Commands on the Inventory')),
                 ('created_by', models.ForeignKey(related_name="{u'class': 'permission', u'app_label': 'main'}(class)s_created+", on_delete=django.db.models.deletion.SET_NULL, default=None, editable=False, to=settings.AUTH_USER_MODEL, null=True)),
                 ('inventory', models.ForeignKey(related_name='permissions', on_delete=django.db.models.deletion.SET_NULL, to='main.Inventory', null=True)),
@@ -286,7 +286,7 @@ class Migration(migrations.Migration):
                 ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                 ('created', models.DateTimeField(default=None, editable=False)),
                 ('modified', models.DateTimeField(default=None, editable=False)),
-                ('ldap_dn', models.CharField(default=b'', max_length=1024)),
+                ('ldap_dn', models.CharField(default='', max_length=1024)),
                 ('user', awx.main.fields.AutoOneToOneField(related_name='profile', editable=False, to=settings.AUTH_USER_MODEL)),
             ],
         ),
@@ -296,7 +296,7 @@ class Migration(migrations.Migration):
                 ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                 ('created', models.DateTimeField(default=None, editable=False)),
                 ('modified', models.DateTimeField(default=None, editable=False)),
-                ('description', models.TextField(default=b'', blank=True)),
+                ('description', models.TextField(default='', blank=True)),
                 ('active', models.BooleanField(default=True, editable=False)),
                 ('name', models.CharField(unique=True, max_length=512)),
                 ('enabled', models.BooleanField(default=True)),
@@ -319,7 +319,7 @@ class Migration(migrations.Migration):
                 ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                 ('created', models.DateTimeField(default=None, editable=False)),
                 ('modified', models.DateTimeField(default=None, editable=False)),
-                ('description', models.TextField(default=b'', blank=True)),
+                ('description', models.TextField(default='', blank=True)),
                 ('active', models.BooleanField(default=True, editable=False)),
                 ('name', models.CharField(max_length=512)),
                 ('created_by', models.ForeignKey(related_name="{u'class': 'team', u'app_label': 'main'}(class)s_created+", on_delete=django.db.models.deletion.SET_NULL, default=None, editable=False, to=settings.AUTH_USER_MODEL, null=True)),
@@ -338,26 +338,26 @@ class Migration(migrations.Migration):
                 ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                 ('created', models.DateTimeField(default=None, editable=False)),
                 ('modified', models.DateTimeField(default=None, editable=False)),
-                ('description', models.TextField(default=b'', blank=True)),
+                ('description', models.TextField(default='', blank=True)),
                 ('active', models.BooleanField(default=True, editable=False)),
                 ('name', models.CharField(max_length=512)),
                 ('old_pk', models.PositiveIntegerField(default=None, null=True, editable=False)),
-                ('launch_type', models.CharField(default=b'manual', max_length=20, editable=False, choices=[(b'manual', 'Manual'), (b'relaunch', 'Relaunch'), (b'callback', 'Callback'), (b'scheduled', 'Scheduled'), (b'dependency', 'Dependency')])),
+                ('launch_type', models.CharField(default='manual', max_length=20, editable=False, choices=[('manual', 'Manual'), ('relaunch', 'Relaunch'), ('callback', 'Callback'), ('scheduled', 'Scheduled'), ('dependency', 'Dependency')])),
                 ('cancel_flag', models.BooleanField(default=False, editable=False)),
-                ('status', models.CharField(default=b'new', max_length=20, editable=False, choices=[(b'new', 'New'), (b'pending', 'Pending'), (b'waiting', 'Waiting'), (b'running', 'Running'), (b'successful', 'Successful'), (b'failed', 'Failed'), (b'error', 'Error'), (b'canceled', 'Canceled')])),
+                ('status', models.CharField(default='new', max_length=20, editable=False, choices=[('new', 'New'), ('pending', 'Pending'), ('waiting', 'Waiting'), ('running', 'Running'), ('successful', 'Successful'), ('failed', 'Failed'), ('error', 'Error'), ('canceled', 'Canceled')])),
                 ('failed', models.BooleanField(default=False, editable=False)),
                 ('started', models.DateTimeField(default=None, null=True, editable=False)),
                 ('finished', models.DateTimeField(default=None, null=True, editable=False)),
                 ('elapsed', models.DecimalField(editable=False, max_digits=12, decimal_places=3)),
-                ('job_args', models.TextField(default=b'', editable=False, blank=True)),
-                ('job_cwd', models.CharField(default=b'', max_length=1024, editable=False, blank=True)),
+                ('job_args', models.TextField(default='', editable=False, blank=True)),
+                ('job_cwd', models.CharField(default='', max_length=1024, editable=False, blank=True)),
                 ('job_env', jsonfield.fields.JSONField(default={}, editable=False, blank=True)),
-                ('job_explanation', models.TextField(default=b'', editable=False, blank=True)),
-                ('start_args', models.TextField(default=b'', editable=False, blank=True)),
-                ('result_stdout_text', models.TextField(default=b'', editable=False, blank=True)),
-                ('result_stdout_file', models.TextField(default=b'', editable=False, blank=True)),
-                ('result_traceback', models.TextField(default=b'', editable=False, blank=True)),
-                ('celery_task_id', models.CharField(default=b'', max_length=100, editable=False, blank=True)),
+                ('job_explanation', models.TextField(default='', editable=False, blank=True)),
+                ('start_args', models.TextField(default='', editable=False, blank=True)),
+                ('result_stdout_text', models.TextField(default='', editable=False, blank=True)),
+                ('result_stdout_file', models.TextField(default='', editable=False, blank=True)),
+                ('result_traceback', models.TextField(default='', editable=False, blank=True)),
+                ('celery_task_id', models.CharField(default='', max_length=100, editable=False, blank=True)),
             ],
         ),
         migrations.CreateModel(
@@ -366,7 +366,7 @@ class Migration(migrations.Migration):
                 ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                 ('created', models.DateTimeField(default=None, editable=False)),
                 ('modified', models.DateTimeField(default=None, editable=False)),
-                ('description', models.TextField(default=b'', blank=True)),
+                ('description', models.TextField(default='', blank=True)),
                 ('active', models.BooleanField(default=True, editable=False)),
                 ('name', models.CharField(max_length=512)),
                 ('old_pk', models.PositiveIntegerField(default=None, null=True, editable=False)),
@@ -374,19 +374,19 @@ class Migration(migrations.Migration):
                 ('last_job_run', models.DateTimeField(default=None, null=True, editable=False)),
                 ('has_schedules', models.BooleanField(default=False, editable=False)),
                 ('next_job_run', models.DateTimeField(default=None, null=True, editable=False)),
-                ('status', models.CharField(default=b'ok', max_length=32, editable=False, choices=[(b'new', 'New'), (b'pending', 'Pending'), (b'waiting', 'Waiting'), (b'running', 'Running'), (b'successful', 'Successful'), (b'failed', 'Failed'), (b'error', 'Error'), (b'canceled', 'Canceled'), (b'never updated', b'Never Updated'), (b'ok', b'OK'), (b'missing', b'Missing'), (b'none', 'No External Source'), (b'updating', 'Updating')])),
+                ('status', models.CharField(default='ok', max_length=32, editable=False, choices=[('new', 'New'), ('pending', 'Pending'), ('waiting', 'Waiting'), ('running', 'Running'), ('successful', 'Successful'), ('failed', 'Failed'), ('error', 'Error'), ('canceled', 'Canceled'), ('never updated', 'Never Updated'), ('ok', 'OK'), ('missing', 'Missing'), ('none', 'No External Source'), ('updating', 'Updating')])),
             ],
         ),
         migrations.CreateModel(
             name='AdHocCommand',
             fields=[
                 ('unifiedjob_ptr', models.OneToOneField(parent_link=True, auto_created=True, primary_key=True, serialize=False, to='main.UnifiedJob')),
-                ('job_type', models.CharField(default=b'run', max_length=64, choices=[(b'run', 'Run'), (b'check', 'Check')])),
-                ('limit', models.CharField(default=b'', max_length=1024, blank=True)),
-                ('module_name', models.CharField(default=b'', max_length=1024, blank=True)),
-                ('module_args', models.TextField(default=b'', blank=True)),
+                ('job_type', models.CharField(default='run', max_length=64, choices=[('run', 'Run'), ('check', 'Check')])),
+                ('limit', models.CharField(default='', max_length=1024, blank=True)),
+                ('module_name', models.CharField(default='', max_length=1024, blank=True)),
+                ('module_args', models.TextField(default='', blank=True)),
                 ('forks', models.PositiveIntegerField(default=0, blank=True)),
-                ('verbosity', models.PositiveIntegerField(default=0, blank=True, choices=[(0, b'0 (Normal)'), (1, b'1 (Verbose)'), (2, b'2 (More Verbose)'), (3, b'3 (Debug)'), (4, b'4 (Connection Debug)'), (5, b'5 (WinRM Debug)')])),
+                ('verbosity', models.PositiveIntegerField(default=0, blank=True, choices=[(0, '0 (Normal)'), (1, '1 (Verbose)'), (2, '2 (More Verbose)'), (3, '3 (Debug)'), (4, '4 (Connection Debug)'), (5, '5 (WinRM Debug)')])),
                 ('become_enabled', models.BooleanField(default=False)),
             ],
             bases=('main.unifiedjob',),
@@ -395,12 +395,12 @@ class Migration(migrations.Migration):
             name='InventorySource',
             fields=[
                 ('unifiedjobtemplate_ptr', models.OneToOneField(parent_link=True, auto_created=True, primary_key=True, serialize=False, to='main.UnifiedJobTemplate')),
-                ('source', models.CharField(default=b'', max_length=32, blank=True, choices=[(b'', 'Manual'), (b'file', 'Local File, Directory or Script'), (b'rax', 'Rackspace Cloud Servers'), (b'ec2', 'Amazon EC2'), (b'gce', 'Google Compute Engine'), (b'azure', 'Microsoft Azure'), (b'vmware', 'VMware vCenter'), (b'openstack', 'OpenStack'), (b'custom', 'Custom Script')])),
-                ('source_path', models.CharField(default=b'', max_length=1024, editable=False, blank=True)),
-                ('source_vars', models.TextField(default=b'', help_text='Inventory source variables in YAML or JSON format.', blank=True)),
-                ('source_regions', models.CharField(default=b'', max_length=1024, blank=True)),
-                ('instance_filters', models.CharField(default=b'', help_text='Comma-separated list of filter expressions (EC2 only). Hosts are imported when ANY of the filters match.', max_length=1024, blank=True)),
-                ('group_by', models.CharField(default=b'', help_text='Limit groups automatically created from inventory source (EC2 only).', max_length=1024, blank=True)),
+                ('source', models.CharField(default='', max_length=32, blank=True, choices=[('', 'Manual'), ('file', 'Local File, Directory or Script'), ('rax', 'Rackspace Cloud Servers'), ('ec2', 'Amazon EC2'), ('gce', 'Google Compute Engine'), ('azure', 'Microsoft Azure'), ('vmware', 'VMware vCenter'), ('openstack', 'OpenStack'), ('custom', 'Custom Script')])),
+                ('source_path', models.CharField(default='', max_length=1024, editable=False, blank=True)),
+                ('source_vars', models.TextField(default='', help_text='Inventory source variables in YAML or JSON format.', blank=True)),
+                ('source_regions', models.CharField(default='', max_length=1024, blank=True)),
+                ('instance_filters', models.CharField(default='', help_text='Comma-separated list of filter expressions (EC2 only). Hosts are imported when ANY of the filters match.', max_length=1024, blank=True)),
+                ('group_by', models.CharField(default='', help_text='Limit groups automatically created from inventory source (EC2 only).', max_length=1024, blank=True)),
                 ('overwrite', models.BooleanField(default=False, help_text='Overwrite local groups and hosts from remote inventory source.')),
                 ('overwrite_vars', models.BooleanField(default=False, help_text='Overwrite local variables from remote inventory source.')),
                 ('update_on_launch', models.BooleanField(default=False)),
@@ -412,12 +412,12 @@ class Migration(migrations.Migration):
             name='InventoryUpdate',
             fields=[
                 ('unifiedjob_ptr', models.OneToOneField(parent_link=True, auto_created=True, primary_key=True, serialize=False, to='main.UnifiedJob')),
-                ('source', models.CharField(default=b'', max_length=32, blank=True, choices=[(b'', 'Manual'), (b'file', 'Local File, Directory or Script'), (b'rax', 'Rackspace Cloud Servers'), (b'ec2', 'Amazon EC2'), (b'gce', 'Google Compute Engine'), (b'azure', 'Microsoft Azure'), (b'vmware', 'VMware vCenter'), (b'openstack', 'OpenStack'), (b'custom', 'Custom Script')])),
-                ('source_path', models.CharField(default=b'', max_length=1024, editable=False, blank=True)),
-                ('source_vars', models.TextField(default=b'', help_text='Inventory source variables in YAML or JSON format.', blank=True)),
-                ('source_regions', models.CharField(default=b'', max_length=1024, blank=True)),
-                ('instance_filters', models.CharField(default=b'', help_text='Comma-separated list of filter expressions (EC2 only). Hosts are imported when ANY of the filters match.', max_length=1024, blank=True)),
-                ('group_by', models.CharField(default=b'', help_text='Limit groups automatically created from inventory source (EC2 only).', max_length=1024, blank=True)),
+                ('source', models.CharField(default='', max_length=32, blank=True, choices=[('', 'Manual'), ('file', 'Local File, Directory or Script'), ('rax', 'Rackspace Cloud Servers'), ('ec2', 'Amazon EC2'), ('gce', 'Google Compute Engine'), ('azure', 'Microsoft Azure'), ('vmware', 'VMware vCenter'), ('openstack', 'OpenStack'), ('custom', 'Custom Script')])),
+                ('source_path', models.CharField(default='', max_length=1024, editable=False, blank=True)),
+                ('source_vars', models.TextField(default='', help_text='Inventory source variables in YAML or JSON format.', blank=True)),
+                ('source_regions', models.CharField(default='', max_length=1024, blank=True)),
+                ('instance_filters', models.CharField(default='', help_text='Comma-separated list of filter expressions (EC2 only). Hosts are imported when ANY of the filters match.', max_length=1024, blank=True)),
+                ('group_by', models.CharField(default='', help_text='Limit groups automatically created from inventory source (EC2 only).', max_length=1024, blank=True)),
                 ('overwrite', models.BooleanField(default=False, help_text='Overwrite local groups and hosts from remote inventory source.')),
                 ('overwrite_vars', models.BooleanField(default=False, help_text='Overwrite local variables from remote inventory source.')),
                 ('license_error', models.BooleanField(default=False, editable=False)),
@@ -428,16 +428,16 @@ class Migration(migrations.Migration):
             name='Job',
             fields=[
                 ('unifiedjob_ptr', models.OneToOneField(parent_link=True, auto_created=True, primary_key=True, serialize=False, to='main.UnifiedJob')),
-                ('job_type', models.CharField(default=b'run', max_length=64, choices=[(b'run', 'Run'), (b'check', 'Check'), (b'scan', 'Scan')])),
-                ('playbook', models.CharField(default=b'', max_length=1024, blank=True)),
+                ('job_type', models.CharField(default='run', max_length=64, choices=[('run', 'Run'), ('check', 'Check'), ('scan', 'Scan')])),
+                ('playbook', models.CharField(default='', max_length=1024, blank=True)),
                 ('forks', models.PositiveIntegerField(default=0, blank=True)),
-                ('limit', models.CharField(default=b'', max_length=1024, blank=True)),
-                ('verbosity', models.PositiveIntegerField(default=0, blank=True, choices=[(0, b'0 (Normal)'), (1, b'1 (Verbose)'), (2, b'2 (More Verbose)'), (3, b'3 (Debug)'), (4, b'4 (Connection Debug)'), (5, b'5 (WinRM Debug)')])),
-                ('extra_vars', models.TextField(default=b'', blank=True)),
-                ('job_tags', models.CharField(default=b'', max_length=1024, blank=True)),
+                ('limit', models.CharField(default='', max_length=1024, blank=True)),
+                ('verbosity', models.PositiveIntegerField(default=0, blank=True, choices=[(0, '0 (Normal)'), (1, '1 (Verbose)'), (2, '2 (More Verbose)'), (3, '3 (Debug)'), (4, '4 (Connection Debug)'), (5, '5 (WinRM Debug)')])),
+                ('extra_vars', models.TextField(default='', blank=True)),
+                ('job_tags', models.CharField(default='', max_length=1024, blank=True)),
                 ('force_handlers', models.BooleanField(default=False)),
-                ('skip_tags', models.CharField(default=b'', max_length=1024, blank=True)),
-                ('start_at_task', models.CharField(default=b'', max_length=1024, blank=True)),
+                ('skip_tags', models.CharField(default='', max_length=1024, blank=True)),
+                ('start_at_task', models.CharField(default='', max_length=1024, blank=True)),
                 ('become_enabled', models.BooleanField(default=False)),
             ],
             options={
@@ -449,18 +449,18 @@ class Migration(migrations.Migration):
             name='JobTemplate',
             fields=[
                 ('unifiedjobtemplate_ptr', models.OneToOneField(parent_link=True, auto_created=True, primary_key=True, serialize=False, to='main.UnifiedJobTemplate')),
-                ('job_type', models.CharField(default=b'run', max_length=64, choices=[(b'run', 'Run'), (b'check', 'Check'), (b'scan', 'Scan')])),
-                ('playbook', models.CharField(default=b'', max_length=1024, blank=True)),
+                ('job_type', models.CharField(default='run', max_length=64, choices=[('run', 'Run'), ('check', 'Check'), ('scan', 'Scan')])),
+                ('playbook', models.CharField(default='', max_length=1024, blank=True)),
                 ('forks', models.PositiveIntegerField(default=0, blank=True)),
-                ('limit', models.CharField(default=b'', max_length=1024, blank=True)),
-                ('verbosity', models.PositiveIntegerField(default=0, blank=True, choices=[(0, b'0 (Normal)'), (1, b'1 (Verbose)'), (2, b'2 (More Verbose)'), (3, b'3 (Debug)'), (4, b'4 (Connection Debug)'), (5, b'5 (WinRM Debug)')])),
-                ('extra_vars', models.TextField(default=b'', blank=True)),
-                ('job_tags', models.CharField(default=b'', max_length=1024, blank=True)),
+                ('limit', models.CharField(default='', max_length=1024, blank=True)),
+                ('verbosity', models.PositiveIntegerField(default=0, blank=True, choices=[(0, '0 (Normal)'), (1, '1 (Verbose)'), (2, '2 (More Verbose)'), (3, '3 (Debug)'), (4, '4 (Connection Debug)'), (5, '5 (WinRM Debug)')])),
+                ('extra_vars', models.TextField(default='', blank=True)),
+                ('job_tags', models.CharField(default='', max_length=1024, blank=True)),
                 ('force_handlers', models.BooleanField(default=False)),
-                ('skip_tags', models.CharField(default=b'', max_length=1024, blank=True)),
-                ('start_at_task', models.CharField(default=b'', max_length=1024, blank=True)),
+                ('skip_tags', models.CharField(default='', max_length=1024, blank=True)),
+                ('start_at_task', models.CharField(default='', max_length=1024, blank=True)),
                 ('become_enabled', models.BooleanField(default=False)),
-                ('host_config_key', models.CharField(default=b'', max_length=1024, blank=True)),
+                ('host_config_key', models.CharField(default='', max_length=1024, blank=True)),
                 ('ask_variables_on_launch', models.BooleanField(default=False)),
                 ('survey_enabled', models.BooleanField(default=False)),
                 ('survey_spec', jsonfield.fields.JSONField(default={}, blank=True)),
@@ -475,9 +475,9 @@ class Migration(migrations.Migration):
             fields=[
                 ('unifiedjobtemplate_ptr', models.OneToOneField(parent_link=True, auto_created=True, primary_key=True, serialize=False, to='main.UnifiedJobTemplate')),
                 ('local_path', models.CharField(help_text='Local path (relative to PROJECTS_ROOT) containing playbooks and related files for this project.', max_length=1024, blank=True)),
-                ('scm_type', models.CharField(default=b'', max_length=8, verbose_name='SCM Type', blank=True, choices=[(b'', 'Manual'), (b'git', 'Git'), (b'hg', 'Mercurial'), (b'svn', 'Subversion')])),
-                ('scm_url', models.CharField(default=b'', max_length=1024, verbose_name='SCM URL', blank=True)),
-                ('scm_branch', models.CharField(default=b'', help_text='Specific branch, tag or commit to checkout.', max_length=256, verbose_name='SCM Branch', blank=True)),
+                ('scm_type', models.CharField(default='', max_length=8, verbose_name='SCM Type', blank=True, choices=[('', 'Manual'), ('git', 'Git'), ('hg', 'Mercurial'), ('svn', 'Subversion')])),
+                ('scm_url', models.CharField(default='', max_length=1024, verbose_name='SCM URL', blank=True)),
+                ('scm_branch', models.CharField(default='', help_text='Specific branch, tag or commit to checkout.', max_length=256, verbose_name='SCM Branch', blank=True)),
                 ('scm_clean', models.BooleanField(default=False)),
                 ('scm_delete_on_update', models.BooleanField(default=False)),
                 ('scm_delete_on_next_update', models.BooleanField(default=False, editable=False)),
@@ -494,9 +494,9 @@ class Migration(migrations.Migration):
             fields=[
                 ('unifiedjob_ptr', models.OneToOneField(parent_link=True, auto_created=True, primary_key=True, serialize=False, to='main.UnifiedJob')),
                 ('local_path', models.CharField(help_text='Local path (relative to PROJECTS_ROOT) containing playbooks and related files for this project.', max_length=1024, blank=True)),
-                ('scm_type', models.CharField(default=b'', max_length=8, verbose_name='SCM Type', blank=True, choices=[(b'', 'Manual'), (b'git', 'Git'), (b'hg', 'Mercurial'), (b'svn', 'Subversion')])),
-                ('scm_url', models.CharField(default=b'', max_length=1024, verbose_name='SCM URL', blank=True)),
-                ('scm_branch', models.CharField(default=b'', help_text='Specific branch, tag or commit to checkout.', max_length=256, verbose_name='SCM Branch', blank=True)),
+                ('scm_type', models.CharField(default='', max_length=8, verbose_name='SCM Type', blank=True, choices=[('', 'Manual'), ('git', 'Git'), ('hg', 'Mercurial'), ('svn', 'Subversion')])),
+                ('scm_url', models.CharField(default='', max_length=1024, verbose_name='SCM URL', blank=True)),
+                ('scm_branch', models.CharField(default='', help_text='Specific branch, tag or commit to checkout.', max_length=256, verbose_name='SCM Branch', blank=True)),
                 ('scm_clean', models.BooleanField(default=False)),
                 ('scm_delete_on_update', models.BooleanField(default=False)),
             ],
@@ -506,8 +506,8 @@ class Migration(migrations.Migration):
             name='SystemJob',
             fields=[
                 ('unifiedjob_ptr', models.OneToOneField(parent_link=True, auto_created=True, primary_key=True, serialize=False, to='main.UnifiedJob')),
-                ('job_type', models.CharField(default=b'', max_length=32, blank=True, choices=[(b'cleanup_jobs', 'Remove jobs older than a certain number of days'), (b'cleanup_activitystream', 'Remove activity stream entries older than a certain number of days'), (b'cleanup_deleted', 'Purge previously deleted items from the database'), (b'cleanup_facts', 'Purge and/or reduce the granularity of system tracking data')])),
-                ('extra_vars', models.TextField(default=b'', blank=True)),
+                ('job_type', models.CharField(default='', max_length=32, blank=True, choices=[('cleanup_jobs', 'Remove jobs older than a certain number of days'), ('cleanup_activitystream', 'Remove activity stream entries older than a certain number of days'), ('cleanup_deleted', 'Purge previously deleted items from the database'), ('cleanup_facts', 'Purge and/or reduce the granularity of system tracking data')])),
+                ('extra_vars', models.TextField(default='', blank=True)),
             ],
             options={
                 'ordering': ('id',),
@@ -518,7 +518,7 @@ class Migration(migrations.Migration):
             name='SystemJobTemplate',
             fields=[
                 ('unifiedjobtemplate_ptr', models.OneToOneField(parent_link=True, auto_created=True, primary_key=True, serialize=False, to='main.UnifiedJobTemplate')),
-                ('job_type', models.CharField(default=b'', max_length=32, blank=True, choices=[(b'cleanup_jobs', 'Remove jobs older than a certain number of days'), (b'cleanup_activitystream', 'Remove activity stream entries older than a certain number of days'), (b'cleanup_deleted', 'Purge previously deleted items from the database'), (b'cleanup_facts', 'Purge and/or reduce the granularity of system tracking data')])),
+                ('job_type', models.CharField(default='', max_length=32, blank=True, choices=[('cleanup_jobs', 'Remove jobs older than a certain number of days'), ('cleanup_activitystream', 'Remove activity stream entries older than a certain number of days'), ('cleanup_deleted', 'Purge previously deleted items from the database'), ('cleanup_facts', 'Purge and/or reduce the granularity of system tracking data')])),
             ],
             bases=('main.unifiedjobtemplate', models.Model),
         ),

awx/main/migrations/0002_squashed_v300_release.py

@@ -105,24 +105,24 @@ def create_system_job_templates(apps, schema_editor):
 
 
 class Migration(migrations.Migration):
-    replaces = [(b'main', '0002_v300_tower_settings_changes'),
-                (b'main', '0003_v300_notification_changes'),
-                (b'main', '0004_v300_fact_changes'),
-                (b'main', '0005_v300_migrate_facts'),
-                (b'main', '0006_v300_active_flag_cleanup'),
-                (b'main', '0007_v300_active_flag_removal'),
-                (b'main', '0008_v300_rbac_changes'),
-                (b'main', '0009_v300_rbac_migrations'),
-                (b'main', '0010_v300_create_system_job_templates'),
-                (b'main', '0011_v300_credential_domain_field'),
-                (b'main', '0012_v300_create_labels'),
-                (b'main', '0013_v300_label_changes'),
-                (b'main', '0014_v300_invsource_cred'),
-                (b'main', '0015_v300_label_changes'),
-                (b'main', '0016_v300_prompting_changes'),
-                (b'main', '0017_v300_prompting_migrations'),
-                (b'main', '0018_v300_host_ordering'),
-                (b'main', '0019_v300_new_azure_credential'),]
+    replaces = [('main', '0002_v300_tower_settings_changes'),
+                ('main', '0003_v300_notification_changes'),
+                ('main', '0004_v300_fact_changes'),
+                ('main', '0005_v300_migrate_facts'),
+                ('main', '0006_v300_active_flag_cleanup'),
+                ('main', '0007_v300_active_flag_removal'),
+                ('main', '0008_v300_rbac_changes'),
+                ('main', '0009_v300_rbac_migrations'),
+                ('main', '0010_v300_create_system_job_templates'),
+                ('main', '0011_v300_credential_domain_field'),
+                ('main', '0012_v300_create_labels'),
+                ('main', '0013_v300_label_changes'),
+                ('main', '0014_v300_invsource_cred'),
+                ('main', '0015_v300_label_changes'),
+                ('main', '0016_v300_prompting_changes'),
+                ('main', '0017_v300_prompting_migrations'),
+                ('main', '0018_v300_host_ordering'),
+                ('main', '0019_v300_new_azure_credential'),]
 
     dependencies = [
         ('taggit', '0002_auto_20150616_2121'),
@@ -143,7 +143,7 @@ class Migration(migrations.Migration):
                 ('description', models.TextField()),
                 ('category', models.CharField(max_length=128)),
                 ('value', models.TextField(blank=True)),
-                ('value_type', models.CharField(max_length=12, choices=[(b'string', 'String'), (b'int', 'Integer'), (b'float', 'Decimal'), (b'json', 'JSON'), (b'bool', 'Boolean'), (b'password', 'Password'), (b'list', 'List')])),
+                ('value_type', models.CharField(max_length=12, choices=[('string', 'String'), ('int', 'Integer'), ('float', 'Decimal'), ('json', 'JSON'), ('bool', 'Boolean'), ('password', 'Password'), ('list', 'List')])),
                 ('user', models.ForeignKey(related_name='settings', default=None, editable=False, to=settings.AUTH_USER_MODEL, null=True)),
             ],
         ),
@@ -154,12 +154,12 @@ class Migration(migrations.Migration):
                 ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                 ('created', models.DateTimeField(default=None, editable=False)),
                 ('modified', models.DateTimeField(default=None, editable=False)),
-                ('status', models.CharField(default=b'pending', max_length=20, editable=False, choices=[(b'pending', 'Pending'), (b'successful', 'Successful'), (b'failed', 'Failed')])),
-                ('error', models.TextField(default=b'', editable=False, blank=True)),
+                ('status', models.CharField(default='pending', max_length=20, editable=False, choices=[('pending', 'Pending'), ('successful', 'Successful'), ('failed', 'Failed')])),
+                ('error', models.TextField(default='', editable=False, blank=True)),
                 ('notifications_sent', models.IntegerField(default=0, editable=False)),
-                ('notification_type', models.CharField(max_length=32, choices=[(b'email', 'Email'), (b'slack', 'Slack'), (b'twilio', 'Twilio'), (b'pagerduty', 'Pagerduty'), (b'hipchat', 'HipChat'), (b'webhook', 'Webhook'), (b'mattermost', 'Mattermost'), (b'irc', 'IRC')])),
-                ('recipients', models.TextField(default=b'', editable=False, blank=True)),
-                ('subject', models.TextField(default=b'', editable=False, blank=True)),
+                ('notification_type', models.CharField(max_length=32, choices=[('email', 'Email'), ('slack', 'Slack'), ('twilio', 'Twilio'), ('pagerduty', 'Pagerduty'), ('hipchat', 'HipChat'), ('webhook', 'Webhook'), ('mattermost', 'Mattermost'), ('rocketchat', 'Rocket.Chat'), ('irc', 'IRC')])),
+                ('recipients', models.TextField(default='', editable=False, blank=True)),
+                ('subject', models.TextField(default='', editable=False, blank=True)),
                 ('body', jsonfield.fields.JSONField(default=dict, blank=True)),
             ],
             options={
@@ -172,9 +172,9 @@ class Migration(migrations.Migration):
                 ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                 ('created', models.DateTimeField(default=None, editable=False)),
                 ('modified', models.DateTimeField(default=None, editable=False)),
-                ('description', models.TextField(default=b'', blank=True)),
+                ('description', models.TextField(default='', blank=True)),
                 ('name', models.CharField(unique=True, max_length=512)),
-                ('notification_type', models.CharField(max_length=32, choices=[(b'email', 'Email'), (b'slack', 'Slack'), (b'twilio', 'Twilio'), (b'pagerduty', 'Pagerduty'), (b'hipchat', 'HipChat'), (b'webhook', 'Webhook'), (b'mattermost', 'Mattermost'), (b'irc', 'IRC')])),
+                ('notification_type', models.CharField(max_length=32, choices=[('email', 'Email'), ('slack', 'Slack'), ('twilio', 'Twilio'), ('pagerduty', 'Pagerduty'), ('hipchat', 'HipChat'), ('webhook', 'Webhook'), ('mattermost', 'Mattermost'), ('rocketchat', 'Rocket.Chat'), ('irc', 'IRC')])),
                 ('notification_configuration', jsonfield.fields.JSONField(default=dict)),
                 ('created_by', models.ForeignKey(related_name="{u'class': 'notificationtemplate', u'app_label': 'main'}(class)s_created+", on_delete=django.db.models.deletion.SET_NULL, default=None, editable=False, to=settings.AUTH_USER_MODEL, null=True)),
                 ('modified_by', models.ForeignKey(related_name="{u'class': 'notificationtemplate', u'app_label': 'main'}(class)s_modified+", on_delete=django.db.models.deletion.SET_NULL, default=None, editable=False, to=settings.AUTH_USER_MODEL, null=True)),
@@ -381,7 +381,7 @@ class Migration(migrations.Migration):
                 ('singleton_name', models.TextField(default=None, unique=True, null=True, db_index=True)),
                 ('members', models.ManyToManyField(related_name='roles', to=settings.AUTH_USER_MODEL)),
                 ('parents', models.ManyToManyField(related_name='children', to='main.Role')),
-                ('implicit_parents', models.TextField(default=b'[]')),
+                ('implicit_parents', models.TextField(default='[]')),
                 ('content_type', models.ForeignKey(default=None, to='contenttypes.ContentType', null=True)),
                 ('object_id', models.PositiveIntegerField(default=None, null=True)),
 
@@ -422,122 +422,122 @@ class Migration(migrations.Migration):
         migrations.AddField(
             model_name='credential',
             name='admin_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=[b'singleton:system_administrator'], to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=['singleton:system_administrator'], to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='credential',
             name='use_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=[b'admin_role'], to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=['admin_role'], to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='credential',
             name='read_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=[b'singleton:system_auditor', b'organization.auditor_role', b'use_role', b'admin_role'], to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=['singleton:system_auditor', 'organization.auditor_role', 'use_role', 'admin_role'], to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='custominventoryscript',
             name='admin_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=b'organization.admin_role', to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role='organization.admin_role', to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='custominventoryscript',
             name='read_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=[b'organization.auditor_role', b'organization.member_role', b'admin_role'], to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=['organization.auditor_role', 'organization.member_role', 'admin_role'], to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='inventory',
             name='admin_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=b'organization.admin_role', to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role='organization.admin_role', to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='inventory',
             name='adhoc_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=b'admin_role', to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role='admin_role', to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='inventory',
             name='update_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=b'admin_role', to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role='admin_role', to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='inventory',
             name='use_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=b'adhoc_role', to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role='adhoc_role', to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='inventory',
             name='read_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=[b'organization.auditor_role',  b'update_role', b'use_role', b'admin_role'], to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=['organization.auditor_role',  'update_role', 'use_role', 'admin_role'], to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='jobtemplate',
             name='admin_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=[b'project.organization.admin_role', b'inventory.organization.admin_role'], to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=['project.organization.admin_role', 'inventory.organization.admin_role'], to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='jobtemplate',
             name='execute_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=[b'admin_role'], to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=['admin_role'], to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='jobtemplate',
             name='read_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=[b'project.organization.auditor_role', b'inventory.organization.auditor_role', b'execute_role', b'admin_role'], to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=['project.organization.auditor_role', 'inventory.organization.auditor_role', 'execute_role', 'admin_role'], to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='organization',
             name='admin_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=b'singleton:system_administrator', to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role='singleton:system_administrator', to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='organization',
             name='auditor_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=b'singleton:system_auditor', to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role='singleton:system_auditor', to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='organization',
             name='member_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=b'admin_role', to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role='admin_role', to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='organization',
             name='read_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=[b'member_role', b'auditor_role'], to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=['member_role', 'auditor_role'], to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='project',
             name='admin_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=[b'organization.admin_role', b'singleton:system_administrator'], to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=['organization.admin_role', 'singleton:system_administrator'], to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='project',
             name='use_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=b'admin_role', to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role='admin_role', to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='project',
             name='update_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=b'admin_role', to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role='admin_role', to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='project',
             name='read_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=[b'organization.auditor_role', b'singleton:system_auditor', b'use_role', b'update_role'], to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=['organization.auditor_role', 'singleton:system_auditor', 'use_role', 'update_role'], to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='team',
             name='admin_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=b'organization.admin_role', to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role='organization.admin_role', to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='team',
             name='member_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=None, to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=None, to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='team',
             name='read_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=[b'admin_role', b'organization.auditor_role', b'member_role'], to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=['admin_role', 'organization.auditor_role', 'member_role'], to='main.Role', null='True'),
         ),
 
         # System Job Templates
@@ -545,18 +545,18 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='systemjob',
             name='job_type',
-            field=models.CharField(default=b'', max_length=32, blank=True, choices=[(b'cleanup_jobs', 'Remove jobs older than a certain number of days'), (b'cleanup_activitystream', 'Remove activity stream entries older than a certain number of days'), (b'cleanup_facts', 'Purge and/or reduce the granularity of system tracking data')]),
+            field=models.CharField(default='', max_length=32, blank=True, choices=[('cleanup_jobs', 'Remove jobs older than a certain number of days'), ('cleanup_activitystream', 'Remove activity stream entries older than a certain number of days'), ('cleanup_facts', 'Purge and/or reduce the granularity of system tracking data')]),
         ),
         migrations.AlterField(
             model_name='systemjobtemplate',
             name='job_type',
-            field=models.CharField(default=b'', max_length=32, blank=True, choices=[(b'cleanup_jobs', 'Remove jobs older than a certain number of days'), (b'cleanup_activitystream', 'Remove activity stream entries older than a certain number of days'), (b'cleanup_facts', 'Purge and/or reduce the granularity of system tracking data')]),
+            field=models.CharField(default='', max_length=32, blank=True, choices=[('cleanup_jobs', 'Remove jobs older than a certain number of days'), ('cleanup_activitystream', 'Remove activity stream entries older than a certain number of days'), ('cleanup_facts', 'Purge and/or reduce the granularity of system tracking data')]),
         ),
         # Credential domain field
         migrations.AddField(
             model_name='credential',
             name='domain',
-            field=models.CharField(default=b'', help_text='The identifier for the domain.', max_length=100, verbose_name='Domain', blank=True),
+            field=models.CharField(default='', help_text='The identifier for the domain.', max_length=100, verbose_name='Domain', blank=True),
         ),
         # Create Labels
         migrations.CreateModel(
@@ -565,7 +565,7 @@ class Migration(migrations.Migration):
                 ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                 ('created', models.DateTimeField(default=None, editable=False)),
                 ('modified', models.DateTimeField(default=None, editable=False)),
-                ('description', models.TextField(default=b'', blank=True)),
+                ('description', models.TextField(default='', blank=True)),
                 ('name', models.CharField(max_length=512)),
                 ('created_by', models.ForeignKey(related_name="{u'class': 'label', u'app_label': 'main'}(class)s_created+", on_delete=django.db.models.deletion.SET_NULL, default=None, editable=False, to=settings.AUTH_USER_MODEL, null=True)),
                 ('modified_by', models.ForeignKey(related_name="{u'class': 'label', u'app_label': 'main'}(class)s_modified+", on_delete=django.db.models.deletion.SET_NULL, default=None, editable=False, to=settings.AUTH_USER_MODEL, null=True)),
@@ -625,7 +625,7 @@ class Migration(migrations.Migration):
         migrations.AddField(
             model_name='credential',
             name='authorize_password',
-            field=models.CharField(default=b'', help_text='Password used by the authorize mechanism.', max_length=1024, blank=True),
+            field=models.CharField(default='', help_text='Password used by the authorize mechanism.', max_length=1024, blank=True),
         ),
         migrations.AlterField(
             model_name='credential',
@@ -640,17 +640,17 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='credential',
             name='kind',
-            field=models.CharField(default=b'ssh', max_length=32, choices=[(b'ssh', 'Machine'), (b'net', 'Network'), (b'scm', 'Source Control'), (b'aws', 'Amazon Web Services'), (b'rax', 'Rackspace'), (b'vmware', 'VMware vCenter'), (b'satellite6', 'Red Hat Satellite 6'), (b'cloudforms', 'Red Hat CloudForms'), (b'gce', 'Google Compute Engine'), (b'azure', 'Microsoft Azure'), (b'openstack', 'OpenStack')]),
+            field=models.CharField(default='ssh', max_length=32, choices=[('ssh', 'Machine'), ('net', 'Network'), ('scm', 'Source Control'), ('aws', 'Amazon Web Services'), ('rax', 'Rackspace'), ('vmware', 'VMware vCenter'), ('satellite6', 'Red Hat Satellite 6'), ('cloudforms', 'Red Hat CloudForms'), ('gce', 'Google Compute Engine'), ('azure', 'Microsoft Azure'), ('openstack', 'OpenStack')]),
         ),
         migrations.AlterField(
             model_name='inventorysource',
             name='source',
-            field=models.CharField(default=b'', max_length=32, blank=True, choices=[(b'', 'Manual'), (b'file', 'Local File, Directory or Script'), (b'rax', 'Rackspace Cloud Servers'), (b'ec2', 'Amazon EC2'), (b'gce', 'Google Compute Engine'), (b'azure', 'Microsoft Azure'), (b'vmware', 'VMware vCenter'), (b'satellite6', 'Red Hat Satellite 6'), (b'cloudforms', 'Red Hat CloudForms'), (b'openstack', 'OpenStack'), (b'custom', 'Custom Script')]),
+            field=models.CharField(default='', max_length=32, blank=True, choices=[('', 'Manual'), ('file', 'Local File, Directory or Script'), ('rax', 'Rackspace Cloud Servers'), ('ec2', 'Amazon EC2'), ('gce', 'Google Compute Engine'), ('azure', 'Microsoft Azure'), ('vmware', 'VMware vCenter'), ('satellite6', 'Red Hat Satellite 6'), ('cloudforms', 'Red Hat CloudForms'), ('openstack', 'OpenStack'), ('custom', 'Custom Script')]),
         ),
         migrations.AlterField(
             model_name='inventoryupdate',
             name='source',
-            field=models.CharField(default=b'', max_length=32, blank=True, choices=[(b'', 'Manual'), (b'file', 'Local File, Directory or Script'), (b'rax', 'Rackspace Cloud Servers'), (b'ec2', 'Amazon EC2'), (b'gce', 'Google Compute Engine'), (b'azure', 'Microsoft Azure'), (b'vmware', 'VMware vCenter'), (b'satellite6', 'Red Hat Satellite 6'), (b'cloudforms', 'Red Hat CloudForms'), (b'openstack', 'OpenStack'), (b'custom', 'Custom Script')]),
+            field=models.CharField(default='', max_length=32, blank=True, choices=[('', 'Manual'), ('file', 'Local File, Directory or Script'), ('rax', 'Rackspace Cloud Servers'), ('ec2', 'Amazon EC2'), ('gce', 'Google Compute Engine'), ('azure', 'Microsoft Azure'), ('vmware', 'VMware vCenter'), ('satellite6', 'Red Hat Satellite 6'), ('cloudforms', 'Red Hat CloudForms'), ('openstack', 'OpenStack'), ('custom', 'Custom Script')]),
         ),
         migrations.AlterField(
             model_name='team',
@@ -702,41 +702,41 @@ class Migration(migrations.Migration):
         migrations.AddField(
             model_name='credential',
             name='client',
-            field=models.CharField(default=b'', help_text='Client Id or Application Id for the credential', max_length=128, blank=True),
+            field=models.CharField(default='', help_text='Client Id or Application Id for the credential', max_length=128, blank=True),
         ),
         migrations.AddField(
             model_name='credential',
             name='secret',
-            field=models.CharField(default=b'', help_text='Secret Token for this credential', max_length=1024, blank=True),
+            field=models.CharField(default='', help_text='Secret Token for this credential', max_length=1024, blank=True),
         ),
         migrations.AddField(
             model_name='credential',
             name='subscription',
-            field=models.CharField(default=b'', help_text='Subscription identifier for this credential', max_length=1024, blank=True),
+            field=models.CharField(default='', help_text='Subscription identifier for this credential', max_length=1024, blank=True),
         ),
         migrations.AddField(
             model_name='credential',
             name='tenant',
-            field=models.CharField(default=b'', help_text='Tenant identifier for this credential', max_length=1024, blank=True),
+            field=models.CharField(default='', help_text='Tenant identifier for this credential', max_length=1024, blank=True),
         ),
         migrations.AlterField(
             model_name='credential',
             name='kind',
-            field=models.CharField(default=b'ssh', max_length=32, choices=[(b'ssh', 'Machine'), (b'net', 'Network'), (b'scm', 'Source Control'), (b'aws', 'Amazon Web Services'), (b'rax', 'Rackspace'), (b'vmware', 'VMware vCenter'), (b'satellite6', 'Satellite 6'), (b'cloudforms', 'CloudForms'), (b'gce', 'Google Compute Engine'), (b'azure', 'Microsoft Azure Classic (deprecated)'), (b'azure_rm', 'Microsoft Azure Resource Manager'), (b'openstack', 'OpenStack')]),
+            field=models.CharField(default='ssh', max_length=32, choices=[('ssh', 'Machine'), ('net', 'Network'), ('scm', 'Source Control'), ('aws', 'Amazon Web Services'), ('rax', 'Rackspace'), ('vmware', 'VMware vCenter'), ('satellite6', 'Satellite 6'), ('cloudforms', 'CloudForms'), ('gce', 'Google Compute Engine'), ('azure', 'Microsoft Azure Classic (deprecated)'), ('azure_rm', 'Microsoft Azure Resource Manager'), ('openstack', 'OpenStack')]),
         ),
         migrations.AlterField(
             model_name='host',
             name='instance_id',
-            field=models.CharField(default=b'', max_length=1024, blank=True),
+            field=models.CharField(default='', max_length=1024, blank=True),
         ),
         migrations.AlterField(
             model_name='inventorysource',
             name='source',
-            field=models.CharField(default=b'', max_length=32, blank=True, choices=[(b'', 'Manual'), (b'file', 'Local File, Directory or Script'), (b'rax', 'Rackspace Cloud Servers'), (b'ec2', 'Amazon EC2'), (b'gce', 'Google Compute Engine'), (b'azure', 'Microsoft Azure Classic (deprecated)'), (b'azure_rm', 'Microsoft Azure Resource Manager'), (b'vmware', 'VMware vCenter'), (b'satellite6', 'Satellite 6'), (b'cloudforms', 'CloudForms'), (b'openstack', 'OpenStack'), (b'custom', 'Custom Script')]),
+            field=models.CharField(default='', max_length=32, blank=True, choices=[('', 'Manual'), ('file', 'Local File, Directory or Script'), ('rax', 'Rackspace Cloud Servers'), ('ec2', 'Amazon EC2'), ('gce', 'Google Compute Engine'), ('azure', 'Microsoft Azure Classic (deprecated)'), ('azure_rm', 'Microsoft Azure Resource Manager'), ('vmware', 'VMware vCenter'), ('satellite6', 'Satellite 6'), ('cloudforms', 'CloudForms'), ('openstack', 'OpenStack'), ('custom', 'Custom Script')]),
         ),
         migrations.AlterField(
             model_name='inventoryupdate',
             name='source',
-            field=models.CharField(default=b'', max_length=32, blank=True, choices=[(b'', 'Manual'), (b'file', 'Local File, Directory or Script'), (b'rax', 'Rackspace Cloud Servers'), (b'ec2', 'Amazon EC2'), (b'gce', 'Google Compute Engine'), (b'azure', 'Microsoft Azure Classic (deprecated)'), (b'azure_rm', 'Microsoft Azure Resource Manager'), (b'vmware', 'VMware vCenter'), (b'satellite6', 'Satellite 6'), (b'cloudforms', 'CloudForms'), (b'openstack', 'OpenStack'), (b'custom', 'Custom Script')]),
+            field=models.CharField(default='', max_length=32, blank=True, choices=[('', 'Manual'), ('file', 'Local File, Directory or Script'), ('rax', 'Rackspace Cloud Servers'), ('ec2', 'Amazon EC2'), ('gce', 'Google Compute Engine'), ('azure', 'Microsoft Azure Classic (deprecated)'), ('azure_rm', 'Microsoft Azure Resource Manager'), ('vmware', 'VMware vCenter'), ('satellite6', 'Satellite 6'), ('cloudforms', 'CloudForms'), ('openstack', 'OpenStack'), ('custom', 'Custom Script')]),
         ),
     ]

awx/main/migrations/0003_squashed_v300_v303_updates.py

@@ -9,20 +9,20 @@ from django.db import migrations, models
 from django.conf import settings
 import awx.main.fields
 
-import _squashed
-from _squashed_30 import SQUASHED_30
+from . import _squashed
+from ._squashed_30 import SQUASHED_30
 
 
 class Migration(migrations.Migration):
-    replaces = [(b'main', '0020_v300_labels_changes'),
-                (b'main', '0021_v300_activity_stream'),
-                (b'main', '0022_v300_adhoc_extravars'),
-                (b'main', '0023_v300_activity_stream_ordering'),
-                (b'main', '0024_v300_jobtemplate_allow_simul'),
-                (b'main', '0025_v300_update_rbac_parents'),
-                (b'main', '0026_v300_credential_unique'),
-                (b'main', '0027_v300_team_migrations'),
-                (b'main', '0028_v300_org_team_cascade')] + _squashed.replaces(SQUASHED_30, applied=True)
+    replaces = [('main', '0020_v300_labels_changes'),
+                ('main', '0021_v300_activity_stream'),
+                ('main', '0022_v300_adhoc_extravars'),
+                ('main', '0023_v300_activity_stream_ordering'),
+                ('main', '0024_v300_jobtemplate_allow_simul'),
+                ('main', '0025_v300_update_rbac_parents'),
+                ('main', '0026_v300_credential_unique'),
+                ('main', '0027_v300_team_migrations'),
+                ('main', '0028_v300_org_team_cascade')] + _squashed.replaces(SQUASHED_30, applied=True)
 
     dependencies = [
         migrations.swappable_dependency(settings.AUTH_USER_MODEL),
@@ -63,22 +63,22 @@ class Migration(migrations.Migration):
         migrations.AddField(
             model_name='adhoccommand',
             name='extra_vars',
-            field=models.TextField(default=b'', blank=True),
+            field=models.TextField(default='', blank=True),
         ),
         migrations.AlterField(
             model_name='credential',
             name='kind',
-            field=models.CharField(default=b'ssh', max_length=32, choices=[(b'ssh', 'Machine'), (b'net', 'Network'), (b'scm', 'Source Control'), (b'aws', 'Amazon Web Services'), (b'rax', 'Rackspace'), (b'vmware', 'VMware vCenter'), (b'satellite6', 'Red Hat Satellite 6'), (b'cloudforms', 'Red Hat CloudForms'), (b'gce', 'Google Compute Engine'), (b'azure', 'Microsoft Azure Classic (deprecated)'), (b'azure_rm', 'Microsoft Azure Resource Manager'), (b'openstack', 'OpenStack')]),
+            field=models.CharField(default='ssh', max_length=32, choices=[('ssh', 'Machine'), ('net', 'Network'), ('scm', 'Source Control'), ('aws', 'Amazon Web Services'), ('rax', 'Rackspace'), ('vmware', 'VMware vCenter'), ('satellite6', 'Red Hat Satellite 6'), ('cloudforms', 'Red Hat CloudForms'), ('gce', 'Google Compute Engine'), ('azure', 'Microsoft Azure Classic (deprecated)'), ('azure_rm', 'Microsoft Azure Resource Manager'), ('openstack', 'OpenStack')]),
         ),
         migrations.AlterField(
             model_name='inventorysource',
             name='source',
-            field=models.CharField(default=b'', max_length=32, blank=True, choices=[(b'', 'Manual'), (b'file', 'Local File, Directory or Script'), (b'rax', 'Rackspace Cloud Servers'), (b'ec2', 'Amazon EC2'), (b'gce', 'Google Compute Engine'), (b'azure', 'Microsoft Azure Classic (deprecated)'), (b'azure_rm', 'Microsoft Azure Resource Manager'), (b'vmware', 'VMware vCenter'), (b'satellite6', 'Red Hat Satellite 6'), (b'cloudforms', 'Red Hat CloudForms'), (b'openstack', 'OpenStack'), (b'custom', 'Custom Script')]),
+            field=models.CharField(default='', max_length=32, blank=True, choices=[('', 'Manual'), ('file', 'Local File, Directory or Script'), ('rax', 'Rackspace Cloud Servers'), ('ec2', 'Amazon EC2'), ('gce', 'Google Compute Engine'), ('azure', 'Microsoft Azure Classic (deprecated)'), ('azure_rm', 'Microsoft Azure Resource Manager'), ('vmware', 'VMware vCenter'), ('satellite6', 'Red Hat Satellite 6'), ('cloudforms', 'Red Hat CloudForms'), ('openstack', 'OpenStack'), ('custom', 'Custom Script')]),
         ),
         migrations.AlterField(
             model_name='inventoryupdate',
             name='source',
-            field=models.CharField(default=b'', max_length=32, blank=True, choices=[(b'', 'Manual'), (b'file', 'Local File, Directory or Script'), (b'rax', 'Rackspace Cloud Servers'), (b'ec2', 'Amazon EC2'), (b'gce', 'Google Compute Engine'), (b'azure', 'Microsoft Azure Classic (deprecated)'), (b'azure_rm', 'Microsoft Azure Resource Manager'), (b'vmware', 'VMware vCenter'), (b'satellite6', 'Red Hat Satellite 6'), (b'cloudforms', 'Red Hat CloudForms'), (b'openstack', 'OpenStack'), (b'custom', 'Custom Script')]),
+            field=models.CharField(default='', max_length=32, blank=True, choices=[('', 'Manual'), ('file', 'Local File, Directory or Script'), ('rax', 'Rackspace Cloud Servers'), ('ec2', 'Amazon EC2'), ('gce', 'Google Compute Engine'), ('azure', 'Microsoft Azure Classic (deprecated)'), ('azure_rm', 'Microsoft Azure Resource Manager'), ('vmware', 'VMware vCenter'), ('satellite6', 'Red Hat Satellite 6'), ('cloudforms', 'Red Hat CloudForms'), ('openstack', 'OpenStack'), ('custom', 'Custom Script')]),
         ),
         # jobtemplate allow simul
         migrations.AddField(
@@ -90,17 +90,17 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='credential',
             name='use_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=[b'organization.admin_role', b'admin_role'], to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=['organization.admin_role', 'admin_role'], to='main.Role', null='True'),
         ),
         migrations.AlterField(
             model_name='team',
             name='member_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=b'admin_role', to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role='admin_role', to='main.Role', null='True'),
         ),
         migrations.AlterField(
             model_name='team',
             name='read_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=[b'organization.auditor_role', b'member_role'], to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=['organization.auditor_role', 'member_role'], to='main.Role', null='True'),
         ),
         # Unique credential
         migrations.AlterUniqueTogether(
@@ -110,7 +110,7 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='credential',
             name='read_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=[b'singleton:system_auditor', b'organization.auditor_role', b'use_role', b'admin_role'], to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=['singleton:system_auditor', 'organization.auditor_role', 'use_role', 'admin_role'], to='main.Role', null='True'),
         ),
         # Team cascade
         migrations.AlterField(

awx/main/migrations/0004_squashed_v310_release.py

@@ -8,8 +8,8 @@ import django.db.models.deletion
 import awx.main.models.workflow
 import awx.main.fields
 
-import _squashed
-from _squashed_30 import SQUASHED_30
+from . import _squashed
+from ._squashed_30 import SQUASHED_30
 
 
 class Migration(migrations.Migration):
@@ -19,7 +19,7 @@ class Migration(migrations.Migration):
     ]
 
     replaces = _squashed.replaces(SQUASHED_30) + [
-        (b'main', '0034_v310_release'),
+        ('main', '0034_v310_release'),
     ]
 
     operations = _squashed.operations(SQUASHED_30) + [
@@ -42,13 +42,13 @@ class Migration(migrations.Migration):
         migrations.AddField(
             model_name='jobevent',
             name='uuid',
-            field=models.CharField(default=b'', max_length=1024, editable=False),
+            field=models.CharField(default='', max_length=1024, editable=False),
         ),
         # Job Parent Event UUID
         migrations.AddField(
             model_name='jobevent',
             name='parent_uuid',
-            field=models.CharField(default=b'', max_length=1024, editable=False),
+            field=models.CharField(default='', max_length=1024, editable=False),
         ),
         # Modify the HA Instance
         migrations.RemoveField(
@@ -63,19 +63,19 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='credential',
             name='become_method',
-            field=models.CharField(default=b'', help_text='Privilege escalation method.', max_length=32, blank=True, choices=[(b'', 'None'), (b'sudo', 'Sudo'), (b'su', 'Su'), (b'pbrun', 'Pbrun'), (b'pfexec', 'Pfexec'), (b'dzdo', 'DZDO'), (b'pmrun', 'Pmrun')]),
+            field=models.CharField(default='', help_text='Privilege escalation method.', max_length=32, blank=True, choices=[('', 'None'), ('sudo', 'Sudo'), ('su', 'Su'), ('pbrun', 'Pbrun'), ('pfexec', 'Pfexec'), ('dzdo', 'DZDO'), ('pmrun', 'Pmrun')]),
         ),
         # Add Workflows
         migrations.AlterField(
             model_name='unifiedjob',
             name='launch_type',
-            field=models.CharField(default=b'manual', max_length=20, editable=False, choices=[(b'manual', 'Manual'), (b'relaunch', 'Relaunch'), (b'callback', 'Callback'), (b'scheduled', 'Scheduled'), (b'dependency', 'Dependency'), (b'workflow', 'Workflow'), (b'sync', 'Sync')]),
+            field=models.CharField(default='manual', max_length=20, editable=False, choices=[('manual', 'Manual'), ('relaunch', 'Relaunch'), ('callback', 'Callback'), ('scheduled', 'Scheduled'), ('dependency', 'Dependency'), ('workflow', 'Workflow'), ('sync', 'Sync')]),
         ),
         migrations.CreateModel(
             name='WorkflowJob',
             fields=[
                 ('unifiedjob_ptr', models.OneToOneField(parent_link=True, auto_created=True, primary_key=True, serialize=False, to='main.UnifiedJob')),
-                ('extra_vars', models.TextField(default=b'', blank=True)),
+                ('extra_vars', models.TextField(default='', blank=True)),
             ],
             options={
                 'ordering': ('id',),
@@ -101,8 +101,8 @@ class Migration(migrations.Migration):
             name='WorkflowJobTemplate',
             fields=[
                 ('unifiedjobtemplate_ptr', models.OneToOneField(parent_link=True, auto_created=True, primary_key=True, serialize=False, to='main.UnifiedJobTemplate')),
-                ('extra_vars', models.TextField(default=b'', blank=True)),
-                ('admin_role', awx.main.fields.ImplicitRoleField(related_name='+', parent_role=b'singleton:system_administrator', to='main.Role', null=b'True')),
+                ('extra_vars', models.TextField(default='', blank=True)),
+                ('admin_role', awx.main.fields.ImplicitRoleField(related_name='+', parent_role='singleton:system_administrator', to='main.Role', null='True')),
             ],
             bases=('main.unifiedjobtemplate', models.Model),
         ),
@@ -176,7 +176,7 @@ class Migration(migrations.Migration):
         migrations.AddField(
             model_name='workflowjobtemplate',
             name='execute_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=[b'admin_role'], to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=['admin_role'], to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='workflowjobtemplate',
@@ -186,7 +186,7 @@ class Migration(migrations.Migration):
         migrations.AddField(
             model_name='workflowjobtemplate',
             name='read_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=[b'singleton:system_auditor', b'organization.auditor_role', b'execute_role', b'admin_role'], to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=['singleton:system_auditor', 'organization.auditor_role', 'execute_role', 'admin_role'], to='main.Role', null='True'),
         ),
         migrations.AddField(
             model_name='workflowjobtemplatenode',
@@ -216,7 +216,7 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='workflowjobtemplate',
             name='admin_role',
-            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=[b'singleton:system_administrator', b'organization.admin_role'], to='main.Role', null=b'True'),
+            field=awx.main.fields.ImplicitRoleField(related_name='+', parent_role=['singleton:system_administrator', 'organization.admin_role'], to='main.Role', null='True'),
         ),
         migrations.AlterField(
             model_name='workflowjobtemplatenode',
@@ -269,23 +269,23 @@ class Migration(migrations.Migration):
         migrations.AddField(
             model_name='unifiedjob',
             name='execution_node',
-            field=models.TextField(default=b'', editable=False, blank=True),
+            field=models.TextField(default='', editable=False, blank=True),
         ),
         # SCM Revision
         migrations.AddField(
             model_name='project',
             name='scm_revision',
-            field=models.CharField(default=b'', editable=False, max_length=1024, blank=True, help_text='The last revision fetched by a project update', verbose_name='SCM Revision'),
+            field=models.CharField(default='', editable=False, max_length=1024, blank=True, help_text='The last revision fetched by a project update', verbose_name='SCM Revision'),
         ),
         migrations.AddField(
             model_name='projectupdate',
             name='job_type',
-            field=models.CharField(default=b'check', max_length=64, choices=[(b'run', 'Run'), (b'check', 'Check')]),
+            field=models.CharField(default='check', max_length=64, choices=[('run', 'Run'), ('check', 'Check')]),
         ),
         migrations.AddField(
             model_name='job',
             name='scm_revision',
-            field=models.CharField(default=b'', editable=False, max_length=1024, blank=True, help_text='The SCM Revision from the Project used for this job, if available', verbose_name='SCM Revision'),
+            field=models.CharField(default='', editable=False, max_length=1024, blank=True, help_text='The SCM Revision from the Project used for this job, if available', verbose_name='SCM Revision'),
         ),
         # Project Playbook Files
         migrations.AddField(
@@ -307,12 +307,12 @@ class Migration(migrations.Migration):
         migrations.AddField(
             model_name='adhoccommandevent',
             name='stdout',
-            field=models.TextField(default=b'', editable=False),
+            field=models.TextField(default='', editable=False),
         ),
         migrations.AddField(
             model_name='adhoccommandevent',
             name='uuid',
-            field=models.CharField(default=b'', max_length=1024, editable=False),
+            field=models.CharField(default='', max_length=1024, editable=False),
         ),
         migrations.AddField(
             model_name='adhoccommandevent',
@@ -327,7 +327,7 @@ class Migration(migrations.Migration):
         migrations.AddField(
             model_name='jobevent',
             name='playbook',
-            field=models.CharField(default=b'', max_length=1024, editable=False),
+            field=models.CharField(default='', max_length=1024, editable=False),
         ),
         migrations.AddField(
             model_name='jobevent',
@@ -337,7 +337,7 @@ class Migration(migrations.Migration):
         migrations.AddField(
             model_name='jobevent',
             name='stdout',
-            field=models.TextField(default=b'', editable=False),
+            field=models.TextField(default='', editable=False),
         ),
         migrations.AddField(
             model_name='jobevent',
@@ -352,7 +352,7 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='adhoccommandevent',
             name='event',
-            field=models.CharField(max_length=100, choices=[(b'runner_on_failed', 'Host Failed'), (b'runner_on_ok', 'Host OK'), (b'runner_on_unreachable', 'Host Unreachable'), (b'runner_on_skipped', 'Host Skipped'), (b'debug', 'Debug'), (b'verbose', 'Verbose'), (b'deprecated', 'Deprecated'), (b'warning', 'Warning'), (b'system_warning', 'System Warning'), (b'error', 'Error')]),
+            field=models.CharField(max_length=100, choices=[('runner_on_failed', 'Host Failed'), ('runner_on_ok', 'Host OK'), ('runner_on_unreachable', 'Host Unreachable'), ('runner_on_skipped', 'Host Skipped'), ('debug', 'Debug'), ('verbose', 'Verbose'), ('deprecated', 'Deprecated'), ('warning', 'Warning'), ('system_warning', 'System Warning'), ('error', 'Error')]),
         ),
         migrations.AlterField(
             model_name='jobevent',
@@ -362,7 +362,7 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='jobevent',
             name='event',
-            field=models.CharField(max_length=100, choices=[(b'runner_on_failed', 'Host Failed'), (b'runner_on_ok', 'Host OK'), (b'runner_on_error', 'Host Failure'), (b'runner_on_skipped', 'Host Skipped'), (b'runner_on_unreachable', 'Host Unreachable'), (b'runner_on_no_hosts', 'No Hosts Remaining'), (b'runner_on_async_poll', 'Host Polling'), (b'runner_on_async_ok', 'Host Async OK'), (b'runner_on_async_failed', 'Host Async Failure'), (b'runner_item_on_ok', 'Item OK'), (b'runner_item_on_failed', 'Item Failed'), (b'runner_item_on_skipped', 'Item Skipped'), (b'runner_retry', 'Host Retry'), (b'runner_on_file_diff', 'File Difference'), (b'playbook_on_start', 'Playbook Started'), (b'playbook_on_notify', 'Running Handlers'), (b'playbook_on_include', 'Including File'), (b'playbook_on_no_hosts_matched', 'No Hosts Matched'), (b'playbook_on_no_hosts_remaining', 'No Hosts Remaining'), (b'playbook_on_task_start', 'Task Started'), (b'playbook_on_vars_prompt', 'Variables Prompted'), (b'playbook_on_setup', 'Gathering Facts'), (b'playbook_on_import_for_host', 'internal: on Import for Host'), (b'playbook_on_not_import_for_host', 'internal: on Not Import for Host'), (b'playbook_on_play_start', 'Play Started'), (b'playbook_on_stats', 'Playbook Complete'), (b'debug', 'Debug'), (b'verbose', 'Verbose'), (b'deprecated', 'Deprecated'), (b'warning', 'Warning'), (b'system_warning', 'System Warning'), (b'error', 'Error')]),
+            field=models.CharField(max_length=100, choices=[('runner_on_failed', 'Host Failed'), ('runner_on_ok', 'Host OK'), ('runner_on_error', 'Host Failure'), ('runner_on_skipped', 'Host Skipped'), ('runner_on_unreachable', 'Host Unreachable'), ('runner_on_no_hosts', 'No Hosts Remaining'), ('runner_on_async_poll', 'Host Polling'), ('runner_on_async_ok', 'Host Async OK'), ('runner_on_async_failed', 'Host Async Failure'), ('runner_item_on_ok', 'Item OK'), ('runner_item_on_failed', 'Item Failed'), ('runner_item_on_skipped', 'Item Skipped'), ('runner_retry', 'Host Retry'), ('runner_on_file_diff', 'File Difference'), ('playbook_on_start', 'Playbook Started'), ('playbook_on_notify', 'Running Handlers'), ('playbook_on_include', 'Including File'), ('playbook_on_no_hosts_matched', 'No Hosts Matched'), ('playbook_on_no_hosts_remaining', 'No Hosts Remaining'), ('playbook_on_task_start', 'Task Started'), ('playbook_on_vars_prompt', 'Variables Prompted'), ('playbook_on_setup', 'Gathering Facts'), ('playbook_on_import_for_host', 'internal: on Import for Host'), ('playbook_on_not_import_for_host', 'internal: on Not Import for Host'), ('playbook_on_play_start', 'Play Started'), ('playbook_on_stats', 'Playbook Complete'), ('debug', 'Debug'), ('verbose', 'Verbose'), ('deprecated', 'Deprecated'), ('warning', 'Warning'), ('system_warning', 'System Warning'), ('error', 'Error')]),
         ),
         migrations.AlterUniqueTogether(
             name='adhoccommandevent',
@@ -505,7 +505,7 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='host',
             name='instance_id',
-            field=models.CharField(default=b'', help_text='The value used by the remote inventory source to uniquely identify the host', max_length=1024, blank=True),
+            field=models.CharField(default='', help_text='The value used by the remote inventory source to uniquely identify the host', max_length=1024, blank=True),
         ),
         migrations.AlterField(
             model_name='project',
@@ -520,7 +520,7 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='project',
             name='scm_type',
-            field=models.CharField(default=b'', choices=[(b'', 'Manual'), (b'git', 'Git'), (b'hg', 'Mercurial'), (b'svn', 'Subversion')], max_length=8, blank=True, help_text='Specifies the source control system used to store the project.', verbose_name='SCM Type'),
+            field=models.CharField(default='', choices=[('', 'Manual'), ('git', 'Git'), ('hg', 'Mercurial'), ('svn', 'Subversion')], max_length=8, blank=True, help_text='Specifies the source control system used to store the project.', verbose_name='SCM Type'),
         ),
         migrations.AlterField(
             model_name='project',
@@ -535,7 +535,7 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='project',
             name='scm_url',
-            field=models.CharField(default=b'', help_text='The location where the project is stored.', max_length=1024, verbose_name='SCM URL', blank=True),
+            field=models.CharField(default='', help_text='The location where the project is stored.', max_length=1024, verbose_name='SCM URL', blank=True),
         ),
         migrations.AlterField(
             model_name='project',
@@ -555,12 +555,12 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='projectupdate',
             name='scm_type',
-            field=models.CharField(default=b'', choices=[(b'', 'Manual'), (b'git', 'Git'), (b'hg', 'Mercurial'), (b'svn', 'Subversion')], max_length=8, blank=True, help_text='Specifies the source control system used to store the project.', verbose_name='SCM Type'),
+            field=models.CharField(default='', choices=[('', 'Manual'), ('git', 'Git'), ('hg', 'Mercurial'), ('svn', 'Subversion')], max_length=8, blank=True, help_text='Specifies the source control system used to store the project.', verbose_name='SCM Type'),
         ),
         migrations.AlterField(
             model_name='projectupdate',
             name='scm_url',
-            field=models.CharField(default=b'', help_text='The location where the project is stored.', max_length=1024, verbose_name='SCM URL', blank=True),
+            field=models.CharField(default='', help_text='The location where the project is stored.', max_length=1024, verbose_name='SCM URL', blank=True),
         ),
         migrations.AlterField(
             model_name='projectupdate',
@@ -600,7 +600,7 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='unifiedjob',
             name='execution_node',
-            field=models.TextField(default=b'', help_text='The Tower node the job executed on.', editable=False, blank=True),
+            field=models.TextField(default='', help_text='The Tower node the job executed on.', editable=False, blank=True),
         ),
         migrations.AlterField(
             model_name='unifiedjob',
@@ -610,7 +610,7 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='unifiedjob',
             name='job_explanation',
-            field=models.TextField(default=b'', help_text="A status field to indicate the state of the job if it wasn't able to run and capture stdout", editable=False, blank=True),
+            field=models.TextField(default='', help_text="A status field to indicate the state of the job if it wasn't able to run and capture stdout", editable=False, blank=True),
         ),
         migrations.AlterField(
             model_name='unifiedjob',

awx/main/migrations/0005_squashed_v310_v313_updates.py

@@ -2,8 +2,8 @@
 from __future__ import unicode_literals
 from django.db import migrations
 
-import _squashed
-from _squashed_31 import SQUASHED_31
+from . import _squashed
+from ._squashed_31 import SQUASHED_31
 
 
 class Migration(migrations.Migration):

awx/main/migrations/0006_v320_release.py

@@ -72,7 +72,7 @@ class Migration(migrations.Migration):
         migrations.AddField(
             model_name='inventory',
             name='kind',
-            field=models.CharField(default=b'', help_text='Kind of inventory being represented.', max_length=32, blank=True, choices=[(b'', 'Hosts have a direct link to this inventory.'), (b'smart', 'Hosts for inventory generated using the host_filter property.')]),
+            field=models.CharField(default='', help_text='Kind of inventory being represented.', max_length=32, blank=True, choices=[('', 'Hosts have a direct link to this inventory.'), ('smart', 'Hosts for inventory generated using the host_filter property.')]),
         ),
         migrations.CreateModel(
             name='SmartInventoryMembership',
@@ -143,7 +143,7 @@ class Migration(migrations.Migration):
         migrations.AddField(
             model_name='inventorysource',
             name='scm_last_revision',
-            field=models.CharField(default=b'', max_length=1024, editable=False, blank=True),
+            field=models.CharField(default='', max_length=1024, editable=False, blank=True),
         ),
         migrations.AddField(
             model_name='inventorysource',
@@ -163,27 +163,27 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='inventorysource',
             name='source',
-            field=models.CharField(default=b'', max_length=32, blank=True, choices=[(b'', 'Manual'), (b'file', 'File, Directory or Script'), (b'scm', 'Sourced from a Project'), (b'ec2', 'Amazon EC2'), (b'gce', 'Google Compute Engine'), (b'azure_rm', 'Microsoft Azure Resource Manager'), (b'vmware', 'VMware vCenter'), (b'satellite6', 'Red Hat Satellite 6'), (b'cloudforms', 'Red Hat CloudForms'), (b'openstack', 'OpenStack'), (b'custom', 'Custom Script')]),
+            field=models.CharField(default='', max_length=32, blank=True, choices=[('', 'Manual'), ('file', 'File, Directory or Script'), ('scm', 'Sourced from a Project'), ('ec2', 'Amazon EC2'), ('gce', 'Google Compute Engine'), ('azure_rm', 'Microsoft Azure Resource Manager'), ('vmware', 'VMware vCenter'), ('satellite6', 'Red Hat Satellite 6'), ('cloudforms', 'Red Hat CloudForms'), ('openstack', 'OpenStack'), ('custom', 'Custom Script')]),
         ),
         migrations.AlterField(
             model_name='inventoryupdate',
             name='source',
-            field=models.CharField(default=b'', max_length=32, blank=True, choices=[(b'', 'Manual'), (b'file', 'File, Directory or Script'), (b'scm', 'Sourced from a Project'), (b'ec2', 'Amazon EC2'), (b'gce', 'Google Compute Engine'), (b'azure_rm', 'Microsoft Azure Resource Manager'), (b'vmware', 'VMware vCenter'), (b'satellite6', 'Red Hat Satellite 6'), (b'cloudforms', 'Red Hat CloudForms'), (b'openstack', 'OpenStack'), (b'custom', 'Custom Script')]),
+            field=models.CharField(default='', max_length=32, blank=True, choices=[('', 'Manual'), ('file', 'File, Directory or Script'), ('scm', 'Sourced from a Project'), ('ec2', 'Amazon EC2'), ('gce', 'Google Compute Engine'), ('azure_rm', 'Microsoft Azure Resource Manager'), ('vmware', 'VMware vCenter'), ('satellite6', 'Red Hat Satellite 6'), ('cloudforms', 'Red Hat CloudForms'), ('openstack', 'OpenStack'), ('custom', 'Custom Script')]),
         ),
         migrations.AlterField(
             model_name='inventorysource',
             name='source_path',
-            field=models.CharField(default=b'', max_length=1024, blank=True),
+            field=models.CharField(default='', max_length=1024, blank=True),
         ),
         migrations.AlterField(
             model_name='inventoryupdate',
             name='source_path',
-            field=models.CharField(default=b'', max_length=1024, blank=True),
+            field=models.CharField(default='', max_length=1024, blank=True),
         ),
         migrations.AlterField(
             model_name='unifiedjob',
             name='launch_type',
-            field=models.CharField(default=b'manual', max_length=20, editable=False, choices=[(b'manual', 'Manual'), (b'relaunch', 'Relaunch'), (b'callback', 'Callback'), (b'scheduled', 'Scheduled'), (b'dependency', 'Dependency'), (b'workflow', 'Workflow'), (b'sync', 'Sync'), (b'scm', 'SCM Update')]),
+            field=models.CharField(default='manual', max_length=20, editable=False, choices=[('manual', 'Manual'), ('relaunch', 'Relaunch'), ('callback', 'Callback'), ('scheduled', 'Scheduled'), ('dependency', 'Dependency'), ('workflow', 'Workflow'), ('sync', 'Sync'), ('scm', 'SCM Update')]),
         ),
         migrations.AddField(
             model_name='inventorysource',
@@ -211,12 +211,12 @@ class Migration(migrations.Migration):
         migrations.AddField(
             model_name='inventorysource',
             name='verbosity',
-            field=models.PositiveIntegerField(default=1, blank=True, choices=[(0, b'0 (WARNING)'), (1, b'1 (INFO)'), (2, b'2 (DEBUG)')]),
+            field=models.PositiveIntegerField(default=1, blank=True, choices=[(0, '0 (WARNING)'), (1, '1 (INFO)'), (2, '2 (DEBUG)')]),
         ),
         migrations.AddField(
             model_name='inventoryupdate',
             name='verbosity',
-            field=models.PositiveIntegerField(default=1, blank=True, choices=[(0, b'0 (WARNING)'), (1, b'1 (INFO)'), (2, b'2 (DEBUG)')]),
+            field=models.PositiveIntegerField(default=1, blank=True, choices=[(0, '0 (WARNING)'), (1, '1 (INFO)'), (2, '2 (DEBUG)')]),
         ),
 
         # Job Templates
@@ -317,7 +317,7 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='inventory',
             name='kind',
-            field=models.CharField(default=b'', help_text='Kind of inventory being represented.', max_length=32, blank=True, choices=[(b'', 'Hosts have a direct link to this inventory.'), (b'smart', 'Hosts for inventory generated using the host_filter property.')]),
+            field=models.CharField(default='', help_text='Kind of inventory being represented.', max_length=32, blank=True, choices=[('', 'Hosts have a direct link to this inventory.'), ('smart', 'Hosts for inventory generated using the host_filter property.')]),
         ),
 
         # Timeout help text update
@@ -378,9 +378,9 @@ class Migration(migrations.Migration):
                 ('id', models.AutoField(verbose_name='ID', serialize=False, auto_created=True, primary_key=True)),
                 ('created', models.DateTimeField(default=None, editable=False)),
                 ('modified', models.DateTimeField(default=None, editable=False)),
-                ('description', models.TextField(default=b'', blank=True)),
+                ('description', models.TextField(default='', blank=True)),
                 ('name', models.CharField(max_length=512)),
-                ('kind', models.CharField(max_length=32, choices=[(b'ssh', 'Machine'), (b'vault', 'Vault'), (b'net', 'Network'), (b'scm', 'Source Control'), (b'cloud', 'Cloud'), (b'insights', 'Insights')])),
+                ('kind', models.CharField(max_length=32, choices=[('ssh', 'Machine'), ('vault', 'Vault'), ('net', 'Network'), ('scm', 'Source Control'), ('cloud', 'Cloud'), ('insights', 'Insights')])),
                 ('managed_by_tower', models.BooleanField(default=False, editable=False)),
                 ('inputs', awx.main.fields.CredentialTypeInputField(default={}, blank=True, help_text='Enter inputs using either JSON or YAML syntax. Use the radio button to toggle between the two. Refer to the Ansible Tower documentation for example syntax.')),
                 ('injectors', awx.main.fields.CredentialTypeInjectorField(default={}, blank=True, help_text='Enter injectors using either JSON or YAML syntax. Use the radio button to toggle between the two. Refer to the Ansible Tower documentation for example syntax.')),
@@ -435,7 +435,7 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='credential',
             name='become_method',
-            field=models.CharField(default=b'', help_text='Privilege escalation method.', max_length=32, blank=True, choices=[(b'', 'None'), (b'sudo', 'Sudo'), (b'su', 'Su'), (b'pbrun', 'Pbrun'), (b'pfexec', 'Pfexec'), (b'dzdo', 'DZDO'), (b'pmrun', 'Pmrun'), (b'runas', 'Runas')]),
+            field=models.CharField(default='', help_text='Privilege escalation method.', max_length=32, blank=True, choices=[('', 'None'), ('sudo', 'Sudo'), ('su', 'Su'), ('pbrun', 'Pbrun'), ('pfexec', 'Pfexec'), ('dzdo', 'DZDO'), ('pmrun', 'Pmrun'), ('runas', 'Runas')]),
         ),
 
         # Connecting activity stream
@@ -496,6 +496,6 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='unifiedjob',
             name='execution_node',
-            field=models.TextField(default=b'', help_text='The node the job executed on.', editable=False, blank=True),
+            field=models.TextField(default='', help_text='The node the job executed on.', editable=False, blank=True),
         ),
     ]

awx/main/migrations/0010_v322_add_ovirt4_tower_inventory.py

@@ -20,11 +20,11 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='inventorysource',
             name='source',
-            field=models.CharField(default=b'', max_length=32, blank=True, choices=[(b'', 'Manual'), (b'file', 'File, Directory or Script'), (b'scm', 'Sourced from a Project'), (b'ec2', 'Amazon EC2'), (b'gce', 'Google Compute Engine'), (b'azure_rm', 'Microsoft Azure Resource Manager'), (b'vmware', 'VMware vCenter'), (b'satellite6', 'Red Hat Satellite 6'), (b'cloudforms', 'Red Hat CloudForms'), (b'openstack', 'OpenStack'), (b'rhv', 'Red Hat Virtualization'), (b'tower', 'Ansible Tower'), (b'custom', 'Custom Script')]),
+            field=models.CharField(default='', max_length=32, blank=True, choices=[('', 'Manual'), ('file', 'File, Directory or Script'), ('scm', 'Sourced from a Project'), ('ec2', 'Amazon EC2'), ('gce', 'Google Compute Engine'), ('azure_rm', 'Microsoft Azure Resource Manager'), ('vmware', 'VMware vCenter'), ('satellite6', 'Red Hat Satellite 6'), ('cloudforms', 'Red Hat CloudForms'), ('openstack', 'OpenStack'), ('rhv', 'Red Hat Virtualization'), ('tower', 'Ansible Tower'), ('custom', 'Custom Script')]),
         ),
         migrations.AlterField(
             model_name='inventoryupdate',
             name='source',
-            field=models.CharField(default=b'', max_length=32, blank=True, choices=[(b'', 'Manual'), (b'file', 'File, Directory or Script'), (b'scm', 'Sourced from a Project'), (b'ec2', 'Amazon EC2'), (b'gce', 'Google Compute Engine'), (b'azure_rm', 'Microsoft Azure Resource Manager'), (b'vmware', 'VMware vCenter'), (b'satellite6', 'Red Hat Satellite 6'), (b'cloudforms', 'Red Hat CloudForms'), (b'openstack', 'OpenStack'), (b'rhv', 'Red Hat Virtualization'), (b'tower', 'Ansible Tower'), (b'custom', 'Custom Script')]),
+            field=models.CharField(default='', max_length=32, blank=True, choices=[('', 'Manual'), ('file', 'File, Directory or Script'), ('scm', 'Sourced from a Project'), ('ec2', 'Amazon EC2'), ('gce', 'Google Compute Engine'), ('azure_rm', 'Microsoft Azure Resource Manager'), ('vmware', 'VMware vCenter'), ('satellite6', 'Red Hat Satellite 6'), ('cloudforms', 'Red Hat CloudForms'), ('openstack', 'OpenStack'), ('rhv', 'Red Hat Virtualization'), ('tower', 'Ansible Tower'), ('custom', 'Custom Script')]),
         ),
     ]

awx/main/migrations/0018_v330_add_additional_stdout_events.py

@@ -21,13 +21,13 @@ class Migration(migrations.Migration):
                 ('created', models.DateTimeField(default=None, editable=False)),
                 ('modified', models.DateTimeField(default=None, editable=False)),
                 ('event_data', awx.main.fields.JSONField(blank=True, default={})),
-                ('uuid', models.CharField(default=b'', editable=False, max_length=1024)),
+                ('uuid', models.CharField(default='', editable=False, max_length=1024)),
                 ('counter', models.PositiveIntegerField(default=0, editable=False)),
-                ('stdout', models.TextField(default=b'', editable=False)),
+                ('stdout', models.TextField(default='', editable=False)),
                 ('verbosity', models.PositiveIntegerField(default=0, editable=False)),
                 ('start_line', models.PositiveIntegerField(default=0, editable=False)),
                 ('end_line', models.PositiveIntegerField(default=0, editable=False)),
-                ('inventory_update', models.ForeignKey(editable=False, on_delete=django.db.models.deletion.CASCADE, related_name='generic_command_events', to='main.InventoryUpdate')),
+                ('inventory_update', models.ForeignKey(editable=False, on_delete=django.db.models.deletion.CASCADE, related_name='inventory_update_events', to='main.InventoryUpdate')),
             ],
             options={
                 'ordering': ('-pk',),
@@ -39,21 +39,21 @@ class Migration(migrations.Migration):
                 ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
                 ('created', models.DateTimeField(default=None, editable=False)),
                 ('modified', models.DateTimeField(default=None, editable=False)),
-                ('event', models.CharField(choices=[(b'runner_on_failed', 'Host Failed'), (b'runner_on_ok', 'Host OK'), (b'runner_on_error', 'Host Failure'), (b'runner_on_skipped', 'Host Skipped'), (b'runner_on_unreachable', 'Host Unreachable'), (b'runner_on_no_hosts', 'No Hosts Remaining'), (b'runner_on_async_poll', 'Host Polling'), (b'runner_on_async_ok', 'Host Async OK'), (b'runner_on_async_failed', 'Host Async Failure'), (b'runner_item_on_ok', 'Item OK'), (b'runner_item_on_failed', 'Item Failed'), (b'runner_item_on_skipped', 'Item Skipped'), (b'runner_retry', 'Host Retry'), (b'runner_on_file_diff', 'File Difference'), (b'playbook_on_start', 'Playbook Started'), (b'playbook_on_notify', 'Running Handlers'), (b'playbook_on_include', 'Including File'), (b'playbook_on_no_hosts_matched', 'No Hosts Matched'), (b'playbook_on_no_hosts_remaining', 'No Hosts Remaining'), (b'playbook_on_task_start', 'Task Started'), (b'playbook_on_vars_prompt', 'Variables Prompted'), (b'playbook_on_setup', 'Gathering Facts'), (b'playbook_on_import_for_host', 'internal: on Import for Host'), (b'playbook_on_not_import_for_host', 'internal: on Not Import for Host'), (b'playbook_on_play_start', 'Play Started'), (b'playbook_on_stats', 'Playbook Complete'), (b'debug', 'Debug'), (b'verbose', 'Verbose'), (b'deprecated', 'Deprecated'), (b'warning', 'Warning'), (b'system_warning', 'System Warning'), (b'error', 'Error')], max_length=100)),
+                ('event', models.CharField(choices=[('runner_on_failed', 'Host Failed'), ('runner_on_ok', 'Host OK'), ('runner_on_error', 'Host Failure'), ('runner_on_skipped', 'Host Skipped'), ('runner_on_unreachable', 'Host Unreachable'), ('runner_on_no_hosts', 'No Hosts Remaining'), ('runner_on_async_poll', 'Host Polling'), ('runner_on_async_ok', 'Host Async OK'), ('runner_on_async_failed', 'Host Async Failure'), ('runner_item_on_ok', 'Item OK'), ('runner_item_on_failed', 'Item Failed'), ('runner_item_on_skipped', 'Item Skipped'), ('runner_retry', 'Host Retry'), ('runner_on_file_diff', 'File Difference'), ('playbook_on_start', 'Playbook Started'), ('playbook_on_notify', 'Running Handlers'), ('playbook_on_include', 'Including File'), ('playbook_on_no_hosts_matched', 'No Hosts Matched'), ('playbook_on_no_hosts_remaining', 'No Hosts Remaining'), ('playbook_on_task_start', 'Task Started'), ('playbook_on_vars_prompt', 'Variables Prompted'), ('playbook_on_setup', 'Gathering Facts'), ('playbook_on_import_for_host', 'internal: on Import for Host'), ('playbook_on_not_import_for_host', 'internal: on Not Import for Host'), ('playbook_on_play_start', 'Play Started'), ('playbook_on_stats', 'Playbook Complete'), ('debug', 'Debug'), ('verbose', 'Verbose'), ('deprecated', 'Deprecated'), ('warning', 'Warning'), ('system_warning', 'System Warning'), ('error', 'Error')], max_length=100)),
                 ('event_data', awx.main.fields.JSONField(blank=True, default={})),
                 ('failed', models.BooleanField(default=False, editable=False)),
                 ('changed', models.BooleanField(default=False, editable=False)),
-                ('uuid', models.CharField(default=b'', editable=False, max_length=1024)),
-                ('playbook', models.CharField(default=b'', editable=False, max_length=1024)),
-                ('play', models.CharField(default=b'', editable=False, max_length=1024)),
-                ('role', models.CharField(default=b'', editable=False, max_length=1024)),
-                ('task', models.CharField(default=b'', editable=False, max_length=1024)),
+                ('uuid', models.CharField(default='', editable=False, max_length=1024)),
+                ('playbook', models.CharField(default='', editable=False, max_length=1024)),
+                ('play', models.CharField(default='', editable=False, max_length=1024)),
+                ('role', models.CharField(default='', editable=False, max_length=1024)),
+                ('task', models.CharField(default='', editable=False, max_length=1024)),
                 ('counter', models.PositiveIntegerField(default=0, editable=False)),
-                ('stdout', models.TextField(default=b'', editable=False)),
+                ('stdout', models.TextField(default='', editable=False)),
                 ('verbosity', models.PositiveIntegerField(default=0, editable=False)),
                 ('start_line', models.PositiveIntegerField(default=0, editable=False)),
                 ('end_line', models.PositiveIntegerField(default=0, editable=False)),
-                ('project_update', models.ForeignKey(editable=False, on_delete=django.db.models.deletion.CASCADE, related_name='generic_command_events', to='main.ProjectUpdate')),
+                ('project_update', models.ForeignKey(editable=False, on_delete=django.db.models.deletion.CASCADE, related_name='project_update_events', to='main.ProjectUpdate')),
             ],
             options={
                 'ordering': ('pk',),
@@ -66,18 +66,30 @@ class Migration(migrations.Migration):
                 ('created', models.DateTimeField(default=None, editable=False)),
                 ('modified', models.DateTimeField(default=None, editable=False)),
                 ('event_data', awx.main.fields.JSONField(blank=True, default={})),
-                ('uuid', models.CharField(default=b'', editable=False, max_length=1024)),
+                ('uuid', models.CharField(default='', editable=False, max_length=1024)),
                 ('counter', models.PositiveIntegerField(default=0, editable=False)),
-                ('stdout', models.TextField(default=b'', editable=False)),
+                ('stdout', models.TextField(default='', editable=False)),
                 ('verbosity', models.PositiveIntegerField(default=0, editable=False)),
                 ('start_line', models.PositiveIntegerField(default=0, editable=False)),
                 ('end_line', models.PositiveIntegerField(default=0, editable=False)),
-                ('system_job', models.ForeignKey(editable=False, on_delete=django.db.models.deletion.CASCADE, related_name='generic_command_events', to='main.SystemJob')),
+                ('system_job', models.ForeignKey(editable=False, on_delete=django.db.models.deletion.CASCADE, related_name='system_job_events', to='main.SystemJob')),
             ],
             options={
                 'ordering': ('-pk',),
             },
         ),
+        migrations.AlterIndexTogether(
+            name='inventoryupdateevent',
+            index_together=set([('inventory_update', 'start_line'), ('inventory_update', 'uuid'), ('inventory_update', 'end_line')]),
+        ),
+        migrations.AlterIndexTogether(
+            name='projectupdateevent',
+            index_together=set([('project_update', 'event'), ('project_update', 'end_line'), ('project_update', 'start_line'), ('project_update', 'uuid')]),
+        ),
+        migrations.AlterIndexTogether(
+            name='systemjobevent',
+            index_together=set([('system_job', 'end_line'), ('system_job', 'uuid'), ('system_job', 'start_line')]),
+        ),
         migrations.RemoveField(
             model_name='unifiedjob',
             name='result_stdout_file',

awx/main/migrations/0021_v330_declare_new_rbac_roles.py

@@ -18,77 +18,77 @@ class Migration(migrations.Migration):
         migrations.AddField(
             model_name='organization',
             name='execute_role',
-            field=awx.main.fields.ImplicitRoleField(null=b'True', on_delete=django.db.models.deletion.CASCADE, parent_role=b'admin_role', related_name='+', to='main.Role'),
+            field=awx.main.fields.ImplicitRoleField(null='True', on_delete=django.db.models.deletion.CASCADE, parent_role='admin_role', related_name='+', to='main.Role'),
         ),
         migrations.AddField(
             model_name='organization',
             name='job_template_admin_role',
-            field=awx.main.fields.ImplicitRoleField(editable=False, null=b'True', on_delete=django.db.models.deletion.CASCADE, parent_role=b'admin_role', related_name='+', to='main.Role'),
+            field=awx.main.fields.ImplicitRoleField(editable=False, null='True', on_delete=django.db.models.deletion.CASCADE, parent_role='admin_role', related_name='+', to='main.Role'),
         ),
         migrations.AddField(
             model_name='organization',
             name='credential_admin_role',
-            field=awx.main.fields.ImplicitRoleField(null=b'True', on_delete=django.db.models.deletion.CASCADE, parent_role=b'admin_role', related_name='+', to='main.Role'),
+            field=awx.main.fields.ImplicitRoleField(null='True', on_delete=django.db.models.deletion.CASCADE, parent_role='admin_role', related_name='+', to='main.Role'),
         ),
         migrations.AddField(
             model_name='organization',
             name='inventory_admin_role',
-            field=awx.main.fields.ImplicitRoleField(null=b'True', on_delete=django.db.models.deletion.CASCADE, parent_role=b'admin_role', related_name='+', to='main.Role'),
+            field=awx.main.fields.ImplicitRoleField(null='True', on_delete=django.db.models.deletion.CASCADE, parent_role='admin_role', related_name='+', to='main.Role'),
         ),
         migrations.AddField(
             model_name='organization',
             name='project_admin_role',
-            field=awx.main.fields.ImplicitRoleField(null=b'True', on_delete=django.db.models.deletion.CASCADE, parent_role=b'admin_role', related_name='+', to='main.Role'),
+            field=awx.main.fields.ImplicitRoleField(null='True', on_delete=django.db.models.deletion.CASCADE, parent_role='admin_role', related_name='+', to='main.Role'),
         ),
         migrations.AddField(
             model_name='organization',
             name='workflow_admin_role',
-            field=awx.main.fields.ImplicitRoleField(null=b'True', on_delete=django.db.models.deletion.CASCADE, parent_role=b'admin_role', related_name='+', to='main.Role'),
+            field=awx.main.fields.ImplicitRoleField(null='True', on_delete=django.db.models.deletion.CASCADE, parent_role='admin_role', related_name='+', to='main.Role'),
         ),
         migrations.AddField(
             model_name='organization',
             name='notification_admin_role',
-            field=awx.main.fields.ImplicitRoleField(null=b'True', on_delete=django.db.models.deletion.CASCADE, parent_role=b'admin_role', related_name='+', to='main.Role'),
+            field=awx.main.fields.ImplicitRoleField(null='True', on_delete=django.db.models.deletion.CASCADE, parent_role='admin_role', related_name='+', to='main.Role'),
         ),
         migrations.AlterField(
             model_name='credential',
             name='admin_role',
-            field=awx.main.fields.ImplicitRoleField(null=b'True', on_delete=django.db.models.deletion.CASCADE, parent_role=[b'singleton:system_administrator', b'organization.credential_admin_role'], related_name='+', to='main.Role'),
+            field=awx.main.fields.ImplicitRoleField(null='True', on_delete=django.db.models.deletion.CASCADE, parent_role=['singleton:system_administrator', 'organization.credential_admin_role'], related_name='+', to='main.Role'),
         ),
         migrations.AlterField(
             model_name='inventory',
             name='admin_role',
-            field=awx.main.fields.ImplicitRoleField(null=b'True', on_delete=django.db.models.deletion.CASCADE, parent_role=b'organization.inventory_admin_role', related_name='+', to='main.Role'),
+            field=awx.main.fields.ImplicitRoleField(null='True', on_delete=django.db.models.deletion.CASCADE, parent_role='organization.inventory_admin_role', related_name='+', to='main.Role'),
         ),
         migrations.AlterField(
             model_name='project',
             name='admin_role',
-            field=awx.main.fields.ImplicitRoleField(null=b'True', on_delete=django.db.models.deletion.CASCADE, parent_role=[b'organization.project_admin_role', b'singleton:system_administrator'], related_name='+', to='main.Role'),
+            field=awx.main.fields.ImplicitRoleField(null='True', on_delete=django.db.models.deletion.CASCADE, parent_role=['organization.project_admin_role', 'singleton:system_administrator'], related_name='+', to='main.Role'),
         ),
         migrations.AlterField(
             model_name='workflowjobtemplate',
             name='admin_role',
-            field=awx.main.fields.ImplicitRoleField(null=b'True', on_delete=django.db.models.deletion.CASCADE, parent_role=[b'singleton:system_administrator', b'organization.workflow_admin_role'], related_name='+', to='main.Role'),
+            field=awx.main.fields.ImplicitRoleField(null='True', on_delete=django.db.models.deletion.CASCADE, parent_role=['singleton:system_administrator', 'organization.workflow_admin_role'], related_name='+', to='main.Role'),
         ),
         migrations.AlterField(
             model_name='workflowjobtemplate',
             name='execute_role',
-            field=awx.main.fields.ImplicitRoleField(null=b'True', on_delete=django.db.models.deletion.CASCADE, parent_role=[b'admin_role', b'organization.execute_role'], related_name='+', to='main.Role'),
+            field=awx.main.fields.ImplicitRoleField(null='True', on_delete=django.db.models.deletion.CASCADE, parent_role=['admin_role', 'organization.execute_role'], related_name='+', to='main.Role'),
         ),
         migrations.AlterField(
             model_name='jobtemplate',
             name='admin_role',
-            field=awx.main.fields.ImplicitRoleField(editable=False, null=b'True', on_delete=django.db.models.deletion.CASCADE, parent_role=[b'project.organization.job_template_admin_role', b'inventory.organization.job_template_admin_role'], related_name='+', to='main.Role'),
+            field=awx.main.fields.ImplicitRoleField(editable=False, null='True', on_delete=django.db.models.deletion.CASCADE, parent_role=['project.organization.job_template_admin_role', 'inventory.organization.job_template_admin_role'], related_name='+', to='main.Role'),
         ),
         migrations.AlterField(
             model_name='jobtemplate',
             name='execute_role',
-            field=awx.main.fields.ImplicitRoleField(null=b'True', on_delete=django.db.models.deletion.CASCADE, parent_role=[b'admin_role', b'project.organization.execute_role', b'inventory.organization.execute_role'], related_name='+', to='main.Role'),
+            field=awx.main.fields.ImplicitRoleField(null='True', on_delete=django.db.models.deletion.CASCADE, parent_role=['admin_role', 'project.organization.execute_role', 'inventory.organization.execute_role'], related_name='+', to='main.Role'),
         ),
         migrations.AlterField(
             model_name='organization',
             name='member_role',
-            field=awx.main.fields.ImplicitRoleField(editable=False, null=b'True', on_delete=django.db.models.deletion.CASCADE, parent_role=[b'admin_role', b'execute_role', b'project_admin_role', b'inventory_admin_role', b'workflow_admin_role', b'notification_admin_role', b'credential_admin_role', b'job_template_admin_role'], related_name='+', to='main.Role'),
+            field=awx.main.fields.ImplicitRoleField(editable=False, null='True', on_delete=django.db.models.deletion.CASCADE, parent_role=['admin_role', 'execute_role', 'project_admin_role', 'inventory_admin_role', 'workflow_admin_role', 'notification_admin_role', 'credential_admin_role', 'job_template_admin_role'], related_name='+', to='main.Role'),
         ),
 
     ]

awx/main/migrations/0025_v330_add_oauth_activity_stream_registrar.py

@@ -35,8 +35,8 @@ class Migration(migrations.Migration):
                 ('skip_authorization', models.BooleanField(default=False)),
                 ('created', models.DateTimeField(auto_now_add=True)),
                 ('updated', models.DateTimeField(auto_now=True)),
-                ('description', models.TextField(blank=True, default=b'')),
-                ('logo_data', models.TextField(default=b'', editable=False, validators=[django.core.validators.RegexValidator(re.compile(b'.*'))])),
+                ('description', models.TextField(blank=True, default='')),
+                ('logo_data', models.TextField(default='', editable=False, validators=[django.core.validators.RegexValidator(re.compile('.*'))])),
                 ('user', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='main_oauth2application', to=settings.AUTH_USER_MODEL)),
             ],
             options={
@@ -52,7 +52,7 @@ class Migration(migrations.Migration):
                 ('scope', models.TextField(blank=True)),
                 ('created', models.DateTimeField(auto_now_add=True)),
                 ('updated', models.DateTimeField(auto_now=True)),
-                ('description', models.CharField(blank=True, default=b'', max_length=200)),
+                ('description', models.CharField(blank=True, default='', max_length=200)),
                 ('last_used', models.DateTimeField(default=None, editable=False, null=True)),
                 ('application', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, to=settings.OAUTH2_PROVIDER_APPLICATION_MODEL)),
                 ('user', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='main_oauth2accesstoken', to=settings.AUTH_USER_MODEL)),
@@ -64,12 +64,12 @@ class Migration(migrations.Migration):
         migrations.AddField(
             model_name='activitystream',
             name='o_auth2_access_token',
-            field=models.ManyToManyField(to='main.OAuth2AccessToken', blank=True, related_name='main_o_auth2_accesstoken'),
+            field=models.ManyToManyField(to='main.OAuth2AccessToken', blank=True),
         ),
         migrations.AddField(
             model_name='activitystream',
             name='o_auth2_application',
-            field=models.ManyToManyField(to='main.OAuth2Application', blank=True, related_name='main_o_auth2_application'),
+            field=models.ManyToManyField(to='main.OAuth2Application', blank=True),
         ),
 
     ]

awx/main/migrations/0033_v330_oauth_help_text.py

@@ -20,7 +20,7 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='oauth2accesstoken',
             name='scope',
-            field=models.TextField(blank=True, default=b'write', help_text="Allowed scopes, further restricts user's permissions."),
+            field=models.TextField(blank=True, default='write', help_text="Allowed scopes, further restricts user's permissions."),
         ),
         migrations.AlterField(
             model_name='oauth2accesstoken',
@@ -30,7 +30,7 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='oauth2application',
             name='authorization_grant_type',
-            field=models.CharField(choices=[(b'authorization-code', 'Authorization code'), (b'implicit', 'Implicit'), (b'password', 'Resource owner password-based'), (b'client-credentials', 'Client credentials')], help_text='The Grant type the user must use for acquire tokens for this application.', max_length=32),
+            field=models.CharField(choices=[('authorization-code', 'Authorization code'), ('implicit', 'Implicit'), ('password', 'Resource owner password-based'), ('client-credentials', 'Client credentials')], help_text='The Grant type the user must use for acquire tokens for this application.', max_length=32),
         ),
         migrations.AlterField(
             model_name='oauth2application',
@@ -40,7 +40,7 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='oauth2application',
             name='client_type',
-            field=models.CharField(choices=[(b'confidential', 'Confidential'), (b'public', 'Public')], help_text='Set to Public or Confidential depending on how secure the client device is.', max_length=32),
+            field=models.CharField(choices=[('confidential', 'Confidential'), ('public', 'Public')], help_text='Set to Public or Confidential depending on how secure the client device is.', max_length=32),
         ),
         migrations.AlterField(
             model_name='oauth2application',

awx/main/migrations/0035_v330_more_oauth2_help_text.py

@@ -16,6 +16,6 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='oauth2accesstoken',
             name='scope',
-            field=models.TextField(blank=True, help_text="Allowed scopes, further restricts user's permissions. Must be a simple space-separated string with allowed scopes ['read', 'write']."),
+            field=models.TextField(blank=True, default='write', help_text="Allowed scopes, further restricts user's permissions. Must be a simple space-separated string with allowed scopes ['read', 'write']."),
         ),
     ]

awx/main/migrations/0040_v330_unifiedjob_controller_node.py

@@ -15,6 +15,6 @@ class Migration(migrations.Migration):
         migrations.AddField(
             model_name='unifiedjob',
             name='controller_node',
-            field=models.TextField(blank=True, default=b'', editable=False, help_text='The instance that managed the isolated execution environment.'),
+            field=models.TextField(blank=True, default='', editable=False, help_text='The instance that managed the isolated execution environment.'),
         ),
     ]

awx/main/migrations/0042_v330_org_member_role_deparent.py

@@ -18,12 +18,12 @@ class Migration(migrations.Migration):
         migrations.AlterField(
             model_name='organization',
             name='member_role',
-            field=awx.main.fields.ImplicitRoleField(editable=False, null=b'True', on_delete=django.db.models.deletion.CASCADE, parent_role=[b'admin_role'], related_name='+', to='main.Role'),
+            field=awx.main.fields.ImplicitRoleField(editable=False, null='True', on_delete=django.db.models.deletion.CASCADE, parent_role=['admin_role'], related_name='+', to='main.Role'),
         ),
         migrations.AlterField(
             model_name='organization',
             name='read_role',
-            field=awx.main.fields.ImplicitRoleField(editable=False, null=b'True', on_delete=django.db.models.deletion.CASCADE, parent_role=[b'member_role', b'auditor_role', b'execute_role', b'project_admin_role', b'inventory_admin_role', b'workflow_admin_role', b'notification_admin_role', b'credential_admin_role', b'job_template_admin_role'], related_name='+', to='main.Role'),
+            field=awx.main.fields.ImplicitRoleField(editable=False, null='True', on_delete=django.db.models.deletion.CASCADE, parent_role=['member_role', 'auditor_role', 'execute_role', 'project_admin_role', 'inventory_admin_role', 'workflow_admin_role', 'notification_admin_role', 'credential_admin_role', 'job_template_admin_role'], related_name='+', to='main.Role'),
         ),
         migrations.RunPython(rebuild_role_hierarchy),
     ]

awx/main/migrations/0043_v330_oauth2accesstoken_modified.py

@@ -15,6 +15,6 @@ class Migration(migrations.Migration):
         migrations.AddField(
             model_name='oauth2accesstoken',
             name='modified',
-            field=models.DateTimeField(editable=False),
+            field=models.DateTimeField(editable=False, auto_now=True),
         ),
     ]