Update CHANGELOG.md

Update some links and notes in the changelog

SUMMARY
Fix typo, remove duplicate change note, fix a wrong link, add link to the ui virtualenv removal

Reviewed-by: Ryan Petrello <None>

.dockerignore

@@ -1,2 +1,3 @@
-.git
 awx/ui/node_modules
+awx/ui_next/node_modules
+Dockerfile

.env

@@ -1,3 +1,3 @@
 PYTHONUNBUFFERED=true
 SELENIUM_DOCKER_TAG=latest
-
+COMPOSE_PROJECT_NAME=tools

.github/BOTMETA.yml

@@ -1,3 +1,4 @@
+---
 files:
     awx/ui/:
         labels: component:ui

.github/ISSUE_TEMPLATE.md

@@ -30,8 +30,9 @@ https://www.ansible.com/security
 
 ##### STEPS TO REPRODUCE
 
-<!-- For bugs, please show exactly how to reproduce the problem. For new
-features, show how the feature would be used.  -->
+<!-- For new features, show how the feature would be used. For bugs, please show
+exactly how to reproduce the problem. Ideally, provide all steps and data needed
+to recreate the bug from a new awx install. -->
 
 ##### EXPECTED RESULTS
 

.github/ISSUE_TEMPLATE/bug_report.md

@@ -3,6 +3,12 @@ name: "\U0001F41B Bug report"
 about: Create a report to help us improve
 
 ---
+<!-- Issues are for **concrete, actionable bugs and feature requests** only - if you're just asking for debugging help or technical support, please use:
+
+- http://webchat.freenode.net/?channels=ansible-awx
+- https://groups.google.com/forum/#!forum/awx-project
+
+We have to limit this because of limited volunteer time to respond to issues! -->
 
 ##### ISSUE TYPE
  - Bug Report

.github/ISSUE_TEMPLATE/feature_request.md

@@ -3,6 +3,12 @@ name: "✨ Feature request"
 about: Suggest an idea for this project
 
 ---
+<!-- Issues are for **concrete, actionable bugs and feature requests** only - if you're just asking for debugging help or technical support, please use:
+
+- http://webchat.freenode.net/?channels=ansible-awx
+- https://groups.google.com/forum/#!forum/awx-project
+
+We have to limit this because of limited volunteer time to respond to issues! -->
 
 ##### ISSUE TYPE
  - Feature Idea

.github/dependabot.yml

@@ -0,0 +1,7 @@
+---
+version: 2
+updates:
+  - package-ecosystem: "pip"
+    directory: "/requirements"
+    schedule:
+      interval: "monthly"

.gitignore

@@ -29,11 +29,18 @@ awx/ui/client/languages
 awx/ui/templates/ui/index.html
 awx/ui/templates/ui/installing.html
 awx/ui_next/node_modules/
+awx/ui_next/src/locales/*/messages.js
 awx/ui_next/coverage/
-awx/ui_next/build/locales/_build
-/tower-license
-/tower-license/**
+awx/ui_next/build
+awx/ui_next/.env.local
+awx/ui_next/instrumented
+rsyslog.pid
 tools/prometheus/data
+tools/docker-compose/ansible/awx_dump.sql
+tools/docker-compose/Dockerfile
+tools/docker-compose/_build
+tools/docker-compose/_sources
+tools/docker-compose/overrides/
 
 # Tower setup playbook testing
 setup/test/roles/postgresql
@@ -84,6 +91,9 @@ awx/awx_test.sqlite3-journal
 # Mac OS X
 *.DS_Store
 
+# VSCode
+.vscode/
+
 # Editors
 *.sw[poj]
 *~
@@ -137,9 +147,14 @@ use_dev_supervisor.txt
 # Ansible module tests
 /awx_collection_test_venv/
 /awx_collection/*.tar.gz
-/awx_collection/galaxy.yml
 /sanity/
+/awx_collection_build/
 
 .idea/*
 *.unison.tmp
 *.#
+/awx/ui_next/.ui-built
+/Dockerfile
+/_build/
+/_build_kube_dev/
+/Dockerfile.kube-dev

.yamllint

@@ -0,0 +1,12 @@
+---
+ignore: |
+  .tox
+  awx/main/tests/data/inventory/plugins/**
+  # vault files
+  awx/main/tests/data/ansible_utils/playbooks/valid/vault.yml
+  awx/ui/test/e2e/tests/smoke-vars.yml
+
+extends: default
+
+rules:
+  line-length: disable

CHANGELOG.md

@@ -0,0 +1,433 @@
+# Changelog
+
+This is a list of high-level changes for each release of AWX. A full list of commits can be found at `https://github.com/ansible/awx/releases/tag/<version>`.
+
+# 18.0.0 (March 23, 2021)
+
+**IMPORTANT INSTALL AND UPGRADE NOTES**
+
+Starting in version 18.0, the [AWX Operator](https://github.com/ansible/awx-operator) is the preferred way to install AWX: https://github.com/ansible/awx/blob/devel/INSTALL.md#installing-awx
+
+If you have a pre-existing installation of AWX that utilizes the Docker-based installation method, this install method has ** notably changed** from 17.x to 18.x.  For details, please see:
+
+- https://groups.google.com/g/awx-project/c/47MjWSUQaOc/m/bCjSDn0eBQAJ
+- https://github.com/ansible/awx/blob/devel/tools/docker-compose
+- https://github.com/ansible/awx/blob/devel/tools/docker-compose/docs/data_migration.md
+
+### Introducing Execution Environments
+
+After a herculean effort from a number of contributors, we're excited to announce that AWX 18.0.0 introduces a new concept called Execution Environments.
+
+Execution Environments are container images which consist of everything necessary to run a playbook within AWX, and which drive the entire management and lifecycle of playbook execution runtime in AWX: https://github.com/ansible/awx/issues/5157.  This means that going forward, AWX no longer utilizes the [bubblewrap](https://github.com/containers/bubblewrap) project for playbook isolation, but instead utilizes a container per playbook run.
+
+Much like custom virtualenvs, custom Execution Environments can be crafted to specify additional Python or system-level dependencies.  Ansible Builder outputs images you can upload to your registry which can *then* be defined in AWX and utilized for playbook runs.
+
+To learn more about Ansible Builder and Execution Environments, see: https://www.ansible.com/blog/introduction-to-ansible-builder
+
+### Other Notable Changes
+
+- Removed `installer` directory.
+  - The Kubernetes installer has been removed in favor of [AWX Operator](https://github.com/ansible/awx-operator).
+  - The "Local Docker" install method has been removed in favor of the development environment. Details can be found at: https://github.com/ansible/awx/blob/devel/tools/docker-compose/README.md
+- Removal of custom virtual environments https://github.com/ansible/awx/pull/9498
+  - Custom virtual environments have been replaced by Execution Environments https://github.com/ansible/awx/pull/9570
+- The default Container Group Pod definition has changed. All custom Pod specs have been reset. https://github.com/ansible/awx/commit/05ef51f710dad8f8036bc5acee4097db4adc0d71
+- Added user interface for the activity stream: https://github.com/ansible/awx/pull/9083
+- Converted many of the top-level list views (Jobs, Teams, Hosts, Inventories, Projects, and more) to a new, permanent table component for substantially increased responsiveness, usability, maintainability, and other 'ility's: https://github.com/ansible/awx/pull/8970, https://github.com/ansible/awx/pull/9182 and many others!
+- Added click-to-expand details for job tables
+- Added search filtering to job output https://github.com/ansible/awx/pull/9208
+- Added the new migration, update, and "installation in progress" page https://github.com/ansible/awx/pull/9123
+- Added the user interface for job settings https://github.com/ansible/awx/pull/8661
+- Runtime errors from jobs are now displayed, along with an explanation for what went wrong, on the output page https://github.com/ansible/awx/pull/8726
+- You can now cancel a running job from its output and details panel https://github.com/ansible/awx/pull/9199
+- Fixed a bug where launch prompt inputs were unexpectedly deposited in the url: https://github.com/ansible/awx/pull/9231
+- Playbook, credential type, and inventory file inputs now support type-ahead and manual type-in! https://github.com/ansible/awx/pull/9120
+- Added ability to relaunch against failed hosts: https://github.com/ansible/awx/pull/9225
+- Added pending workflow approval count to the application header https://github.com/ansible/awx/pull/9334
+- Added user interface for management jobs: https://github.com/ansible/awx/pull/9224
+- Added toast message to show notification template test result to notification templates list https://github.com/ansible/awx/pull/9318
+- Replaced CodeMirror with AceEditor for editing template variables and notification templates https://github.com/ansible/awx/pull/9281
+- Added support for filtering and pagination on job output https://github.com/ansible/awx/pull/9208
+- Added support for html in custom login text https://github.com/ansible/awx/pull/9519
+
+# 17.1.0 (March 9, 2021)
+- Addressed a security issue in AWX (CVE-2021-20253)
+- Fixed a bug permissions error related to redis in K8S-based deployments: https://github.com/ansible/awx/issues/9401
+
+# 17.0.1 (January 26, 2021)
+- Fixed pgdocker directory permissions issue with Local Docker installer: https://github.com/ansible/awx/pull/9152
+- Fixed a bug in the UI which caused toggle settings to not be changed when clicked: https://github.com/ansible/awx/pull/9093
+
+# 17.0.0 (January 22, 2021)
+- AWX now requires PostgreSQL 12 by default: https://github.com/ansible/awx/pull/8943
+  **Note:** users who encounter permissions errors at upgrade time should `chown -R ~/.awx/pgdocker` to ensure it's owned by the user running the install playbook
+- Added support for region name for OpenStack inventory: https://github.com/ansible/awx/issues/5080
+- Added the ability to chain undefined attributes in custom notification templates: https://github.com/ansible/awx/issues/8677
+- Dramatically simplified the `image_build` role: https://github.com/ansible/awx/pull/8980
+- Fixed a bug which can cause schema migrations to fail at install time: https://github.com/ansible/awx/issues/9077
+- Fixed a bug which caused the `is_superuser` user property to be out of date in certain circumstances: https://github.com/ansible/awx/pull/8833
+- Fixed a bug which sometimes results in race conditions on setting access: https://github.com/ansible/awx/pull/8580
+- Fixed a bug which sometimes causes an unexpected delay in stdout for some playbooks: https://github.com/ansible/awx/issues/9085
+- (UI) Added support for credential password prompting on job launch: https://github.com/ansible/awx/pull/9028
+- (UI) Added the ability to configure LDAP settings in the UI: https://github.com/ansible/awx/issues/8291
+- (UI) Added a sync button to the Project detail view: https://github.com/ansible/awx/issues/8847
+- (UI) Added a form for configuring Google Outh 2.0 settings: https://github.com/ansible/awx/pull/8762
+- (UI) Added searchable keys and related keys to the Credentials list: https://github.com/ansible/awx/issues/8603
+- (UI) Added support for advanced search and copying to Notification Templates: https://github.com/ansible/awx/issues/7879
+- (UI) Added support for prompting on workflow nodes: https://github.com/ansible/awx/issues/5913
+- (UI) Added support for session timeouts: https://github.com/ansible/awx/pull/8250
+- (UI) Fixed a bug that broke websocket streaming for the insecure ws:// protocol: https://github.com/ansible/awx/pull/8877
+- (UI) Fixed a bug in the user interface when a translation for the browser's preferred locale isn't available: https://github.com/ansible/awx/issues/8884
+- (UI) Fixed bug where navigating from one survey question form directly to another wasn't reloading the form: https://github.com/ansible/awx/issues/7522
+- (UI) Fixed a bug which can cause an uncaught error while launching a Job Template: https://github.com/ansible/awx/issues/8936
+- Updated autobahn to address CVE-2020-35678
+
+## 16.0.0 (December 10, 2020)
+- AWX now ships with a reimagined user interface.  **Please read this before upgrading:** https://groups.google.com/g/awx-project/c/KuT5Ao92HWo
+- Removed support for syncing inventory from Red Hat CloudForms - https://github.com/ansible/awx/commit/0b701b3b2
+- Removed support for Mercurial-based project updates - https://github.com/ansible/awx/issues/7932
+- Upgraded NodeJS to actively maintained LTS 14.15.1 - https://github.com/ansible/awx/pull/8766
+- Added Git-LFS to the default image build - https://github.com/ansible/awx/pull/8700
+- Added the ability to specify `metadata.labels` in the podspec for container groups - https://github.com/ansible/awx/issues/8486
+- Added support for Kubernetes pod annotations - https://github.com/ansible/awx/pull/8434
+- Added the ability to label the web container in local Docker installs - https://github.com/ansible/awx/pull/8449
+- Added additional metadata (as an extra var) to playbook runs to report the SCM branch name - https://github.com/ansible/awx/pull/8433
+- Fixed a bug that caused k8s installations to fail due to an incorrect Helm repo - https://github.com/ansible/awx/issues/8715
+- Fixed a bug that prevented certain Workflow Approval resources from being deleted - https://github.com/ansible/awx/pull/8612
+- Fixed a bug that prevented the deletion of inventories stuck in "pending deletion" state - https://github.com/ansible/awx/issues/8525
+- Fixed a display bug in webhook notifications with certain unicode characters - https://github.com/ansible/awx/issues/7400
+- Improved support for exporting dependent objects (Inventory Hosts and Groups) in the `awx export` CLI tool - https://github.com/ansible/awx/commit/607bc0788
+
+## 15.0.1 (October 20, 2020)
+- Added several optimizations to improve performance for a variety of high-load simultaneous job launch use cases https://github.com/ansible/awx/pull/8403
+- Added the ability to source roles and collections from requirements.yaml files (not just requirements.yml) - https://github.com/ansible/awx/issues/4540
+- awx.awx collection modules now provide a clearer error message for incompatible versions of awxkit - https://github.com/ansible/awx/issues/8127
+- Fixed a bug in notification messages that contain certain unicode characters - https://github.com/ansible/awx/issues/7400
+- Fixed a bug that prevents the deletion of Workflow Approval records - https://github.com/ansible/awx/issues/8305
+- Fixed a bug that broke the selection of webhook credentials - https://github.com/ansible/awx/issues/7892
+- Fixed a bug which can cause confusing behavior for social auth logins across distinct browser tabs - https://github.com/ansible/awx/issues/8154
+- Fixed several bugs in the output of Workflow Job Templates using the `awx export` tool - https://github.com/ansible/awx/issues/7798 https://github.com/ansible/awx/pull/7847
+- Fixed a race condition that can lead to missing hosts when running parallel inventory syncs - https://github.com/ansible/awx/issues/5571
+- Fixed an HTTP 500 error when certain LDAP group parameters aren't properly set - https://github.com/ansible/awx/issues/7622
+- Updated a few dependencies in response to several CVEs:
+    * CVE-2020-7720
+    * CVE-2020-7743
+    * CVE-2020-7676
+
+## 15.0.0 (September 30, 2020)
+- Added improved support for fetching Ansible collections from private Galaxy content sources (such as https://github.com/ansible/galaxy_ng) - https://github.com/ansible/awx/issues/7813
+  **Note:** as part of this change, new Organizations created in the AWX API will _no longer_ automatically synchronize roles and collections from galaxy.ansible.com by default.  More details on this change can be found at:  https://github.com/ansible/awx/issues/8341#issuecomment-707310633
+- AWX now utilizes a version of certifi that auto-discovers certificates in the system certificate store - https://github.com/ansible/awx/pull/8242
+- Added support for arbitrary custom inventory plugin configuration: https://github.com/ansible/awx/issues/5150
+- Added an optional setting to disable the auto-creation of organizations and teams on successful SAML login. - https://github.com/ansible/awx/pull/8069
+- Added a number of optimizations to AWX's callback receiver to improve the speed of stdout processing for simultaneous playbooks runs - https://github.com/ansible/awx/pull/8193 https://github.com/ansible/awx/pull/8191
+- Added the ability to use `!include` and `!import` constructors when constructing YAML for use with the AWX CLI - https://github.com/ansible/awx/issues/8135
+- Fixed a bug that prevented certain users from being able to edit approval nodes in Workflows - https://github.com/ansible/awx/pull/8253
+- Fixed a bug that broke password prompting for credentials in certain cases - https://github.com/ansible/awx/issues/8202
+- Fixed a bug which can cause PostgreSQL deadlocks when running many parallel playbooks against large shared inventories - https://github.com/ansible/awx/issues/8145
+- Fixed a bug which can cause delays in AWX's task manager when large numbers of simultaneous jobs are scheduled - https://github.com/ansible/awx/issues/7655
+- Fixed a bug which can cause certain scheduled jobs - those that run every X minute(s) or hour(s) - to fail to run at the proper time - https://github.com/ansible/awx/issues/8071
+- Fixed a performance issue for playbooks that store large amounts of data using the `set_stats` module - https://github.com/ansible/awx/issues/8006
+- Fixed a bug related to AWX's handling of the auth_path argument for the HashiVault KeyValue credential plugin - https://github.com/ansible/awx/pull/7991
+- Fixed a bug that broke support for Remote Archive SCM Type project syncs on platforms that utilize Python2 - https://github.com/ansible/awx/pull/8057
+- Updated to the latest version of Django Rest Framework to address CVE-2020-25626
+- Updated to the latest version of Django to address CVE-2020-24583 and CVE-2020-24584
+- Updated to the latest verson of channels_redis to address a bug that slowly causes Daphne processes to leak memory over time - https://github.com/django/channels_redis/issues/212
+
+## 14.1.0 (Aug 25, 2020)
+- AWX images can now be built on ARM64 - https://github.com/ansible/awx/pull/7607
+- Added the Remote Archive SCM Type to support using immutable artifacts and releases (such as tarballs and zip files) as projects - https://github.com/ansible/awx/issues/7954
+- Deprecated official support for Mercurial-based project updates - https://github.com/ansible/awx/issues/7932
+- Added resource import/export support to the official AWX collection - https://github.com/ansible/awx/issues/7329
+- Added the ability to import YAML-based resources (instead of just JSON) when using the AWX CLI - https://github.com/ansible/awx/pull/7808
+- Users upgrading from older versions of AWX may encounter an issue that causes their postgres container to restart in a loop (https://github.com/ansible/awx/issues/7854) - if you encounter this, bring your containers down and then back up (e.g., `docker-compose down && docker-compose up -d`) after upgrading to 14.1.0.
+- Updated the AWX CLI to export labels associated with Workflow Job Templates - https://github.com/ansible/awx/pull/7847
+- Updated to the latest python-ldap to address a bug - https://github.com/ansible/awx/issues/7868
+- Upgraded git-python to fix a bug that caused workflows to sometimes fail - https://github.com/ansible/awx/issues/6119
+- Worked around a bug in the channels_redis library that slowly causes Daphne processes to leak memory over time - https://github.com/django/channels_redis/issues/212
+- Fixed a bug in the AWX CLI that prevented Workflow nodes from importing properly - https://github.com/ansible/awx/issues/7793
+- Fixed a bug in the awx.awx collection release process that templated the wrong version - https://github.com/ansible/awx/issues/7870
+- Fixed a bug that caused errors rendering stdout that contained UTF-16 surrogate pairs - https://github.com/ansible/awx/pull/7918
+
+## 14.0.0 (Aug 6, 2020)
+- As part of our commitment to inclusivity in open source, we recently took some time to audit AWX's source code and user interface and replace certain terminology with more inclusive language.  Strictly speaking, this isn't a bug or a feature, but we think it's important and worth calling attention to:
+    * https://github.com/ansible/awx/commit/78229f58715fbfbf88177e54031f532543b57acc
+    * https://www.redhat.com/en/blog/making-open-source-more-inclusive-eradicating-problematic-language
+- Installing roles and collections via requirements.yml as part of Project Updates now requires at least Ansible 2.9 - https://github.com/ansible/awx/issues/7769
+- Deprecated the use of the `PRIMARY_GALAXY_USERNAME` and `PRIMARY_GALAXY_PASSWORD` settings. We recommend using tokens to access Galaxy or Automation Hub.
+- Added local caching for downloaded roles and collections so they are not re-downloaded on nodes where they are up to date with the project - https://github.com/ansible/awx/issues/5518
+- Added the ability to associate K8S/OpenShift credentials to Job Template for playbook interaction with the `community.kubernetes` collection - https://github.com/ansible/awx/issues/5735
+- Added the ability to include HTML in the Custom Login Info presented on the login page - https://github.com/ansible/awx/issues/7600
+- Fixed https://access.redhat.com/security/cve/cve-2020-14327 - Server-side request forgery on credentials
+- Fixed https://access.redhat.com/security/cve/cve-2020-14328 - Server-side request forgery on webhooks
+- Fixed https://access.redhat.com/security/cve/cve-2020-14329 - Sensitive data exposure on labels
+- Fixed https://access.redhat.com/security/cve/cve-2020-14337 - Named URLs allow for testing the presence or absence of objects
+- Fixed a number of bugs in the user interface related to an upgrade of jQuery:
+     * https://github.com/ansible/awx/issues/7530
+     * https://github.com/ansible/awx/issues/7546
+     * https://github.com/ansible/awx/issues/7534
+     * https://github.com/ansible/awx/issues/7606
+- Fixed a bug that caused the `-f yaml` flag of the AWX CLI to not print properly formatted YAML - https://github.com/ansible/awx/issues/7795
+- Fixed a bug in the installer that caused errors when `docker_registry_password` was set - https://github.com/ansible/awx/issues/7695
+- Fixed a permissions error that prevented certain users from starting AWX services - https://github.com/ansible/awx/issues/7545
+- Fixed a bug that allows superusers to run unsafe Jinja code when defining custom Credential Types - https://github.com/ansible/awx/pull/7584/
+- Fixed a bug that prevented users from creating (or editing) custom Credential Types containing boolean fields - https://github.com/ansible/awx/issues/7483
+- Fixed a bug that prevented users with postgres usernames containing uppercase letters from restoring backups succesfully - https://github.com/ansible/awx/pull/7519
+- Fixed a bug which allowed the creation (in the Tower API) of Groups and Hosts with the same name - https://github.com/ansible/awx/issues/4680
+
+## 13.0.0 (Jun 23, 2020)
+- Added import and export commands to the official AWX CLI, replacing send and receive from the old tower-cli (https://github.com/ansible/awx/pull/6125).
+- Removed scripts as a means of running inventory updates of built-in types (https://github.com/ansible/awx/pull/6911)
+- Ansible 2.8 is now partially unsupported; some inventory source types are known to no longer work.
+- Fixed an issue where the vmware inventory source ssl_verify source variable was not recognized (https://github.com/ansible/awx/pull/7360)
+- Fixed a bug that caused redis' listen socket to have too-permissive file permissions (https://github.com/ansible/awx/pull/7317)
+- Fixed a bug that caused rsyslogd's configuration file to have world-readable file permissions, potentially leaking secrets (CVE-2020-10782)
+
+## 12.0.0 (Jun 9, 2020)
+- Removed memcached as a dependency of AWX (https://github.com/ansible/awx/pull/7240)
+- Moved to a single container image build instead of separate awx_web and awx_task images. The container image is just `awx` (https://github.com/ansible/awx/pull/7228)
+- Official AWX container image builds now use a two-stage container build process that notably reduces the size of our published images (https://github.com/ansible/awx/pull/7017)
+- Removed support for HipChat notifications ([EoL announcement](https://www.atlassian.com/partnerships/slack/faq#faq-98b17ca3-247f-423b-9a78-70a91681eff0)); all previously-created HipChat notification templates will be deleted due to this removal.
+- Fixed a bug which broke AWX installations with oc version 4.3 (https://github.com/ansible/awx/pull/6948/)
+- Fixed a performance issue that caused notable delay of stdout processing for playbooks run against large numbers of hosts (https://github.com/ansible/awx/issues/6991)
+- Fixed a bug that caused CyberArk AIM credential plugin looks to hang forever in some environments (https://github.com/ansible/awx/issues/6986)
+- Fixed a bug that caused ANY/ALL converage settings not to properly save when editing approval nodes in the UI (https://github.com/ansible/awx/issues/6998)
+- Fixed a bug that broke support for the satellite6_group_prefix source variable (https://github.com/ansible/awx/issues/7031)
+- Fixed a bug that prevented changes to workflow node convergence settings when approval nodes were in use (https://github.com/ansible/awx/issues/7063)
+- Fixed a bug that caused notifications to fail on newer version of Mattermost (https://github.com/ansible/awx/issues/7264)
+- Fixed a bug (by upgrading to 0.8.1 of the foreman collection) that prevented host_filters from working properly with Foreman-based inventory (https://github.com/ansible/awx/issues/7225)
+- Fixed a bug that prevented the usage of the Conjur credential plugin with secrets that contain spaces (https://github.com/ansible/awx/issues/7191)
+- Fixed a bug in awx-manage run_wsbroadcast --status in kubernetes (https://github.com/ansible/awx/pull/7009)
+- Fixed a bug that broke notification toggles for system jobs in the UI (https://github.com/ansible/awx/pull/7042)
+- Fixed a bug that broke local pip installs of awxkit (https://github.com/ansible/awx/issues/7107)
+- Fixed a bug that prevented PagerDuty notifications from sending for workflow job template approvals (https://github.com/ansible/awx/issues/7094)
+- Fixed a bug that broke external log aggregation support for URL paths that include the = character (such as the tokens for SumoLogic) (https://github.com/ansible/awx/issues/7139)
+- Fixed a bug that prevented organization admins from removing labels from workflow job templates (https://github.com/ansible/awx/pull/7143)
+
+## 11.2.0 (Apr 29, 2020)
+
+- Inventory updates now use collection-based plugins by default (in Ansible 2.9+):
+    - amazon.aws.aws_ec2
+    - community.vmware.vmware_vm_inventory
+    - azure.azcollection.azure_rm
+    - google.cloud.gcp_compute
+    - theforeman.foreman.foreman
+    - openstack.cloud.openstack
+    - ovirt.ovirt_collection.ovirt
+    - awx.awx.tower
+- Added support for Approle and LDAP/AD mechanisms to the Hashicorp Vault credential plugin (https://github.com/ansible/awx/issues/5076)
+- Added Project (Domain Name) support for the OpenStack Keystone v3 API (https://github.com/ansible/awx/issues/6831)
+- Added a new setting for raising log verbosity for rsyslogd (https://github.com/ansible/awx/pull/6818)
+- Added the ability to monitor stdout in the CLI for running jobs and workflow jobs (https://github.com/ansible/awx/issues/6165)
+- Fixed a bug which prevented the AWX CLI from properly installing with newer versions of pip (https://github.com/ansible/awx/issues/6870)
+- Fixed a bug which broke AWX's external logging support when configured with HTTPS endpoints that utilize self-signed certificates (https://github.com/ansible/awx/issues/6851)
+- Fixed a local docker installer bug that mistakenly attempted to upgrade PostgreSQL when an external pg_hostname is specified (https://github.com/ansible/awx/pull/5398)
+- Fixed a race condition that caused task container crashes when pods are quickly brought down and back up (https://github.com/ansible/awx/issues/6750)
+- Fixed a bug that caused 404 errors when attempting to view the second page of the workflow approvals view (https://github.com/ansible/awx/issues/6803)
+- Fixed a bug that prevented the use of ANSIBLE_SSH_ARGS for ad-hoc-commands (https://github.com/ansible/awx/pull/6811)
+- Fixed a bug that broke AWX installs/upgrades on Red Hat OpenShift (https://github.com/ansible/awx/issues/6791)
+
+
+## 11.1.0 (Apr 22, 2020)
+- Changed rsyslogd to persist queued events to disk (to prevent a risk of out-of-memory errors) (https://github.com/ansible/awx/issues/6746)
+- Added the ability to configure the destination and maximum disk size of rsyslogd spool (in the event of a log aggregator outage) (https://github.com/ansible/awx/pull/6763)
+- Added the ability to discover playbooks in project clones from symlinked directories (https://github.com/ansible/awx/pull/6773)
+- Fixed a bug that caused certain log aggregator settings to break logging integration (https://github.com/ansible/awx/issues/6760)
+- Fixed a bug that caused playbook execution in container groups to sometimes unexpectedly deadlock (https://github.com/ansible/awx/issues/6692)
+- Improved stability of the new redis clustering implementation (https://github.com/ansible/awx/pull/6739 https://github.com/ansible/awx/pull/6720)
+- Improved stability of the new rsyslogd-based logging implementation (https://github.com/ansible/awx/pull/6796)
+
+## 11.0.0 (Apr 16, 2020)
+- As of AWX 11.0.0, Kubernetes-based deployments use a Deployment rather than a StatefulSet.
+- Reimplemented external logging support using rsyslogd to improve reliability and address a number of issues (https://github.com/ansible/awx/issues/5155)
+- Changed activity stream logs to include summary fields for related objects (https://github.com/ansible/awx/issues/1761)
+- Added code to more gracefully attempt to reconnect to redis if it restarts/becomes unavailable (https://github.com/ansible/awx/pull/6670)
+- Fixed a bug that caused REFRESH_TOKEN_EXPIRE_SECONDS to not properly be respected for OAuth2.0 refresh tokens generated by AWX (https://github.com/ansible/awx/issues/6630)
+- Fixed a bug that broke schedules containing RRULES with very old DTSTART dates (https://github.com/ansible/awx/pull/6550)
+- Fixed a bug that broke installs on older versions of Ansible packaged with certain Linux distributions (https://github.com/ansible/awx/issues/5501)
+- Fixed a bug that caused the activity stream to sometimes report the incorrect actor when associating user membership on SAML login (https://github.com/ansible/awx/pull/6525)
+- Fixed a bug in AWX's Grafana notification support when annotation tags are omitted (https://github.com/ansible/awx/issues/6580)
+- Fixed a bug that prevented some users from searching for Source Control credentials in the AWX user interface (https://github.com/ansible/awx/issues/6600)
+- Fixed a bug that prevented disassociating orphaned users from credentials (https://github.com/ansible/awx/pull/6554)
+- Updated Twisted to address CVE-2020-10108 and CVE-2020-10109.
+
+## 10.0.0 (Mar 30, 2020)
+- As of AWX 10.0.0, the official AWX CLI no longer supports Python 2 (it requires at least Python 3.6) (https://github.com/ansible/awx/pull/6327)
+- AWX no longer relies on RabbitMQ; Redis is added as a new dependency (https://github.com/ansible/awx/issues/5443)
+- Altered AWX's event tables to allow more than ~2 billion total events (https://github.com/ansible/awx/issues/6010)
+- Improved the performance (time to execute, and memory consumption) of the periodic job cleanup system job (https://github.com/ansible/awx/pull/6166)
+- Updated Job Templates so they now have an explicit Organization field (it is no longer inferred from the associated Project) (https://github.com/ansible/awx/issues/3903)
+- Updated social-auth-core to address an upcoming GitHub API deprecation (https://github.com/ansible/awx/issues/5970)
+- Updated to ansible-runner 1.4.6 to address various bugs.
+- Updated Django to address CVE-2020-9402
+- Updated pyyaml version to address CVE-2017-18342
+- Fixed a bug which prevented the new `scm_branch` field from being used in custom notification templates (https://github.com/ansible/awx/issues/6258)
+- Fixed a race condition that sometimes causes success/failure notifications to include an incomplete list of hosts (https://github.com/ansible/awx/pull/6290)
+- Fixed a bug that can cause certain setting pages to lose unsaved form edits when a playbook is launched (https://github.com/ansible/awx/issues/5265)
+- Fixed a bug that can prevent the "Use TLS/SSL" field from properly saving when editing email notification templates (https://github.com/ansible/awx/issues/6383)
+- Fixed a race condition that sometimes broke event/stdout processing for jobs launched in container groups (https://github.com/ansible/awx/issues/6280)
+
+## 9.3.0 (Mar 12, 2020)
+- Added the ability to specify an OAuth2 token description in the AWX CLI (https://github.com/ansible/awx/issues/6122)
+- Added support for K8S service account annotations to the installer (https://github.com/ansible/awx/pull/6007)
+- Added support for K8S imagePullSecrets to the installer (https://github.com/ansible/awx/pull/5989)
+- Launching jobs (and workflows) using the --monitor flag in the AWX CLI now returns a non-zero exit code on job failure (https://github.com/ansible/awx/issues/5920)
+- Improved UI performance for various job views when many simultaneous users are logged into AWX (https://github.com/ansible/awx/issues/5883)
+- Updated to the latest version of Django to address a few open CVEs (https://github.com/ansible/awx/pull/6080)
+- Fixed a critical bug which can cause AWX to hang and stop launching playbooks after a periodic of time (https://github.com/ansible/awx/issues/5617)
+- Fixed a bug which caused delays in project update stdout for certain large SCM clones (as of Ansible 2.9+) (https://github.com/ansible/awx/pull/6254)
+- Fixed a bug which caused certain smart inventory filters to mistakenly return duplicate hosts (https://github.com/ansible/awx/pull/5972)
+- Fixed an unclear server error when creating smart inventories with the AWX collection (https://github.com/ansible/awx/issues/6250)
+- Fixed a bug that broke Grafana notification support (https://github.com/ansible/awx/issues/6137)
+- Fixed a UI bug which prevent users with read access to an organization from editing credentials for that organization (https://github.com/ansible/awx/pull/6241)
+- Fixed a bug which prevent workflow approval records from recording a `started` and `elapsed` date (https://github.com/ansible/awx/issues/6202)
+- Fixed a bug which caused workflow nodes to have a confusing option for `verbosity` (https://github.com/ansible/awx/issues/6196)
+- Fixed an RBAC bug which prevented projects and inventory schedules from being created by certain users in certain contexts (https://github.com/ansible/awx/issues/5717)
+- Fixed a bug that caused `role_path` in a project's config to not be respected due to an error processing `/etc/ansible/ansible.cfg` (https://github.com/ansible/awx/pull/6038)
+- Fixed a bug that broke inventory updates for installs with custom home directories for the awx user (https://github.com/ansible/awx/pull/6152)
+- Fixed a bug that broke fact data collection when AWX encounters invalid/unexpected fact data (https://github.com/ansible/awx/issues/5935)
+
+
+## 9.2.0 (Feb 12, 2020)
+- Added the ability to configure the convergence behavior of workflow nodes https://github.com/ansible/awx/issues/3054
+- AWX now allows for a configurable global limit for fork count (per-job run).  The default maximum is 200. https://github.com/ansible/awx/pull/5604
+- Added the ability to specify AZURE_PUBLIC_CLOUD (for e.g., Azure Government KeyVault support) for the Azure credential plugin https://github.com/ansible/awx/issues/5138
+- Added support for several additional parameters for Satellite dynamic inventory https://github.com/ansible/awx/pull/5598
+- Added a new field to jobs for tracking the date/time a job is cancelled https://github.com/ansible/awx/pull/5610
+- Made a series of additional optimizations to the callback receiver to further improve stdout write speed for running playbooks https://github.com/ansible/awx/pull/5677 https://github.com/ansible/awx/pull/5739
+- Updated AWX to be compatible with Helm 3.x (https://github.com/ansible/awx/pull/5776)
+- Optimized AWX's job dependency/scheduling code to drastically improve processing time in scenarios where there are many pending jobs scheduled simultaneously https://github.com/ansible/awx/issues/5154
+- Fixed a bug which could cause SCM authentication details (basic auth passwords) to be reported to external loggers in certain failure scenarios (e.g., when a git clone fails and ansible itself prints an error message to stdout) https://github.com/ansible/awx/pull/5812
+- Fixed a k8s installer bug that caused installs to fail in certain situations https://github.com/ansible/awx/issues/5574
+- Fixed a number of issues that caused analytics gathering and reporting to run more often than necessary https://github.com/ansible/awx/pull/5721
+- Fixed a bug in the AWX CLI that prevented JSON-type settings from saving properly https://github.com/ansible/awx/issues/5528
+- Improved support for fetching custom virtualenv dependencies when AWX is installed behind a proxy https://github.com/ansible/awx/pull/5805
+- Updated the bundled version of openstacksdk to address a known issue https://github.com/ansible/awx/issues/5821
+- Updated the bundled vmware_inventory plugin to the latest version to address a bug https://github.com/ansible/awx/pull/5668
+- Fixed a bug that can cause inventory updates to fail to properly save their output when run within a workflow https://github.com/ansible/awx/pull/5666
+- Removed a number of pre-computed fields from the Host and Group models to improve AWX performance.  As part of this change, inventory group UIs throughout the interface no longer display status icons https://github.com/ansible/awx/pull/5448
+
+## 9.1.1 (Jan 14, 2020)
+
+- Fixed a bug that caused database migrations on Kubernetes installs to hang https://github.com/ansible/awx/pull/5579
+- Upgraded Python-level app dependencies in AWX virtual environment https://github.com/ansible/awx/pull/5407
+- Running jobs no longer block associated inventory updates https://github.com/ansible/awx/pull/5519
+- Fixed invalid_response SAML error https://github.com/ansible/awx/pull/5577
+- Optimized the callback receiver to drastically improve the write speed of stdout for parallel jobs (https://github.com/ansible/awx/pull/5618)
+
+## 9.1.0 (Dec 17, 2019)
+- Added a command to generate a new SECRET_KEY and rekey the secrets in the database
+- Removed project update locking when jobs using it are running
+- Fixed slow queries for /api/v2/instances and /api/v2/instance_groups when smart inventories are used
+- Fixed a partial password disclosure when special characters existed in the RabbitMQ password (CVE-2019-19342)
+- Fixed hang in error handling for source control checkouts
+- Fixed an error on subsequent job runs that override the branch of a project on an instance that did not have a prior project checkout
+- Fixed an issue where jobs launched in isolated or container groups would incorrectly timeout
+- Fixed an incorrect link to instance groups documentation in the user interface
+- Fixed editing of inventory on Workflow templates
+- Fixed multiple issues with OAuth2 token cleanup system jobs
+- Fixed a bug that broke email notifications for workflow approval/deny https://github.com/ansible/awx/issues/5401
+- Updated SAML implementation to automatically login if authorization already exists
+- Updated AngularJS to 1.7.9 for CVE-2019-10768
+
+## 9.0.1 (Nov 4, 2019)
+
+- Fixed a bug in the installer that broke certain types of k8s installs https://github.com/ansible/awx/issues/5205
+
+## 9.0.0 (Oct 31, 2019)
+
+- Updated AWX images to use centos:8 as the parent image.
+- Updated to ansible-runner 1.4.4 to address various bugs.
+- Added oc and kubectl to the AWX images to support new container-based execution introduced in 8.0.0.
+- Added some optimizations to speed up the deletion of large Inventory Groups.
+- Fixed a bug that broke webhook launches for Job Templates that define a survey (https://github.com/ansible/awx/issues/5062).
+- Fixed a bug in the CLI which incorrectly parsed launch time arguments for `awx job_templates launch` and `awx workflow_job_templates launch` (https://github.com/ansible/awx/issues/5093).
+- Fixed a bug that caused inventory updates using "sourced from a project" to stop working (https://github.com/ansible/awx/issues/4750).
+- Fixed a bug that caused Slack notifications to sometimes show the wrong bot avatar (https://github.com/ansible/awx/pull/5125).
+- Fixed a bug that prevented the use of digits in AWX's URL settings (https://github.com/ansible/awx/issues/5081).
+
+## 8.0.0 (Oct 21, 2019)
+
+- The Ansible Tower Ansible modules have been migrated to a new official Ansible AWX collection: https://galaxy.ansible.com/awx/AWX
+  Please note that this functionality is only supported in Ansible 2.9+
+- AWX now supports the ability to launch jobs from external webhooks (GitHub and GitLab integration are supported).
+- AWX now supports Container Groups, a new feature that allows you to schedule and run playbooks on single-use kubernetes pods on-demand.
+- AWX now supports sending notifications when Workflow steps are approved, denied, or time out.
+- AWX now records the user who approved or denied Workflow steps.
+- AWX now supports fetching Ansible Collections from private galaxy servers.
+- AWX now checks the user's ansible.cfg for paths where role/collections may live when running project updates.
+- AWX now uses PostgreSQL 10 by default.
+- AWX now warns more loudly about underlying AMQP connectivity issues (https://github.com/ansible/awx/pull/4857).
+- Added a few optimizations to drastically improve dashboard performance for larger AWX installs (installs with several hundred thousand jobs or more).
+- Updated to the latest version of Ansible's VMWare inventory script (which adds support for vmware_guest_facts).
+- Deprecated /api/v2/inventory_scripts/ (this endpoint - and the Custom Inventory Script feature - will be removed in a future release of AWX).
+- Fixed a bug which prevented Organization Admins from removing users from their own Organization (https://github.com/ansible/awx/issues/2979)
+- Fixed a bug which sometimes caused cluster nodes to fail to re-join with a cryptic error, "No instance found with the current cluster host id" (https://github.com/ansible/awx/issues/4294)
+- Fixed a bug that prevented the use of launch-time passphrases when using credential plugins (https://github.com/ansible/awx/pull/4807)
+- Fixed a bug that caused notifications assigned at the Organization level not to take effect for Workflows in that Organization (https://github.com/ansible/awx/issues/4712)
+- Fixed a bug which caused a notable amount of CPU overhead on RabbitMQ health checks (https://github.com/ansible/awx/pull/5009)
+- Fixed a bug which sometimes caused the <return> key to stop functioning in <textarea> elements (https://github.com/ansible/awx/issues/4192)
+- Fixed a bug which caused request contention when the same OAuth2.0 token was used in multiple simultaneous requests (https://github.com/ansible/awx/issues/4694)
+- Fixed a bug related to parsing multiple choice survey options (https://github.com/ansible/awx/issues/4452).
+- Fixed a bug that caused single-sign-on icons on the login page to fail to render in certain Windows browsers (https://github.com/ansible/awx/issues/3924)
+- Fixed a number of bugs that caused certain OAuth2 settings to not be properly respected, such as REFRESH_TOKEN_EXPIRE_SECONDS.
+- Fixed a number of bugs in the AWX CLI, including a bug which sometimes caused long lines of stdout output to be unexpectedly truncated.
+- Fixed a number of bugs on the job details UI which sometimes caused auto-scrolling stdout to become stuck.
+- Fixed a bug which caused LDAP authentication to fail if the TLD of the server URL contained digits (https://github.com/ansible/awx/issues/3646)
+- Fixed a bug which broke HashiCorp Vault integration on older versions of HashiCorp Vault.
+
+## 7.0.0 (Sept 4, 2019)
+
+- AWX now detects and installs Ansible Collections defined in your project (note - this feature only works in Ansible 2.9+) (https://github.com/ansible/awx/issues/2534)
+- AWX now includes an official command line client.  Keep an eye out for a follow-up email on this mailing list for information on how to install it and try it out.
+- Added the ability to provide a specific SCM branch on jobs (https://github.com/ansible/awx/issues/282)
+- Added support for Workflow Approval Nodes, a new feature which allows you to add "pause and wait for approval" steps into your workflows (https://github.com/ansible/awx/issues/1206)
+- Added the ability to specify a specific HTTP method for webhook notifications (POST vs PUT) (https://github.com/ansible/awx/pull/4124)
+- Added the ability to specify a username and password for HTTP Basic Authorization for webhook notifications (https://github.com/ansible/awx/pull/4124)
+- Added support for customizing the text content of notifications (https://github.com/ansible/awx/issues/79)
+- Added the ability to enable and disable hosts in dynamic inventory (https://github.com/ansible/awx/pull/4420)
+- Added the description (if any) to the Job Template list (https://github.com/ansible/awx/issues/4359)
+- Added new metrics for instance hostnames and pending jobs to the /api/v2/metrics/ endpoint (https://github.com/ansible/awx/pull/4375)
+- Changed AWX's on/off toggle buttons to a non-text based style to simplify internationalization (https://github.com/ansible/awx/pull/4425)
+- Events emitted by ansible for adhoc commands are now sent to the external log aggregrator (https://github.com/ansible/awx/issues/4545)
+- Fixed a bug which allowed a user to make an organization credential in another organization without permissions to that organization (https://github.com/ansible/awx/pull/4483)
+- Fixed a bug that caused `extra_vars` on workflows to break when edited (https://github.com/ansible/awx/issues/4293)
+- Fixed a slow SQL query that caused performance issues when large numbers of groups exist (https://github.com/ansible/awx/issues/4461)
+- Fixed a few minor bugs in survey field validation (https://github.com/ansible/awx/pull/4509) (https://github.com/ansible/awx/pull/4479)
+- Fixed a bug that sometimes resulted in orphaned `ansible_runner_pi` directories in `/tmp` after playbook execution (https://github.com/ansible/awx/pull/4409)
+- Fixed a bug that caused the `is_system_auditor` flag in LDAP configuration to not work (https://github.com/ansible/awx/pull/4396)
+- Fixed a bug which caused schedules to disappear from the UI when toggled off (https://github.com/ansible/awx/pull/4378)
+- Fixed a bug that sometimes caused stdout content to contain extraneous blank lines in newer versions of Ansible (https://github.com/ansible/awx/pull/4391)
+- Updated to the latest Django security release, 2.2.4 (https://github.com/ansible/awx/pull/4410) (https://www.djangoproject.com/weblog/2019/aug/01/security-releases/)
+- Updated the default version of git to a version that includes support for x509 certificates (https://github.com/ansible/awx/issues/4362)
+- Removed the deprecated `credential` field from `/api/v2/workflow_job_templates/N/` (as part of the `/api/v1/` removal in prior AWX versions - https://github.com/ansible/awx/pull/4490).
+
+## 6.1.0 (Jul 18, 2019)
+
+- Updated AWX to use Django 2.2.2.
+- Updated the provided openstacksdk version to support new functionality (such as Nova scheduler_hints)
+- Added the ability to specify a custom cacert for the HashiCorp Vault credential plugin
+- Fixed a number of bugs related to path lookups for the HashiCorp Vault credential plugin
+- Fixed a bug which prevented signed SSH certificates from working, including the HashiCorp Vault Signed SSH backend
+- Fixed a bug which prevented custom logos from displaying on the login page (as a result of a new Content Security Policy in 6.0.0)
+- Fixed a bug which broke websocket connectivity in Apple Safari (as a result of a new Content Security Policy in 6.0.0)
+- Fixed a bug on the job output page that occasionally caused the "up" and "down" buttons to not load additional output
+- Fixed a bug on the job output page that caused quoted task names to display incorrectly
+
+## 6.0.0 (Jul 1, 2019)
+
+- Removed support for "Any" notification templates and their API endpoints e.g., /api/v2/job_templates/N/notification_templates/any/ (https://github.com/ansible/awx/issues/4022)
+- Fixed a bug which prevented credentials from properly being applied to inventory sources (https://github.com/ansible/awx/issues/4059)
+- Fixed a bug which can cause the task dispatcher to hang indefinitely when external logging support (e.g., Splunk, Logstash) is enabled (https://github.com/ansible/awx/issues/4181)
+- Fixed a bug which causes slow stdout display when running jobs against smart inventories. (https://github.com/ansible/awx/issues/3106)
+- Fixed a bug that caused SSL verification flags to fail to be respected for LDAP authentication in certain environments. (https://github.com/ansible/awx/pull/4190)
+- Added a simple Content Security Policy (https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) to restrict access to third-party resources in the browser. (https://github.com/ansible/awx/pull/4167)
+- Updated ovirt4 library dependencies to work with newer versions of oVirt (https://github.com/ansible/awx/issues/4138)
+
+## 5.0.0 (Jun 21, 2019)
+
+- Bump Django Rest Framework from 3.7.7 to 3.9.4
+- Bump setuptools / pip dependencies
+- Fixed bug where Recent Notification list would not appear
+- Added notifications on job start
+- Default to Ansible 2.8

CONTRIBUTING.md

@@ -11,24 +11,15 @@ Have questions about this document or anything not covered here? Come chat with
   * [Prerequisites](#prerequisites)
     * [Docker](#docker)
     * [Docker compose](#docker-compose)
-    * [Node and npm](#node-and-npm)
-  * [Build the environment](#build-the-environment)
+    * [Frontend Development](#frontend-development)
+  * [Build and Run the Development Environment](#build-and-run-the-development-environment)
     * [Fork and clone the AWX repo](#fork-and-clone-the-awx-repo)
-    * [Create local settings](#create-local-settings)
-    * [Build the base image](#build-the-base-image)
-    * [Build the user interface](#build-the-user-interface)
-  * [Running the environment](#running-the-environment)
-    * [Start the containers](#start-the-containers)
-    * [Start from the container shell](#start-from-the-container-shell)
-  * [Post Build Steps](#post-build-steps)
-    * [Start a shell](#start-a-shell)
-    * [Create a superuser](#create-a-superuser)
-    * [Load the data](#load-the-data)
-    * [Building API Documentation](#build-api-documentation)
+  * [Building API Documentation](#building-api-documentation)
   * [Accessing the AWX web interface](#accessing-the-awx-web-interface)
   * [Purging containers and images](#purging-containers-and-images)
 * [What should I work on?](#what-should-i-work-on)
 * [Submitting Pull Requests](#submitting-pull-requests)
+* [PR Checks run by Zuul](#pr-checks-run-by-zuul)
 * [Reporting Issues](#reporting-issues)
 
 ## Things to know prior to submitting code
@@ -42,7 +33,7 @@ Have questions about this document or anything not covered here? Come chat with
 
 ## Setting up your development environment
 
-The AWX development environment workflow and toolchain is based on Docker, and the docker-compose tool, to provide dependencies, services, and databases necessary to run all of the components. It also binds the local source tree into the development container, making it possible to observe and test changes in real time.
+The AWX development environment workflow and toolchain uses Docker and the docker-compose tool, to provide dependencies, services, and databases necessary to run all of the components. It also bind-mounts the local source tree into the development container, making it possible to observe and test changes in real time.
 
 ### Prerequisites
 
@@ -55,215 +46,37 @@ respectively.
 
 For Linux platforms, refer to the following from Docker:
 
-**Fedora**
+* **Fedora** - https://docs.docker.com/engine/installation/linux/docker-ce/fedora/
 
-> https://docs.docker.com/engine/installation/linux/docker-ce/fedora/
+* **CentOS** - https://docs.docker.com/engine/installation/linux/docker-ce/centos/
 
-**CentOS**
+* **Ubuntu** - https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/
 
-> https://docs.docker.com/engine/installation/linux/docker-ce/centos/
+* **Debian** - https://docs.docker.com/engine/installation/linux/docker-ce/debian/
 
-**Ubuntu**
+* **Arch** - https://wiki.archlinux.org/index.php/Docker
 
-> https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/
+#### Docker Compose
 
-**Debian**
-
-> https://docs.docker.com/engine/installation/linux/docker-ce/debian/
-
-**Arch**
-
-> https://wiki.archlinux.org/index.php/Docker
-
-#### Docker compose
-
-If you're not using Docker for Mac, or Docker for Windows, you may need, or choose to, install the Docker compose Python module separately, in which case you'll need to run the following:
+If you're not using Docker for Mac, or Docker for Windows, you may need, or choose to, install the `docker-compose` Python module separately.
 
 ```bash
-(host)$ pip install docker-compose
+(host)$ pip3 install docker-compose
 ```
 
 #### Frontend Development
 
-See [the ui development documentation](awx/ui/README.md).
-
-
-### Build the environment
+See [the ui development documentation](awx/ui_next/CONTRIBUTING.md).
 
 #### Fork and clone the AWX repo
 
 If you have not done so already, you'll need to fork the AWX repo on GitHub. For more on how to do this, see [Fork a Repo](https://help.github.com/articles/fork-a-repo/).
 
-#### Create local settings
+### Build and Run the Development Environment
 
-AWX will import the file `awx/settings/local_settings.py` and combine it with defaults in `awx/settings/defaults.py`. This file is required for starting the development environment and startup will fail if it's not provided.
+See the [README.md](./tools/docker-compose/README.md) for docs on how to build the awx_devel image and run the development environment.  
 
-An example is provided. Make a copy of it, and edit as needed (the defaults are usually fine):
-
-```bash
-(host)$ cp awx/settings/local_settings.py.docker_compose awx/settings/local_settings.py
-```
-
-#### Build the base image
-
-The AWX base container image (defined in `tools/docker-compose/Dockerfile`) contains basic OS dependencies and symbolic links into the development environment that make running the services easy.
-
-Run the following to build the image:
-
-```bash
-(host)$ make docker-compose-build
-```
-
-**NOTE**
-
-> The image will need to be rebuilt, if the Python requirements or OS dependencies change.
-
-Once the build completes, you will have a `ansible/awx_devel` image in your local image cache. Use the `docker images` command to view it, as follows:
-
-```bash
-(host)$ docker images
-
-REPOSITORY                                   TAG                 IMAGE ID            CREATED             SIZE
-ansible/awx_devel                            latest              ba9ec3e8df74        26 minutes ago      1.42GB
-```
-
-#### Build the user interface
-
-Run the following to build the AWX UI:
-
-```bash
-(host) $ make ui-devel
-```
-See [the ui development documentation](awx/ui/README.md) for more information on using the frontend development, build, and test tooling.
-
-### Running the environment
-
-#### Start the containers
-
-Start the development containers by running the following:
-
-```bash
-(host)$ make docker-compose
-```
-
-The above utilizes the image built in the previous step, and will automatically start all required services and dependent containers. Once the containers launch, your session will be attached to the *awx* container, and you'll be able to watch log messages and events in real time. You will see messages from Django and the front end build process.
-
-If you start a second terminal session, you can take a look at the running containers using the `docker ps` command. For example:
-
-```bash
-# List running containers
-(host)$ docker ps
-
-$ docker ps
-CONTAINER ID        IMAGE                                              COMMAND                  CREATED             STATUS              PORTS                                                                                                                                      NAMES
-aa4a75d6d77b        gcr.io/ansible-tower-engineering/awx_devel:devel   "/tini -- /bin/sh ..."   23 seconds ago      Up 15 seconds       0.0.0.0:5555->5555/tcp, 0.0.0.0:7899-7999->7899-7999/tcp, 0.0.0.0:8013->8013/tcp, 0.0.0.0:8043->8043/tcp, 22/tcp, 0.0.0.0:8080->8080/tcp   tools_awx_1
-e4c0afeb548c        postgres:10                                       "docker-entrypoint..."   26 seconds ago      Up 23 seconds       5432/tcp                                                                                                                                   tools_postgres_1
-0089699d5afd        tools_logstash                                     "/docker-entrypoin..."   26 seconds ago      Up 25 seconds                                                                                                                                                  tools_logstash_1
-4d4ff0ced266        memcached:alpine                                   "docker-entrypoint..."   26 seconds ago      Up 25 seconds       0.0.0.0:11211->11211/tcp                                                                                                                   tools_memcached_1
-92842acd64cd        rabbitmq:3-management                              "docker-entrypoint..."   26 seconds ago      Up 24 seconds       4369/tcp, 5671-5672/tcp, 15671/tcp, 25672/tcp, 0.0.0.0:15672->15672/tcp                                                                    tools_rabbitmq_1
-```
-**NOTE**
-
-> The Makefile assumes that the image you built is tagged with your current branch. This allows you to build images for different contexts or branches. When starting the containers, you can choose a specific branch by setting `COMPOSE_TAG=<branch name>` in your environment.
-
-> For example, you might be working in a feature branch, but you want to run the containers using the `devel` image you built previously. To do that, start the containers using the following command: `$ COMPOSE_TAG=devel make docker-compose`
-
-##### Wait for migrations to complete
-
-The first time you start the environment, database migrations need to run in order to build the PostgreSQL database. It will take few moments, but eventually you will see output in your terminal session that looks like the following:
-
-```bash
-awx_1        | Operations to perform:
-awx_1        |   Synchronize unmigrated apps: solo, api, staticfiles, debug_toolbar, messages, channels, django_extensions, ui, rest_framework, polymorphic
-awx_1        |   Apply all migrations: sso, taggit, sessions, sites, kombu_transport_django, social_auth, contenttypes, auth, conf, main
-awx_1        | Synchronizing apps without migrations:
-awx_1        |   Creating tables...
-awx_1        |     Running deferred SQL...
-awx_1        |   Installing custom SQL...
-awx_1        | Running migrations:
-awx_1        |   Rendering model states... DONE
-awx_1        |   Applying contenttypes.0001_initial... OK
-awx_1        |   Applying contenttypes.0002_remove_content_type_name... OK
-awx_1        |   Applying auth.0001_initial... OK
-awx_1        |   Applying auth.0002_alter_permission_name_max_length... OK
-awx_1        |   Applying auth.0003_alter_user_email_max_length... OK
-awx_1        |   Applying auth.0004_alter_user_username_opts... OK
-awx_1        |   Applying auth.0005_alter_user_last_login_null... OK
-awx_1        |   Applying auth.0006_require_contenttypes_0002... OK
-awx_1        |   Applying taggit.0001_initial... OK
-awx_1        |   Applying taggit.0002_auto_20150616_2121... OK
-awx_1        |   Applying main.0001_initial... OK
-awx_1        |   Applying main.0002_squashed_v300_release... OK
-awx_1        |   Applying main.0003_squashed_v300_v303_updates... OK
-awx_1        |   Applying main.0004_squashed_v310_release... OK
-awx_1        |   Applying conf.0001_initial... OK
-awx_1        |   Applying conf.0002_v310_copy_tower_settings... OK
-...
-```
-
-Once migrations are completed, you can begin using AWX.
-
-#### Start from the container shell
-
-Often times you'll want to start the development environment without immediately starting all of the services in the *awx* container, and instead be taken directly to a shell. You can do this with the following:
-
-```bash
-(host)$ make docker-compose-test
-```
-
-Using `docker exec`, this will create a session in the running *awx* container, and place you at a command prompt, where you can run shell commands inside the container.
-
-If you want to start and use the development environment, you'll first need to bootstrap it by running the following command:
-
-```bash
-(container)# /bootstrap_development.sh
-```
-
-The above will do all the setup tasks, including running database migrations, so it may take a couple minutes.
-
-Now you can start each service individually, or start all services in a pre-configured tmux session like so:
-
-```bash
-(container)# cd /awx_devel
-(container)# make server
-```
-
-### Post Build Steps
-
-Before you can log in and use the system, you will need to create an admin user. Optionally, you may also want to load some demo data.
-
-##### Start a shell
-
-To create the admin user, and load demo data, you first need to start a shell session on the *awx* container. In a new terminal session, use the `docker exec` command as follows to start the shell session:
-
-```bash
-(host)$ docker exec -it tools_awx_1 bash
-```
-This creates a session in the *awx* containers, just as if you were using `ssh`, and allows you execute commands within the running container.
-
-##### Create an admin user
-
-Before you can log into AWX, you need to create an admin user. With this user you will be able to create more users, and begin configuring the server. From within the container shell, run the following command:
-
-```bash
-(container)# awx-manage createsuperuser
-```
-You will be prompted for a username, an email address, and a password, and you will be asked to confirm the password. The email address is not important, so just enter something that looks like an email address. Remember the username and password, as you will use them to log into the web interface for the first time.
-
-##### Load demo data
-
-You can optionally load some demo data. This will create a demo project, inventory, and job template. From within the container shell, run the following to load the data:
-
-```bash
-(container)# awx-manage create_preload_data
-```
-
-**NOTE**
-
-> This information will persist in the database running in the `tools_postgres_1` container, until the container is removed. You may periodically need to recreate
-this container, and thus the database, if the database schema changes in an upstream commit.
-
-##### Building API Documentation
+### Building API Documentation
 
 AWX includes support for building [Swagger/OpenAPI
 documentation](https://swagger.io).  To build the documentation locally, run:
@@ -281,7 +94,7 @@ is an example of one such service.
 
 You can now log into the AWX web interface at [https://localhost:8043](https://localhost:8043), and access the API directly at [https://localhost:8043/api/](https://localhost:8043/api/).
 
-To log in use the admin user and password you created above in [Create an admin user](#create-an-admin-user).
+[Create an admin user](./tools/docker-compose/README.md#create-an-admin-user) if needed.
 
 ### Purging containers and images
 
@@ -332,7 +145,7 @@ Sometimes it might take us a while to fully review your PR. We try to keep the `
 
 All submitted PRs will have the linter and unit tests run against them via Zuul, and the status reported in the PR.
 
-## PR Checks ran by Zuul
+## PR Checks run by Zuul
 Zuul jobs for awx are defined in the [zuul-jobs](https://github.com/ansible/zuul-jobs) repo.
 
 Zuul runs the following checks that must pass:

DATA_MIGRATION.md

@@ -2,96 +2,8 @@
 
 ## Introduction
 
-Upgrades using Django migrations are not expected to work in AWX.  As a result, to upgrade to a new version, it is necessary to export resources from the old AWX node and import them into a freshly-installed node with the new version.  The recommended way to do this is to use the tower-cli send/receive feature.
+Early versions of AWX did not support seamless upgrades between major versions and required the use of a backup and restore tool to perform upgrades.
 
-This tool does __not__ support export/import of the following:
-* Logs/history
-* Credential passwords
-* LDAP/AWX config
+Users who wish to upgrade modern AWX installations should follow the instructions at:
 
-### Install & Configure Tower-CLI
-
-In terminal, pip install tower-cli (if you do not have pip already, install [here](https://pip.pypa.io/en/stable/installing/)):
-```
-$ pip install --upgrade ansible-tower-cli
-```
-
-The AWX host URL, user, and password must be set for the AWX instance to be exported:
-```
-$ tower-cli config host http://<old-awx-host.example.com>
-$ tower-cli config username <user>
-$ tower-cli config password <pass>
-```
-
-For more information on installing tower-cli look [here](http://tower-cli.readthedocs.io/en/latest/quickstart.html).
-
-
-### Export Resources
-
-Export all objects
-
-```$ tower-cli receive --all > assets.json```
-
-
-
-### Teardown Old AWX
-
-Clean up remnants of the old AWX install:
-
-```docker rm -f $(docker ps -aq)```     # remove all old awx containers
-
-```make clean-ui```              # clean up ui artifacts
-
-
-### Install New AWX version
-
-If you are installing AWX as a dev container, pull down the latest code or version you want from GitHub, build
-the image locally, then start the container
-
-```
-git pull                                        # retrieve latest AWX changes from repository
-make docker-compose-build                       # build AWX image
-make docker-compose                             # run container
-```
-For other install methods, refer to the [Install.md](https://github.com/ansible/awx/blob/devel/INSTALL.md). 
- 
-
-### Import Resources
-
-
-Configure tower-cli for your new AWX host as shown earlier.  Import from a JSON file named assets.json
-
-```
-$ tower-cli config host http://<new-awx-host.example.com>
-$ tower-cli config username <user>
-$ tower-cli config password <pass>
-$ tower-cli send assets.json
-```
-
---------------------------------------------------------------------------------
-
-## Additional Info
-
-If you have two running AWX hosts, it is possible to copy all assets from one instance to another
-
-```$ tower-cli receive --tower-host old-awx-host.example.com --all | tower-cli send --tower-host new-awx-host.example.com```
-
-
-
-#### More Granular Exports:
-
-Export all credentials
-
-```$ tower-cli receive --credential all > credentials.json```
-> Note: This exports the credentials with blank strings for passwords and secrets
-
-Export a credential named "My Credential"
-
-```$ tower-cli receive --credential "My Credential"```
-
-#### More Granular Imports:
-
-
-You could import anything except an organization defined in a JSON file named assets.json
-
-```$ tower-cli send --prevent organization assets.json```
+https://github.com/ansible/awx/blob/devel/INSTALL.md#upgrading-from-previous-versions

INSTALL.md

@@ -1,623 +1,149 @@
+Table of Contents
+=================
+
+   * [Installing AWX](#installing-awx)
+      * [The AWX Operator](#the-awx-operator)
+         * [Quickstart with minikube](#quickstart-with-minikube)
+            * [Starting minikube](#starting-minikube)
+            * [Deploying the AWX Operator](#deploying-the-awx-operator)
+               * [Verifying the Operator Deployment](#verifying-the-operator-deployment)
+            * [Deploy AWX](#deploy-awx)
+               * [Accessing AWX](#accessing-awx)
+   * [Installing the AWX CLI](#installing-the-awx-cli)
+      * [Building the CLI Documentation](#building-the-cli-documentation)
+
+
 # Installing AWX
 
-This document provides a guide for installing AWX.
+:warning: NOTE |
+--- |
+If you're installing an older release of AWX (prior to 18.0), these instructions have changed.  Take a look at your version specific instructions, e.g., for AWX 17.0.1, see: [https://github.com/ansible/awx/blob/17.0.1/INSTALL.md](https://github.com/ansible/awx/blob/17.0.1/INSTALL.md)
+If you're attempting to migrate an older Docker-based AWX installation, see: [Migrating Data from Local Docker](https://github.com/ansible/awx/blob/devel/tools/docker-compose/docs/data_migration.md) |
 
-## Table of contents
+## The AWX Operator
 
-- [Getting started](#getting-started)
-  - [Clone the repo](#clone-the-repo)
-  - [AWX branding](#awx-branding)
-  - [Prerequisites](#prerequisites)
-  - [System Requirements](#system-requirements)
-  - [AWX Tunables](#awx-tunables)
-  - [Choose a deployment platform](#choose-a-deployment-platform)
-  - [Official vs Building Images](#official-vs-building-images)
-- [OpenShift](#openshift)
-  - [Prerequisites](#prerequisites-1)
-    - [Deploying to Minishift](#deploying-to-minishift)
-  - [Pre-build steps](#pre-build-steps)
-  - [PostgreSQL](#postgresql)
-  - [Start the build](#start-the-build)
-  - [Post build](#post-build)
-  - [Accessing AWX](#accessing-awx)
-- [Kubernetes](#kubernetes)
-  - [Prerequisites](#prerequisites-2)
-  - [Pre-build steps](#pre-build-steps-1)
-  - [Configuring Helm](#configuring-helm)
-  - [Start the build](#start-the-build-1)
-  - [Accessing AWX](#accessing-awx-1)
-  - [SSL Termination](#ssl-termination)
-- [Docker Compose](#docker-compose)
-  - [Prerequisites](#prerequisites-3)
-  - [Pre-build steps](#pre-build-steps-2)
-    - [Deploying to a remote host](#deploying-to-a-remote-host)
-    - [Inventory variables](#inventory-variables)
-      - [Docker registry](#docker-registry)
-      - [PostgreSQL](#postgresql-1)
-      - [Proxy settings](#proxy-settings)
-  - [Start the build](#start-the-build-2)
-  - [Post build](#post-build-2)
-  - [Accessing AWX](#accessing-awx-2)
+Starting in version 18.0, the [AWX Operator](https://github.com/ansible/awx-operator) is the preferred way to install AWX.
 
-## Getting started
+AWX can also alternatively be installed and [run in Docker](./tools/docker-compose/README.md), but this install path is only recommended for development/test-oriented deployments, and has no official published release.
 
-### Clone the repo
+### Quickstart with minikube
 
-If you have not already done so, you will need to clone, or create a local copy, of the [AWX repo](https://github.com/ansible/awx). For more on how to clone the repo, view [git clone help](https://git-scm.com/docs/git-clone).
+If you don't have an existing OpenShift or Kubernetes cluster, minikube is a fast and easy way to get up and running.
 
-Once you have a local copy, run commands within the root of the project tree.
+To install minikube, follow the steps in their [documentation](https://minikube.sigs.k8s.io/docs/start/).
 
-### AWX branding
+#### Starting minikube
 
-You can optionally install the AWX branding assets from the [awx-logos repo](https://github.com/ansible/awx-logos). Prior to installing, please review and agree to the [trademark guidelines](https://github.com/ansible/awx-logos/blob/master/TRADEMARKS.md).
-
-To install the assets, clone the `awx-logos` repo so that it is next to your `awx` clone. As you progress through the installation steps, you'll be setting variables in the [inventory](./installer/inventory) file. To include the assets in the build, set `awx_official=true`.
-
-### Prerequisites
-
-Before you can run a deployment, you'll need the following installed in your local environment:
-
-- [Ansible](http://docs.ansible.com/ansible/latest/intro_installation.html) Requires Version 2.4+
-- [Docker](https://docs.docker.com/engine/installation/)
-    + A recent version
-- [docker](https://pypi.org/project/docker/) Python module
-    + This is incompatible with `docker-py`. If you have previously installed `docker-py`, please uninstall it.
-    + We use this module instead of `docker-py` because it is what the `docker-compose` Python module requires.
-- [GNU Make](https://www.gnu.org/software/make/)
-- [Git](https://git-scm.com/) Requires Version 1.8.4+
-- [Node 10.x LTS version](https://nodejs.org/en/download/)
-- [NPM 6.x LTS](https://docs.npmjs.com/)
-
-### System Requirements
-
-The system that runs the AWX service will need to satisfy the following requirements
-
-- At least 4GB of memory
-- At least 2 cpu cores
-- At least 20GB of space
-- Running Docker, Openshift, or Kubernetes
-- If you choose to use an external PostgreSQL database, please note that the minimum version is 9.6+.
-
-### AWX Tunables
-
-**TODO** add tunable bits
-
-### Choose a deployment platform
-
-We currently support running AWX as a containerized application using Docker images deployed to either an OpenShift cluster, a Kubernetes cluster, or docker-compose. The remainder of this document will walk you through the process of building the images, and deploying them to either platform.
-
-The [installer](./installer) directory contains an [inventory](./installer/inventory) file, and a playbook, [install.yml](./installer/install.yml). You'll begin by setting variables in the inventory file according to the platform you wish to use, and then you'll start the image build and deployment process by running the playbook.
-
-In the sections below, you'll find deployment details and instructions for each platform:
-- [OpenShift](#openshift)
-- [Kubernetes](#kubernetes)
-- [Docker Compose](#docker-compose).
-
-### Official vs Building Images
-
-When installing AWX you have the option of building your own images or using the images provided on DockerHub (see [awx_web](https://hub.docker.com/r/ansible/awx_web/) and [awx_task](https://hub.docker.com/r/ansible/awx_task/))
-
-This is controlled by the following variables in the `inventory` file
+Once you have installed minikube, run the following command to start it. You may wish to customize these options.
 
 ```
-dockerhub_base=ansible
-dockerhub_version=latest
+$ minikube start --cpus=4 --memory=8g --addons=ingress
 ```
 
-If these variables are present then all deployments will use these hosted images. If the variables are not present then the images will be built during the install.
+#### Deploying the AWX Operator
 
-*dockerhub_base*
-
-> The base location on DockerHub where the images are hosted (by default this pulls container images named `ansible/awx_web:tag` and `ansible/awx_task:tag`)
-
-*dockerhub_version*
-
-> Multiple versions are provided. `latest` always pulls the most recent. You may also select version numbers at different granularities: 1, 1.0, 1.0.1, 1.0.0.123
-
-## OpenShift
-
-### Prerequisites
-
-To complete a deployment to OpenShift, you will obviously need access to an OpenShift cluster. For demo and testing purposes, you can use [Minishift](https://github.com/minishift/minishift) to create a single node cluster running inside a virtual machine.
-
-When using OpenShift for deploying AWX make sure you have correct privileges to add the security context 'privileged', otherwise the installation will fail. The privileged context is needed because of the use of [the bubblewrap tool](https://github.com/containers/bubblewrap) to add an additional layer of security when using containers.
-
-You will also need to have the `oc` command in your PATH. The `install.yml` playbook will call out to `oc` when logging into, and creating objects on the cluster.
-
-The default resource requests per-deployment requires:
-
-> Memory: 6GB
-> CPU: 3 cores
-
-This can be tuned by overriding the variables found in [/installer/roles/kubernetes/defaults/main.yml](/installer/roles/kubernetes/defaults/main.yml). Special care should be taken when doing this as undersized instances will experience crashes and resource exhaustion.
-
-For more detail on how resource requests are formed see: [https://docs.openshift.com/container-platform/latest/dev_guide/compute_resources.html#dev-compute-resources](https://docs.openshift.com/container-platform/latest/dev_guide/compute_resources.html#dev-compute-resources)
-
-### Pre-build steps
-
-Before starting the build process, review the [inventory](./installer/inventory) file, and uncomment and provide values for the following variables found in the `[all:vars]` section:
-
-*openshift_host*
-
-> IP address or hostname of the OpenShift cluster. If you're using Minishift, this will be the value returned by `minishift ip`.
-
-
-*openshift_skip_tls_verify*
-
-> Boolean. Set to True if using self-signed certs.
-
-*openshift_project*
-
-> Name of the OpenShift project that will be created, and used as the namespace for the AWX app. Defaults to *awx*.
-
-*openshift_user*
-
-> Username of the OpenShift user that will create the project, and deploy the application. Defaults to *developer*.
-
-*openshift_pg_emptydir*
-
-> Boolean. Set to True to use an emptyDir volume when deploying the PostgreSQL pod. Note: This should only be used for demo and testing purposes.
-
-*docker_registry*
-
-> IP address and port, or URL, for accessing a registry that the OpenShift cluster can access. Defaults to *172.30.1.1:5000*, the internal registry delivered with Minishift. This is not needed if you are using official hosted images.
-
-*docker_registry_repository*
-
-> Namespace to use when pushing and pulling images to and from the registry. Generally this will match the project name. It defaults to *awx*. This is not needed if you are using official hosted images.
-
-*docker_registry_username*
-
-> Username of the user that will push images to the registry. Will generally match the *openshift_user* value. Defaults to *developer*. This is not needed if you are using official hosted images.
-
-#### Deploying to Minishift
-
-Install Minishift by following the [installation guide](https://docs.openshift.org/latest/minishift/getting-started/installing.html).
-
-The recommended minimum resources for your Minishift VM:
-
-```bash
-$ minishift start --cpus=4 --memory=8GB
-```
-
-The Minishift VM contains a Docker daemon, which you can use to build the AWX images. This is generally the approach you should take, and we recommend doing so. To use this instance, run the following command to setup your environment:
-
-```bash
-# Set DOCKER environment variable to point to the Minishift VM
-$ eval $(minishift docker-env)
-```
-
-**Note**
-
-> If you choose to not use the Docker instance running inside the VM, and build the images externally, you will have to enable the OpenShift cluster to access the images. This involves pushing the images to an external Docker registry, and granting the cluster access to it, or exposing the internal registry, and pushing the images into it.
-
-#### PostgreSQL
-
-By default, AWX will deploy a PostgreSQL pod inside of your cluster. You will need to create a [Persistent Volume Claim](https://docs.openshift.org/latest/dev_guide/persistent_volumes.html) which is named `postgresql` by default, and can be overridden by setting the `openshift_pg_pvc_name` variable. For testing and demo purposes, you may set `openshift_pg_emptydir=yes`.
-
-If you wish to use an external database, in the inventory file, set the value of `pg_hostname`, and update `pg_username`, `pg_password`, `pg_admin_password`, `pg_database`, and `pg_port` with the connection information. When setting `pg_hostname` the installer will assume you have configured the database in that location and will not launch the postgresql pod.
-
-### Start the build
-
-To start the build, you will pass two *extra* variables on the command line. The first is *openshift_password*, which is the password for the *openshift_user*, and the second is *docker_registry_password*, which is the password associated with *docker_registry_username*.
-
-If you're using the OpenShift internal registry, then you'll pass an access token for the *docker_registry_password* value, rather than a password. The `oc whoami -t` command will generate the required token, as long as you're logged into the cluster via `oc cluster login`.
-
-To start the build and deployment, run the following (docker_registry_password is optional if using official images):
-
-```bash
-# Start the build and deployment
-$ ansible-playbook -i inventory install.yml -e openshift_password=developer  -e docker_registry_password=$(oc whoami -t)
-```
-
-### Post build
-
-After the playbook run completes, check the status of the deployment by running `oc get pods`:
-
-```bash
-# View the running pods
-$ oc get pods
-
-NAME                   READY     STATUS    RESTARTS   AGE
-awx-3886581826-5mv0l   4/4       Running   0          8s
-postgresql-1-l85fh     1/1       Running   0          20m
+For a comprehensive overview of features, see [README.md](https://github.com/ansible/awx-operator/blob/devel/README.md) in the awx-operator repo. The following steps are the bare minimum to get AWX up and running. 
 
 ```
-
-In the above example, the name of the AWX pod is `awx-3886581826-5mv0l`. Before accessing the AWX web interface, setup tasks and database migrations need to complete. These tasks are running in the `awx_task` container inside the AWX pod. To monitor their status, tail the container's STDOUT by running the following command, replacing the AWX pod name with the pod name from your environment:
-
-```bash
-# Follow the awx_task log output
-$ oc logs -f awx-3886581826-5mv0l -c awx-celery
+$ minikube kubectl -- apply -f https://raw.githubusercontent.com/ansible/awx-operator/devel/deploy/awx-operator.yaml
 ```
 
-You will see the following indicating that database migrations are running:
+##### Verifying the Operator Deployment
 
-```bash
-Using /etc/ansible/ansible.cfg as config file
-127.0.0.1 | SUCCESS => {
-    "changed": false,
-    "db": "awx"
-}
-Operations to perform:
-  Synchronize unmigrated apps: solo, api, staticfiles, messages, channels, django_extensions, ui, rest_framework, polymorphic
-  Apply all migrations: sso, taggit, sessions, sites, kombu_transport_django, social_auth, contenttypes, auth, conf, main
-Synchronizing apps without migrations:
-  Creating tables...
-    Running deferred SQL...
-  Installing custom SQL...
-Running migrations:
-  Rendering model states... DONE
-  Applying contenttypes.0001_initial... OK
-  Applying contenttypes.0002_remove_content_type_name... OK
-  Applying auth.0001_initial... OK
-  Applying auth.0002_alter_permission_name_max_length... OK
-  Applying auth.0003_alter_user_email_max_length... OK
-  Applying auth.0004_alter_user_username_opts... OK
-  Applying auth.0005_alter_user_last_login_null... OK
-  Applying auth.0006_require_contenttypes_0002... OK
-  Applying taggit.0001_initial... OK
-  Applying taggit.0002_auto_20150616_2121... OK
-  ...
+After a few seconds, the operator should be up and running. Verify it by running the following command:
+
+```
+$ minikube kubectl get pods
+NAME                           READY   STATUS    RESTARTS   AGE
+awx-operator-7c78bfbfd-xb6th   1/1     Running   0          11s
 ```
 
-When you see output similar to the following, you'll know that database migrations have completed, and you can access the web interface:
+#### Deploy AWX
 
-```bash
-Python 2.7.5 (default, Nov  6 2016, 00:28:07)
-[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2
-Type "help", "copyright", "credits" or "license" for more information.
-(InteractiveConsole)
+Once the Operator is running, you can now deploy AWX by creating a simple YAML file:
 
->>> <User: admin>
->>> Default organization added.
-Demo Credential, Inventory, and Job Template added.
-Successfully registered instance awx-3886581826-5mv0l
-(changed: True)
-Creating instance group tower
-Added instance awx-3886581826-5mv0l to tower
+```
+$ cat myawx.yml
+---
+apiVersion: awx.ansible.com/v1beta1
+kind: AWX
+metadata:
+  name: awx
+spec:
+  tower_ingress_type: Ingress
 ```
 
-Once database migrations complete, the web interface will be accessible.
+And then creating the AWX object in the Kubernetes API:
 
-### Accessing AWX
-
-The AWX web interface is running in the AWX pod, behind the `awx-web-svc` service. To view the service, and its port value, run the following command:
-
-```bash
-# View available services
-$ oc get services
-
-NAME          CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
-awx-web-svc   172.30.111.74   <nodes>       8052:30083/TCP   37m
-postgresql    172.30.102.9    <none>        5432/TCP         38m
+```
+$ minikube kubectl apply -- -f myawx.yml
+awx.awx.ansible.com/awx created
 ```
 
-The deployment process creates a route, `awx-web-svc`, to expose the service. How the ingres is actually created will vary depending on your environment, and how the cluster is configured. You can view the route, and the external IP address and hostname assigned to it, by running the following command:
+After creating the AWX object in the Kubernetes API, the operator will begin running its reconciliation loop. 
 
-```bash
-# View available routes
-$ oc get routes
+To see what's going on, you can tail the logs of the operator pod (note that your pod name will be different):
 
-NAME          HOST/PORT                             PATH      SERVICES      PORT      TERMINATION   WILDCARD
-awx-web-svc   awx-web-svc-awx.192.168.64.2.nip.io             awx-web-svc   http      edge/Allow    None
+```
+$ minikube kubectl logs -- -f awx-operator-7c78bfbfd-xb6th
 ```
 
-The above example is taken from a Minishift instance. From a web browser, use `https` to access the `HOST/PORT` value from your environment. Using the above example, the URL to access the server would be [https://awx-web-svc-awx.192.168.64.2.nip.io](https://awx-web-svc-awx.192.168.64.2.nip.io).
+After a few seconds, you will see the database and application pods show up. On a fresh system, it may take a few minutes for the container images to download.
 
-Once you access the AWX server, you will be prompted with a login dialog. The default administrator username is `admin`, and the password is `password`.
-
-## Kubernetes
-
-### Prerequisites
-
-A Kubernetes deployment will require you to have access to a Kubernetes cluster as well as the following tools:
-
-- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/)
-- [helm](https://docs.helm.sh/using_helm/#quickstart-guide)
-
-The installation program will reference `kubectl` directly. `helm` is only necessary if you are letting the installer configure PostgreSQL for you.
-
-The default resource requests per-pod requires:
-
-> Memory: 6GB
-> CPU: 3 cores
-
-This can be tuned by overriding the variables found in [/installer/roles/kubernetes/defaults/main.yml](/installer/roles/kubernetes/defaults/main.yml). Special care should be taken when doing this as undersized instances will experience crashes and resource exhaustion.
-
-For more detail on how resource requests are formed see: [https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/](https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/)
-
-### Pre-build steps
-
-Before starting the build process, review the [inventory](./installer/inventory) file, and uncomment and provide values for the following variables found in the `[all:vars]` section uncommenting when necessary. Make sure the openshift and standalone docker sections are commented out:
-
-*kubernetes_context*
-
-> Prior to running the installer, make sure you've configured the context for the cluster you'll be installing to. This is how the installer knows which cluster to connect to and what authentication to use
-
-*kubernetes_namespace*
-
-> Name of the Kubernetes namespace where the AWX resources will be installed. This will be created if it doesn't exist
-
-*docker_registry_*
-
-> These settings should be used if building your own base images. You'll need access to an external registry and are responsible for making sure your kube cluster can talk to it and use it. If these are undefined and the dockerhub_ configuration settings are uncommented then the images will be pulled from dockerhub instead
-
-### Configuring Helm
-
-If you want the AWX installer to manage creating the database pod (rather than installing and configuring postgres on your own). Then you will need to have a working `helm` installation, you can find details here: [https://docs.helm.sh/using_helm/#quickstart-guide](https://docs.helm.sh/using_helm/#quickstart-guide).
-
-Newer Kubernetes clusters with RBAC enabled will need to make sure a service account is created, make sure to follow the instructions here [https://docs.helm.sh/using_helm/#role-based-access-control](https://docs.helm.sh/using_helm/#role-based-access-control)
-
-### Start the build
-
-After making changes to the `inventory` file use `ansible-playbook` to begin the install
-
-```bash
-$ ansible-playbook -i inventory install.yml
+```
+$ minikube kubectl get pods
+NAME                           READY   STATUS    RESTARTS   AGE
+awx-5ffbfd489c-bvtvf           3/3     Running   0          2m54s
+awx-operator-7c78bfbfd-xb6th   1/1     Running   0          6m42s
+awx-postgres-0                 1/1     Running   0          2m58s
 ```
 
-### Post build
+##### Accessing AWX
 
-After the playbook run completes, check the status of the deployment by running `kubectl get pods --namespace awx` (replace awx with the namespace you used):
+To access the AWX UI, you'll need to grab the service url from minikube:
 
-```bash
-# View the running pods, it may take a few minutes for everything to be marked in the Running state
-$ kubectl get pods --namespace awx
-NAME                             READY     STATUS    RESTARTS   AGE
-awx-2558692395-2r8ss             4/4       Running   0          29s
-awx-postgresql-355348841-kltkn   1/1       Running   0          1m
+```
+$ minikube service awx-service --url
+http://192.168.59.2:31868
 ```
 
-### Accessing AWX
+On fresh installs, you will see the "AWX is currently upgrading." page until database migrations finish.
 
-The AWX web interface is running in the AWX pod behind the `awx-web-svc` service:
+Once you are redirected to the login screen, you can now log in by obtaining the generated admin password (note: do not copy the trailing `%`):
 
-```bash
-# View available services
-$ kubectl get svc --namespace awx
-NAME             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
-awx-postgresql   ClusterIP   10.7.250.208   <none>        5432/TCP       2m
-awx-web-svc      NodePort    10.7.241.35    <none>        80:30177/TCP   1m
+```
+$ minikube kubectl -- get secret awx-admin-password -o jsonpath='{.data.password}' | base64 --decode
+b6ChwVmqEiAsil2KSpH4xGaZPeZvWnWj%
 ```
 
-The deployment process creates an `Ingress` named `awx-web-svc` also. Some kubernetes cloud providers will automatically handle routing configuration when an Ingress is created others may require that you more explicitly configure it. You can see what kubernetes knows about things with:
+Now you can log in at the URL above with the username "admin" and the password above. Happy Automating!
 
-```bash
- kubectl get ing --namespace awx
-NAME          HOSTS     ADDRESS          PORTS     AGE
-awx-web-svc   *         35.227.x.y       80        3m
-```
 
-If your provider is able to allocate an IP Address from the Ingress controller then you can navigate to the address and access the AWX interface. For some providers it can take a few minutes to allocate and make this accessible. For other providers it may require you to manually intervene.
+# Installing the AWX CLI
 
-### SSL Termination
+`awx` is the official command-line client for AWX.  It:
 
-Unlike Openshift's `Route` the Kubernetes `Ingress` doesn't yet handle SSL termination. As such the default configuration will only expose AWX through HTTP on port 80. You are responsible for configuring SSL support until support is added (either to Kubernetes or AWX itself).
+* Uses naming and structure consistent with the AWX HTTP API
+* Provides consistent output formats with optional machine-parsable formats
+* To the extent possible, auto-detects API versions, available endpoints, and
+  feature support across multiple versions of AWX.
 
+Potential uses include:
 
-## Docker-Compose
+* Configuring and launching jobs/playbooks
+* Checking on the status and output of job runs
+* Managing objects like organizations, users, teams, etc...
 
-### Prerequisites
+The preferred way to install the AWX CLI is through pip directly from PyPI:
 
-- [Docker](https://docs.docker.com/engine/installation/) on the host where AWX will be deployed. After installing Docker, the Docker service must be started (depending on your OS, you may have to add the local user that uses Docker to the ``docker`` group, refer to the documentation for details)
-- [docker-compose](https://pypi.org/project/docker-compose/) Python module.
-    + This also installs the `docker` Python module, which is incompatible with `docker-py`. If you have previously installed `docker-py`, please uninstall it.
-- [Docker Compose](https://docs.docker.com/compose/install/).
+    pip3 install awxkit
+    awx --help
 
-### Pre-build steps
+## Building the CLI Documentation
 
-#### Deploying to a remote host
+To build the docs, spin up a real AWX server, `pip3 install sphinx sphinxcontrib-autoprogram`, and run:
 
-By default, the delivered [installer/inventory](./installer/inventory) file will deploy AWX to the local host. It is possible, however, to deploy to a remote host. The [installer/install.yml](./installer/install.yml) playbook can be used to build images on the local host, and ship the built images to, and run deployment tasks on, a remote host. To do this, modify the [installer/inventory](./installer/inventory) file, by commenting out `localhost`, and adding the remote host.
-
-For example, suppose you wish to build images locally on your CI/CD host, and deploy them to a remote host named *awx-server*. To do this, add *awx-server* to the [installer/inventory](./installer/inventory) file, and comment out or remove `localhost`, as demonstrated by the following:
-
-```yaml
-# localhost ansible_connection=local
-awx-server
-
-[all:vars]
-...
-```
-
-In the above example, image build tasks will be delegated to `localhost`, which is typically where the clone of the AWX project exists. Built images will be archived, copied to remote host, and imported into the remote Docker image cache. Tasks to start the AWX containers will then execute on the remote host.
-
-If you choose to use the official images then the remote host will be the one to pull those images.
-
-**Note**
-
-> You may also want to set additional variables to control how Ansible connects to the host. For more information about this, view [Behavioral Inventory Parameters](http://docs.ansible.com/ansible/latest/intro_inventory.html#id12).
-
-> As mentioned above, in [Prerequisites](#prerequisites-1), the prerequisites are required on the remote host.
-
-> When deploying to a remote host, the playbook does not execute tasks with the `become` option. For this reason, make sure the user that connects to the remote host has privileges to run the `docker` command. This typically means that non-privileged users need to be part of the `docker` group.
-
-
-#### Inventory variables
-
-Before starting the build process, review the [inventory](./installer/inventory) file, and uncomment and provide values for the following variables found in the `[all:vars]` section:
-
-*postgres_data_dir*
-
-> If you're using the default PostgreSQL container (see [PostgreSQL](#postgresql-1) below), provide a path that can be mounted to the container, and where the database can be persisted.
-
-*host_port*
-
-> Provide a port number that can be mapped from the Docker daemon host to the web server running inside the AWX container. Defaults to *80*.
-
-*host_port_ssl*
-
-> Provide a port number that can be mapped from the Docker daemon host to the web server running inside the AWX container for SSL support. Defaults to *443*, only works if you also set `ssl_certificate` (see below).
-
-*ssl_certificate*
-
-> Optionally, provide the path to a file that contains a certificate and its private key.
-
-*docker_compose_dir*
-
-> When using docker-compose, the `docker-compose.yml` file will be created there (default `/tmp/awxcompose`).
-
-*custom_venv_dir*
-
-> Adds the custom venv environments from the local host to be passed into the containers at install.
-
-*ca_trust_dir*
-
-> If you're using a non trusted CA, provide a path where the untrusted Certs are stored on your Host.
-
-#### Docker registry
-
-If you wish to tag and push built images to a Docker registry, set the following variables in the inventory file:
-
-*docker_registry*
-
-> IP address and port, or URL, for accessing a registry.
-
-*docker_registry_repository*
-
-> Namespace to use when pushing and pulling images to and from the registry. Defaults to *awx*.
-
-*docker_registry_username*
-
-> Username of the user that will push images to the registry. Defaults to *developer*.
-
-*docker_remove_local_images*
-
-> Due to the way that the docker_image module behaves, images will not be pushed to a remote repository if they are present locally.  Set this to delete local versions of the images that will be pushed to the remote.  This will fail if containers are currently running from those images.
-
-**Note**
-
-> These settings are ignored if using official images
-
-
-#### Proxy settings
-
-*http_proxy*
-
-> IP address and port, or URL, for using an http_proxy.
-
-*https_proxy*
-
-> IP address and port, or URL, for using an https_proxy.
-
-*no_proxy*
-
-> Exclude IP address or URL from the proxy.
-
-#### PostgreSQL
-
-AWX requires access to a PostgreSQL database, and by default, one will be created and deployed in a container, and data will be persisted to a host volume. In this scenario, you must set the value of `postgres_data_dir` to a path that can be mounted to the container. When the container is stopped, the database files will still exist in the specified path.
-
-If you wish to use an external database, in the inventory file, set the value of `pg_hostname`, and update `pg_username`, `pg_password`, `pg_admin_password`, `pg_database`, and `pg_port` with the connection information.
-
-### Start the build
-
-If you are not pushing images to a Docker registry, start the build by running the following:
-
-```bash
-# Set the working directory to installer
-$ cd installer
-
-# Run the Ansible playbook
-$ ansible-playbook -i inventory install.yml
-```
-
-If you're pushing built images to a repository, then use the `-e` option to pass the registry password as follows, replacing *password* with the password of the username assigned to `docker_registry_username` (note that you will also need to remove `dockerhub_base` and `dockerhub_version` from the inventory file):
-
-```bash
-# Set the working directory to installer
-$ cd installer
-
-# Run the Ansible playbook
-$ ansible-playbook -i inventory -e docker_registry_password=password install.yml
-```
-
-### Post build
-
-After the playbook run completes, Docker will report up to 5 running containers. If you chose to use an existing PostgresSQL database, then it will report 4. You can view the running containers using the `docker ps` command, as follows:
-
-```bash
-CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                NAMES
-e240ed8209cd        awx_task:1.0.0.8    "/tini -- /bin/sh ..."   2 minutes ago       Up About a minute   8052/tcp                             awx_task
-1cfd02601690        awx_web:1.0.0.8     "/tini -- /bin/sh ..."   2 minutes ago       Up About a minute   0.0.0.0:443->8052/tcp                 awx_web
-55a552142bcd        memcached:alpine    "docker-entrypoint..."   2 minutes ago       Up 2 minutes        11211/tcp                            memcached
-84011c072aad        rabbitmq:3          "docker-entrypoint..."   2 minutes ago       Up 2 minutes        4369/tcp, 5671-5672/tcp, 25672/tcp   rabbitmq
-97e196120ab3        postgres:9.6        "docker-entrypoint..."   2 minutes ago       Up 2 minutes        5432/tcp                             postgres
-```
-
-If you're deploying using Docker Compose, container names will be prefixed by the name of the folder where the docker-compose.yml file is created (by default, `awx`).
-
-Immediately after the containers start, the *awx_task* container will perform required setup tasks, including database migrations. These tasks need to complete before the web interface can be accessed. To monitor the progress, you can follow the container's STDOUT by running the following:
-
-```bash
-# Tail the the awx_task log
-$ docker logs -f awx_task
-```
-
-You will see output similar to the following:
-
-```bash
-Using /etc/ansible/ansible.cfg as config file
-127.0.0.1 | SUCCESS => {
-    "changed": false,
-    "db": "awx"
-}
-Operations to perform:
-  Synchronize unmigrated apps: solo, api, staticfiles, messages, channels, django_extensions, ui, rest_framework, polymorphic
-  Apply all migrations: sso, taggit, sessions, sites, kombu_transport_django, social_auth, contenttypes, auth, conf, main
-Synchronizing apps without migrations:
-  Creating tables...
-    Running deferred SQL...
-  Installing custom SQL...
-Running migrations:
-  Rendering model states... DONE
-  Applying contenttypes.0001_initial... OK
-  Applying contenttypes.0002_remove_content_type_name... OK
-  Applying auth.0001_initial... OK
-  Applying auth.0002_alter_permission_name_max_length... OK
-  Applying auth.0003_alter_user_email_max_length... OK
-  Applying auth.0004_alter_user_username_opts... OK
-  Applying auth.0005_alter_user_last_login_null... OK
-  Applying auth.0006_require_contenttypes_0002... OK
-  Applying taggit.0001_initial... OK
-  Applying taggit.0002_auto_20150616_2121... OK
-  Applying main.0001_initial... OK
-...
-```
-
-Once migrations complete, you will see the following log output, indicating that migrations have completed:
-
-```bash
-Python 2.7.5 (default, Nov  6 2016, 00:28:07)
-[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux2
-Type "help", "copyright", "credits" or "license" for more information.
-(InteractiveConsole)
-
->>> <User: admin>
->>> Default organization added.
-Demo Credential, Inventory, and Job Template added.
-Successfully registered instance awx
-(changed: True)
-Creating instance group tower
-Added instance awx to tower
-(changed: True)
-...
-```
-
-### Accessing AWX
-
-The AWX web server is accessible on the deployment host, using the *host_port* value set in the *inventory* file. The default URL is [http://localhost](http://localhost).
-
-You will prompted with a login dialog. The default administrator username is `admin`, and the password is `password`.
-
-### Maintenance using docker-compose
-
-After the installation, maintenance operations with docker-compose can be done by using the  `docker-compose.yml` file created at the location pointed by `docker_compose_dir`.
-
-Among the possible operations, you may:
-
-- Stop AWX : `docker-compose stop`
-- Upgrade AWX : `docker-compose pull && docker-compose up --force-recreate`
-
-See the [docker-compose documentation](https://docs.docker.com/compose/) for details.
+    ~ cd awxkit/awxkit/cli/docs
+    ~ TOWER_HOST=https://awx.example.org TOWER_USERNAME=example TOWER_PASSWORD=secret make clean html
+    ~ cd build/html/ && python -m http.server
+    Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ..

ISSUES.md

@@ -31,7 +31,7 @@ If your issue isn't considered high priority, then please be patient as it may t
 
 `state:needs_info` The issue needs more information. This could be more debug output, more specifics out the system such as version information. Any detail that is currently preventing this issue from moving forward. This should be considered a blocked state.
 
-`state:needs_review` The the issue/pull request needs to be reviewed by other maintainers and contributors. This is usually used when there is a question out to another maintainer or when a person is less familar with an area of the code base the issue is for.
+`state:needs_review` The issue/pull request needs to be reviewed by other maintainers and contributors. This is usually used when there is a question out to another maintainer or when a person is less familar with an area of the code base the issue is for.
 
 `state:needs_revision` More commonly used on pull requests, this state represents that there are changes that are being waited on.
 

MANIFEST.in

@@ -4,12 +4,13 @@ recursive-include awx *.mo
 recursive-include awx/static *
 recursive-include awx/templates *.html
 recursive-include awx/api/templates *.md *.html
-recursive-include awx/ui/templates *.html
-recursive-include awx/ui/static *
+recursive-include awx/ui_next/build *.html
+recursive-include awx/ui_next/build *
 recursive-include awx/playbooks *.yml
 recursive-include awx/lib/site-packages *
 recursive-include awx/plugins *.ps1
 recursive-include requirements *.txt
+recursive-include requirements *.yml
 recursive-include config *
 recursive-include docs/licenses *
 recursive-exclude awx devonly.py*

Makefile

@@ -6,26 +6,32 @@ PACKER ?= packer
 PACKER_BUILD_OPTS ?= -var 'official=$(OFFICIAL)' -var 'aw_repo_url=$(AW_REPO_URL)'
 NODE ?= node
 NPM_BIN ?= npm
+CHROMIUM_BIN=/tmp/chrome-linux/chrome
 DEPS_SCRIPT ?= packaging/bundle/deps.py
 GIT_BRANCH ?= $(shell git rev-parse --abbrev-ref HEAD)
 MANAGEMENT_COMMAND ?= awx-manage
 IMAGE_REPOSITORY_AUTH ?=
 IMAGE_REPOSITORY_BASE ?= https://gcr.io
 VERSION := $(shell cat VERSION)
+PYCURL_SSL_LIBRARY ?= openssl
 
 # NOTE: This defaults the container image version to the branch that's active
 COMPOSE_TAG ?= $(GIT_BRANCH)
 COMPOSE_HOST ?= $(shell hostname)
 
-VENV_BASE ?= /venv
-COLLECTION_VENV ?= /awx_devel/awx_collection_test_venv
+VENV_BASE ?= /var/lib/awx/venv/
 SCL_PREFIX ?=
 CELERY_SCHEDULE_FILE ?= /var/lib/awx/beat.db
 
 DEV_DOCKER_TAG_BASE ?= gcr.io/ansible-tower-engineering
+DEVEL_IMAGE_NAME ?= $(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG)
+
 # Python packages to install only from source (not from binary wheels)
 # Comma separated list
-SRC_ONLY_PKGS ?= cffi,pycparser,psycopg2,twilio
+SRC_ONLY_PKGS ?= cffi,pycparser,psycopg2,twilio,pycurl
+# These should be upgraded in the AWX and Ansible venv before attempting
+# to install the actual requirements
+VENV_BOOTSTRAP ?= pip==19.3.1 setuptools==41.6.0
 
 # Determine appropriate shasum command
 UNAME_S := $(shell uname -s)
@@ -52,34 +58,14 @@ WHEEL_COMMAND ?= bdist_wheel
 SDIST_TAR_FILE ?= $(SDIST_TAR_NAME).tar.gz
 WHEEL_FILE ?= $(WHEEL_NAME)-py2-none-any.whl
 
-# UI flag files
-UI_DEPS_FLAG_FILE = awx/ui/.deps_built
-UI_RELEASE_DEPS_FLAG_FILE = awx/ui/.release_deps_built
-UI_RELEASE_FLAG_FILE = awx/ui/.release_built
-
 I18N_FLAG_FILE = .i18n_built
 
 .PHONY: awx-link clean clean-tmp clean-venv requirements requirements_dev \
-	develop refresh adduser migrate dbchange runserver \
+	develop refresh adduser migrate dbchange \
 	receiver test test_unit test_coverage coverage_html \
-	dev_build release_build release_clean sdist \
-	ui-docker-machine ui-docker ui-release ui-devel \
-	ui-test ui-deps ui-test-ci VERSION
-
-# remove ui build artifacts
-clean-ui: clean-languages
-	rm -rf awx/ui/static/
-	rm -rf awx/ui/node_modules/
-	rm -rf awx/ui/test/unit/reports/
-	rm -rf awx/ui/test/spec/reports/
-	rm -rf awx/ui/test/e2e/reports/
-	rm -rf awx/ui/client/languages/
-	rm -rf awx/ui_next/node_modules/
-	rm -rf awx/ui_next/coverage/
-	rm -rf awx/ui_next/build/locales/_build/
-	rm -f $(UI_DEPS_FLAG_FILE)
-	rm -f $(UI_RELEASE_DEPS_FLAG_FILE)
-	rm -f $(UI_RELEASE_FLAG_FILE)
+	dev_build release_build sdist \
+	ui-release ui-devel \
+	VERSION docker-compose-sources
 
 clean-tmp:
 	rm -rf tmp/
@@ -100,7 +86,7 @@ clean-languages:
 	find . -type f -regex ".*\.mo$$" -delete
 
 # Remove temporary build files, compiled Python files.
-clean: clean-ui clean-api clean-dist
+clean: clean-ui clean-api clean-awxkit clean-dist
 	rm -rf awx/public
 	rm -rf awx/lib/site-packages
 	rm -rf awx/job_status
@@ -116,6 +102,10 @@ clean-api:
 	find . -type d -name "__pycache__" -delete
 	rm -f awx/awx_test.sqlite3*
 	rm -rf requirements/vendor
+	rm -rf awx/projects
+
+clean-awxkit:
+	rm -rf awxkit/*.egg-info awxkit/.tox awxkit/build/*
 
 # convenience target to assert environment variables are defined
 guard-%:
@@ -124,79 +114,38 @@ guard-%:
 	    exit 1; \
 	fi
 
-virtualenv: virtualenv_ansible virtualenv_awx
-
-virtualenv_ansible:
-	if [ "$(VENV_BASE)" ]; then \
-		if [ ! -d "$(VENV_BASE)" ]; then \
-			mkdir $(VENV_BASE); \
-		fi; \
-		if [ ! -d "$(VENV_BASE)/ansible" ]; then \
-			virtualenv -p python --system-site-packages $(VENV_BASE)/ansible && \
-			$(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) --ignore-installed six packaging appdirs && \
-			$(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) --ignore-installed setuptools==36.0.1 && \
-			$(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) --ignore-installed pip==9.0.1; \
-		fi; \
-	fi
-
-virtualenv_ansible_py3:
-	if [ "$(VENV_BASE)" ]; then \
-		if [ ! -d "$(VENV_BASE)" ]; then \
-			mkdir $(VENV_BASE); \
-		fi; \
-		if [ ! -d "$(VENV_BASE)/ansible" ]; then \
-			$(PYTHON) -m venv --system-site-packages $(VENV_BASE)/ansible; \
-		fi; \
-	fi
+virtualenv: virtualenv_awx
 
+# flit is needed for offline install of certain packages, specifically ptyprocess
+# it is needed for setup, but not always recognized as a setup dependency
+# similar to pip, setuptools, and wheel, these are all needed here as a bootstrapping issues
 virtualenv_awx:
 	if [ "$(VENV_BASE)" ]; then \
 		if [ ! -d "$(VENV_BASE)" ]; then \
 			mkdir $(VENV_BASE); \
 		fi; \
 		if [ ! -d "$(VENV_BASE)/awx" ]; then \
-			$(PYTHON) -m venv --system-site-packages $(VENV_BASE)/awx; \
-			$(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) --ignore-installed docutils==0.14; \
+			virtualenv -p $(PYTHON) $(VENV_BASE)/awx; \
+			$(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) $(VENV_BOOTSTRAP); \
 		fi; \
 	fi
 
-requirements_ansible: virtualenv_ansible
-	if [[ "$(PIP_OPTIONS)" == *"--no-index"* ]]; then \
-	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_local.txt | $(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) --ignore-installed -r /dev/stdin ; \
-	else \
-	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_git.txt | $(VENV_BASE)/ansible/bin/pip install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) --ignore-installed -r /dev/stdin ; \
-	fi
-	$(VENV_BASE)/ansible/bin/pip uninstall --yes -r requirements/requirements_ansible_uninstall.txt
-
-requirements_ansible_py3: virtualenv_ansible_py3
-	if [[ "$(PIP_OPTIONS)" == *"--no-index"* ]]; then \
-	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_local.txt | $(VENV_BASE)/ansible/bin/pip3 install $(PIP_OPTIONS) --ignore-installed -r /dev/stdin ; \
-	else \
-	    cat requirements/requirements_ansible.txt requirements/requirements_ansible_git.txt | $(VENV_BASE)/ansible/bin/pip3 install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) --ignore-installed -r /dev/stdin ; \
-	fi
-	$(VENV_BASE)/ansible/bin/pip3 uninstall --yes -r requirements/requirements_ansible_uninstall.txt
-
-requirements_ansible_dev:
-	if [ "$(VENV_BASE)" ]; then \
-		$(VENV_BASE)/ansible/bin/pip install pytest mock; \
-	fi
-
 # Install third-party requirements needed for AWX's environment.
+# this does not use system site packages intentionally
 requirements_awx: virtualenv_awx
 	if [[ "$(PIP_OPTIONS)" == *"--no-index"* ]]; then \
-	    cat requirements/requirements.txt requirements/requirements_local.txt | $(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) --ignore-installed -r /dev/stdin ; \
+	    cat requirements/requirements.txt requirements/requirements_local.txt | $(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) -r /dev/stdin ; \
 	else \
-	    cat requirements/requirements.txt requirements/requirements_git.txt | $(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) --ignore-installed -r /dev/stdin ; \
+	    cat requirements/requirements.txt requirements/requirements_git.txt | $(VENV_BASE)/awx/bin/pip install $(PIP_OPTIONS) --no-binary $(SRC_ONLY_PKGS) -r /dev/stdin ; \
 	fi
-	echo "include-system-site-packages = true" >> $(VENV_BASE)/awx/lib/python$(PYTHON_VERSION)/pyvenv.cfg
 	$(VENV_BASE)/awx/bin/pip uninstall --yes -r requirements/requirements_tower_uninstall.txt
 
 requirements_awx_dev:
 	$(VENV_BASE)/awx/bin/pip install -r requirements/requirements_dev.txt
 
-requirements: requirements_ansible requirements_awx
+requirements: requirements_awx
 
-requirements_dev: requirements_awx requirements_ansible_py3 requirements_awx_dev requirements_ansible_dev
+requirements_dev: requirements_awx requirements_awx_dev
 
 requirements_test: requirements
 
@@ -249,28 +198,6 @@ migrate:
 dbchange:
 	$(MANAGEMENT_COMMAND) makemigrations
 
-server_noattach:
-	tmux new-session -d -s awx 'exec make uwsgi'
-	tmux rename-window 'AWX'
-	tmux select-window -t awx:0
-	tmux split-window -v 'exec make dispatcher'
-	tmux new-window 'exec make daphne'
-	tmux select-window -t awx:1
-	tmux rename-window 'WebSockets'
-	tmux split-window -h 'exec make runworker'
-	tmux split-window -v 'exec make nginx'
-	tmux new-window 'exec make receiver'
-	tmux select-window -t awx:2
-	tmux rename-window 'Extra Services'
-	tmux select-window -t awx:0
-
-server: server_noattach
-	tmux -2 attach-session -t awx
-
-# Use with iterm2's native tmux protocol support
-servercc: server_noattach
-	tmux -2 -CC attach-session -t awx
-
 supervisor:
 	@if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
@@ -283,11 +210,27 @@ collectstatic:
 	fi; \
 	mkdir -p awx/public/static && $(PYTHON) manage.py collectstatic --clear --noinput > /dev/null 2>&1
 
+UWSGI_DEV_RELOAD_COMMAND ?= supervisorctl restart tower-processes:awx-dispatcher tower-processes:awx-receiver
+
 uwsgi: collectstatic
 	@if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
 	fi; \
-    uwsgi -b 32768 --socket 127.0.0.1:8050 --module=awx.wsgi:application --home=/venv/awx --chdir=/awx_devel/ --vacuum --processes=5 --harakiri=120 --master --no-orphans --py-autoreload 1 --max-requests=1000 --stats /tmp/stats.socket --lazy-apps --logformat "%(addr) %(method) %(uri) - %(proto) %(status)" --hook-accepting1="exec:supervisorctl restart tower-processes:awx-dispatcher tower-processes:awx-receiver"
+	uwsgi -b 32768 \
+	    --socket 127.0.0.1:8050 \
+	    --module=awx.wsgi:application \
+	    --home=/var/lib/awx/venv/awx \
+	    --chdir=/awx_devel/ \
+	    --vacuum \
+	    --processes=5 \
+	    --harakiri=120 --master \
+	    --no-orphans \
+	    --py-autoreload 1 \
+	    --max-requests=1000 \
+	    --stats /tmp/stats.socket \
+	    --lazy-apps \
+	    --logformat "%(addr) %(method) %(uri) - %(proto) %(status)" \
+	    --hook-accepting1="exec: $(UWSGI_DEV_RELOAD_COMMAND)"
 
 daphne:
 	@if [ "$(VENV_BASE)" ]; then \
@@ -295,18 +238,11 @@ daphne:
 	fi; \
 	daphne -b 127.0.0.1 -p 8051 awx.asgi:channel_layer
 
-runworker:
+wsbroadcast:
 	@if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
 	fi; \
-	$(PYTHON) manage.py runworker --only-channels websocket.*
-
-# Run the built-in development webserver (by default on http://localhost:8013).
-runserver:
-	@if [ "$(VENV_BASE)" ]; then \
-		. $(VENV_BASE)/awx/bin/activate; \
-	fi; \
-	$(PYTHON) manage.py runserver
+	$(PYTHON) manage.py run_wsbroadcast
 
 # Run to start the background task dispatcher for development.
 dispatcher:
@@ -363,51 +299,64 @@ swagger: reports
 check: flake8 pep8 # pyflakes pylint
 
 awx-link:
-	cp -R /tmp/awx.egg-info /awx_devel/ || true
-	sed -i "s/placeholder/$(shell git describe --long | sed 's/\./\\./g')/" /awx_devel/awx.egg-info/PKG-INFO
-	cp -f /tmp/awx.egg-link /venv/awx/lib/python$(PYTHON_VERSION)/site-packages/awx.egg-link
+	[ -d "/awx_devel/awx.egg-info" ] || python3 /awx_devel/setup.py egg_info_dev
+	cp -f /tmp/awx.egg-link /var/lib/awx/venv/awx/lib/python$(PYTHON_VERSION)/site-packages/awx.egg-link
 
 TEST_DIRS ?= awx/main/tests/unit awx/main/tests/functional awx/conf/tests awx/sso/tests
 
 # Run all API unit tests.
 test:
-	@if [ "$(VENV_BASE)" ]; then \
+	if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
 	fi; \
 	PYTHONDONTWRITEBYTECODE=1 py.test -p no:cacheprovider -n auto $(TEST_DIRS)
-	cd awxkit && $(VENV_BASE)/awx/bin/tox -re py2,py3
-	awx-manage check_migrations --dry-run --check  -n 'vNNN_missing_migration_file'
-
-prepare_collection_venv:
-	rm -rf $(COLLECTION_VENV)
-	mkdir $(COLLECTION_VENV)
-	$(VENV_BASE)/awx/bin/pip install --target=$(COLLECTION_VENV) git+https://github.com/ansible/tower-cli.git
+	cmp VERSION awxkit/VERSION || "VERSION and awxkit/VERSION *must* match"
+	cd awxkit && $(VENV_BASE)/awx/bin/tox -re py3
+	awx-manage check_migrations --dry-run --check  -n 'missing_migration_file'
 
 COLLECTION_TEST_DIRS ?= awx_collection/test/awx
+COLLECTION_TEST_TARGET ?=
 COLLECTION_PACKAGE ?= awx
 COLLECTION_NAMESPACE ?= awx
+COLLECTION_INSTALL = ~/.ansible/collections/ansible_collections/$(COLLECTION_NAMESPACE)/$(COLLECTION_PACKAGE)
 
 test_collection:
-	@if [ "$(VENV_BASE)" ]; then \
+	rm -f $(shell ls -d $(VENV_BASE)/awx/lib/python* | head -n 1)/no-global-site-packages.txt
+	if [ "$(VENV_BASE)" ]; then \
 		. $(VENV_BASE)/awx/bin/activate; \
-	fi; \
-	PYTHONPATH=$(COLLECTION_VENV):/awx_devel/awx_collection:$PYTHONPATH py.test $(COLLECTION_TEST_DIRS)
+	fi && \
+	pip install ansible && \
+	py.test $(COLLECTION_TEST_DIRS) -v
+	# The python path needs to be modified so that the tests can find Ansible within the container
+	# First we will use anything expility set as PYTHONPATH
+	# Second we will load any libraries out of the virtualenv (if it's unspecified that should be ok because python should not load out of an empty directory)
+	# Finally we will add the system path so that the tests can find the ansible libraries
 
 flake8_collection:
 	flake8 awx_collection/  # Different settings, in main exclude list
 
-test_collection_all: prepare_collection_venv test_collection flake8_collection
+test_collection_all: test_collection flake8_collection
 
-test_collection_sanity:
-	rm -rf sanity
-	mkdir -p sanity/ansible_collections/awx
-	cp -Ra awx_collection sanity/ansible_collections/awx/awx  # symlinks do not work
-	cd sanity/ansible_collections/awx/awx && git init && git add . # requires both this file structure and a git repo, so there you go
-	cd sanity/ansible_collections/awx/awx && ansible-test sanity --test validate-modules
+# WARNING: symlinking a collection is fundamentally unstable
+# this is for rapid development iteration with playbooks, do not use with other test targets
+symlink_collection:
+	rm -rf $(COLLECTION_INSTALL)
+	mkdir -p ~/.ansible/collections/ansible_collections/$(COLLECTION_NAMESPACE)  # in case it does not exist
+	ln -s $(shell pwd)/awx_collection $(COLLECTION_INSTALL)
 
 build_collection:
-	ansible-playbook -i localhost, awx_collection/template_galaxy.yml -e collection_package=$(COLLECTION_PACKAGE) -e collection_namespace=$(COLLECTION_NAMESPACE) -e collection_version=$(VERSION)
-	ansible-galaxy collection build awx_collection --output-path=awx_collection
+	ansible-playbook -i localhost, awx_collection/tools/template_galaxy.yml -e collection_package=$(COLLECTION_PACKAGE) -e collection_namespace=$(COLLECTION_NAMESPACE) -e collection_version=$(VERSION) -e '{"awx_template_version":false}'
+	ansible-galaxy collection build awx_collection_build --force --output-path=awx_collection_build
+
+install_collection: build_collection
+	rm -rf $(COLLECTION_INSTALL)
+	ansible-galaxy collection install awx_collection_build/$(COLLECTION_NAMESPACE)-$(COLLECTION_PACKAGE)-$(VERSION).tar.gz
+
+test_collection_sanity: install_collection
+	cd $(COLLECTION_INSTALL) && ansible-test sanity
+
+test_collection_integration: install_collection
+	cd $(COLLECTION_INSTALL) && ansible-test integration $(COLLECTION_TEST_TARGET)
 
 test_unit:
 	@if [ "$(VENV_BASE)" ]; then \
@@ -438,147 +387,50 @@ bulk_data:
 	fi; \
 	$(PYTHON) tools/data_generators/rbac_dummy_data_generator.py --preset=$(DATA_GEN_PRESET)
 
-# l10n TASKS
-# --------------------------------------
-
-# check for UI po files
-HAVE_PO := $(shell ls awx/ui/po/*.po 2>/dev/null)
-check-po:
-ifdef HAVE_PO
-	# Should be 'Language: zh-CN' but not 'Language: zh_CN' in zh_CN.po
-	for po in awx/ui/po/*.po ; do \
-	    echo $$po; \
-	    mo="awx/ui/po/`basename $$po .po`.mo"; \
-	    msgfmt --check --verbose $$po -o $$mo; \
-	    if test "$$?" -ne 0 ; then \
-	        exit -1; \
-	    fi; \
-	    rm $$mo; \
-	    name=`echo "$$po" | grep '-'`; \
-	    if test "x$$name" != x ; then \
-	        right_name=`echo $$language | sed -e 's/-/_/'`; \
-	        echo "ERROR: WRONG $$name CORRECTION: $$right_name"; \
-	        exit -1; \
-	    fi; \
-	    language=`grep '^"Language:' "$$po" | grep '_'`; \
-	    if test "x$$language" != x ; then \
-	        right_language=`echo $$language | sed -e 's/_/-/'`; \
-	        echo "ERROR: WRONG $$language CORRECTION: $$right_language in $$po"; \
-	        exit -1; \
-	    fi; \
-	done;
-else
-	@echo No PO files
-endif
-
-# generate UI .pot
-pot: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run pot
-
-# generate django .pot .po
-LANG = "en-us"
-messages:
-	@if [ "$(VENV_BASE)" ]; then \
-		. $(VENV_BASE)/awx/bin/activate; \
-	fi; \
-	$(PYTHON) manage.py makemessages -l $(LANG) --keep-pot
-
-# generate l10n .json .mo
-languages: $(I18N_FLAG_FILE)
-
-$(I18N_FLAG_FILE): $(UI_RELEASE_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run languages
-	$(PYTHON) tools/scripts/compilemessages.py
-	touch $(I18N_FLAG_FILE)
-
-# End l10n TASKS
-# --------------------------------------
-
-# UI RELEASE TASKS
-# --------------------------------------
-ui-release: $(UI_RELEASE_FLAG_FILE)
-
-$(UI_RELEASE_FLAG_FILE): $(I18N_FLAG_FILE) $(UI_RELEASE_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run build-release
-	touch $(UI_RELEASE_FLAG_FILE)
-
-$(UI_RELEASE_DEPS_FLAG_FILE):
-	PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=1 $(NPM_BIN) --unsafe-perm --prefix awx/ui ci --no-save awx/ui
-	touch $(UI_RELEASE_DEPS_FLAG_FILE)
-
-# END UI RELEASE TASKS
-# --------------------------------------
 
 # UI TASKS
 # --------------------------------------
-ui-deps: $(UI_DEPS_FLAG_FILE)
 
-$(UI_DEPS_FLAG_FILE):
-	@if [ -f ${UI_RELEASE_DEPS_FLAG_FILE} ]; then \
-		rm -rf awx/ui/node_modules; \
-		rm -f ${UI_RELEASE_DEPS_FLAG_FILE}; \
-	fi; \
-	$(NPM_BIN) --unsafe-perm --prefix awx/ui ci --no-save awx/ui
-	touch $(UI_DEPS_FLAG_FILE)
+UI_BUILD_FLAG_FILE = awx/ui_next/.ui-built
 
-ui-docker-machine: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run ui-docker-machine -- $(MAKEFLAGS)
+clean-ui:
+	rm -rf node_modules
+	rm -rf awx/ui_next/node_modules
+	rm -rf awx/ui_next/build
+	rm -rf awx/ui_next/src/locales/_build
+	rm -rf $(UI_BUILD_FLAG_FILE)
 
-# Native docker. Builds UI and raises BrowserSync & filesystem polling.
-ui-docker: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run ui-docker -- $(MAKEFLAGS)
+awx/ui_next/node_modules:
+	$(NPM_BIN) --prefix awx/ui_next --loglevel warn --ignore-scripts install
 
-# Builds UI with development UI without raising browser-sync or filesystem polling.
-ui-devel: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run build-devel -- $(MAKEFLAGS)
+$(UI_BUILD_FLAG_FILE):
+	$(NPM_BIN) --prefix awx/ui_next --loglevel warn run compile-strings
+	$(NPM_BIN) --prefix awx/ui_next --loglevel warn run build
+	mkdir -p awx/public/static/css
+	mkdir -p awx/public/static/js
+	mkdir -p awx/public/static/media
+	cp -r awx/ui_next/build/static/css/* awx/public/static/css
+	cp -r awx/ui_next/build/static/js/* awx/public/static/js
+	cp -r awx/ui_next/build/static/media/* awx/public/static/media
+	touch $@
 
-ui-test: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run test
+ui-release: awx/ui_next/node_modules $(UI_BUILD_FLAG_FILE)
 
-ui-lint: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) run --prefix awx/ui jshint
-	$(NPM_BIN) run --prefix awx/ui lint
+ui-devel: awx/ui_next/node_modules
+	@$(MAKE) -B $(UI_BUILD_FLAG_FILE)
 
-# A standard go-to target for API developers to use building the frontend
-ui: clean-ui ui-devel
+ui-devel-instrumented: awx/ui_next/node_modules
+	$(NPM_BIN) --prefix awx/ui_next --loglevel warn run start-instrumented
 
-ui-test-ci: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run test:ci
-	$(NPM_BIN) --prefix awx/ui run unit
+ui-devel-test: awx/ui_next/node_modules
+	$(NPM_BIN) --prefix awx/ui_next --loglevel warn run start
 
-jshint: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) run --prefix awx/ui jshint
-	$(NPM_BIN) run --prefix awx/ui lint
-
-ui-zuul-lint-and-test: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) run --prefix awx/ui jshint
-	$(NPM_BIN) run --prefix awx/ui lint
-	$(NPM_BIN) --prefix awx/ui run test:ci
-	$(NPM_BIN) --prefix awx/ui run unit
-
-# END UI TASKS
-# --------------------------------------
-
-# UI NEXT TASKS
-# --------------------------------------
-
-ui-next-lint:
+ui-zuul-lint-and-test:
 	$(NPM_BIN) --prefix awx/ui_next install
 	$(NPM_BIN) run --prefix awx/ui_next lint
 	$(NPM_BIN) run --prefix awx/ui_next prettier-check
+	$(NPM_BIN) run --prefix awx/ui_next test -- --coverage --watchAll=false
 
-ui-next-test:
-	$(NPM_BIN) --prefix awx/ui_next install
-	$(NPM_BIN) run --prefix awx/ui_next test
-
-ui-next-zuul-lint-and-test:
-	$(NPM_BIN) --prefix awx/ui_next install
-	$(NPM_BIN) run --prefix awx/ui_next lint
-	$(NPM_BIN) run --prefix awx/ui_next prettier-check
-	$(NPM_BIN) run --prefix awx/ui_next test
-
-# END UI NEXT TASKS
-# --------------------------------------
 
 # Build a pip-installable package into dist/ with a timestamped version number.
 dev_build:
@@ -615,76 +467,71 @@ docker-auth:
 		echo "$(IMAGE_REPOSITORY_AUTH)" | docker login -u oauth2accesstoken --password-stdin $(IMAGE_REPOSITORY_BASE); \
 	fi;
 
-# Docker isolated rampart
-docker-compose-isolated:
-	CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml -f tools/docker-isolated-override.yml up
+# This directory is bind-mounted inside of the development container and
+# needs to be pre-created for permissions to be set correctly. Otherwise,
+# Docker will create this directory as root.
+awx/projects:
+	@mkdir -p $@
 
-# Docker Compose Development environment
-docker-compose: docker-auth
-	CURRENT_UID=$(shell id -u) OS="$(shell docker info | grep 'Operating System')" TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml up --no-recreate awx
+COMPOSE_UP_OPTS ?=
+CLUSTER_NODE_COUNT ?= 1
 
-docker-compose-cluster: docker-auth
-	CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose-cluster.yml up
+docker-compose-sources:
+	ansible-playbook -i tools/docker-compose/inventory tools/docker-compose/ansible/sources.yml \
+	    -e awx_image=$(DEV_DOCKER_TAG_BASE)/awx_devel \
+	    -e awx_image_tag=$(COMPOSE_TAG) \
+	    -e cluster_node_count=$(CLUSTER_NODE_COUNT)
 
-docker-compose-credential-plugins: docker-auth
+docker-compose: docker-auth awx/projects docker-compose-sources
+	docker-compose -f tools/docker-compose/_sources/docker-compose.yml $(COMPOSE_UP_OPTS) up
+
+docker-compose-credential-plugins: docker-auth awx/projects docker-compose-sources
 	echo -e "\033[0;31mTo generate a CyberArk Conjur API key: docker exec -it tools_conjur_1 conjurctl account create quick-start\033[0m"
-	CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml -f tools/docker-credential-plugins-override.yml up --no-recreate awx
+	docker-compose -f tools/docker-compose/_sources/docker-compose.yml -f tools/docker-credential-plugins-override.yml up --no-recreate awx
 
-docker-compose-test: docker-auth
-	cd tools && CURRENT_UID=$(shell id -u) OS="$(shell docker info | grep 'Operating System')" TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm --service-ports awx /bin/bash
+docker-compose-test: docker-auth awx/projects docker-compose-sources
+	docker-compose -f tools/docker-compose/_sources/docker-compose.yml run --rm --service-ports awx_1 /bin/bash
 
-docker-compose-runtest:
-	cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm --service-ports awx /start_tests.sh
+docker-compose-runtest: awx/projects docker-compose-sources
+	docker-compose -f tools/docker-compose/_sources/docker-compose.yml run --rm --service-ports awx_1 /start_tests.sh
 
-docker-compose-build-swagger:
-	cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm --service-ports awx /start_tests.sh swagger
+docker-compose-build-swagger: awx/projects docker-compose-sources
+	docker-compose -f tools/docker-compose/_sources/docker-compose.yml run --rm --service-ports --no-deps awx_1 /start_tests.sh swagger
 
 detect-schema-change: genschema
 	curl https://s3.amazonaws.com/awx-public-ci-files/schema.json -o reference-schema.json
 	# Ignore differences in whitespace with -b
 	diff -u -b reference-schema.json schema.json
 
-docker-compose-clean:
-	cd tools && CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose run --rm -w /awx_devel --service-ports awx make clean
-	cd tools && TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose rm -sf
-
-docker-compose-build: awx-devel-build
+docker-compose-clean: awx/projects
+	docker-compose -f tools/docker-compose/_sources/docker-compose.yml rm -sf
 
 # Base development image build
-awx-devel-build:
-	docker build -t ansible/awx_devel -f tools/docker-compose/Dockerfile \
-		--cache-from=$(DEV_DOCKER_TAG_BASE)/awx_devel:devel \
-		--cache-from=$(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG) .
-	docker tag ansible/awx_devel $(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG)
-	#docker push $(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG)
+docker-compose-build:
+	ansible-playbook tools/ansible/dockerfile.yml -e build_dev=True
+	DOCKER_BUILDKIT=1 docker build -t $(DEVEL_IMAGE_NAME) \
+	    --build-arg BUILDKIT_INLINE_CACHE=1 \
+	    --cache-from=$(DEV_DOCKER_TAG_BASE)/awx_devel:$(COMPOSE_TAG) .
 
-# For use when developing on "isolated" AWX deployments
-docker-compose-isolated-build: awx-devel-build
-	docker build -t ansible/awx_isolated -f tools/docker-isolated/Dockerfile .
-	docker tag ansible/awx_isolated $(DEV_DOCKER_TAG_BASE)/awx_isolated:$(COMPOSE_TAG)
-	#docker push $(DEV_DOCKER_TAG_BASE)/awx_isolated:$(COMPOSE_TAG)
-
-MACHINE?=default
 docker-clean:
-	eval $$(docker-machine env $(MACHINE))
 	$(foreach container_id,$(shell docker ps -f name=tools_awx -aq),docker stop $(container_id); docker rm -f $(container_id);)
-	-docker images | grep "awx_devel" | awk '{print $$1 ":" $$2}' | xargs docker rmi
+	docker images | grep "awx_devel" | awk '{print $$1 ":" $$2}' | xargs docker rmi
+
+docker-clean-volumes: docker-compose-clean
+	docker volume rm tools_awx_db
 
 docker-refresh: docker-clean docker-compose
 
 # Docker Development Environment with Elastic Stack Connected
-docker-compose-elk: docker-auth
-	CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml -f tools/elastic/docker-compose.logstash-link.yml -f tools/elastic/docker-compose.elastic-override.yml up --no-recreate
+docker-compose-elk: docker-auth awx/projects docker-compose-sources
+	docker-compose -f tools/docker-compose/_sources/docker-compose.yml -f tools/elastic/docker-compose.logstash-link.yml -f tools/elastic/docker-compose.elastic-override.yml up --no-recreate
 
-docker-compose-cluster-elk: docker-auth
-	TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose-cluster.yml -f tools/elastic/docker-compose.logstash-link-cluster.yml -f tools/elastic/docker-compose.elastic-override.yml up --no-recreate
+docker-compose-cluster-elk: docker-auth awx/projects docker-compose-sources
+	docker-compose -f tools/docker-compose/_sources/docker-compose.yml -f tools/elastic/docker-compose.logstash-link-cluster.yml -f tools/elastic/docker-compose.elastic-override.yml up --no-recreate
 
 prometheus:
 	docker run -u0 --net=tools_default --link=`docker ps | egrep -o "tools_awx(_run)?_([^ ]+)?"`:awxweb --volume `pwd`/tools/prometheus:/prometheus --name prometheus -d -p 0.0.0.0:9090:9090 prom/prometheus --web.enable-lifecycle --config.file=/prometheus/prometheus.yml
 
-minishift-dev:
-	ansible-playbook -i localhost, -e devtree_directory=$(CURDIR) tools/clusterdevel/start_minishift_dev.yml
-
 clean-elk:
 	docker stop tools_kibana_1
 	docker stop tools_logstash_1
@@ -694,7 +541,38 @@ clean-elk:
 	docker rm tools_kibana_1
 
 psql-container:
-	docker run -it --net tools_default --rm postgres:10 sh -c 'exec psql -h "postgres" -p "5432" -U postgres'
+	docker run -it --net tools_default --rm postgres:12 sh -c 'exec psql -h "postgres" -p "5432" -U postgres'
 
 VERSION:
 	@echo "awx: $(VERSION)"
+
+Dockerfile: tools/ansible/roles/dockerfile/templates/Dockerfile.j2
+	ansible-playbook tools/ansible/dockerfile.yml
+
+Dockerfile.kube-dev: tools/ansible/roles/dockerfile/templates/Dockerfile.j2
+	ansible-playbook tools/ansible/dockerfile.yml \
+	    -e dockerfile_name=Dockerfile.kube-dev \
+	    -e kube_dev=True \
+	    -e template_dest=_build_kube_dev
+
+awx-kube-dev-build: Dockerfile.kube-dev
+	docker build -f Dockerfile.kube-dev \
+	    --build-arg BUILDKIT_INLINE_CACHE=1 \
+	    -t $(DEV_DOCKER_TAG_BASE)/awx_kube_devel:$(COMPOSE_TAG) .
+
+
+# Translation TASKS
+# --------------------------------------
+
+# generate UI .pot
+pot: $(UI_BUILD_FLAG_FILE)
+	$(NPM_BIN) --prefix awx/ui_next --loglevel warn run extract-strings
+	$(NPM_BIN) --prefix awx/ui_next --loglevel warn run extract-template
+
+# generate API django .pot .po
+LANG = "en-us"
+messages:
+	@if [ "$(VENV_BASE)" ]; then \
+		. $(VENV_BASE)/awx/bin/activate; \
+	fi; \
+	$(PYTHON) manage.py makemessages -l $(LANG) --keep-pot

README.md

@@ -1,4 +1,5 @@
-[![Gated by Zuul](https://zuul-ci.org/gated.svg)](https://ansible.softwarefactory-project.io/zuul/status)
+[![Gated by Zuul](https://zuul-ci.org/gated.svg)](https://ansible.softwarefactory-project.io/zuul/status) [![Code of Conduct](https://img.shields.io/badge/code%20of%20conduct-Ansible-yellow.svg)](https://docs.ansible.com/ansible/latest/community/code_of_conduct.html) [![Apache v2 License](https://img.shields.io/badge/license-Apache%202.0-brightgreen.svg)](https://github.com/ansible/awx/blob/devel/LICENSE.md) [![AWX Mailing List](https://img.shields.io/badge/mailing%20list-AWX-orange.svg)](https://groups.google.com/g/awx-project)
+[![IRC Chat](https://img.shields.io/badge/IRC-%23ansible--awx-blueviolet.svg)](https://webchat.freenode.net/#ansible-awx)
 
 <img src="https://raw.githubusercontent.com/ansible/awx-logos/master/awx/ui/client/assets/logo-login.svg?sanitize=true" width=200 alt="AWX" />
 
@@ -16,20 +17,20 @@ Contributing
 ------------
 
 - Refer to the [Contributing guide](./CONTRIBUTING.md) to get started developing, testing, and building AWX.
-- All code submissions are done through pull requests against the `devel` branch.
-- All contributors must use git commit --signoff for any commit to be merged, and agree that usage of --signoff constitutes agreement with the terms of [DCO 1.1](./DCO_1_1.md)
-- Take care to make sure no merge commits are in the submission, and use `git rebase` vs `git merge` for this reason.
-- If submitting a large code change, it's a good idea to join the `#ansible-awx` channel on irc.freenode.net, and talk about what you would like to do or add first. This not only helps everyone know what's going on, it also helps save time and effort, if the community decides some changes are needed.
+- All code submissions are made through pull requests against the `devel` branch.
+- All contributors must use git commit --signoff for any commit to be merged and agree that usage of --signoff constitutes agreement with the terms of [DCO 1.1](./DCO_1_1.md)
+- Take care to make sure no merge commits are in the submission, and use `git rebase` vs. `git merge` for this reason.
+- If submitting a large code change, it's a good idea to join the `#ansible-awx` channel on irc.freenode.net and talk about what you would like to do or add first. This not only helps everyone know what's going on, but it also helps save time and effort if the community decides some changes are needed.
 
 Reporting Issues
 ----------------
 
-If you're experiencing a problem that you feel is a bug in AWX, or have ideas for how to improve AWX, we encourage you to open an issue, and share your feedback. But before opening a new issue, we ask that you please take a look at our [Issues guide](./ISSUES.md).
+If you're experiencing a problem that you feel is a bug in AWX or have ideas for improving AWX, we encourage you to open an issue and share your feedback. But before opening a new issue, we ask that you please take a look at our [Issues guide](./ISSUES.md).
 
 Code of Conduct
 ---------------
 
-We ask all of our community members and contributors to adhere to the [Ansible code of conduct](http://docs.ansible.com/ansible/latest/community/code_of_conduct.html). If you have questions, or need assistance, please reach out to our community team at [codeofconduct@ansible.com](mailto:codeofconduct@ansible.com)   
+We ask all of our community members and contributors to adhere to the [Ansible code of conduct](http://docs.ansible.com/ansible/latest/community/code_of_conduct.html). If you have questions or need assistance, please reach out to our community team at [codeofconduct@ansible.com](mailto:codeofconduct@ansible.com)   
 
 Get Involved
 ------------
@@ -38,9 +39,3 @@ We welcome your feedback and ideas. Here's how to reach us with feedback and que
 
 - Join the `#ansible-awx` channel on irc.freenode.net
 - Join the [mailing list](https://groups.google.com/forum/#!forum/awx-project) 
-
-License
--------
-
-[Apache v2](./LICENSE.md)
-

VERSION

@@ -1 +1 @@
-9.0.1
+18.0.0

awx/__init__.py

@@ -24,31 +24,19 @@ except ImportError: # pragma: no cover
 import hashlib
 
 try:
-    import django
-    from django.db.backends.base import schema
-    from django.db.backends.utils import names_digest
+    import django  # noqa: F401
     HAS_DJANGO = True
 except ImportError:
     HAS_DJANGO = False
+else:
+    from django.db.backends.base import schema
+    from django.db.models import indexes
+    from django.db.backends.utils import names_digest
 
 
 if HAS_DJANGO is True:
-    # This line exists to make sure we don't regress on FIPS support if we
-    # upgrade Django; if you're upgrading Django and see this error,
-    # update the version check below, and confirm that FIPS still works.
-    # If operating in a FIPS environment, `hashlib.md5()` will raise a `ValueError`,
-    # but will support the `usedforsecurity` keyword on RHEL and Centos systems.
-
-    # Keep an eye on https://code.djangoproject.com/ticket/28401
-    target_version = '2.2.4'
-    if django.__version__ != target_version:
-        raise RuntimeError(
-            "Django version other than {target} detected: {current}. "
-            "Overriding `names_digest` is known to work for Django {target} "
-            "and may not work in other Django versions.".format(target=target_version,
-                                                                current=django.__version__)
-        )
 
+    # See upgrade blocker note in requirements/README.md
     try:
         names_digest('foo', 'bar', 'baz', length=8)
     except ValueError:
@@ -63,6 +51,7 @@ if HAS_DJANGO is True:
             return h.hexdigest()[:length]
 
         schema.names_digest = names_digest
+        indexes.names_digest = names_digest
 
 
 def find_commands(management_dir):
@@ -86,7 +75,14 @@ def oauth2_getattribute(self, attr):
     # Custom method to override
     # oauth2_provider.settings.OAuth2ProviderSettings.__getattribute__
     from django.conf import settings
-    val = settings.OAUTH2_PROVIDER.get(attr)
+    val = None
+    if 'migrate' not in sys.argv:
+        # certain Django OAuth Toolkit migrations actually reference
+        # setting lookups for references to model classes (e.g.,
+        # oauth2_settings.REFRESH_TOKEN_MODEL)
+        # If we're doing an OAuth2 setting lookup *while running* a migration,
+        # don't do our usual "Configure Tower in Tower" database setting lookup
+        val = settings.OAUTH2_PROVIDER.get(attr)
     if val is None:
         val = object.__getattribute__(self, attr)
     return val

awx/api/conf.py

@@ -16,6 +16,7 @@ register(
     help_text=_('Number of seconds that a user is inactive before they will need to login again.'),
     category=_('Authentication'),
     category_slug='authentication',
+    unit=_('seconds'),
 )
 register(
     'SESSIONS_PER_USER',
@@ -49,6 +50,7 @@ register(
                 'in the number of seconds.'),
     category=_('Authentication'),
     category_slug='authentication',
+    unit=_('seconds'),
 )
 register(
     'ALLOW_OAUTH2_FOR_EXTERNAL_USERS',
@@ -62,3 +64,15 @@ register(
     category=_('Authentication'),
     category_slug='authentication',
 )
+register(
+    'LOGIN_REDIRECT_OVERRIDE',
+    field_class=fields.CharField,
+    allow_blank=True,
+    required=False,
+    default='',
+    label=_('Login redirect override URL'),
+    help_text=_('URL to which unauthorized users will be redirected to log in. '
+                'If blank, users will be sent to the Tower login page.'),
+    category=_('Authentication'),
+    category_slug='authentication',
+)

awx/api/filters.py

@@ -9,7 +9,7 @@ from functools import reduce
 # Django
 from django.core.exceptions import FieldError, ValidationError
 from django.db import models
-from django.db.models import Q
+from django.db.models import Q, CharField, IntegerField, BooleanField
 from django.db.models.fields import FieldDoesNotExist
 from django.db.models.fields.related import ForeignObjectRel, ManyToManyField, ForeignKey
 from django.contrib.contenttypes.models import ContentType
@@ -63,19 +63,19 @@ class TypeFilterBackend(BaseFilterBackend):
             raise ParseError(*e.args)
 
 
-def get_field_from_path(model, path):
+def get_fields_from_path(model, path):
     '''
     Given a Django ORM lookup path (possibly over multiple models)
-    Returns the last field in the line, and also the revised lookup path
+    Returns the fields in the line, and also the revised lookup path
     ex., given
         model=Organization
         path='project__timeout'
-    returns tuple of field at the end of the line as well as a corrected
-    path, for special cases we do substitutions
-        (<IntegerField for timeout>, 'project__timeout')
+    returns tuple of fields traversed as well and a corrected path,
+    for special cases we do substitutions
+        ([<IntegerField for timeout>], 'project__timeout')
     '''
     # Store of all the fields used to detect repeats
-    field_set = set([])
+    field_list = []
     new_parts = []
     for name in path.split('__'):
         if model is None:
@@ -111,13 +111,24 @@ def get_field_from_path(model, path):
                 raise PermissionDenied(_('Filtering on %s is not allowed.' % name))
             elif getattr(field, '__prevent_search__', False):
                 raise PermissionDenied(_('Filtering on %s is not allowed.' % name))
-        if field in field_set:
+        if field in field_list:
             # Field traversed twice, could create infinite JOINs, DoSing Tower
             raise ParseError(_('Loops not allowed in filters, detected on field {}.').format(field.name))
-        field_set.add(field)
+        field_list.append(field)
         model = getattr(field, 'related_model', None)
 
-    return field, '__'.join(new_parts)
+    return field_list, '__'.join(new_parts)
+
+
+def get_field_from_path(model, path):
+    '''
+    Given a Django ORM lookup path (possibly over multiple models)
+    Returns the last field in the line, and the revised lookup path
+    ex.
+        (<IntegerField for timeout>, 'project__timeout')
+    '''
+    field_list, new_path = get_fields_from_path(model, path)
+    return (field_list[-1], new_path)
 
 
 class FieldLookupBackend(BaseFilterBackend):
@@ -133,7 +144,11 @@ class FieldLookupBackend(BaseFilterBackend):
                          'regex', 'iregex', 'gt', 'gte', 'lt', 'lte', 'in',
                          'isnull', 'search')
 
-    def get_field_from_lookup(self, model, lookup):
+    # A list of fields that we know can be filtered on without the possiblity
+    # of introducing duplicates
+    NO_DUPLICATES_ALLOW_LIST = (CharField, IntegerField, BooleanField)
+
+    def get_fields_from_lookup(self, model, lookup):
 
         if '__' in lookup and lookup.rsplit('__', 1)[-1] in self.SUPPORTED_LOOKUPS:
             path, suffix = lookup.rsplit('__', 1)
@@ -147,11 +162,16 @@ class FieldLookupBackend(BaseFilterBackend):
         # FIXME: Could build up a list of models used across relationships, use
         # those lookups combined with request.user.get_queryset(Model) to make
         # sure user cannot query using objects he could not view.
-        field, new_path = get_field_from_path(model, path)
+        field_list, new_path = get_fields_from_path(model, path)
 
         new_lookup = new_path
         new_lookup = '__'.join([new_path, suffix])
-        return field, new_lookup
+        return field_list, new_lookup
+
+    def get_field_from_lookup(self, model, lookup):
+        '''Method to match return type of single field, if needed.'''
+        field_list, new_lookup = self.get_fields_from_lookup(model, lookup)
+        return (field_list[-1], new_lookup)
 
     def to_python_related(self, value):
         value = force_text(value)
@@ -182,7 +202,10 @@ class FieldLookupBackend(BaseFilterBackend):
         except UnicodeEncodeError:
             raise ValueError("%r is not an allowed field name. Must be ascii encodable." % lookup)
 
-        field, new_lookup = self.get_field_from_lookup(model, lookup)
+        field_list, new_lookup = self.get_fields_from_lookup(model, lookup)
+        field = field_list[-1]
+
+        needs_distinct = (not all(isinstance(f, self.NO_DUPLICATES_ALLOW_LIST) for f in field_list))
 
         # Type names are stored without underscores internally, but are presented and
         # and serialized over the API containing underscores so we remove `_`
@@ -211,10 +234,10 @@ class FieldLookupBackend(BaseFilterBackend):
             for rm_field in related_model._meta.fields:
                 if rm_field.name in ('username', 'first_name', 'last_name', 'email', 'name', 'description', 'playbook'):
                     new_lookups.append('{}__{}__icontains'.format(new_lookup[:-8], rm_field.name))
-            return value, new_lookups
+            return value, new_lookups, needs_distinct
         else:
             value = self.value_to_python_for_field(field, value)
-        return value, new_lookup
+        return value, new_lookup, needs_distinct
 
     def filter_queryset(self, request, queryset, view):
         try:
@@ -225,6 +248,7 @@ class FieldLookupBackend(BaseFilterBackend):
             chain_filters = []
             role_filters = []
             search_filters = {}
+            needs_distinct = False
             # Can only have two values: 'AND', 'OR'
             # If 'AND' is used, an iterm must satisfy all condition to show up in the results.
             # If 'OR' is used, an item just need to satisfy one condition to appear in results.
@@ -233,6 +257,11 @@ class FieldLookupBackend(BaseFilterBackend):
                 if key in self.RESERVED_NAMES:
                     continue
 
+                # HACK: make `created` available via API for the Django User ORM model
+                # so it keep compatiblity with other objects which exposes the `created` attr.
+                if queryset.model._meta.object_name == 'User' and key.startswith('created'):
+                    key = key.replace('created', 'date_joined')
+
                 # HACK: Make job event filtering by host name mostly work even
                 # when not capturing job event hosts M2M.
                 if queryset.model._meta.object_name == 'JobEvent' and key.startswith('hosts__name'):
@@ -256,9 +285,12 @@ class FieldLookupBackend(BaseFilterBackend):
                         search_filter_relation = 'AND'
                         values = reduce(lambda list1, list2: list1 + list2, [i.split(',') for i in values])
                     for value in values:
-                        search_value, new_keys = self.value_to_python(queryset.model, key, force_text(value))
+                        search_value, new_keys, _ = self.value_to_python(queryset.model, key, force_text(value))
                         assert isinstance(new_keys, list)
                         search_filters[search_value] = new_keys
+                    # by definition, search *only* joins across relations,
+                    # so it _always_ needs a .distinct()
+                    needs_distinct = True
                     continue
 
                 # Custom chain__ and or__ filters, mutually exclusive (both can
@@ -282,7 +314,9 @@ class FieldLookupBackend(BaseFilterBackend):
                 for value in values:
                     if q_int:
                         value = int(value)
-                    value, new_key = self.value_to_python(queryset.model, key, value)
+                    value, new_key, distinct = self.value_to_python(queryset.model, key, value)
+                    if distinct:
+                        needs_distinct = True
                     if q_chain:
                         chain_filters.append((q_not, new_key, value))
                     elif q_or:
@@ -332,7 +366,9 @@ class FieldLookupBackend(BaseFilterBackend):
                     else:
                         q = Q(**{k:v})
                     queryset = queryset.filter(q)
-                queryset = queryset.filter(*args).distinct()
+                queryset = queryset.filter(*args)
+                if needs_distinct:
+                    queryset = queryset.distinct()
             return queryset
         except (FieldError, FieldDoesNotExist, ValueError, TypeError) as e:
             raise ParseError(e.args[0])

awx/api/generics.py

@@ -5,10 +5,12 @@
 import inspect
 import logging
 import time
+import uuid
 import urllib.parse
 
 # Django
 from django.conf import settings
+from django.core.cache import cache
 from django.db import connection
 from django.db.models.fields import FieldDoesNotExist
 from django.db.models.fields.related import OneToOneRel
@@ -43,9 +45,11 @@ from awx.main.utils import (
     get_search_fields,
     getattrd,
     get_object_or_400,
-    decrypt_field
+    decrypt_field,
+    get_awx_version,
 )
 from awx.main.utils.db import get_all_field_names
+from awx.main.views import ApiErrorView
 from awx.api.serializers import ResourceAccessListElementSerializer, CopySerializer, UserSerializer
 from awx.api.versioning import URLPathVersioning
 from awx.api.metadata import SublistAttachDetatchMetadata, Metadata
@@ -154,11 +158,11 @@ class APIView(views.APIView):
             self.queries_before = len(connection.queries)
 
         # If there are any custom headers in REMOTE_HOST_HEADERS, make sure
-        # they respect the proxy whitelist
+        # they respect the allowed proxy list
         if all([
-            settings.PROXY_IP_WHITELIST,
-            request.environ.get('REMOTE_ADDR') not in settings.PROXY_IP_WHITELIST,
-            request.environ.get('REMOTE_HOST') not in settings.PROXY_IP_WHITELIST
+            settings.PROXY_IP_ALLOWED_LIST,
+            request.environ.get('REMOTE_ADDR') not in settings.PROXY_IP_ALLOWED_LIST,
+            request.environ.get('REMOTE_HOST') not in settings.PROXY_IP_ALLOWED_LIST
         ]):
             for custom_header in settings.REMOTE_HOST_HEADERS:
                 if custom_header.startswith('HTTP_'):
@@ -183,6 +187,30 @@ class APIView(views.APIView):
         '''
         Log warning for 400 requests.  Add header with elapsed time.
         '''
+        from awx.main.utils import get_licenser
+        from awx.main.utils.licensing import OpenLicense
+        #
+        # If the URL was rewritten, and we get a 404, we should entirely
+        # replace the view in the request context with an ApiErrorView()
+        # Without this change, there will be subtle differences in the BrowseableAPIRenderer
+        #
+        # These differences could provide contextual clues which would allow
+        # anonymous users to determine if usernames were valid or not
+        # (e.g., if an anonymous user visited `/api/v2/users/valid/`, and got a 404,
+        # but also saw that the page heading said "User Detail", they might notice
+        # that's a difference in behavior from a request to `/api/v2/users/not-valid/`, which
+        # would show a page header of "Not Found").  Changing the view here
+        # guarantees that the rendered response will look exactly like the response
+        # when you visit a URL that has no matching URL paths in `awx.api.urls`.
+        #
+        if response.status_code == 404 and 'awx.named_url_rewritten' in request.environ:
+            self.headers.pop('Allow', None)
+            response = super(APIView, self).finalize_response(request, response, *args, **kwargs)
+            view = ApiErrorView()
+            setattr(view, 'request', request)
+            response.renderer_context['view'] = view
+            return response
+
         if response.status_code >= 400:
             status_msg = "status %s received by user %s attempting to access %s from %s" % \
                          (response.status_code, request.user, request.path, request.META.get('REMOTE_ADDR', None))
@@ -192,9 +220,12 @@ class APIView(views.APIView):
                 response.data['detail'] += ' To establish a login session, visit /api/login/.'
                 logger.info(status_msg)
             else:
-                logger.warn(status_msg)
+                logger.warning(status_msg)
         response = super(APIView, self).finalize_response(request, response, *args, **kwargs)
         time_started = getattr(self, 'time_started', None)
+        response['X-API-Product-Version'] = get_awx_version()
+        response['X-API-Product-Name'] = 'AWX' if isinstance(get_licenser(), OpenLicense) else 'Red Hat Ansible Tower'
+        
         response['X-API-Node'] = settings.CLUSTER_HOST_ID
         if time_started:
             time_elapsed = time.time() - self.time_started
@@ -548,6 +579,15 @@ class SubListCreateAPIView(SubListAPIView, ListCreateAPIView):
         })
         return d
 
+    def get_queryset(self):
+        if hasattr(self, 'parent_key'):
+            # Prefer this filtering because ForeignKey allows us more assumptions
+            parent = self.get_parent_object()
+            self.check_parent_access(parent)
+            qs = self.request.user.get_queryset(self.model)
+            return qs.filter(**{self.parent_key: parent})
+        return super(SubListCreateAPIView, self).get_queryset()
+
     def create(self, request, *args, **kwargs):
         # If the object ID was not specified, it probably doesn't exist in the
         # DB yet. We want to see if we can create it.  The URL may choose to
@@ -821,7 +861,7 @@ class CopyAPIView(GenericAPIView):
 
     @staticmethod
     def _decrypt_model_field_if_needed(obj, field_name, field_val):
-        if field_name in getattr(type(obj), 'REENCRYPTION_BLACKLIST_AT_COPY', []):
+        if field_name in getattr(type(obj), 'REENCRYPTION_BLOCKLIST_AT_COPY', []):
             return field_val
         if isinstance(obj, Credential) and field_name == 'inputs':
             for secret in obj.credential_type.secret_fields:
@@ -867,7 +907,7 @@ class CopyAPIView(GenericAPIView):
                 field_val = getattr(obj, field.name)
             except AttributeError:
                 continue
-            # Adjust copy blacklist fields here.
+            # Adjust copy blocked fields here.
             if field.name in fields_to_discard or field.name in [
                 'id', 'pk', 'polymorphic_ctype', 'unifiedjobtemplate_ptr', 'created_by', 'modified_by'
             ] or field.name.endswith('_role'):
@@ -964,6 +1004,11 @@ class CopyAPIView(GenericAPIView):
         if hasattr(new_obj, 'admin_role') and request.user not in new_obj.admin_role.members.all():
             new_obj.admin_role.members.add(request.user)
         if sub_objs:
+            # store the copied object dict into cache, because it's
+            # often too large for postgres' notification bus
+            # (which has a default maximum message size of 8k)
+            key = 'deep-copy-{}'.format(str(uuid.uuid4()))
+            cache.set(key, sub_objs, timeout=3600)
             permission_check_func = None
             if hasattr(type(self), 'deep_copy_permission_check_func'):
                 permission_check_func = (
@@ -971,7 +1016,7 @@ class CopyAPIView(GenericAPIView):
                 )
             trigger_delayed_deep_copy(
                 self.model.__module__, self.model.__name__,
-                obj.pk, new_obj.pk, request.user.pk, sub_objs,
+                obj.pk, new_obj.pk, request.user.pk, key,
                 permission_check_func=permission_check_func
             )
         serializer = self._get_copy_return_serializer(new_obj)

awx/api/metadata.py

@@ -2,6 +2,7 @@
 # All Rights Reserved.
 
 from collections import OrderedDict
+from uuid import UUID
 
 # Django
 from django.core.exceptions import PermissionDenied
@@ -20,9 +21,10 @@ from rest_framework.fields import JSONField as DRFJSONField
 from rest_framework.request import clone_request
 
 # AWX
+from awx.api.fields import ChoiceNullField
 from awx.main.fields import JSONField, ImplicitRoleField
-from awx.main.models import InventorySource, NotificationTemplate
-from awx.main.scheduler.kubernetes import PodManager
+from awx.main.models import NotificationTemplate
+from awx.main.tasks import AWXReceptorJob
 
 
 class Metadata(metadata.SimpleMetadata):
@@ -37,7 +39,7 @@ class Metadata(metadata.SimpleMetadata):
             'min_length', 'max_length',
             'min_value', 'max_value',
             'category', 'category_slug',
-            'defined_in_file'
+            'defined_in_file', 'unit',
         ]
 
         for attr in text_attrs:
@@ -59,7 +61,8 @@ class Metadata(metadata.SimpleMetadata):
                 'type': _('Data type for this {}.'),
                 'url': _('URL for this {}.'),
                 'related': _('Data structure with URLs of related resources.'),
-                'summary_fields': _('Data structure with name/description for related resources.'),
+                'summary_fields': _('Data structure with name/description for related resources.  '
+                                    'The output for some objects may be limited for performance reasons.'),
                 'created': _('Timestamp when this {} was created.'),
                 'modified': _('Timestamp when this {} was last modified.'),
             }
@@ -84,6 +87,8 @@ class Metadata(metadata.SimpleMetadata):
         # FIXME: Still isn't showing all default values?
         try:
             default = field.get_default()
+            if type(default) is UUID:
+                default = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
             if field.field_name == 'TOWER_URL_BASE' and default == 'https://towerhost':
                 default = '{}://{}'.format(self.request.scheme, self.request.get_host())
             field_info['default'] = default
@@ -96,25 +101,20 @@ class Metadata(metadata.SimpleMetadata):
             field_info['children'] = self.get_serializer_info(field)
 
         if not isinstance(field, (RelatedField, ManyRelatedField)) and hasattr(field, 'choices'):
-            field_info['choices'] = [(choice_value, choice_name) for choice_value, choice_name in field.choices.items()]
+            choices = [
+                (choice_value, choice_name) for choice_value, choice_name in field.choices.items()
+            ]
+            if not any(choice in ('', None) for choice, _ in choices):
+                if field.allow_blank:
+                    choices = [("", "---------")] + choices
+                if field.allow_null and not isinstance(field, ChoiceNullField):
+                    choices = [(None, "---------")] + choices
+            field_info['choices'] = choices
 
         # Indicate if a field is write-only.
         if getattr(field, 'write_only', False):
             field_info['write_only'] = True
 
-        # Special handling of inventory source_region choices that vary based on
-        # selected inventory source.
-        if field.field_name == 'source_regions':
-            for cp in ('azure_rm', 'ec2', 'gce'):
-                get_regions = getattr(InventorySource, 'get_%s_region_choices' % cp)
-                field_info['%s_region_choices' % cp] = get_regions()
-
-        # Special handling of group_by choices for EC2.
-        if field.field_name == 'group_by':
-            for cp in ('ec2',):
-                get_group_by_choices = getattr(InventorySource, 'get_%s_group_by_choices' % cp)
-                field_info['%s_group_by_choices' % cp] = get_group_by_choices()
-
         # Special handling of notification configuration where the required properties
         # are conditional on the type selected.
         if field.field_name == 'notification_configuration':
@@ -209,7 +209,7 @@ class Metadata(metadata.SimpleMetadata):
                     continue
 
                 if field == "pod_spec_override":
-                    meta['default'] = PodManager().pod_definition
+                    meta['default'] = AWXReceptorJob().pod_definition
 
                 # Add type choices if available from the serializer.
                 if field == 'type' and hasattr(serializer, 'get_type_choices'):

awx/api/renderers.py

@@ -7,6 +7,24 @@ from prometheus_client.parser import text_string_to_metric_families
 # Django REST Framework
 from rest_framework import renderers
 from rest_framework.request import override_method
+from rest_framework.utils import encoders
+
+
+class SurrogateEncoder(encoders.JSONEncoder):
+
+    def encode(self, obj):
+        ret = super(SurrogateEncoder, self).encode(obj)
+        try:
+            ret.encode()
+        except UnicodeEncodeError as e:
+            if 'surrogates not allowed' in e.reason:
+                ret = ret.encode('utf-8', 'replace').decode()
+        return ret
+
+
+class DefaultJSONRenderer(renderers.JSONRenderer):
+
+    encoder_class = SurrogateEncoder
 
 
 class BrowsableAPIRenderer(renderers.BrowsableAPIRenderer):

awx/api/templates/api/_schedule_detail.md

@@ -4,7 +4,6 @@ The following lists the expected format and details of our rrules:
 * DTSTART is expected to be in UTC
 * INTERVAL is required
 * SECONDLY is not supported
-* TZID is not supported
 * RRULE must precede the rule statements
 * BYDAY is supported but not BYDAY with a numerical prefix
 * BYYEARDAY and BYWEEKNO are not supported

awx/api/templates/api/dashboard_jobs_graph_view.md

@@ -8,7 +8,7 @@ The `period` of the data can be adjusted with:
 
     ?period=month
 
-Where `month` can be replaced with `week`, or `day`.  `month` is the default.
+Where `month` can be replaced with `week`, `two_weeks`, or `day`.  `month` is the default.
 
 The type of job can be filtered with:
 

awx/api/templates/api/inventory_source_cancel.md

@@ -1,7 +1,7 @@
 # Cancel Inventory Update
 
 Make a GET request to this resource to determine if the inventory update can be
-cancelled.  The response will include the following field:
+canceled.  The response will include the following field:
 
 * `can_cancel`: Indicates whether this update can be canceled (boolean,
   read-only)

awx/api/templates/api/job_cancel.md

@@ -1,7 +1,7 @@
 {% ifmeth GET %}
-# Determine if a Job can be cancelled
+# Determine if a Job can be canceled
 
-Make a GET request to this resource to determine if the job can be cancelled.
+Make a GET request to this resource to determine if the job can be canceled.
 The response will include the following field:
 
 * `can_cancel`: Indicates whether this job can be canceled (boolean, read-only)

awx/api/templates/api/project_update_cancel.md

@@ -1,7 +1,7 @@
 # Cancel Project Update
 
 Make a GET request to this resource to determine if the project update can be
-cancelled.  The response will include the following field:
+canceled.  The response will include the following field:
 
 * `can_cancel`: Indicates whether this update can be canceled (boolean,
   read-only)

awx/api/templates/api/setting_logging_test.md

@@ -1 +1,2 @@
 # Test Logging Configuration
+

awx/api/urls/execution_environments.py

@@ -0,0 +1,20 @@
+from django.conf.urls import url
+
+from awx.api.views import (
+    ExecutionEnvironmentList,
+    ExecutionEnvironmentDetail,
+    ExecutionEnvironmentJobTemplateList,
+    ExecutionEnvironmentCopy,
+    ExecutionEnvironmentActivityStreamList,
+)
+
+
+urls = [
+    url(r'^$', ExecutionEnvironmentList.as_view(), name='execution_environment_list'),
+    url(r'^(?P<pk>[0-9]+)/$', ExecutionEnvironmentDetail.as_view(), name='execution_environment_detail'),
+    url(r'^(?P<pk>[0-9]+)/unified_job_templates/$', ExecutionEnvironmentJobTemplateList.as_view(), name='execution_environment_job_template_list'),
+    url(r'^(?P<pk>[0-9]+)/copy/$', ExecutionEnvironmentCopy.as_view(), name='execution_environment_copy'),
+    url(r'^(?P<pk>[0-9]+)/activity_stream/$', ExecutionEnvironmentActivityStreamList.as_view(), name='execution_environment_activity_stream_list'),
+]
+
+__all__ = ['urls']

awx/api/urls/oauth2_root.py

@@ -1,11 +1,15 @@
 # Copyright (c) 2017 Ansible, Inc.
 # All Rights Reserved.
+from datetime import timedelta
 
+from django.utils.timezone import now
+from django.conf import settings
 from django.conf.urls import url
 
 from oauthlib import oauth2
 from oauth2_provider import views
 
+from awx.main.models import RefreshToken
 from awx.api.views import (
     ApiOAuthAuthorizationRootView,
 )
@@ -14,6 +18,21 @@ from awx.api.views import (
 class TokenView(views.TokenView):
 
     def create_token_response(self, request):
+        # Django OAuth2 Toolkit has a bug whereby refresh tokens are *never*
+        # properly expired (ugh):
+        #
+        # https://github.com/jazzband/django-oauth-toolkit/issues/746
+        #
+        # This code detects and auto-expires them on refresh grant
+        # requests.
+        if request.POST.get('grant_type') == 'refresh_token' and 'refresh_token' in request.POST:
+            refresh_token = RefreshToken.objects.filter(
+                token=request.POST['refresh_token']
+            ).first()
+            if refresh_token:
+                expire_seconds = settings.OAUTH2_PROVIDER.get('REFRESH_TOKEN_EXPIRE_SECONDS', 0)
+                if refresh_token.created + timedelta(seconds=expire_seconds) < now():
+                    return request.build_absolute_uri(), {}, 'The refresh token has expired.', '403'
         try:
             return super(TokenView, self).create_token_response(request)
         except oauth2.AccessDeniedError as e:

awx/api/urls/organization.py

@@ -9,7 +9,9 @@ from awx.api.views import (
     OrganizationUsersList,
     OrganizationAdminsList,
     OrganizationInventoriesList,
+    OrganizationExecutionEnvironmentsList,
     OrganizationProjectsList,
+    OrganizationJobTemplatesList,
     OrganizationWorkflowJobTemplatesList,
     OrganizationTeamsList,
     OrganizationCredentialList,
@@ -20,6 +22,7 @@ from awx.api.views import (
     OrganizationNotificationTemplatesSuccessList,
     OrganizationNotificationTemplatesApprovalList,
     OrganizationInstanceGroupsList,
+    OrganizationGalaxyCredentialsList,
     OrganizationObjectRolesList,
     OrganizationAccessList,
     OrganizationApplicationList,
@@ -32,7 +35,9 @@ urls = [
     url(r'^(?P<pk>[0-9]+)/users/$', OrganizationUsersList.as_view(), name='organization_users_list'),
     url(r'^(?P<pk>[0-9]+)/admins/$', OrganizationAdminsList.as_view(), name='organization_admins_list'),
     url(r'^(?P<pk>[0-9]+)/inventories/$', OrganizationInventoriesList.as_view(), name='organization_inventories_list'),
+    url(r'^(?P<pk>[0-9]+)/execution_environments/$', OrganizationExecutionEnvironmentsList.as_view(), name='organization_execution_environments_list'),
     url(r'^(?P<pk>[0-9]+)/projects/$', OrganizationProjectsList.as_view(), name='organization_projects_list'),
+    url(r'^(?P<pk>[0-9]+)/job_templates/$', OrganizationJobTemplatesList.as_view(), name='organization_job_templates_list'),
     url(r'^(?P<pk>[0-9]+)/workflow_job_templates/$', OrganizationWorkflowJobTemplatesList.as_view(), name='organization_workflow_job_templates_list'),
     url(r'^(?P<pk>[0-9]+)/teams/$', OrganizationTeamsList.as_view(), name='organization_teams_list'),
     url(r'^(?P<pk>[0-9]+)/credentials/$', OrganizationCredentialList.as_view(), name='organization_credential_list'),
@@ -47,6 +52,7 @@ urls = [
     url(r'^(?P<pk>[0-9]+)/notification_templates_approvals/$', OrganizationNotificationTemplatesApprovalList.as_view(),
         name='organization_notification_templates_approvals_list'),
     url(r'^(?P<pk>[0-9]+)/instance_groups/$', OrganizationInstanceGroupsList.as_view(), name='organization_instance_groups_list'),
+    url(r'^(?P<pk>[0-9]+)/galaxy_credentials/$', OrganizationGalaxyCredentialsList.as_view(), name='organization_galaxy_credentials_list'),
     url(r'^(?P<pk>[0-9]+)/object_roles/$', OrganizationObjectRolesList.as_view(), name='organization_object_roles_list'),
     url(r'^(?P<pk>[0-9]+)/access_list/$', OrganizationAccessList.as_view(), name='organization_access_list'),
     url(r'^(?P<pk>[0-9]+)/applications/$', OrganizationApplicationList.as_view(), name='organization_applications_list'),

awx/api/urls/urls.py

@@ -15,6 +15,7 @@ from awx.api.views import (
     ApiV2PingView,
     ApiV2ConfigView,
     ApiV2SubscriptionView,
+    ApiV2AttachView,
     AuthView,
     UserMeList,
     DashboardView,
@@ -23,9 +24,7 @@ from awx.api.views import (
     UnifiedJobList,
     HostAnsibleFactsDetail,
     JobCredentialsList,
-    JobExtraCredentialsList,
     JobTemplateCredentialsList,
-    JobTemplateExtraCredentialsList,
     SchedulePreview,
     ScheduleZoneInfo,
     OAuth2ApplicationList,
@@ -34,13 +33,16 @@ from awx.api.views import (
     OAuth2ApplicationDetail,
 )
 
-from awx.api.views.metrics import MetricsView
+from awx.api.views.metrics import (
+    MetricsView,
+)
 
 from .organization import urls as organization_urls
 from .user import urls as user_urls
 from .project import urls as project_urls
 from .project_update import urls as project_update_urls
 from .inventory import urls as inventory_urls
+from .execution_environments import urls as execution_environment_urls
 from .team import urls as team_urls
 from .host import urls as host_urls
 from .group import urls as group_urls
@@ -81,9 +83,7 @@ v2_urls = [
     url(r'^credential_types/', include(credential_type_urls)),
     url(r'^credential_input_sources/', include(credential_input_source_urls)),
     url(r'^hosts/(?P<pk>[0-9]+)/ansible_facts/$', HostAnsibleFactsDetail.as_view(), name='host_ansible_facts_detail'),
-    url(r'^jobs/(?P<pk>[0-9]+)/extra_credentials/$', JobExtraCredentialsList.as_view(), name='job_extra_credentials_list'),
     url(r'^jobs/(?P<pk>[0-9]+)/credentials/$', JobCredentialsList.as_view(), name='job_credentials_list'),
-    url(r'^job_templates/(?P<pk>[0-9]+)/extra_credentials/$', JobTemplateExtraCredentialsList.as_view(), name='job_template_extra_credentials_list'),
     url(r'^job_templates/(?P<pk>[0-9]+)/credentials/$', JobTemplateCredentialsList.as_view(), name='job_template_credentials_list'),
     url(r'^schedules/preview/$', SchedulePreview.as_view(), name='schedule_rrule'),
     url(r'^schedules/zoneinfo/$', ScheduleZoneInfo.as_view(), name='schedule_zoneinfo'),
@@ -96,6 +96,7 @@ v2_urls = [
     url(r'^ping/$', ApiV2PingView.as_view(), name='api_v2_ping_view'),
     url(r'^config/$', ApiV2ConfigView.as_view(), name='api_v2_config_view'),
     url(r'^config/subscriptions/$', ApiV2SubscriptionView.as_view(), name='api_v2_subscription_view'),
+    url(r'^config/attach/$', ApiV2AttachView.as_view(), name='api_v2_attach_view'),
     url(r'^auth/$', AuthView.as_view()),
     url(r'^me/$', UserMeList.as_view(), name='user_me_list'),
     url(r'^dashboard/$', DashboardView.as_view(), name='dashboard_view'),
@@ -106,6 +107,7 @@ v2_urls = [
     url(r'^schedules/', include(schedule_urls)),
     url(r'^organizations/', include(organization_urls)),
     url(r'^users/', include(user_urls)),
+    url(r'^execution_environments/', include(execution_environment_urls)),
     url(r'^projects/', include(project_urls)),
     url(r'^project_updates/', include(project_update_urls)),
     url(r'^teams/', include(team_urls)),

awx/api/views/__init__.py

@@ -12,7 +12,9 @@ import socket
 import sys
 import time
 from base64 import b64encode
-from collections import OrderedDict, Iterable
+from collections import OrderedDict
+
+from urllib3.exceptions import ConnectTimeoutError
 
 
 # Django
@@ -72,17 +74,17 @@ from awx.api.generics import (
     SubListDestroyAPIView
 )
 from awx.api.versioning import reverse
-from awx.conf.license import get_license
 from awx.main import models
 from awx.main.utils import (
     camelcase_to_underscore,
     extract_ansible_vars,
-    get_awx_version,
+    get_awx_http_client_headers,
     get_object_or_400,
     getattrd,
     get_pk_from_dict,
     schedule_task_manager,
-    ignore_inventory_computed_fields
+    ignore_inventory_computed_fields,
+    set_environ
 )
 from awx.main.utils.encryption import encrypt_value
 from awx.main.utils.filters import SmartFilter
@@ -110,7 +112,9 @@ from awx.api.views.organization import ( # noqa
     OrganizationInventoriesList,
     OrganizationUsersList,
     OrganizationAdminsList,
+    OrganizationExecutionEnvironmentsList,
     OrganizationProjectsList,
+    OrganizationJobTemplatesList,
     OrganizationWorkflowJobTemplatesList,
     OrganizationTeamsList,
     OrganizationActivityStreamList,
@@ -121,6 +125,7 @@ from awx.api.views.organization import ( # noqa
     OrganizationNotificationTemplatesSuccessList,
     OrganizationNotificationTemplatesApprovalList,
     OrganizationInstanceGroupsList,
+    OrganizationGalaxyCredentialsList,
     OrganizationAccessList,
     OrganizationObjectRolesList,
 )
@@ -149,6 +154,7 @@ from awx.api.views.root import ( # noqa
     ApiV2PingView,
     ApiV2ConfigView,
     ApiV2SubscriptionView,
+    ApiV2AttachView,
 )
 from awx.api.views.webhooks import ( # noqa
     WebhookKeyView,
@@ -170,6 +176,15 @@ def api_exception_handler(exc, context):
         exc = ParseError(exc.args[0])
     if isinstance(context['view'], UnifiedJobStdout):
         context['view'].renderer_classes = [renderers.BrowsableAPIRenderer, JSONRenderer]
+    if isinstance(exc, APIException):
+        req = context['request']._request
+        if 'awx.named_url_rewritten' in req.environ and not str(getattr(exc, 'status_code', 0)).startswith('2'):
+            # if the URL was rewritten, and it's not a 2xx level status code,
+            # revert the request.path to its original value to avoid leaking
+            # any context about the existance of resources
+            req.path = req.environ['awx.named_url_rewritten']
+            if exc.status_code == 403:
+                exc = NotFound(detail=_('Not found.'))
     return exception_handler(exc, context)
 
 
@@ -205,20 +220,15 @@ class DashboardView(APIView):
                                             'failed': ec2_inventory_failed.count()}
 
         user_groups = get_user_queryset(request.user, models.Group)
-        groups_job_failed = (
-            models.Group.objects.filter(hosts_with_active_failures__gt=0) | models.Group.objects.filter(groups_with_active_failures__gt=0)
-        ).count()
         groups_inventory_failed = models.Group.objects.filter(inventory_sources__last_job_failed=True).count()
         data['groups'] = {'url': reverse('api:group_list', request=request),
-                          'failures_url': reverse('api:group_list', request=request) + "?has_active_failures=True",
                           'total': user_groups.count(),
-                          'job_failed': groups_job_failed,
                           'inventory_failed': groups_inventory_failed}
 
         user_hosts = get_user_queryset(request.user, models.Host)
-        user_hosts_failed = user_hosts.filter(has_active_failures=True)
+        user_hosts_failed = user_hosts.filter(last_job_host_summary__failed=True)
         data['hosts'] = {'url': reverse('api:host_list', request=request),
-                         'failures_url': reverse('api:host_list', request=request) + "?has_active_failures=True",
+                         'failures_url': reverse('api:host_list', request=request) + "?last_job_host_summary__failed=True",
                          'total': user_hosts.count(),
                          'failed': user_hosts_failed.count()}
 
@@ -233,8 +243,8 @@ class DashboardView(APIView):
         git_failed_projects = git_projects.filter(last_job_failed=True)
         svn_projects = user_projects.filter(scm_type='svn')
         svn_failed_projects = svn_projects.filter(last_job_failed=True)
-        hg_projects = user_projects.filter(scm_type='hg')
-        hg_failed_projects = hg_projects.filter(last_job_failed=True)
+        archive_projects = user_projects.filter(scm_type='archive')
+        archive_failed_projects = archive_projects.filter(last_job_failed=True)
         data['scm_types'] = {}
         data['scm_types']['git'] = {'url': reverse('api:project_list', request=request) + "?scm_type=git",
                                     'label': 'Git',
@@ -246,11 +256,11 @@ class DashboardView(APIView):
                                     'failures_url': reverse('api:project_list', request=request) + "?scm_type=svn&last_job_failed=True",
                                     'total': svn_projects.count(),
                                     'failed': svn_failed_projects.count()}
-        data['scm_types']['hg'] = {'url': reverse('api:project_list', request=request) + "?scm_type=hg",
-                                   'label': 'Mercurial',
-                                   'failures_url': reverse('api:project_list', request=request) + "?scm_type=hg&last_job_failed=True",
-                                   'total': hg_projects.count(),
-                                   'failed': hg_failed_projects.count()}
+        data['scm_types']['archive'] = {'url': reverse('api:project_list', request=request) + "?scm_type=archive",
+                                        'label': 'Remote Archive',
+                                        'failures_url': reverse('api:project_list', request=request) + "?scm_type=archive&last_job_failed=True",
+                                        'total': archive_projects.count(),
+                                        'failed': archive_failed_projects.count()}
 
         user_list = get_user_queryset(request.user, models.User)
         team_list = get_user_queryset(request.user, models.Team)
@@ -301,6 +311,9 @@ class DashboardJobsGraphView(APIView):
         if period == 'month':
             end_date = start_date - dateutil.relativedelta.relativedelta(months=1)
             interval = 'days'
+        elif period == 'two_weeks':
+            end_date = start_date - dateutil.relativedelta.relativedelta(weeks=2)
+            interval = 'days'
         elif period == 'week':
             end_date = start_date - dateutil.relativedelta.relativedelta(weeks=1)
             interval = 'days'
@@ -384,7 +397,7 @@ class InstanceGroupDetail(RelatedJobsPreventDeleteMixin, RetrieveUpdateDestroyAP
     permission_classes = (InstanceGroupTowerPermission,)
 
     def update_raw_data(self, data):
-        if self.get_object().is_containerized:
+        if self.get_object().is_container_group:
             data.pop('policy_instance_percentage', None)
             data.pop('policy_instance_minimum', None)
             data.pop('policy_instance_list', None)
@@ -673,6 +686,52 @@ class TeamAccessList(ResourceAccessList):
     parent_model = models.Team
 
 
+class ExecutionEnvironmentList(ListCreateAPIView):
+
+    always_allow_superuser = False
+    model = models.ExecutionEnvironment
+    serializer_class = serializers.ExecutionEnvironmentSerializer
+    swagger_topic = "Execution Environments"
+
+
+class ExecutionEnvironmentDetail(RetrieveUpdateDestroyAPIView):
+
+    always_allow_superuser = False
+    model = models.ExecutionEnvironment
+    serializer_class = serializers.ExecutionEnvironmentSerializer
+    swagger_topic = "Execution Environments"
+
+
+class ExecutionEnvironmentJobTemplateList(SubListAPIView):
+
+    model = models.UnifiedJobTemplate
+    serializer_class = serializers.UnifiedJobTemplateSerializer
+    parent_model = models.ExecutionEnvironment
+    relationship = 'unifiedjobtemplates'
+
+
+class ExecutionEnvironmentCopy(CopyAPIView):
+    
+    model = models.ExecutionEnvironment
+    copy_return_serializer_class = serializers.ExecutionEnvironmentSerializer
+
+
+class ExecutionEnvironmentActivityStreamList(SubListAPIView):
+
+    model = models.ActivityStream
+    serializer_class = serializers.ActivityStreamSerializer
+    parent_model = models.ExecutionEnvironment
+    relationship = 'activitystream_set'
+    search_fields = ('changes',)
+
+    def get_queryset(self):
+        parent = self.get_parent_object()
+        self.check_parent_access(parent)
+
+        qs = self.request.user.get_queryset(self.model)
+        return qs.filter(execution_environment=parent)
+
+
 class ProjectList(ListCreateAPIView):
 
     model = models.Project
@@ -1096,7 +1155,7 @@ class UserRolesList(SubListAttachDetachAPIView):
 
         credential_content_type = ContentType.objects.get_for_model(models.Credential)
         if role.content_type == credential_content_type:
-            if role.content_object.organization and user not in role.content_object.organization.member_role:
+            if 'disassociate' not in request.data and role.content_object.organization and user not in role.content_object.organization.member_role:
                 data = dict(msg=_("You cannot grant credential access to a user not in the credentials' organization"))
                 return Response(data, status=status.HTTP_400_BAD_REQUEST)
 
@@ -1341,6 +1400,13 @@ class CredentialDetail(RetrieveUpdateDestroyAPIView):
     model = models.Credential
     serializer_class = serializers.CredentialSerializer
 
+    def destroy(self, request, *args, **kwargs):
+        instance = self.get_object()
+        if instance.managed_by_tower:
+            raise PermissionDenied(detail=_("Deletion not allowed for managed credentials"))
+        return super(CredentialDetail, self).destroy(request, *args, **kwargs)
+
+
 
 class CredentialActivityStreamList(SubListAPIView):
 
@@ -1386,6 +1452,7 @@ class CredentialExternalTest(SubDetailAPIView):
 
     model = models.Credential
     serializer_class = serializers.EmptySerializer
+    obj_permission_type = 'use'
 
     def post(self, request, *args, **kwargs):
         obj = self.get_object()
@@ -1400,10 +1467,18 @@ class CredentialExternalTest(SubDetailAPIView):
             obj.credential_type.plugin.backend(**backend_kwargs)
             return Response({}, status=status.HTTP_202_ACCEPTED)
         except requests.exceptions.HTTPError as exc:
-            message = 'HTTP {}\n{}'.format(exc.response.status_code, exc.response.text)
+            message = 'HTTP {}'.format(exc.response.status_code)
             return Response({'inputs': message}, status=status.HTTP_400_BAD_REQUEST)
         except Exception as exc:
-            return Response({'inputs': str(exc)}, status=status.HTTP_400_BAD_REQUEST)
+            message = exc.__class__.__name__
+            args = getattr(exc, 'args', [])
+            for a in args:
+                if isinstance(
+                    getattr(a, 'reason', None),
+                    ConnectTimeoutError
+                ):
+                    message = str(a.reason)
+            return Response({'inputs': message}, status=status.HTTP_400_BAD_REQUEST)
 
 
 class CredentialInputSourceDetail(RetrieveUpdateDestroyAPIView):
@@ -1452,10 +1527,18 @@ class CredentialTypeExternalTest(SubDetailAPIView):
             obj.plugin.backend(**backend_kwargs)
             return Response({}, status=status.HTTP_202_ACCEPTED)
         except requests.exceptions.HTTPError as exc:
-            message = 'HTTP {}\n{}'.format(exc.response.status_code, exc.response.text)
+            message = 'HTTP {}'.format(exc.response.status_code)
             return Response({'inputs': message}, status=status.HTTP_400_BAD_REQUEST)
         except Exception as exc:
-            return Response({'inputs': str(exc)}, status=status.HTTP_400_BAD_REQUEST)
+            message = exc.__class__.__name__
+            args = getattr(exc, 'args', [])
+            for a in args:
+                if isinstance(
+                    getattr(a, 'reason', None),
+                    ConnectTimeoutError
+                ):
+                    message = str(a.reason)
+            return Response({'inputs': message}, status=status.HTTP_400_BAD_REQUEST)
 
 
 class HostRelatedSearchMixin(object):
@@ -1611,7 +1694,8 @@ class HostInsights(GenericAPIView):
 
     def _call_insights_api(self, url, session, headers):
         try:
-            res = session.get(url, headers=headers, timeout=120)
+            with set_environ(**settings.AWX_TASK_ENV):
+                res = session.get(url, headers=headers, timeout=120)
         except requests.exceptions.SSLError:
             raise BadGateway(_('SSLError while trying to connect to {}').format(url))
         except requests.exceptions.Timeout:
@@ -1643,18 +1727,6 @@ class HostInsights(GenericAPIView):
 
         return session
 
-    def _get_headers(self):
-        license = get_license(show_key=False).get('license_type', 'UNLICENSED')
-        headers = {
-            'Content-Type': 'application/json',
-            'User-Agent': '{} {} ({})'.format(
-                'AWX' if license == 'open' else 'Red Hat Ansible Tower',
-                get_awx_version(),
-                license
-            )
-        }
-
-        return headers
 
     def _get_platform_info(self, host, session, headers):
         url = '{}/api/inventory/v1/hosts?insights_id={}'.format(
@@ -1721,7 +1793,7 @@ class HostInsights(GenericAPIView):
         username = cred.get_input('username', default='')
         password = cred.get_input('password', default='')
         session = self._get_session(username, password)
-        headers = self._get_headers()
+        headers = get_awx_http_client_headers()
 
         data = self._get_insights(host, session, headers)
         return Response(data, status=status.HTTP_200_OK)
@@ -2162,7 +2234,7 @@ class InventorySourceHostsList(HostRelatedSearchMixin, SubListDestroyAPIView):
                     host__inventory_sources=inv_source
                 ).delete()
                 r = super(InventorySourceHostsList, self).perform_list_destroy(instance_list)
-        update_inventory_computed_fields.delay(inv_source.inventory_id, True)
+        update_inventory_computed_fields.delay(inv_source.inventory_id)
         return r
 
 
@@ -2189,7 +2261,7 @@ class InventorySourceGroupsList(SubListDestroyAPIView):
                     group__inventory_sources=inv_source
                 ).delete()
                 r = super(InventorySourceGroupsList, self).perform_list_destroy(instance_list)
-        update_inventory_computed_fields.delay(inv_source.inventory_id, True)
+        update_inventory_computed_fields.delay(inv_source.inventory_id)
         return r
 
 
@@ -2351,70 +2423,24 @@ class JobTemplateLaunch(RetrieveAPIView):
         old field structure to launch endpoint
         TODO: delete this method with future API version changes
         '''
-        ignored_fields = {}
         modern_data = data.copy()
 
         id_fd = '{}_id'.format('inventory')
         if 'inventory' not in modern_data and id_fd in modern_data:
             modern_data['inventory'] = modern_data[id_fd]
 
-        # Automatically convert legacy launch credential arguments into a list of `.credentials`
-        if 'credentials' in modern_data and 'extra_credentials' in modern_data:
-            raise ParseError({"error": _(
-                "'credentials' cannot be used in combination with 'extra_credentials'."
-            )})
-
-        if 'extra_credentials' in modern_data:
-            # make a list of the current credentials
-            existing_credentials = obj.credentials.all()
-            template_credentials = list(existing_credentials)  # save copy of existing
-            new_credentials = []
-            if 'extra_credentials' in modern_data:
-                existing_credentials = [
-                    cred for cred in existing_credentials
-                    if cred.credential_type.kind not in ('cloud', 'net')
-                ]
-                prompted_value = modern_data.pop('extra_credentials')
-
-                # validate type, since these are not covered by a serializer
-                if not isinstance(prompted_value, Iterable):
-                    msg = _(
-                        "Incorrect type. Expected a list received {}."
-                    ).format(prompted_value.__class__.__name__)
-                    raise ParseError({'extra_credentials': [msg], 'credentials': [msg]})
-
-                # add the deprecated credential specified in the request
-                if not isinstance(prompted_value, Iterable) or isinstance(prompted_value, str):
-                    prompted_value = [prompted_value]
-
-                # If user gave extra_credentials, special case to use exactly
-                # the given list without merging with JT credentials
-                if prompted_value:
-                    obj._deprecated_credential_launch = True  # signal to not merge credentials
-                new_credentials.extend(prompted_value)
-
-            # combine the list of "new" and the filtered list of "old"
-            new_credentials.extend([cred.pk for cred in existing_credentials])
-            if new_credentials:
-                # If provided list doesn't contain the pre-existing credentials
-                # defined on the template, add them back here
-                for cred_obj in template_credentials:
-                    if cred_obj.pk not in new_credentials:
-                        new_credentials.append(cred_obj.pk)
-                modern_data['credentials'] = new_credentials
-
         # credential passwords were historically provided as top-level attributes
         if 'credential_passwords' not in modern_data:
             modern_data['credential_passwords'] = data.copy()
 
-        return (modern_data, ignored_fields)
+        return modern_data
 
 
     def post(self, request, *args, **kwargs):
         obj = self.get_object()
 
         try:
-            modern_data, ignored_fields = self.modernize_launch_payload(
+            modern_data = self.modernize_launch_payload(
                 data=request.data, obj=obj
             )
         except ParseError as exc:
@@ -2424,8 +2450,6 @@ class JobTemplateLaunch(RetrieveAPIView):
         if not serializer.is_valid():
             return Response(serializer.errors, status=status.HTTP_400_BAD_REQUEST)
 
-        ignored_fields.update(serializer._ignored_fields)
-
         if not request.user.can_access(models.JobLaunchConfig, 'add', serializer.validated_data, template=obj):
             raise PermissionDenied()
 
@@ -2441,11 +2465,11 @@ class JobTemplateLaunch(RetrieveAPIView):
             data = OrderedDict()
             if isinstance(new_job, models.WorkflowJob):
                 data['workflow_job'] = new_job.id
-                data['ignored_fields'] = self.sanitize_for_response(ignored_fields)
+                data['ignored_fields'] = self.sanitize_for_response(serializer._ignored_fields)
                 data.update(serializers.WorkflowJobSerializer(new_job, context=self.get_serializer_context()).to_representation(new_job))
             else:
                 data['job'] = new_job.id
-                data['ignored_fields'] = self.sanitize_for_response(ignored_fields)
+                data['ignored_fields'] = self.sanitize_for_response(serializer._ignored_fields)
                 data.update(serializers.JobSerializer(new_job, context=self.get_serializer_context()).to_representation(new_job))
             headers = {'Location': new_job.get_absolute_url(request)}
             return Response(data, status=status.HTTP_201_CREATED, headers=headers)
@@ -2561,7 +2585,7 @@ class JobTemplateSurveySpec(GenericAPIView):
                 if not isinstance(val, allow_types):
                     return Response(dict(error=_("'{field_name}' in survey question {idx} expected to be {type_label}.").format(
                         field_name=field_name, type_label=type_label, **context
-                    )))
+                    )), status=status.HTTP_400_BAD_REQUEST)
             if survey_item['variable'] in variable_set:
                 return Response(dict(error=_("'variable' '%(item)s' duplicated in survey question %(survey)s.") % {
                     'item': survey_item['variable'], 'survey': str(idx)}), status=status.HTTP_400_BAD_REQUEST)
@@ -2576,7 +2600,7 @@ class JobTemplateSurveySpec(GenericAPIView):
                     "'{survey_item[type]}' in survey question {idx} is not one of '{allowed_types}' allowed question types."
                 ).format(
                     allowed_types=', '.join(JobTemplateSurveySpec.ALLOWED_TYPES.keys()), **context
-                )))
+                )), status=status.HTTP_400_BAD_REQUEST)
             if 'default' in survey_item and survey_item['default'] != '':
                 if not isinstance(survey_item['default'], JobTemplateSurveySpec.ALLOWED_TYPES[qtype]):
                     type_label = 'string'
@@ -2594,7 +2618,7 @@ class JobTemplateSurveySpec(GenericAPIView):
                     if survey_item[key] is not None and (not isinstance(survey_item[key], int)):
                         return Response(dict(error=_(
                             "The {min_or_max} limit in survey question {idx} expected to be integer."
-                        ).format(min_or_max=key, **context)))
+                        ).format(min_or_max=key, **context)), status=status.HTTP_400_BAD_REQUEST)
             # if it's a multiselect or multiple choice, it must have coices listed
             # choices and defualts must come in as strings seperated by /n characters.
             if qtype == 'multiselect' or qtype == 'multiplechoice':
@@ -2604,7 +2628,7 @@ class JobTemplateSurveySpec(GenericAPIView):
                 else:
                     return Response(dict(error=_(
                         "Survey question {idx} of type {survey_item[type]} must specify choices.".format(**context)
-                    )))
+                    )), status=status.HTTP_400_BAD_REQUEST)
                 # If there is a default string split it out removing extra /n characters.
                 # Note: There can still be extra newline characters added in the API, these are sanitized out using .strip()
                 if 'default' in survey_item:
@@ -2618,11 +2642,11 @@ class JobTemplateSurveySpec(GenericAPIView):
                         if len(list_of_defaults) > 1:
                             return Response(dict(error=_(
                                 "Multiple Choice (Single Select) can only have one default value.".format(**context)
-                            )))
+                            )), status=status.HTTP_400_BAD_REQUEST)
                     if any(item not in survey_item['choices'] for item in list_of_defaults):
                         return Response(dict(error=_(
                             "Default choice must be answered from the choices listed.".format(**context)
-                        )))
+                        )), status=status.HTTP_400_BAD_REQUEST)
 
             # Process encryption substitution
             if ("default" in survey_item and isinstance(survey_item['default'], str) and
@@ -2719,28 +2743,12 @@ class JobTemplateCredentialsList(SubListCreateAttachDetachAPIView):
             return {"error": _("Cannot assign multiple {credential_type} credentials.").format(
                 credential_type=sub.unique_hash(display=True))}
         kind = sub.credential_type.kind
-        if kind not in ('ssh', 'vault', 'cloud', 'net'):
+        if kind not in ('ssh', 'vault', 'cloud', 'net', 'kubernetes'):
             return {'error': _('Cannot assign a Credential of kind `{}`.').format(kind)}
 
         return super(JobTemplateCredentialsList, self).is_valid_relation(parent, sub, created)
 
 
-class JobTemplateExtraCredentialsList(JobTemplateCredentialsList):
-
-    deprecated = True
-
-    def get_queryset(self):
-        sublist_qs = super(JobTemplateExtraCredentialsList, self).get_queryset()
-        sublist_qs = sublist_qs.filter(credential_type__kind__in=['cloud', 'net'])
-        return sublist_qs
-
-    def is_valid_relation(self, parent, sub, created=False):
-        valid = super(JobTemplateExtraCredentialsList, self).is_valid_relation(parent, sub, created)
-        if sub.credential_type.kind not in ('cloud', 'net'):
-            return {'error': _('Extra credentials must be network or cloud.')}
-        return valid
-
-
 class JobTemplateLabelList(DeleteLastUnattachLabelMixin, SubListCreateAttachDetachAPIView):
 
     model = models.Label
@@ -3079,7 +3087,7 @@ class WorkflowJobTemplateNodeCreateApproval(RetrieveAPIView):
             approval_template,
             context=self.get_serializer_context()
         ).data
-        return Response(data, status=status.HTTP_200_OK)
+        return Response(data, status=status.HTTP_201_CREATED)
 
     def check_permissions(self, request):
         obj = self.get_object().workflow_job_template
@@ -3280,7 +3288,7 @@ class WorkflowJobRelaunch(GenericAPIView):
             jt = obj.job_template
             if not jt:
                 raise ParseError(_('Cannot relaunch slice workflow job orphaned from job template.'))
-            elif not jt.inventory or min(jt.inventory.hosts.count(), jt.job_slice_count) != obj.workflow_nodes.count():
+            elif not obj.inventory or min(obj.inventory.hosts.count(), jt.job_slice_count) != obj.workflow_nodes.count():
                 raise ParseError(_('Cannot relaunch sliced workflow job after slice count has changed.'))
         new_workflow_job = obj.create_relaunch_workflow_job()
         new_workflow_job.signal_start()
@@ -3557,16 +3565,6 @@ class JobCredentialsList(SubListAPIView):
     relationship = 'credentials'
 
 
-class JobExtraCredentialsList(JobCredentialsList):
-
-    deprecated = True
-
-    def get_queryset(self):
-        sublist_qs = super(JobExtraCredentialsList, self).get_queryset()
-        sublist_qs = sublist_qs.filter(credential_type__kind__in=['cloud', 'net'])
-        return sublist_qs
-
-
 class JobLabelList(SubListAPIView):
 
     model = models.Label
@@ -3831,6 +3829,12 @@ class JobEventHostsList(HostRelatedSearchMixin, SubListAPIView):
     relationship = 'hosts'
     name = _('Job Event Hosts List')
 
+    def get_queryset(self):
+        parent_event = self.get_parent_object()
+        self.check_parent_access(parent_event)
+        qs = self.request.user.get_queryset(self.model).filter(job_events_as_primary_host=parent_event)
+        return qs
+
 
 class BaseJobEventsList(NoTruncateMixin, SubListAPIView):
 
@@ -3853,8 +3857,7 @@ class HostJobEventsList(BaseJobEventsList):
     def get_queryset(self):
         parent_obj = self.get_parent_object()
         self.check_parent_access(parent_obj)
-        qs = self.request.user.get_queryset(self.model).filter(
-            Q(host=parent_obj) | Q(hosts=parent_obj)).distinct()
+        qs = self.request.user.get_queryset(self.model).filter(host=parent_obj)
         return qs
 
 
@@ -3870,9 +3873,7 @@ class JobJobEventsList(BaseJobEventsList):
     def get_queryset(self):
         job = self.get_parent_object()
         self.check_parent_access(job)
-        qs = job.job_events
-        qs = qs.select_related('host')
-        qs = qs.prefetch_related('hosts', 'children')
+        qs = job.job_events.select_related('host').order_by('start_line')
         return qs.all()
 
 
@@ -4296,7 +4297,9 @@ class NotificationTemplateDetail(RetrieveUpdateDestroyAPIView):
         obj = self.get_object()
         if not request.user.can_access(self.model, 'delete', obj):
             return Response(status=status.HTTP_404_NOT_FOUND)
-        if obj.notifications.filter(status='pending').exists():
+
+        hours_old = now() - dateutil.relativedelta.relativedelta(hours=8)
+        if obj.notifications.filter(status='pending', created__gt=hours_old).exists():
             return Response({"error": _("Delete not allowed while there are pending notifications")},
                             status=status.HTTP_405_METHOD_NOT_ALLOWED)
         return super(NotificationTemplateDetail, self).delete(request, *args, **kwargs)
@@ -4315,7 +4318,7 @@ class NotificationTemplateTest(GenericAPIView):
         msg = "Tower Notification Test {} {}".format(obj.id, settings.TOWER_URL_BASE)
         if obj.notification_type in ('email', 'pagerduty'):
             body = "Ansible Tower Test Notification {} {}".format(obj.id, settings.TOWER_URL_BASE)
-        elif obj.notification_type == 'webhook':
+        elif obj.notification_type in ('webhook', 'grafana'):
             body = '{{"body": "Ansible Tower Test Notification {} {}"}}'.format(obj.id, settings.TOWER_URL_BASE)
         else:
             body = {"body": "Ansible Tower Test Notification {} {}".format(obj.id, settings.TOWER_URL_BASE)}
@@ -4426,7 +4429,7 @@ class RoleUsersList(SubListAttachDetachAPIView):
 
         credential_content_type = ContentType.objects.get_for_model(models.Credential)
         if role.content_type == credential_content_type:
-            if role.content_object.organization and user not in role.content_object.organization.member_role:
+            if 'disassociate' not in request.data and role.content_object.organization and user not in role.content_object.organization.member_role:
                 data = dict(msg=_("You cannot grant credential access to a user not in the credentials' organization"))
                 return Response(data, status=status.HTTP_400_BAD_REQUEST)
 

awx/api/views/inventory.py

@@ -134,7 +134,8 @@ class InventoryDetail(RelatedJobsPreventDeleteMixin, ControlledByScmMixin, Retri
 
         # Do not allow changes to an Inventory kind.
         if kind is not None and obj.kind != kind:
-            return self.http_method_not_allowed(request, *args, **kwargs)
+            return Response(dict(error=_('You cannot turn a regular inventory into a "smart" inventory.')),
+                            status=status.HTTP_405_METHOD_NOT_ALLOWED)
         return super(InventoryDetail, self).update(request, *args, **kwargs)
 
     def destroy(self, request, *args, **kwargs):

awx/api/views/metrics.py

@@ -22,7 +22,7 @@ from awx.api.generics import (
 )
 
 
-logger = logging.getLogger('awx.main.analytics')
+logger = logging.getLogger('awx.analytics')
 
 
 class MetricsView(APIView):
@@ -39,3 +39,4 @@ class MetricsView(APIView):
         if (request.user.is_superuser or request.user.is_system_auditor):
             return Response(metrics().decode('UTF-8'))
         raise PermissionDenied()
+

awx/api/views/mixin.py

@@ -4,10 +4,7 @@
 import dateutil
 import logging
 
-from django.db.models import (
-    Count,
-    F,
-)
+from django.db.models import Count
 from django.db import transaction
 from django.shortcuts import get_object_or_404
 from django.utils.timezone import now
@@ -175,28 +172,18 @@ class OrganizationCountsMixin(object):
 
         inv_qs = Inventory.accessible_objects(self.request.user, 'read_role')
         project_qs = Project.accessible_objects(self.request.user, 'read_role')
+        jt_qs = JobTemplate.accessible_objects(self.request.user, 'read_role')
 
         # Produce counts of Foreign Key relationships
-        db_results['inventories'] = inv_qs\
-            .values('organization').annotate(Count('organization')).order_by('organization')
+        db_results['inventories'] = inv_qs.values('organization').annotate(Count('organization')).order_by('organization')
 
         db_results['teams'] = Team.accessible_objects(
             self.request.user, 'read_role').values('organization').annotate(
             Count('organization')).order_by('organization')
 
-        JT_project_reference = 'project__organization'
-        JT_inventory_reference = 'inventory__organization'
-        db_results['job_templates_project'] = JobTemplate.accessible_objects(
-            self.request.user, 'read_role').exclude(
-            project__organization=F(JT_inventory_reference)).values(JT_project_reference).annotate(
-            Count(JT_project_reference)).order_by(JT_project_reference)
+        db_results['job_templates'] = jt_qs.values('organization').annotate(Count('organization')).order_by('organization')
 
-        db_results['job_templates_inventory'] = JobTemplate.accessible_objects(
-            self.request.user, 'read_role').values(JT_inventory_reference).annotate(
-            Count(JT_inventory_reference)).order_by(JT_inventory_reference)
-
-        db_results['projects'] = project_qs\
-            .values('organization').annotate(Count('organization')).order_by('organization')
+        db_results['projects'] = project_qs.values('organization').annotate(Count('organization')).order_by('organization')
 
         # Other members and admins of organization are always viewable
         db_results['users'] = org_qs.annotate(
@@ -212,11 +199,7 @@ class OrganizationCountsMixin(object):
                 'admins': 0, 'projects': 0}
 
         for res, count_qs in db_results.items():
-            if res == 'job_templates_project':
-                org_reference = JT_project_reference
-            elif res == 'job_templates_inventory':
-                org_reference = JT_inventory_reference
-            elif res == 'users':
+            if res == 'users':
                 org_reference = 'id'
             else:
                 org_reference = 'organization'
@@ -229,14 +212,6 @@ class OrganizationCountsMixin(object):
                         continue
                     count_context[org_id][res] = entry['%s__count' % org_reference]
 
-        # Combine the counts for job templates by project and inventory
-        for org in org_id_list:
-            org_id = org['id']
-            count_context[org_id]['job_templates'] = 0
-            for related_path in ['job_templates_project', 'job_templates_inventory']:
-                if related_path in count_context[org_id]:
-                    count_context[org_id]['job_templates'] += count_context[org_id].pop(related_path)
-
         full_context['related_field_counts'] = count_context
 
         return full_context

awx/api/views/organization.py

@@ -7,12 +7,15 @@ import logging
 # Django
 from django.db.models import Count
 from django.contrib.contenttypes.models import ContentType
+from django.utils.translation import ugettext_lazy as _
 
 # AWX
 from awx.main.models import (
     ActivityStream,
     Inventory,
+    Host,
     Project,
+    ExecutionEnvironment,
     JobTemplate,
     WorkflowJobTemplate,
     Organization,
@@ -21,6 +24,7 @@ from awx.main.models import (
     User,
     Team,
     InstanceGroup,
+    Credential
 )
 from awx.api.generics import (
     ListCreateAPIView,
@@ -28,6 +32,7 @@ from awx.api.generics import (
     SubListAPIView,
     SubListCreateAttachDetachAPIView,
     SubListAttachDetachAPIView,
+    SubListCreateAPIView,
     ResourceAccessList,
     BaseUsersList,
 )
@@ -35,14 +40,15 @@ from awx.api.generics import (
 from awx.api.serializers import (
     OrganizationSerializer,
     InventorySerializer,
-    ProjectSerializer,
     UserSerializer,
     TeamSerializer,
     ActivityStreamSerializer,
     RoleSerializer,
     NotificationTemplateSerializer,
-    WorkflowJobTemplateSerializer,
     InstanceGroupSerializer,
+    ExecutionEnvironmentSerializer,
+    ProjectSerializer, JobTemplateSerializer, WorkflowJobTemplateSerializer,
+    CredentialSerializer
 )
 from awx.api.views.mixin import (
     RelatedJobsPreventDeleteMixin,
@@ -94,7 +100,8 @@ class OrganizationDetail(RelatedJobsPreventDeleteMixin, RetrieveUpdateDestroyAPI
         org_counts['projects'] = Project.accessible_objects(**access_kwargs).filter(
             organization__id=org_id).count()
         org_counts['job_templates'] = JobTemplate.accessible_objects(**access_kwargs).filter(
-            project__organization__id=org_id).count()
+            organization__id=org_id).count()
+        org_counts['hosts'] = Host.objects.org_active_count(org_id)
 
         full_context['related_field_counts'] = {}
         full_context['related_field_counts'][org_id] = org_counts
@@ -128,21 +135,37 @@ class OrganizationAdminsList(BaseUsersList):
     ordering = ('username',)
 
 
-class OrganizationProjectsList(SubListCreateAttachDetachAPIView):
+class OrganizationProjectsList(SubListCreateAPIView):
 
     model = Project
     serializer_class = ProjectSerializer
     parent_model = Organization
-    relationship = 'projects'
     parent_key = 'organization'
 
 
-class OrganizationWorkflowJobTemplatesList(SubListCreateAttachDetachAPIView):
+class OrganizationExecutionEnvironmentsList(SubListCreateAttachDetachAPIView):
+
+    model = ExecutionEnvironment
+    serializer_class = ExecutionEnvironmentSerializer
+    parent_model = Organization
+    relationship = 'executionenvironments'
+    parent_key = 'organization'
+    swagger_topic = "Execution Environments"
+
+
+class OrganizationJobTemplatesList(SubListCreateAPIView):
+
+    model = JobTemplate
+    serializer_class = JobTemplateSerializer
+    parent_model = Organization
+    parent_key = 'organization'
+
+
+class OrganizationWorkflowJobTemplatesList(SubListCreateAPIView):
 
     model = WorkflowJobTemplate
     serializer_class = WorkflowJobTemplateSerializer
     parent_model = Organization
-    relationship = 'workflows'
     parent_key = 'organization'
 
 
@@ -208,6 +231,20 @@ class OrganizationInstanceGroupsList(SubListAttachDetachAPIView):
     relationship = 'instance_groups'
 
 
+class OrganizationGalaxyCredentialsList(SubListAttachDetachAPIView):
+
+    model = Credential
+    serializer_class = CredentialSerializer
+    parent_model = Organization
+    relationship = 'galaxy_credentials'
+
+    def is_valid_relation(self, parent, sub, created=False):
+        if sub.kind != 'galaxy_api_token':
+            return {'msg': _(
+                f"Credential must be a Galaxy credential, not {sub.credential_type.name}."
+            )}
+
+
 class OrganizationAccessList(ResourceAccessList):
 
     model = User # needs to be User for AccessLists's

awx/api/views/root.py

@@ -1,9 +1,10 @@
 # Copyright (c) 2018 Ansible, Inc.
 # All Rights Reserved.
 
+import base64
+import json
 import logging
 import operator
-import json
 from collections import OrderedDict
 
 from django.conf import settings
@@ -20,6 +21,8 @@ from rest_framework import status
 import requests
 
 from awx.api.generics import APIView
+from awx.conf.registry import settings_registry
+from awx.main.analytics import all_collectors
 from awx.main.ha import is_ha_environment
 from awx.main.utils import (
     get_awx_version,
@@ -27,8 +30,8 @@ from awx.main.utils import (
     get_custom_venv_choices,
     to_python_boolean,
 )
+from awx.main.utils.licensing import validate_entitlement_manifest
 from awx.api.versioning import reverse, drf_reverse
-from awx.conf.license import get_license
 from awx.main.constants import PRIVILEGE_ESCALATION_METHODS
 from awx.main.models import (
     Project,
@@ -37,6 +40,7 @@ from awx.main.models import (
     InstanceGroup,
     JobTemplate,
 )
+from awx.main.utils import set_environ
 
 logger = logging.getLogger('awx.api.views.root')
 
@@ -60,6 +64,7 @@ class ApiRootView(APIView):
         data['oauth2'] = drf_reverse('api:oauth_authorization_root_view')
         data['custom_logo'] = settings.CUSTOM_LOGO
         data['custom_login_info'] = settings.CUSTOM_LOGIN_INFO
+        data['login_redirect_override'] = settings.LOGIN_REDIRECT_OVERRIDE
         return Response(data)
 
 
@@ -95,6 +100,7 @@ class ApiVersionRootView(APIView):
         data['dashboard'] = reverse('api:dashboard_view', request=request)
         data['organizations'] = reverse('api:organization_list', request=request)
         data['users'] = reverse('api:user_list', request=request)
+        data['execution_environments'] = reverse('api:execution_environment_list', request=request)
         data['projects'] = reverse('api:project_list', request=request)
         data['project_updates'] = reverse('api:project_update_list', request=request)
         data['teams'] = reverse('api:team_list', request=request)
@@ -174,7 +180,7 @@ class ApiV2PingView(APIView):
 class ApiV2SubscriptionView(APIView):
 
     permission_classes = (IsAuthenticated,)
-    name = _('Configuration')
+    name = _('Subscriptions')
     swagger_topic = 'System Configuration'
 
     def check_permissions(self, request):
@@ -185,31 +191,87 @@ class ApiV2SubscriptionView(APIView):
     def post(self, request):
         from awx.main.utils.common import get_licenser
         data = request.data.copy()
-        if data.get('rh_password') == '$encrypted$':
-            data['rh_password'] = settings.REDHAT_PASSWORD
+        if data.get('subscriptions_password') == '$encrypted$':
+            data['subscriptions_password'] = settings.SUBSCRIPTIONS_PASSWORD
         try:
-            user, pw = data.get('rh_username'), data.get('rh_password')
-            validated = get_licenser().validate_rh(user, pw)
+            user, pw = data.get('subscriptions_username'), data.get('subscriptions_password')
+            with set_environ(**settings.AWX_TASK_ENV):
+                validated = get_licenser().validate_rh(user, pw)
             if user:
-                settings.REDHAT_USERNAME = data['rh_username']
+                settings.SUBSCRIPTIONS_USERNAME = data['subscriptions_username']
             if pw:
-                settings.REDHAT_PASSWORD = data['rh_password']
+                settings.SUBSCRIPTIONS_PASSWORD = data['subscriptions_password']
         except Exception as exc:
-            msg = _("Invalid License")
+            msg = _("Invalid Subscription")
             if (
                 isinstance(exc, requests.exceptions.HTTPError) and
                 getattr(getattr(exc, 'response', None), 'status_code', None) == 401
             ):
                 msg = _("The provided credentials are invalid (HTTP 401).")
-            if isinstance(exc, (ValueError, OSError)) and exc.args:
+            elif isinstance(exc, requests.exceptions.ProxyError):
+                msg = _("Unable to connect to proxy server.")
+            elif isinstance(exc, requests.exceptions.ConnectionError):
+                msg = _("Could not connect to subscription service.")
+            elif isinstance(exc, (ValueError, OSError)) and exc.args:
                 msg = exc.args[0]
-            logger.exception(smart_text(u"Invalid license submitted."),
-                             extra=dict(actor=request.user.username))
+            else:
+                logger.exception(smart_text(u"Invalid subscription submitted."),
+                                 extra=dict(actor=request.user.username))
             return Response({"error": msg}, status=status.HTTP_400_BAD_REQUEST)
 
         return Response(validated)
 
 
+class ApiV2AttachView(APIView):
+
+    permission_classes = (IsAuthenticated,)
+    name = _('Attach Subscription')
+    swagger_topic = 'System Configuration'
+
+    def check_permissions(self, request):
+        super(ApiV2AttachView, self).check_permissions(request)
+        if not request.user.is_superuser and request.method.lower() not in {'options', 'head'}:
+            self.permission_denied(request)  # Raises PermissionDenied exception.
+
+    def post(self, request):
+        data = request.data.copy()
+        pool_id = data.get('pool_id', None)
+        if not pool_id:
+            return Response({"error": _("No subscription pool ID provided.")}, status=status.HTTP_400_BAD_REQUEST)
+        user = getattr(settings, 'SUBSCRIPTIONS_USERNAME', None)
+        pw = getattr(settings, 'SUBSCRIPTIONS_PASSWORD', None)
+        if pool_id and user and pw:
+            from awx.main.utils.common import get_licenser
+            data = request.data.copy()
+            try:
+                with set_environ(**settings.AWX_TASK_ENV):
+                    validated = get_licenser().validate_rh(user, pw)
+            except Exception as exc:
+                msg = _("Invalid Subscription")
+                if (
+                    isinstance(exc, requests.exceptions.HTTPError) and
+                    getattr(getattr(exc, 'response', None), 'status_code', None) == 401
+                ):
+                    msg = _("The provided credentials are invalid (HTTP 401).")
+                elif isinstance(exc, requests.exceptions.ProxyError):
+                    msg = _("Unable to connect to proxy server.")
+                elif isinstance(exc, requests.exceptions.ConnectionError):
+                    msg = _("Could not connect to subscription service.")
+                elif isinstance(exc, (ValueError, OSError)) and exc.args:
+                    msg = exc.args[0]
+                else:
+                    logger.exception(smart_text(u"Invalid subscription submitted."),
+                                     extra=dict(actor=request.user.username))
+                return Response({"error": msg}, status=status.HTTP_400_BAD_REQUEST)
+        for sub in validated:
+            if sub['pool_id'] == pool_id:
+                sub['valid_key'] = True
+                settings.LICENSE = sub
+                return Response(sub)
+
+        return Response({"error": _("Error processing subscription metadata.")}, status=status.HTTP_400_BAD_REQUEST)
+
+
 class ApiV2ConfigView(APIView):
 
     permission_classes = (IsAuthenticated,)
@@ -224,15 +286,11 @@ class ApiV2ConfigView(APIView):
     def get(self, request, format=None):
         '''Return various sitewide configuration settings'''
 
-        if request.user.is_superuser or request.user.is_system_auditor:
-            license_data = get_license(show_key=True)
-        else:
-            license_data = get_license(show_key=False)
+        from awx.main.utils.common import get_licenser
+        license_data = get_licenser().validate()
+
         if not license_data.get('valid_key', False):
             license_data = {}
-        if license_data and 'features' in license_data and 'activity_streams' in license_data['features']:
-            # FIXME: Make the final setting value dependent on the feature?
-            license_data['features']['activity_streams'] &= settings.ACTIVITY_STREAM_ENABLED
 
         pendo_state = settings.PENDO_TRACKING_STATE if settings.PENDO_TRACKING_STATE in ('off', 'anonymous', 'detailed') else 'off'
 
@@ -243,6 +301,7 @@ class ApiV2ConfigView(APIView):
             ansible_version=get_ansible_version(),
             eula=render_to_string("eula.md") if license_data.get('license_type', 'UNLICENSED') != 'open' else '',
             analytics_status=pendo_state,
+            analytics_collectors=all_collectors(),
             become_methods=PRIVILEGE_ESCALATION_METHODS,
         )
 
@@ -270,9 +329,10 @@ class ApiV2ConfigView(APIView):
 
         return Response(data)
 
+
     def post(self, request):
         if not isinstance(request.data, dict):
-            return Response({"error": _("Invalid license data")}, status=status.HTTP_400_BAD_REQUEST)
+            return Response({"error": _("Invalid subscription data")}, status=status.HTTP_400_BAD_REQUEST)
         if "eula_accepted" not in request.data:
             return Response({"error": _("Missing 'eula_accepted' property")}, status=status.HTTP_400_BAD_REQUEST)
         try:
@@ -289,24 +349,47 @@ class ApiV2ConfigView(APIView):
             logger.info(smart_text(u"Invalid JSON submitted for license."),
                         extra=dict(actor=request.user.username))
             return Response({"error": _("Invalid JSON")}, status=status.HTTP_400_BAD_REQUEST)
-        try:
-            from awx.main.utils.common import get_licenser
-            license_data = json.loads(data_actual)
-            license_data_validated = get_licenser(**license_data).validate()
-        except Exception:
-            logger.warning(smart_text(u"Invalid license submitted."),
-                           extra=dict(actor=request.user.username))
-            return Response({"error": _("Invalid License")}, status=status.HTTP_400_BAD_REQUEST)
+
+        from awx.main.utils.common import get_licenser
+        license_data = json.loads(data_actual)
+        if 'license_key' in license_data:
+            return Response({"error": _('Legacy license submitted. A subscription manifest is now required.')}, status=status.HTTP_400_BAD_REQUEST)
+        if 'manifest' in license_data:
+            try:
+                json_actual = json.loads(base64.b64decode(license_data['manifest']))
+                if 'license_key' in json_actual:
+                    return Response(
+                        {"error": _('Legacy license submitted. A subscription manifest is now required.')},
+                        status=status.HTTP_400_BAD_REQUEST
+                    )
+            except Exception:
+                pass
+            try:
+                license_data = validate_entitlement_manifest(license_data['manifest'])
+            except ValueError as e:
+                return Response({"error": str(e)}, status=status.HTTP_400_BAD_REQUEST)
+            except Exception:
+                logger.exception('Invalid manifest submitted. {}')
+                return Response({"error": _('Invalid manifest submitted.')}, status=status.HTTP_400_BAD_REQUEST)
+
+            try:
+                license_data_validated = get_licenser().license_from_manifest(license_data)
+            except Exception:
+                logger.warning(smart_text(u"Invalid subscription submitted."),
+                               extra=dict(actor=request.user.username))
+                return Response({"error": _("Invalid License")}, status=status.HTTP_400_BAD_REQUEST)
+        else:
+            license_data_validated = get_licenser().validate()
 
         # If the license is valid, write it to the database.
         if license_data_validated['valid_key']:
-            settings.LICENSE = license_data
-            settings.TOWER_URL_BASE = "{}://{}".format(request.scheme, request.get_host())
+            if not settings_registry.is_setting_read_only('TOWER_URL_BASE'):
+                settings.TOWER_URL_BASE = "{}://{}".format(request.scheme, request.get_host())
             return Response(license_data_validated)
 
-        logger.warning(smart_text(u"Invalid license submitted."),
+        logger.warning(smart_text(u"Invalid subscription submitted."),
                        extra=dict(actor=request.user.username))
-        return Response({"error": _("Invalid license")}, status=status.HTTP_400_BAD_REQUEST)
+        return Response({"error": _("Invalid subscription")}, status=status.HTTP_400_BAD_REQUEST)
 
     def delete(self, request):
         try:

awx/asgi.py

@@ -2,14 +2,14 @@
 # All Rights Reserved.
 import os
 import logging
+import django
 from awx import __version__ as tower_version
-
 # Prepare the AWX environment.
 from awx import prepare_env, MODE
+from channels.routing import get_default_application  # noqa
 prepare_env() # NOQA
 
-from django.core.wsgi import get_wsgi_application  # NOQA
-from channels.asgi import get_channel_layer
+
 
 """
 ASGI config for AWX project.
@@ -25,13 +25,14 @@ if MODE == 'production':
     try:
         fd = open("/var/lib/awx/.tower_version", "r")
         if fd.read().strip() != tower_version:
-            raise Exception()
-    except Exception:
+            raise ValueError()
+    except FileNotFoundError:
+        pass
+    except ValueError as e:
         logger.error("Missing or incorrect metadata for Tower version.  Ensure Tower was installed using the setup playbook.")
-        raise Exception("Missing or incorrect metadata for Tower version.  Ensure Tower was installed using the setup playbook.")
+        raise Exception("Missing or incorrect metadata for Tower version.  Ensure Tower was installed using the setup playbook.") from e
 
 
 os.environ.setdefault("DJANGO_SETTINGS_MODULE", "awx.settings")
-
-
-channel_layer = get_channel_layer()
+django.setup()
+channel_layer = get_default_application()

awx/conf/fields.py

@@ -11,9 +11,10 @@ from django.utils.translation import ugettext_lazy as _
 
 # Django REST Framework
 from rest_framework.fields import (  # noqa
-    BooleanField, CharField, ChoiceField, DictField, EmailField,
+    BooleanField, CharField, ChoiceField, DictField, DateTimeField, EmailField,
     IntegerField, ListField, NullBooleanField
 )
+from rest_framework.serializers import PrimaryKeyRelatedField  # noqa
 
 logger = logging.getLogger('awx.conf.fields')
 
@@ -172,9 +173,9 @@ class URLField(CharField):
                         netloc = '{}:{}'.format(netloc, url_parts.port)
                     if url_parts.username:
                         if url_parts.password:
-                            netloc = '{}:{}@{}' % (url_parts.username, url_parts.password, netloc)
+                            netloc = '{}:{}@{}'.format(url_parts.username, url_parts.password, netloc)
                         else:
-                            netloc = '{}@{}' % (url_parts.username, netloc)
+                            netloc = '{}@{}'.format(url_parts.username, netloc)
                     value = urlparse.urlunsplit([url_parts.scheme, netloc, url_parts.path, url_parts.query, url_parts.fragment])
             except Exception:
                 raise  # If something fails here, just fall through and let the validators check it.

awx/conf/license.py

@@ -1,19 +1,14 @@
 # Copyright (c) 2016 Ansible, Inc.
 # All Rights Reserved.
 
-# Tower
-from awx.main.utils.common import get_licenser
-
 __all__ = ['get_license']
 
 
 def _get_validated_license_data():
+    from awx.main.utils import get_licenser
     return get_licenser().validate()
 
 
-def get_license(show_key=False):
+def get_license():
     """Return a dictionary representing the active license on this Tower instance."""
-    license_data = _get_validated_license_data()
-    if not show_key:
-        license_data.pop('license_key', None)
-    return license_data
+    return _get_validated_license_data()

awx/conf/migrations/0004_v320_reencrypt.py

@@ -2,7 +2,6 @@
 from __future__ import unicode_literals
 
 from django.db import migrations
-from awx.conf.migrations import _reencrypt
 
 
 class Migration(migrations.Migration):
@@ -12,5 +11,8 @@ class Migration(migrations.Migration):
     ]
 
     operations = [
-        migrations.RunPython(_reencrypt.replace_aesecb_fernet),
+        # This list is intentionally empty.
+        # Tower 3.2 included several data migrations that are no longer
+        # necessary (this list is now empty because Tower 3.2 is past EOL and
+        # cannot be directly upgraded to modern versions of Tower)
     ]

awx/conf/migrations/0005_v330_rename_two_session_settings.py

@@ -23,4 +23,4 @@ class Migration(migrations.Migration):
     operations = [
         migrations.RunPython(copy_session_settings, reverse_copy_session_settings),
     ]
-    
+

awx/conf/migrations/0007_v380_rename_more_settings.py

@@ -0,0 +1,19 @@
+# -*- coding: utf-8 -*-
+from __future__ import unicode_literals
+from django.db import migrations
+from awx.conf.migrations import _rename_setting
+
+
+def copy_allowed_ips(apps, schema_editor):
+    _rename_setting.rename_setting(apps, schema_editor, old_key='PROXY_IP_WHITELIST', new_key='PROXY_IP_ALLOWED_LIST')
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('conf', '0006_v331_ldap_group_type'),
+    ]
+
+    operations = [
+        migrations.RunPython(copy_allowed_ips),
+    ]

awx/conf/migrations/0008_subscriptions.py

@@ -0,0 +1,26 @@
+# Generated by Django 2.2.11 on 2020-08-04 15:19
+
+import logging
+
+from django.db import migrations
+
+from awx.conf.migrations._subscriptions import clear_old_license, prefill_rh_credentials
+
+logger = logging.getLogger('awx.conf.migrations')
+
+
+def _noop(apps, schema_editor):
+    pass
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('conf', '0007_v380_rename_more_settings'),
+    ]
+
+
+    operations = [
+        migrations.RunPython(clear_old_license, _noop),
+        migrations.RunPython(prefill_rh_credentials, _noop)
+    ]

awx/conf/migrations/_reencrypt.py

@@ -1,30 +1,13 @@
 import base64
 import hashlib
 
-from django.utils.encoding import smart_str
-
 from cryptography.hazmat.backends import default_backend
 from cryptography.hazmat.primitives.ciphers import Cipher
 from cryptography.hazmat.primitives.ciphers.algorithms import AES
 from cryptography.hazmat.primitives.ciphers.modes import ECB
 
-from awx.conf import settings_registry
 
-
-__all__ = ['replace_aesecb_fernet', 'get_encryption_key', 'encrypt_field',
-           'decrypt_value', 'decrypt_value', 'should_decrypt_field']
-
-
-def replace_aesecb_fernet(apps, schema_editor):
-    from awx.main.utils.encryption import encrypt_field
-    Setting = apps.get_model('conf', 'Setting')
-
-    for setting in Setting.objects.filter().order_by('pk'):
-        if settings_registry.is_setting_encrypted(setting.key):
-            if should_decrypt_field(setting.value):
-                setting.value = decrypt_field(setting, 'value')
-                setting.value = encrypt_field(setting, 'value')
-            setting.save()
+__all__ = ['get_encryption_key', 'decrypt_field']
 
 
 def get_encryption_key(field_name, pk=None):
@@ -76,38 +59,3 @@ def decrypt_field(instance, field_name, subfield=None):
     key = get_encryption_key(field_name, getattr(instance, 'pk', None))
 
     return decrypt_value(key, value)
-
-
-def encrypt_field(instance, field_name, ask=False, subfield=None, skip_utf8=False):
-    '''
-    Return content of the given instance and field name encrypted.
-    '''
-    value = getattr(instance, field_name)
-    if isinstance(value, dict) and subfield is not None:
-        value = value[subfield]
-    if not value or value.startswith('$encrypted$') or (ask and value == 'ASK'):
-        return value
-    if skip_utf8:
-        utf8 = False
-    else:
-        utf8 = type(value) == str
-    value = smart_str(value)
-    key = get_encryption_key(field_name, getattr(instance, 'pk', None))
-    encryptor = Cipher(AES(key), ECB(), default_backend()).encryptor()
-    block_size = 16
-    while len(value) % block_size != 0:
-        value += '\x00'
-    encrypted = encryptor.update(value) + encryptor.finalize()
-    b64data = base64.b64encode(encrypted)
-    tokens = ['$encrypted', 'AES', b64data]
-    if utf8:
-        # If the value to encrypt is utf-8, we need to add a marker so we
-        # know to decode the data when it's decrypted later
-        tokens.insert(1, 'UTF8')
-    return '$'.join(tokens)
-
-
-def should_decrypt_field(value):
-    if hasattr(value, 'startswith'):
-        return value.startswith('$encrypted$') and '$AESCBC$' not in value
-    return False

awx/conf/migrations/_subscriptions.py

@@ -0,0 +1,34 @@
+# -*- coding: utf-8 -*-
+import logging
+from django.utils.timezone import now
+from awx.main.utils.encryption import decrypt_field, encrypt_field
+
+logger = logging.getLogger('awx.conf.settings')
+
+__all__ = ['clear_old_license', 'prefill_rh_credentials']
+    
+
+def clear_old_license(apps, schema_editor):
+    Setting = apps.get_model('conf', 'Setting')
+    Setting.objects.filter(key='LICENSE').delete()
+
+
+def _migrate_setting(apps, old_key, new_key, encrypted=False):
+    Setting = apps.get_model('conf', 'Setting')
+    if not Setting.objects.filter(key=old_key).exists():
+        return
+    new_setting = Setting.objects.create(key=new_key,
+                                         created=now(),
+                                         modified=now()
+                                         )
+    if encrypted:
+        new_setting.value = decrypt_field(Setting.objects.filter(key=old_key).first(), 'value')
+        new_setting.value = encrypt_field(new_setting, 'value')
+    else:
+        new_setting.value = getattr(Setting.objects.filter(key=old_key).first(), 'value')
+    new_setting.save()
+
+
+def prefill_rh_credentials(apps, schema_editor):
+    _migrate_setting(apps, 'REDHAT_USERNAME', 'SUBSCRIPTIONS_USERNAME', encrypted=False)
+    _migrate_setting(apps, 'REDHAT_PASSWORD', 'SUBSCRIPTIONS_PASSWORD', encrypted=True)

awx/conf/models.py

@@ -78,14 +78,6 @@ class Setting(CreatedModifiedModel):
     def get_cache_id_key(self, key):
         return '{}_ID'.format(key)
 
-    def display_value(self):
-        if self.key == 'LICENSE' and 'license_key' in self.value:
-            # don't log the license key in activity stream
-            value = self.value.copy()
-            value['license_key'] = '********'
-            return value
-        return self.value
-
 
 import awx.conf.signals  # noqa
 

awx/conf/registry.py

@@ -129,12 +129,14 @@ class SettingsRegistry(object):
         placeholder = field_kwargs.pop('placeholder', empty)
         encrypted = bool(field_kwargs.pop('encrypted', False))
         defined_in_file = bool(field_kwargs.pop('defined_in_file', False))
+        unit = field_kwargs.pop('unit', None)
         if getattr(field_kwargs.get('child', None), 'source', None) is not None:
             field_kwargs['child'].source = None
         field_instance = field_class(**field_kwargs)
         field_instance.category_slug = category_slug
         field_instance.category = category
         field_instance.depends_on = depends_on
+        field_instance.unit = unit
         if placeholder is not empty:
             field_instance.placeholder = placeholder
         field_instance.defined_in_file = defined_in_file

awx/conf/settings.py

@@ -1,14 +1,10 @@
 # Python
-from collections import namedtuple
 import contextlib
 import logging
-import re
 import sys
 import threading
 import time
-import traceback
-import urllib.parse
-from io import StringIO
+import os
 
 # Django
 from django.conf import LazySettings
@@ -16,12 +12,14 @@ from django.conf import settings, UserSettingsHolder
 from django.core.cache import cache as django_cache
 from django.core.exceptions import ImproperlyConfigured
 from django.db import transaction, connection
-from django.db.utils import Error as DBError
+from django.db.utils import Error as DBError, ProgrammingError
 from django.utils.functional import cached_property
 
 # Django REST Framework
 from rest_framework.fields import empty, SkipField
 
+import cachetools
+
 # Tower
 from awx.main.utils import encrypt_field, decrypt_field
 from awx.conf import settings_registry
@@ -33,21 +31,23 @@ from awx.conf.migrations._reencrypt import decrypt_field as old_decrypt_field
 
 logger = logging.getLogger('awx.conf.settings')
 
+SETTING_MEMORY_TTL = 5 if 'callback_receiver' in ' '.join(sys.argv) else 0
+
 # Store a special value to indicate when a setting is not set in the database.
 SETTING_CACHE_NOTSET = '___notset___'
 
-# Cannot store None in memcached; use a special value instead to indicate None.
+# Cannot store None in cache; use a special value instead to indicate None.
 # If the special value for None is the same as the "not set" value, then a value
 # of None will be equivalent to the setting not being set (and will raise an
 # AttributeError if there is no other default defined).
 # SETTING_CACHE_NONE = '___none___'
 SETTING_CACHE_NONE = SETTING_CACHE_NOTSET
 
-# Cannot store empty list/tuple in memcached; use a special value instead to
+# Cannot store empty list/tuple in cache; use a special value instead to
 # indicate an empty list.
 SETTING_CACHE_EMPTY_LIST = '___[]___'
 
-# Cannot store empty dict in memcached; use a special value instead to indicate
+# Cannot store empty dict in cache; use a special value instead to indicate
 # an empty dict.
 SETTING_CACHE_EMPTY_DICT = '___{}___'
 
@@ -60,15 +60,6 @@ SETTING_CACHE_DEFAULTS = True
 __all__ = ['SettingsWrapper', 'get_settings_to_cache', 'SETTING_CACHE_NOTSET']
 
 
-def normalize_broker_url(value):
-    parts = value.rsplit('@', 1)
-    match = re.search('(amqp://[^:]+:)(.*)', parts[0])
-    if match:
-        prefix, password = match.group(1), match.group(2)
-        parts[0] = prefix + urllib.parse.quote(password)
-    return '@'.join(parts)
-
-
 @contextlib.contextmanager
 def _ctit_db_wrapper(trans_safe=False):
     '''
@@ -88,43 +79,21 @@ def _ctit_db_wrapper(trans_safe=False):
                     logger.debug('Obtaining database settings in spite of broken transaction.')
                     transaction.set_rollback(False)
         yield
-    except DBError:
-        # We want the _full_ traceback with the context
-        # First we get the current call stack, which constitutes the "top",
-        # it has the context up to the point where the context manager is used
-        top_stack = StringIO()
-        traceback.print_stack(file=top_stack)
-        top_lines = top_stack.getvalue().strip('\n').split('\n')
-        top_stack.close()
-        # Get "bottom" stack from the local error that happened
-        # inside of the "with" block this wraps
-        exc_type, exc_value, exc_traceback = sys.exc_info()
-        bottom_stack = StringIO()
-        traceback.print_tb(exc_traceback, file=bottom_stack)
-        bottom_lines = bottom_stack.getvalue().strip('\n').split('\n')
-        # Glue together top and bottom where overlap is found
-        bottom_cutoff = 0
-        for i, line in enumerate(bottom_lines):
-            if line in top_lines:
-                # start of overlapping section, take overlap from bottom
-                top_lines = top_lines[:top_lines.index(line)]
-                bottom_cutoff = i
-                break
-        bottom_lines = bottom_lines[bottom_cutoff:]
-        tb_lines = top_lines + bottom_lines
-
-        tb_string = '\n'.join(
-            ['Traceback (most recent call last):'] +
-            tb_lines +
-            ['{}: {}'.format(exc_type.__name__, str(exc_value))]
-        )
-        bottom_stack.close()
-        # Log the combined stack
+    except DBError as exc:
         if trans_safe:
-            if 'check_migrations' not in sys.argv:
-                logger.debug('Database settings are not available, using defaults, error:\n{}'.format(tb_string))
+            if 'migrate' not in sys.argv and 'check_migrations' not in sys.argv:
+                level = logger.exception
+                if isinstance(exc, ProgrammingError):
+                    if 'relation' in str(exc) and 'does not exist' in str(exc):
+                        # this generally means we can't fetch Tower configuration
+                        # because the database hasn't actually finished migrating yet;
+                        # this is usually a sign that a service in a container (such as ws_broadcast)
+                        # has come up *before* the database has finished migrating, and
+                        # especially that the conf.settings table doesn't exist yet
+                        level = logger.debug
+                level('Database settings are not available, using defaults.')
         else:
-            logger.debug('Error modifying something related to database settings.\n{}'.format(tb_string))
+            logger.exception('Error modifying something related to database settings.')
     finally:
         if trans_safe and is_atomic and rollback_set:
             transaction.set_rollback(rollback_set)
@@ -136,6 +105,15 @@ def filter_sensitive(registry, key, value):
     return value
 
 
+class TransientSetting(object):
+
+    __slots__ = ('pk', 'value')
+
+    def __init__(self, pk, value):
+        self.pk = pk
+        self.value = value
+
+
 class EncryptedCacheProxy(object):
 
     def __init__(self, cache, registry, encrypter=None, decrypter=None):
@@ -163,7 +141,6 @@ class EncryptedCacheProxy(object):
     def get(self, key, **kwargs):
         value = self.cache.get(key, **kwargs)
         value = self._handle_encryption(self.decrypter, key, value)
-        logger.debug('cache get(%r, %r) -> %r', key, empty, filter_sensitive(self.registry, key, value))
         return value
 
     def set(self, key, value, log=True, **kwargs):
@@ -186,8 +163,6 @@ class EncryptedCacheProxy(object):
             self.set(key, value, log=False, **kwargs)
 
     def _handle_encryption(self, method, key, value):
-        TransientSetting = namedtuple('TransientSetting', ['pk', 'value'])
-
         if value is not empty and self.registry.is_setting_encrypted(key):
             # If the setting exists in the database, we'll use its primary key
             # as part of the AES key when encrypting/decrypting
@@ -273,6 +248,7 @@ class SettingsWrapper(UserSettingsHolder):
         # These values have to be stored via self.__dict__ in this way to get
         # around the magic __setattr__ method on this class (which is used to
         # store API-assigned settings in the database).
+        self.__dict__['__forks__'] = {}
         self.__dict__['default_settings'] = default_settings
         self.__dict__['_awx_conf_settings'] = self
         self.__dict__['_awx_conf_preload_expires'] = None
@@ -281,6 +257,26 @@ class SettingsWrapper(UserSettingsHolder):
         self.__dict__['cache'] = EncryptedCacheProxy(cache, registry)
         self.__dict__['registry'] = registry
 
+        # record the current pid so we compare it post-fork for
+        # processes like the dispatcher and callback receiver
+        self.__dict__['pid'] = os.getpid()
+
+    def __clean_on_fork__(self):
+        pid = os.getpid()
+        # if the current pid does *not* match the value on self, it means
+        # that value was copied on fork, and we're now in a *forked* process;
+        # the *first* time we enter this code path (on setting access),
+        # forcibly close DB/cache sockets and set a marker so we don't run
+        # this code again _in this process_
+        #
+        if pid != self.__dict__['pid'] and pid not in self.__dict__['__forks__']:
+            self.__dict__['__forks__'][pid] = True
+            # It's important to close these post-fork, because we
+            # don't want the forked processes to inherit the open sockets
+            # for the DB and cache connections (that way lies race conditions)
+            connection.close()
+            django_cache.close()
+
     @cached_property
     def all_supported_settings(self):
         return self.registry.get_registered_settings()
@@ -436,6 +432,7 @@ class SettingsWrapper(UserSettingsHolder):
     def SETTINGS_MODULE(self):
         return self._get_default('SETTINGS_MODULE')
 
+    @cachetools.cached(cache=cachetools.TTLCache(maxsize=2048, ttl=SETTING_MEMORY_TTL))
     def __getattr__(self, name):
         value = empty
         if name in self.all_supported_settings:
@@ -443,22 +440,13 @@ class SettingsWrapper(UserSettingsHolder):
                 value = self._get_local(name)
         if value is not empty:
             return value
-        value = self._get_default(name)
-        # sometimes users specify RabbitMQ passwords that contain
-        # unescaped : and @ characters that confused urlparse, e.g.,
-        # amqp://guest:a@ns:ibl3#@localhost:5672//
-        #
-        # detect these scenarios, and automatically escape the user's
-        # password so it just works
-        if name == 'BROKER_URL':
-            value = normalize_broker_url(value)
-        return value
+        return self._get_default(name)
 
     def _set_local(self, name, value):
         field = self.registry.get_setting_field(name)
         if field.read_only:
             logger.warning('Attempt to set read only setting "%s".', name)
-            raise ImproperlyConfigured('Setting "%s" is read only.'.format(name))
+            raise ImproperlyConfigured('Setting "{}" is read only.'.format(name))
 
         try:
             data = field.to_representation(value)
@@ -489,7 +477,7 @@ class SettingsWrapper(UserSettingsHolder):
         field = self.registry.get_setting_field(name)
         if field.read_only:
             logger.warning('Attempt to delete read only setting "%s".', name)
-            raise ImproperlyConfigured('Setting "%s" is read only.'.format(name))
+            raise ImproperlyConfigured('Setting "{}" is read only.'.format(name))
         for setting in Setting.objects.filter(key=name, user__isnull=True):
             setting.delete()
             # pre_delete handler will delete from cache.

awx/conf/tests/functional/test_api.py

@@ -325,17 +325,3 @@ def test_setting_singleton_delete_no_read_only_fields(api_request, dummy_setting
         )
         assert response.data['FOO_BAR'] == 23
 
-
-@pytest.mark.django_db
-def test_setting_logging_test(api_request):
-    with mock.patch('awx.conf.views.AWXProxyHandler.perform_test') as mock_func:
-        api_request(
-            'post',
-            reverse('api:setting_logging_test'),
-            data={'LOG_AGGREGATOR_HOST': 'http://foobar', 'LOG_AGGREGATOR_TYPE': 'logstash'}
-        )
-        call = mock_func.call_args_list[0]
-        args, kwargs = call
-        given_settings = kwargs['custom_settings']
-        assert given_settings.LOG_AGGREGATOR_HOST == 'http://foobar'
-        assert given_settings.LOG_AGGREGATOR_TYPE == 'logstash'

awx/conf/tests/unit/test_registry.py

@@ -29,9 +29,10 @@ def reg(request):
     # as "defined in a settings file".  This is analogous to manually
     # specifying a setting on the filesystem (e.g., in a local_settings.py in
     # development, or in /etc/tower/conf.d/<something>.py)
-    defaults = request.node.get_marker('defined_in_file')
-    if defaults:
-        settings.configure(**defaults.kwargs)
+    for marker in request.node.own_markers:
+        if marker.name == 'defined_in_file':
+            settings.configure(**marker.kwargs)
+
     settings._wrapped = SettingsWrapper(settings._wrapped,
                                         cache,
                                         registry)

awx/conf/tests/unit/test_settings.py

@@ -41,13 +41,16 @@ def settings(request):
     cache = LocMemCache(str(uuid4()), {})  # make a new random cache each time
     settings = LazySettings()
     registry = SettingsRegistry(settings)
+    defaults = {}
 
     # @pytest.mark.defined_in_file can be used to mark specific setting values
     # as "defined in a settings file".  This is analogous to manually
     # specifying a setting on the filesystem (e.g., in a local_settings.py in
     # development, or in /etc/tower/conf.d/<something>.py)
-    in_file_marker = request.node.get_marker('defined_in_file')
-    defaults = in_file_marker.kwargs if in_file_marker else {}
+    for marker in request.node.own_markers:
+        if marker.name == 'defined_in_file':
+            defaults = marker.kwargs
+
     defaults['DEFAULTS_SNAPSHOT'] = {}
     settings.configure(**defaults)
     settings._wrapped = SettingsWrapper(settings._wrapped,
@@ -63,15 +66,6 @@ def test_unregistered_setting(settings):
     assert settings.cache.get('DEBUG') is None
 
 
-def test_cached_settings_unicode_is_auto_decoded(settings):
-    # https://github.com/linsomniac/python-memcached/issues/79
-    # https://github.com/linsomniac/python-memcached/blob/288c159720eebcdf667727a859ef341f1e908308/memcache.py#L961
-
-    value = 'Iñtërnâtiônàlizætiøn'  # this simulates what python-memcached does on cache.set()
-    settings.cache.set('DEBUG', value)
-    assert settings.cache.get('DEBUG') == 'Iñtërnâtiônàlizætiøn'
-
-
 def test_read_only_setting(settings):
     settings.registry.register(
         'AWX_READ_ONLY',
@@ -251,31 +245,6 @@ def test_setting_from_db(settings, mocker):
         assert settings.cache.get('AWX_SOME_SETTING') == 'FROM_DB'
 
 
-@pytest.mark.parametrize('encrypted', (True, False))
-def test_setting_from_db_with_unicode(settings, mocker, encrypted):
-    settings.registry.register(
-        'AWX_SOME_SETTING',
-        field_class=fields.CharField,
-        category=_('System'),
-        category_slug='system',
-        default='DEFAULT',
-        encrypted=encrypted
-    )
-    # this simulates a bug in python-memcached; see https://github.com/linsomniac/python-memcached/issues/79
-    value = 'Iñtërnâtiônàlizætiøn'
-
-    setting_from_db = mocker.Mock(id=1, key='AWX_SOME_SETTING', value=value)
-    mocks = mocker.Mock(**{
-        'order_by.return_value': mocker.Mock(**{
-            '__iter__': lambda self: iter([setting_from_db]),
-            'first.return_value': setting_from_db
-        }),
-    })
-    with mocker.patch('awx.conf.models.Setting.objects.filter', return_value=mocks):
-        assert settings.AWX_SOME_SETTING == 'Iñtërnâtiônàlizætiøn'
-        assert settings.cache.get('AWX_SOME_SETTING') == 'Iñtërnâtiônàlizætiøn'
-
-
 @pytest.mark.defined_in_file(AWX_SOME_SETTING='DEFAULT')
 def test_read_only_setting_assignment(settings):
     "read-only settings cannot be overwritten"

awx/conf/views.py

@@ -3,15 +3,20 @@
 
 # Python
 import collections
+import logging
+import subprocess
 import sys
+import socket
+from socket import SHUT_RDWR
 
 # Django
+from django.db import connection
 from django.conf import settings
 from django.http import Http404
 from django.utils.translation import ugettext_lazy as _
 
 # Django REST Framework
-from rest_framework.exceptions import PermissionDenied, ValidationError
+from rest_framework.exceptions import PermissionDenied
 from rest_framework.response import Response
 from rest_framework import serializers
 from rest_framework import status
@@ -26,7 +31,6 @@ from awx.api.generics import (
 from awx.api.permissions import IsSuperUser
 from awx.api.versioning import reverse
 from awx.main.utils import camelcase_to_underscore
-from awx.main.utils.handlers import AWXProxyHandler, LoggingConnectivityException
 from awx.main.tasks import handle_setting_changes
 from awx.conf.models import Setting
 from awx.conf.serializers import SettingCategorySerializer, SettingSingletonSerializer
@@ -127,7 +131,8 @@ class SettingSingletonDetail(RetrieveUpdateDestroyAPIView):
                 setting.save(update_fields=['value'])
                 settings_change_list.append(key)
         if settings_change_list:
-            handle_setting_changes.delay(settings_change_list)
+            connection.on_commit(lambda: handle_setting_changes.delay(settings_change_list))
+
 
     def destroy(self, request, *args, **kwargs):
         instance = self.get_object()
@@ -142,7 +147,7 @@ class SettingSingletonDetail(RetrieveUpdateDestroyAPIView):
             setting.delete()
             settings_change_list.append(setting.key)
         if settings_change_list:
-            handle_setting_changes.delay(settings_change_list)
+            connection.on_commit(lambda: handle_setting_changes.delay(settings_change_list))
 
         # When TOWER_URL_BASE is deleted from the API, reset it to the hostname
         # used to make the request as a default.
@@ -161,40 +166,53 @@ class SettingLoggingTest(GenericAPIView):
     filter_backends = []
 
     def post(self, request, *args, **kwargs):
-        defaults = dict()
-        for key in settings_registry.get_registered_settings(category_slug='logging'):
-            try:
-                defaults[key] = settings_registry.get_setting_field(key).get_default()
-            except serializers.SkipField:
-                defaults[key] = None
-        obj = type('Settings', (object,), defaults)()
-        serializer = self.get_serializer(obj, data=request.data)
-        serializer.is_valid(raise_exception=True)
-        # Special validation specific to logging test.
-        errors = {}
-        for key in ['LOG_AGGREGATOR_TYPE', 'LOG_AGGREGATOR_HOST']:
-            if not request.data.get(key, ''):
-                errors[key] = 'This field is required.'
-        if errors:
-            raise ValidationError(errors)
-
-        if request.data.get('LOG_AGGREGATOR_PASSWORD', '').startswith('$encrypted$'):
-            serializer.validated_data['LOG_AGGREGATOR_PASSWORD'] = getattr(
-                settings, 'LOG_AGGREGATOR_PASSWORD', ''
-            )
+        # Error if logging is not enabled
+        enabled = getattr(settings, 'LOG_AGGREGATOR_ENABLED', False)
+        if not enabled:
+            return Response({'error': 'Logging not enabled'}, status=status.HTTP_409_CONFLICT)
+        
+        # Send test message to configured logger based on db settings
+        try:
+            default_logger = settings.LOG_AGGREGATOR_LOGGERS[0]
+            if default_logger != 'awx':
+                default_logger = f'awx.analytics.{default_logger}'
+        except IndexError:
+            default_logger = 'awx'
+        logging.getLogger(default_logger).error('AWX Connection Test Message')
+        
+        hostname = getattr(settings, 'LOG_AGGREGATOR_HOST', None)
+        protocol = getattr(settings, 'LOG_AGGREGATOR_PROTOCOL', None)
 
         try:
-            class MockSettings:
-                pass
-            mock_settings = MockSettings()
-            for k, v in serializer.validated_data.items():
-                setattr(mock_settings, k, v)
-            AWXProxyHandler().perform_test(custom_settings=mock_settings)
-            if mock_settings.LOG_AGGREGATOR_PROTOCOL.upper() == 'UDP':
-                return Response(status=status.HTTP_201_CREATED)
-        except LoggingConnectivityException as e:
-            return Response({'error': str(e)}, status=status.HTTP_500_INTERNAL_SERVER_ERROR)
-        return Response(status=status.HTTP_200_OK)
+            subprocess.check_output(
+                ['rsyslogd', '-N1', '-f', '/var/lib/awx/rsyslog/rsyslog.conf'],
+                stderr=subprocess.STDOUT
+            )
+        except subprocess.CalledProcessError as exc:
+            return Response({'error': exc.output}, status=status.HTTP_400_BAD_REQUEST)
+        
+        # Check to ensure port is open at host
+        if protocol in ['udp', 'tcp']:
+            port = getattr(settings, 'LOG_AGGREGATOR_PORT', None)
+            # Error if port is not set when using UDP/TCP
+            if not port:
+                return Response({'error': 'Port required for ' + protocol}, status=status.HTTP_400_BAD_REQUEST)
+        else:
+            # if http/https by this point, domain is reacheable
+            return Response(status=status.HTTP_202_ACCEPTED)
+
+        if protocol == 'udp':
+            s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+        else:
+            s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+        try:
+            s.settimeout(.5)
+            s.connect((hostname, int(port)))
+            s.shutdown(SHUT_RDWR)
+            s.close()
+            return Response(status=status.HTTP_202_ACCEPTED)
+        except Exception as e:
+            return Response({'error': str(e)}, status=status.HTTP_400_BAD_REQUEST)
 
 
 # Create view functions for all of the class-based views to simplify inclusion

awx/main/access.py

@@ -11,7 +11,6 @@ from functools import reduce
 from django.conf import settings
 from django.db.models import Q, Prefetch
 from django.contrib.auth.models import User
-from django.contrib.contenttypes.models import ContentType
 from django.utils.translation import ugettext_lazy as _
 from django.core.exceptions import ObjectDoesNotExist
 
@@ -30,9 +29,9 @@ from awx.main.utils import (
 )
 from awx.main.models import (
     ActivityStream, AdHocCommand, AdHocCommandEvent, Credential, CredentialType,
-    CredentialInputSource, CustomInventoryScript, Group, Host, Instance, InstanceGroup,
-    Inventory, InventorySource, InventoryUpdate, InventoryUpdateEvent, Job, JobEvent,
-    JobHostSummary, JobLaunchConfig, JobTemplate, Label, Notification,
+    CredentialInputSource, CustomInventoryScript, ExecutionEnvironment, Group, Host, Instance,
+    InstanceGroup, Inventory, InventorySource, InventoryUpdate, InventoryUpdateEvent, Job,
+    JobEvent, JobHostSummary, JobLaunchConfig, JobTemplate, Label, Notification,
     NotificationTemplate, Organization, Project, ProjectUpdate,
     ProjectUpdateEvent, Role, Schedule, SystemJob, SystemJobEvent,
     SystemJobTemplate, Team, UnifiedJob, UnifiedJobTemplate, WorkflowJob,
@@ -307,7 +306,7 @@ class BaseAccess(object):
 
         return True  # User has access to both, permission check passed
 
-    def check_license(self, add_host_name=None, feature=None, check_expiration=True):
+    def check_license(self, add_host_name=None, feature=None, check_expiration=True, quiet=False):
         validation_info = get_licenser().validate()
         if validation_info.get('license_type', 'UNLICENSED') == 'open':
             return
@@ -317,8 +316,10 @@ class BaseAccess(object):
             validation_info['time_remaining'] = 99999999
             validation_info['grace_period_remaining'] = 99999999
 
-        report_violation = lambda message: logger.error(message)
-
+        if quiet:
+            report_violation = lambda message: None
+        else:
+            report_violation = lambda message: logger.warning(message)
         if (
             validation_info.get('trial', False) is True or
             validation_info['instance_count'] == 10  # basic 10 license
@@ -332,14 +333,14 @@ class BaseAccess(object):
             report_violation(_("License has expired."))
 
         free_instances = validation_info.get('free_instances', 0)
-        available_instances = validation_info.get('available_instances', 0)
+        instance_count = validation_info.get('instance_count', 0)
 
         if add_host_name:
             host_exists = Host.objects.filter(name=add_host_name).exists()
             if not host_exists and free_instances == 0:
-                report_violation(_("License count of %s instances has been reached.") % available_instances)
+                report_violation(_("License count of %s instances has been reached.") % instance_count)
             elif not host_exists and free_instances < 0:
-                report_violation(_("License count of %s instances has been exceeded.") % available_instances)
+                report_violation(_("License count of %s instances has been exceeded.") % instance_count)
         elif not add_host_name and free_instances < 0:
             report_violation(_("Host count exceeds available instances."))
 
@@ -403,14 +404,6 @@ class BaseAccess(object):
                 # Cannot copy manual project without errors
                 user_capabilities[display_method] = False
                 continue
-            elif display_method in ['start', 'schedule'] and isinstance(obj, Group):  # TODO: remove in 3.3
-                try:
-                    if obj.deprecated_inventory_source and not obj.deprecated_inventory_source._can_update():
-                        user_capabilities[display_method] = False
-                        continue
-                except Group.deprecated_inventory_source.RelatedObjectDoesNotExist:
-                    user_capabilities[display_method] = False
-                    continue
             elif display_method in ['start', 'schedule'] and isinstance(obj, (Project)):
                 if obj.scm_type == '':
                     user_capabilities[display_method] = False
@@ -502,7 +495,7 @@ class NotificationAttachMixin(BaseAccess):
             # due to this special case, we use symmetrical logic with attach permission
             return self._can_attach(notification_template=sub_obj, resource_obj=obj)
         return super(NotificationAttachMixin, self).can_unattach(
-            obj, sub_obj, relationship, relationship, data=data
+            obj, sub_obj, relationship, data=data
         )
 
 
@@ -648,8 +641,8 @@ class UserAccess(BaseAccess):
                 # in these cases only superusers can modify orphan users
                 return False
             return not obj.roles.all().exclude(
-                content_type=ContentType.objects.get_for_model(User)
-            ).filter(ancestors__in=self.user.roles.all()).exists()
+                ancestors__in=self.user.roles.all()
+            ).exists()
         else:
             return self.is_all_org_admin(obj)
 
@@ -784,10 +777,14 @@ class OrganizationAccess(NotificationAttachMixin, BaseAccess):
 
     @check_superuser
     def can_change(self, obj, data):
+        if data and data.get('default_environment'):
+            ee = get_object_from_data('default_environment', ExecutionEnvironment, data)
+            if not self.user.can_access(ExecutionEnvironment, 'read', ee):
+                return False
+
         return self.user in obj.admin_role
 
     def can_delete(self, obj):
-        self.check_license(check_expiration=False)
         is_change_possible = self.can_change(obj, None)
         if not is_change_possible:
             return False
@@ -907,7 +904,7 @@ class HostAccess(BaseAccess):
     model = Host
     select_related = ('created_by', 'modified_by', 'inventory',
                       'last_job__job_template', 'last_job_host_summary__job',)
-    prefetch_related = ('groups',)
+    prefetch_related = ('groups', 'inventory_sources')
 
     def filtered_queryset(self):
         return self.model.objects.filter(inventory__in=Inventory.accessible_pk_qs(self.user, 'read_role'))
@@ -1111,11 +1108,6 @@ class CredentialTypeAccess(BaseAccess):
     def can_use(self, obj):
         return True
 
-    def get_method_capability(self, method, obj, parent_obj):
-        if obj.managed_by_tower:
-            return False
-        return super(CredentialTypeAccess, self).get_method_capability(method, obj, parent_obj)
-
     def filtered_queryset(self):
         return self.model.objects.all()
 
@@ -1190,6 +1182,8 @@ class CredentialAccess(BaseAccess):
     def get_user_capabilities(self, obj, **kwargs):
         user_capabilities = super(CredentialAccess, self).get_user_capabilities(obj, **kwargs)
         user_capabilities['use'] = self.can_use(obj)
+        if getattr(obj, 'managed_by_tower', False) is True:
+            user_capabilities['edit'] = user_capabilities['delete'] = False
         return user_capabilities
 
 
@@ -1319,6 +1313,54 @@ class TeamAccess(BaseAccess):
                                                     *args, **kwargs)
 
 
+class ExecutionEnvironmentAccess(BaseAccess):
+    """
+    I can see an execution environment when:
+     - I'm a superuser
+     - I'm a member of the same organization
+     - it is a global ExecutionEnvironment
+    I can create/change an execution environment when:
+     - I'm a superuser
+     - I'm an admin for the organization(s)
+    """
+
+    model = ExecutionEnvironment
+    select_related = ('organization',)
+    prefetch_related = ('organization__admin_role', 'organization__execution_environment_admin_role')
+
+    def filtered_queryset(self):
+        return ExecutionEnvironment.objects.filter(
+            Q(organization__in=Organization.accessible_pk_qs(self.user, 'read_role')) |
+            Q(organization__isnull=True)
+        ).distinct()
+
+    @check_superuser
+    def can_add(self, data):
+        if not data:  # So the browseable API will work
+            return Organization.accessible_objects(self.user, 'execution_environment_admin_role').exists()
+        return self.check_related('organization', Organization, data, mandatory=True,
+                                  role_field='execution_environment_admin_role')
+
+    def can_change(self, obj, data):
+        if obj.managed_by_tower:
+            raise PermissionDenied
+        if self.user.is_superuser:
+            return True
+        if obj and obj.organization_id is None:
+            raise PermissionDenied
+        if self.user not in obj.organization.execution_environment_admin_role:
+            raise PermissionDenied
+        if data and 'organization' in data:
+            new_org = get_object_from_data('organization', Organization, data, obj=obj)
+            if not new_org or self.user not in new_org.execution_environment_admin_role:
+                return False
+        return self.check_related('organization', Organization, data, obj=obj, mandatory=True,
+                                  role_field='execution_environment_admin_role')
+
+    def can_delete(self, obj):
+        return self.can_change(obj, None)
+
+
 class ProjectAccess(NotificationAttachMixin, BaseAccess):
     '''
     I can see projects when:
@@ -1348,14 +1390,29 @@ class ProjectAccess(NotificationAttachMixin, BaseAccess):
     def can_add(self, data):
         if not data:  # So the browseable API will work
             return Organization.accessible_objects(self.user, 'project_admin_role').exists()
-        return (self.check_related('organization', Organization, data, role_field='project_admin_role', mandatory=True) and
-                self.check_related('credential', Credential, data, role_field='use_role'))
+
+        if data.get('default_environment'):
+            ee = get_object_from_data('default_environment', ExecutionEnvironment, data)
+            if not self.user.can_access(ExecutionEnvironment, 'read', ee):
+                return False
+
+        return (
+            self.check_related('organization', Organization, data, role_field='project_admin_role', mandatory=True) and
+            self.check_related('credential', Credential, data, role_field='use_role')
+        )
 
     @check_superuser
     def can_change(self, obj, data):
-        return (self.check_related('organization', Organization, data, obj=obj, role_field='project_admin_role') and
-                self.user in obj.admin_role and
-                self.check_related('credential', Credential, data, obj=obj, role_field='use_role'))
+        if data and data.get('default_environment'):
+            ee = get_object_from_data('default_environment', ExecutionEnvironment, data, obj=obj)
+            if not self.user.can_access(ExecutionEnvironment, 'read', ee):
+                return False
+
+        return (
+            self.check_related('organization', Organization, data, obj=obj, role_field='project_admin_role') and
+            self.user in obj.admin_role and
+            self.check_related('credential', Credential, data, obj=obj, role_field='use_role')
+        )
 
     @check_superuser
     def can_start(self, obj, validate_license=True):
@@ -1409,7 +1466,7 @@ class JobTemplateAccess(NotificationAttachMixin, BaseAccess):
     '''
 
     model = JobTemplate
-    select_related = ('created_by', 'modified_by', 'inventory', 'project',
+    select_related = ('created_by', 'modified_by', 'inventory', 'project', 'organization',
                       'next_schedule',)
     prefetch_related = (
         'instance_groups',
@@ -1433,16 +1490,11 @@ class JobTemplateAccess(NotificationAttachMixin, BaseAccess):
         Users who are able to create deploy jobs can also run normal and check (dry run) jobs.
         '''
         if not data:  # So the browseable API will work
-            return (
-                Project.accessible_objects(self.user, 'use_role').exists() or
-                Inventory.accessible_objects(self.user, 'use_role').exists())
+            return Project.accessible_objects(self.user, 'use_role').exists()
 
         # if reference_obj is provided, determine if it can be copied
         reference_obj = data.get('reference_obj', None)
 
-        if 'survey_enabled' in data and data['survey_enabled']:
-            self.check_license(feature='surveys')
-
         if self.user.is_superuser:
             return True
 
@@ -1465,6 +1517,10 @@ class JobTemplateAccess(NotificationAttachMixin, BaseAccess):
             if self.user not in inventory.use_role:
                 return False
 
+        ee = get_value(ExecutionEnvironment, 'execution_environment')
+        if ee and not self.user.can_access(ExecutionEnvironment, 'read', ee):
+            return False
+
         project = get_value(Project, 'project')
         # If the user has admin access to the project (as an org admin), should
         # be able to proceed without additional checks.
@@ -1502,22 +1558,28 @@ class JobTemplateAccess(NotificationAttachMixin, BaseAccess):
         return self.user in obj.execute_role
 
     def can_change(self, obj, data):
-        data_for_change = data
         if self.user not in obj.admin_role and not self.user.is_superuser:
             return False
-        if data is not None:
-            data = dict(data)
+        if data is None:
+            return True
 
-            if self.changes_are_non_sensitive(obj, data):
-                if 'survey_enabled' in data and obj.survey_enabled != data['survey_enabled'] and data['survey_enabled']:
-                    self.check_license(feature='surveys')
-                return True
+        data = dict(data)
 
-            for required_field in ('inventory', 'project'):
-                required_obj = getattr(obj, required_field, None)
-                if required_field not in data_for_change and required_obj is not None:
-                    data_for_change[required_field] = required_obj.pk
-        return self.can_read(obj) and (self.can_add(data_for_change) if data is not None else True)
+        if self.changes_are_non_sensitive(obj, data):
+            return True
+
+        if data.get('execution_environment'):
+            ee = get_object_from_data('execution_environment', ExecutionEnvironment, data)
+            if not self.user.can_access(ExecutionEnvironment, 'read', ee):
+                return False
+
+        for required_field, cls in (('inventory', Inventory), ('project', Project)):
+            is_mandatory = True
+            if not getattr(obj, '{}_id'.format(required_field)):
+                is_mandatory = False
+            if not self.check_related(required_field, cls, data, obj=obj, role_field='use_role', mandatory=is_mandatory):
+                return False
+        return True
 
     def changes_are_non_sensitive(self, obj, data):
         '''
@@ -1525,8 +1587,7 @@ class JobTemplateAccess(NotificationAttachMixin, BaseAccess):
         thus can be made by a job template administrator which may not have access
         to the any inventory, project, or credentials associated with the template.
         '''
-        # We are white listing fields that can
-        field_whitelist = [
+        allowed_fields = [
             'name', 'description', 'forks', 'limit', 'verbosity', 'extra_vars',
             'job_tags', 'force_handlers', 'skip_tags', 'ask_variables_on_launch',
             'ask_tags_on_launch', 'ask_job_type_on_launch', 'ask_skip_tags_on_launch',
@@ -1541,7 +1602,7 @@ class JobTemplateAccess(NotificationAttachMixin, BaseAccess):
             if k not in [x.name for x in obj._meta.concrete_fields]:
                 continue
             if hasattr(obj, k) and getattr(obj, k) != v:
-                if k not in field_whitelist and v != getattr(obj, '%s_id' % k, None) \
+                if k not in allowed_fields and v != getattr(obj, '%s_id' % k, None) \
                         and not (hasattr(obj, '%s_id' % k) and getattr(obj, '%s_id' % k) is None and v == ''): # Equate '' to None in the case of foreign keys
                     return False
         return True
@@ -1552,9 +1613,9 @@ class JobTemplateAccess(NotificationAttachMixin, BaseAccess):
     @check_superuser
     def can_attach(self, obj, sub_obj, relationship, data, skip_sub_obj_read_check=False):
         if relationship == "instance_groups":
-            if not obj.project.organization:
+            if not obj.organization:
                 return False
-            return self.user.can_access(type(sub_obj), "read", sub_obj) and self.user in obj.project.organization.admin_role
+            return self.user.can_access(type(sub_obj), "read", sub_obj) and self.user in obj.organization.admin_role
         if relationship == 'credentials' and isinstance(sub_obj, Credential):
             return self.user in obj.admin_role and self.user in sub_obj.use_role
         return super(JobTemplateAccess, self).can_attach(
@@ -1585,6 +1646,7 @@ class JobAccess(BaseAccess):
     select_related = ('created_by', 'modified_by', 'job_template', 'inventory',
                       'project', 'project_update',)
     prefetch_related = (
+        'organization',
         'unified_job_template',
         'instance_group',
         'credentials__credential_type',
@@ -1605,42 +1667,19 @@ class JobAccess(BaseAccess):
 
         return qs.filter(
             Q(job_template__in=JobTemplate.accessible_objects(self.user, 'read_role')) |
-            Q(inventory__organization__in=org_access_qs) |
-            Q(project__organization__in=org_access_qs)).distinct()
-
-    def related_orgs(self, obj):
-        orgs = []
-        if obj.inventory and obj.inventory.organization:
-            orgs.append(obj.inventory.organization)
-        if obj.project and obj.project.organization and obj.project.organization not in orgs:
-            orgs.append(obj.project.organization)
-        return orgs
-
-    def org_access(self, obj, role_types=['admin_role']):
-        orgs = self.related_orgs(obj)
-        for org in orgs:
-            for role_type in role_types:
-                role = getattr(org, role_type)
-                if self.user in role:
-                    return True
-        return False
+            Q(organization__in=org_access_qs)).distinct()
 
     def can_add(self, data, validate_license=True):
-        if validate_license:
-            self.check_license()
-
-        if not data:  # So the browseable API will work
-            return True
-        return self.user.is_superuser
+        raise NotImplementedError('Direct job creation not possible in v2 API')
 
     def can_change(self, obj, data):
-        return (obj.status == 'new' and
-                self.can_read(obj) and
-                self.can_add(data, validate_license=False))
+        raise NotImplementedError('Direct job editing not supported in v2 API')
 
     @check_superuser
     def can_delete(self, obj):
-        return self.org_access(obj)
+        if not obj.organization:
+            return False
+        return self.user in obj.organization.admin_role
 
     def can_start(self, obj, validate_license=True):
         if validate_license:
@@ -1660,6 +1699,7 @@ class JobAccess(BaseAccess):
         except JobLaunchConfig.DoesNotExist:
             config = None
 
+        # Standard permissions model
         if obj.job_template and (self.user not in obj.job_template.execute_role):
             return False
 
@@ -1674,24 +1714,17 @@ class JobAccess(BaseAccess):
                 if JobLaunchConfigAccess(self.user).can_add({'reference_obj': config}):
                     return True
 
-        org_access = bool(obj.inventory) and self.user in obj.inventory.organization.inventory_admin_role
-        project_access = obj.project is None or self.user in obj.project.admin_role
-        credential_access = all([self.user in cred.use_role for cred in obj.credentials.all()])
+        # Standard permissions model without job template involved
+        if obj.organization and self.user in obj.organization.execute_role:
+            return True
+        elif not (obj.job_template or obj.organization):
+            raise PermissionDenied(_('Job has been orphaned from its job template and organization.'))
+        elif obj.job_template and config is not None:
+            raise PermissionDenied(_('Job was launched with prompted fields you do not have access to.'))
+        elif obj.job_template and config is None:
+            raise PermissionDenied(_('Job was launched with unknown prompted fields. Organization admin permissions required.'))
 
-        # job can be relaunched if user could make an equivalent JT
-        ret = org_access and credential_access and project_access
-        if not ret and self.save_messages and not self.messages:
-            if not obj.job_template:
-                pretext = _('Job has been orphaned from its job template.')
-            elif config is None:
-                pretext = _('Job was launched with unknown prompted fields.')
-            else:
-                pretext = _('Job was launched with prompted fields.')
-            if credential_access:
-                self.messages['detail'] = '{} {}'.format(pretext, _(' Organization level permissions required.'))
-            else:
-                self.messages['detail'] = '{} {}'.format(pretext, _(' You do not have permission to related resources.'))
-        return ret
+        return False
 
     def get_method_capability(self, method, obj, parent_obj):
         if method == 'start':
@@ -1704,10 +1737,16 @@ class JobAccess(BaseAccess):
     def can_cancel(self, obj):
         if not obj.can_cancel:
             return False
-        # Delete access allows org admins to stop running jobs
-        if self.user == obj.created_by or self.can_delete(obj):
+        # Users may always cancel their own jobs
+        if self.user == obj.created_by:
             return True
-        return obj.job_template is not None and self.user in obj.job_template.admin_role
+        # Users with direct admin to JT may cancel jobs started by anyone
+        if obj.job_template and self.user in obj.job_template.admin_role:
+            return True
+        # If orphaned, allow org JT admins to stop running jobs
+        if not obj.job_template and obj.organization and self.user in obj.organization.job_template_admin_role:
+            return True
+        return False
 
 
 class SystemJobTemplateAccess(BaseAccess):
@@ -1942,11 +1981,11 @@ class WorkflowJobNodeAccess(BaseAccess):
 # TODO: notification attachments?
 class WorkflowJobTemplateAccess(NotificationAttachMixin, BaseAccess):
     '''
-    I can only see/manage Workflow Job Templates if I'm a super user
+    I can see/manage Workflow Job Templates based on object roles
     '''
 
     model = WorkflowJobTemplate
-    select_related = ('created_by', 'modified_by', 'next_schedule',
+    select_related = ('created_by', 'modified_by', 'organization', 'next_schedule',
                       'admin_role', 'execute_role', 'read_role',)
 
     def filtered_queryset(self):
@@ -1964,9 +2003,10 @@ class WorkflowJobTemplateAccess(NotificationAttachMixin, BaseAccess):
         if not data:  # So the browseable API will work
             return Organization.accessible_objects(self.user, 'workflow_admin_role').exists()
 
-        # will check this if surveys are added to WFJT
-        if 'survey_enabled' in data and data['survey_enabled']:
-            self.check_license(feature='surveys')
+        if data.get('execution_environment'):
+            ee = get_object_from_data('execution_environment', ExecutionEnvironment, data)
+            if not self.user.can_access(ExecutionEnvironment, 'read', ee):
+                return False
 
         return (
             self.check_related('organization', Organization, data, role_field='workflow_admin_role', mandatory=True) and
@@ -2017,6 +2057,11 @@ class WorkflowJobTemplateAccess(NotificationAttachMixin, BaseAccess):
         if self.user.is_superuser:
             return True
 
+        if data and data.get('execution_environment'):
+            ee = get_object_from_data('execution_environment', ExecutionEnvironment, data)
+            if not self.user.can_access(ExecutionEnvironment, 'read', ee):
+                return False
+
         return (
             self.check_related('organization', Organization, data, role_field='workflow_admin_role', obj=obj) and
             self.check_related('inventory', Inventory, data, role_field='use_role', obj=obj) and
@@ -2036,7 +2081,7 @@ class WorkflowJobAccess(BaseAccess):
        I can also cancel it if I started it
     '''
     model = WorkflowJob
-    select_related = ('created_by', 'modified_by',)
+    select_related = ('created_by', 'modified_by', 'organization',)
 
     def filtered_queryset(self):
         return WorkflowJob.objects.filter(
@@ -2238,7 +2283,7 @@ class JobEventAccess(BaseAccess):
     '''
 
     model = JobEvent
-    prefetch_related = ('hosts', 'job__job_template', 'host',)
+    prefetch_related = ('job__job_template', 'host',)
 
     def filtered_queryset(self):
         return self.model.objects.filter(
@@ -2330,6 +2375,7 @@ class UnifiedJobTemplateAccess(BaseAccess):
     prefetch_related = (
         'last_job',
         'current_job',
+        'organization',
         'credentials__credential_type',
         Prefetch('labels', queryset=Label.objects.all().order_by('name')),
     )
@@ -2369,6 +2415,7 @@ class UnifiedJobAccess(BaseAccess):
     prefetch_related = (
         'created_by',
         'modified_by',
+        'organization',
         'unified_job_node__workflow_job',
         'unified_job_template',
         'instance_group',
@@ -2399,8 +2446,7 @@ class UnifiedJobAccess(BaseAccess):
             Q(unified_job_template_id__in=UnifiedJobTemplate.accessible_pk_qs(self.user, 'read_role')) |
             Q(inventoryupdate__inventory_source__inventory__id__in=inv_pk_qs) |
             Q(adhoccommand__inventory__id__in=inv_pk_qs) |
-            Q(job__inventory__organization__in=org_auditor_qs) |
-            Q(job__project__organization__in=org_auditor_qs)
+            Q(organization__in=org_auditor_qs)
         )
         return qs
 
@@ -2427,6 +2473,9 @@ class ScheduleAccess(BaseAccess):
     def can_add(self, data):
         if not JobLaunchConfigAccess(self.user).can_add(data):
             return False
+        if not data:
+            return Role.objects.filter(role_field__in=['update_role', 'execute_role'], ancestors__in=self.user.roles.all()).exists()
+
         return self.check_related('unified_job_template', UnifiedJobTemplate, data, role_field='execute_role', mandatory=True)
 
     @check_superuser
@@ -2514,13 +2563,16 @@ class NotificationAccess(BaseAccess):
 
 class LabelAccess(BaseAccess):
     '''
-    I can see/use a Label if I have permission to associated organization
+    I can see/use a Label if I have permission to associated organization, or to a JT that the label is on
     '''
     model = Label
     prefetch_related = ('modified_by', 'created_by', 'organization',)
 
     def filtered_queryset(self):
-        return self.model.objects.all()
+        return self.model.objects.filter(
+            Q(organization__in=Organization.accessible_pk_qs(self.user, 'read_role')) |
+            Q(unifiedjobtemplate_labels__in=UnifiedJobTemplate.accessible_pk_qs(self.user, 'read_role'))
+        )
 
     @check_superuser
     def can_add(self, data):
@@ -2785,6 +2837,9 @@ class WorkflowApprovalTemplateAccess(BaseAccess):
         else:
             return (self.check_related('workflow_approval_template', UnifiedJobTemplate, role_field='admin_role'))
 
+    def can_change(self, obj, data):
+        return self.user.can_access(WorkflowJobTemplate, 'change', obj.workflow_job_template, data={})
+
     def can_start(self, obj, validate_license=False):
         # for copying WFJTs that contain approval nodes
         if self.user.is_superuser:

awx/main/analytics/__init__.py

@@ -1 +1 @@
-from .core import register, gather, ship, table_version  # noqa
+from .core import all_collectors, expensive_collectors, register, gather, ship  # noqa

awx/main/analytics/broadcast_websocket.py

@@ -0,0 +1,169 @@
+import datetime
+import asyncio
+import logging
+import aioredis
+import redis
+import re
+
+from prometheus_client import (
+    generate_latest,
+    Gauge,
+    Counter,
+    Enum,
+    CollectorRegistry,
+    parser,
+)
+
+from django.conf import settings
+
+
+BROADCAST_WEBSOCKET_REDIS_KEY_NAME = 'broadcast_websocket_stats'
+
+
+logger = logging.getLogger('awx.analytics.broadcast_websocket')
+
+
+def dt_to_seconds(dt):
+    return int((dt - datetime.datetime(1970,1,1)).total_seconds())
+
+
+def now_seconds():
+    return dt_to_seconds(datetime.datetime.now())
+
+
+def safe_name(s):
+    # Replace all non alpha-numeric characters with _
+    return re.sub('[^0-9a-zA-Z]+', '_', s)
+
+
+# Second granularity; Per-minute
+class FixedSlidingWindow():
+    def __init__(self, start_time=None):
+        self.buckets = dict()
+        self.start_time = start_time or now_seconds()
+
+    def cleanup(self, now_bucket=None):
+        now_bucket = now_bucket or now_seconds()
+        if self.start_time + 60 < now_bucket:
+            self.start_time = now_bucket - 60
+
+            # Delete old entries
+            for k in list(self.buckets.keys()):
+                if k < self.start_time:
+                    del self.buckets[k]
+
+    def record(self, ts=None):
+        now_bucket = ts or dt_to_seconds(datetime.datetime.now())
+
+        val = self.buckets.get(now_bucket, 0)
+        self.buckets[now_bucket] = val + 1
+
+        self.cleanup(now_bucket)
+
+    def render(self, ts=None):
+        self.cleanup(now_bucket=ts)
+        return sum(self.buckets.values()) or 0
+
+
+class BroadcastWebsocketStatsManager():
+    def __init__(self, event_loop, local_hostname):
+        self._local_hostname = local_hostname
+
+        self._event_loop = event_loop
+        self._stats = dict()
+        self._redis_key = BROADCAST_WEBSOCKET_REDIS_KEY_NAME
+
+    def new_remote_host_stats(self, remote_hostname):
+        self._stats[remote_hostname] = BroadcastWebsocketStats(self._local_hostname,
+                                                               remote_hostname)
+        return self._stats[remote_hostname]
+
+    def delete_remote_host_stats(self, remote_hostname):
+        del self._stats[remote_hostname]
+
+    async def run_loop(self):
+        try:
+            redis_conn = await aioredis.create_redis_pool(settings.BROKER_URL)
+            while True:
+                stats_data_str = ''.join(stat.serialize() for stat in self._stats.values())
+                await redis_conn.set(self._redis_key, stats_data_str)
+
+                await asyncio.sleep(settings.BROADCAST_WEBSOCKET_STATS_POLL_RATE_SECONDS)
+        except Exception as e:
+            logger.warn(e)
+            await asyncio.sleep(settings.BROADCAST_WEBSOCKET_STATS_POLL_RATE_SECONDS)
+            self.start()
+
+    def start(self):
+        self.async_task = self._event_loop.create_task(self.run_loop())
+        return self.async_task
+
+    @classmethod
+    def get_stats_sync(cls):
+        '''
+        Stringified verion of all the stats
+        '''
+        redis_conn = redis.Redis.from_url(settings.BROKER_URL)
+        stats_str = redis_conn.get(BROADCAST_WEBSOCKET_REDIS_KEY_NAME) or b''
+        return parser.text_string_to_metric_families(stats_str.decode('UTF-8'))
+
+
+class BroadcastWebsocketStats():
+    def __init__(self, local_hostname, remote_hostname):
+        self._local_hostname = local_hostname
+        self._remote_hostname = remote_hostname
+        self._registry = CollectorRegistry()
+
+        # TODO: More robust replacement
+        self.name = safe_name(self._local_hostname)
+        self.remote_name = safe_name(self._remote_hostname)
+
+        self._messages_received_total = Counter(f'awx_{self.remote_name}_messages_received_total',
+                                                'Number of messages received, to be forwarded, by the broadcast websocket system',
+                                                registry=self._registry)
+        self._messages_received = Gauge(f'awx_{self.remote_name}_messages_received',
+                                        'Number forwarded messages received by the broadcast websocket system, for the duration of the current connection',
+                                        registry=self._registry)
+        self._connection = Enum(f'awx_{self.remote_name}_connection',
+                                'Websocket broadcast connection',
+                                states=['disconnected', 'connected'],
+                                registry=self._registry)
+        self._connection.state('disconnected')
+        self._connection_start = Gauge(f'awx_{self.remote_name}_connection_start',
+                                       'Time the connection was established',
+                                       registry=self._registry)
+
+        self._messages_received_per_minute = Gauge(f'awx_{self.remote_name}_messages_received_per_minute',
+                                                   'Messages received per minute',
+                                                   registry=self._registry)
+        self._internal_messages_received_per_minute = FixedSlidingWindow()
+
+    def unregister(self):
+        self._registry.unregister(f'awx_{self.remote_name}_messages_received')
+        self._registry.unregister(f'awx_{self.remote_name}_connection')
+
+    def record_message_received(self):
+        self._internal_messages_received_per_minute.record()
+        self._messages_received.inc()
+        self._messages_received_total.inc()
+
+    def record_connection_established(self):
+        self._connection.state('connected')
+        self._connection_start.set_to_current_time()
+        self._messages_received.set(0)
+
+    def record_connection_lost(self):
+        self._connection.state('disconnected')
+
+    def get_connection_duration(self):
+        return (datetime.datetime.now() - self._connection_established_ts).total_seconds()
+
+    def render(self):
+        msgs_per_min = self._internal_messages_received_per_minute.render()
+        self._messages_received_per_minute.set(msgs_per_min)
+
+    def serialize(self):
+        self.render()
+
+        registry_data = generate_latest(self._registry).decode('UTF-8')
+        return registry_data

awx/main/analytics/collectors.py

@@ -1,3 +1,4 @@
+import io
 import os
 import os.path
 import platform
@@ -6,13 +7,14 @@ from django.db import connection
 from django.db.models import Count
 from django.conf import settings
 from django.utils.timezone import now
+from django.utils.translation import ugettext_lazy as _
 
 from awx.conf.license import get_license
 from awx.main.utils import (get_awx_version, get_ansible_version,
                             get_custom_venv_choices, camelcase_to_underscore)
 from awx.main import models
 from django.contrib.sessions.models import Session
-from awx.main.analytics import register, table_version
+from awx.main.analytics import register
 
 '''
 This module is used to define metrics collected by awx.main.analytics.gather()
@@ -31,9 +33,9 @@ data _since_ the last report date - i.e., new data in the last 24 hours)
 '''
 
 
-@register('config', '1.0')
-def config(since):
-    license_info = get_license(show_key=False)
+@register('config', '1.2', description=_('General platform configuration.'))
+def config(since, **kwargs):
+    license_info = get_license()
     install_type = 'traditional'
     if os.environ.get('container') == 'oci':
         install_type = 'openshift'
@@ -52,7 +54,8 @@ def config(since):
         'tower_version': get_awx_version(),
         'ansible_version': get_ansible_version(),
         'license_type': license_info.get('license_type', 'UNLICENSED'),
-        'free_instances': license_info.get('free instances', 0),
+        'free_instances': license_info.get('free_instances', 0),
+        'total_licensed_instances': license_info.get('instance_count', 0),
         'license_expiry': license_info.get('time_remaining', 0),
         'pendo_tracking': settings.PENDO_TRACKING_STATE,
         'authentication_backends': settings.AUTHENTICATION_BACKENDS,
@@ -62,8 +65,8 @@ def config(since):
     }
 
 
-@register('counts', '1.0')
-def counts(since):
+@register('counts', '1.0', description=_('Counts of objects such as organizations, inventories, and projects'))
+def counts(since, **kwargs):
     counts = {}
     for cls in (models.Organization, models.Team, models.User,
                 models.Inventory, models.Credential, models.Project,
@@ -97,8 +100,8 @@ def counts(since):
     return counts
 
     
-@register('org_counts', '1.0')
-def org_counts(since):
+@register('org_counts', '1.0', description=_('Counts of users and teams by organization'))
+def org_counts(since, **kwargs):
     counts = {}
     for org in models.Organization.objects.annotate(num_users=Count('member_role__members', distinct=True), 
                                                     num_teams=Count('teams', distinct=True)).values('name', 'id', 'num_users', 'num_teams'):
@@ -109,8 +112,8 @@ def org_counts(since):
     return counts
     
     
-@register('cred_type_counts', '1.0')
-def cred_type_counts(since):
+@register('cred_type_counts', '1.0', description=_('Counts of credentials by credential type'))
+def cred_type_counts(since, **kwargs):
     counts = {}
     for cred_type in models.CredentialType.objects.annotate(num_credentials=Count(
                                                             'credentials', distinct=True)).values('name', 'id', 'managed_by_tower', 'num_credentials'):  
@@ -121,28 +124,33 @@ def cred_type_counts(since):
     return counts
     
     
-@register('inventory_counts', '1.0')
-def inventory_counts(since):
+@register('inventory_counts', '1.2', description=_('Inventories, their inventory sources, and host counts'))
+def inventory_counts(since, **kwargs):
     counts = {}
     for inv in models.Inventory.objects.filter(kind='').annotate(num_sources=Count('inventory_sources', distinct=True), 
                                                                  num_hosts=Count('hosts', distinct=True)).only('id', 'name', 'kind'):
+        source_list = []
+        for source in inv.inventory_sources.filter().annotate(num_hosts=Count('hosts', distinct=True)).values('name','source', 'num_hosts'):
+            source_list.append(source)
         counts[inv.id] = {'name': inv.name,
                           'kind': inv.kind,
                           'hosts': inv.num_hosts,
-                          'sources': inv.num_sources
+                          'sources': inv.num_sources,
+                          'source_list': source_list
                           }
 
     for smart_inv in models.Inventory.objects.filter(kind='smart'):
         counts[smart_inv.id] = {'name': smart_inv.name,
                                 'kind': smart_inv.kind,
-                                'num_hosts': smart_inv.hosts.count(),
-                                'num_sources': smart_inv.inventory_sources.count()
+                                'hosts': smart_inv.hosts.count(),
+                                'sources': 0,
+                                'source_list': []
                                 }
     return counts
 
 
-@register('projects_by_scm_type', '1.0')
-def projects_by_scm_type(since):
+@register('projects_by_scm_type', '1.0', description=_('Counts of projects by source control type'))
+def projects_by_scm_type(since, **kwargs):
     counts = dict(
         (t[0] or 'manual', 0)
         for t in models.Project.SCM_TYPE_CHOICES
@@ -160,12 +168,14 @@ def _get_isolated_datetime(last_check):
     return last_check
 
 
-@register('instance_info', '1.0')
-def instance_info(since, include_hostnames=False):
+@register('instance_info', '1.0', description=_('Cluster topology and capacity'))
+def instance_info(since, include_hostnames=False, **kwargs):
     info = {}
     instances = models.Instance.objects.values_list('hostname').values(
         'uuid', 'version', 'capacity', 'cpu', 'memory', 'managed_by_policy', 'hostname', 'last_isolated_check', 'enabled')
     for instance in instances:
+        consumed_capacity = sum(x.task_impact for x in models.UnifiedJob.objects.filter(execution_node=instance['hostname'],
+                                status__in=('running', 'waiting')))
         instance_info = {
             'uuid': instance['uuid'],
             'version': instance['version'],
@@ -174,7 +184,9 @@ def instance_info(since, include_hostnames=False):
             'memory': instance['memory'],
             'managed_by_policy': instance['managed_by_policy'],
             'last_isolated_check': _get_isolated_datetime(instance['last_isolated_check']),
-            'enabled': instance['enabled']
+            'enabled': instance['enabled'],
+            'consumed_capacity': consumed_capacity,
+            'remaining_capacity': instance['capacity'] - consumed_capacity
         }
         if include_hostnames is True:
             instance_info['hostname'] = instance['hostname']
@@ -182,8 +194,7 @@ def instance_info(since, include_hostnames=False):
     return info
 
 
-@register('job_counts', '1.0')
-def job_counts(since):
+def job_counts(since, **kwargs):
     counts = {}
     counts['total_jobs'] = models.UnifiedJob.objects.exclude(launch_type='sync').count()
     counts['status'] = dict(models.UnifiedJob.objects.exclude(launch_type='sync').values_list('status').annotate(Count('status')).order_by())
@@ -192,8 +203,7 @@ def job_counts(since):
     return counts
     
     
-@register('job_instance_counts', '1.0')
-def job_instance_counts(since):
+def job_instance_counts(since, **kwargs):
     counts = {}
     job_types = models.UnifiedJob.objects.exclude(launch_type='sync').values_list(
         'execution_node', 'launch_type').annotate(job_launch_type=Count('launch_type')).order_by()
@@ -207,34 +217,79 @@ def job_instance_counts(since):
     return counts
 
 
-@register('query_info', '1.0')
-def query_info(since, collection_type):
+@register('query_info', '1.0', description=_('Metadata about the analytics collected'))
+def query_info(since, collection_type, until, **kwargs):
     query_info = {}
     query_info['last_run'] = str(since)
-    query_info['current_time'] = str(now())
+    query_info['current_time'] = str(until)
     query_info['collection_type'] = collection_type
     return query_info
 
 
-# Copies Job Events from db to a .csv to be shipped
-@table_version('events_table.csv', '1.0')
-@table_version('unified_jobs_table.csv', '1.0')
-@table_version('unified_job_template_table.csv', '1.0')
-def copy_tables(since, full_path):
-    def _copy_table(table, query, path):
-        file_path = os.path.join(path, table + '_table.csv')
-        file = open(file_path, 'w', encoding='utf-8')
-        with connection.cursor() as cursor:
-            cursor.copy_expert(query, file)
-            file.close()
-        return file_path
+'''
+The event table can be *very* large, and we have a 100MB upload limit.
 
+Split large table dumps at dump time into a series of files.
+'''
+MAX_TABLE_SIZE = 200 * 1048576
+
+
+class FileSplitter(io.StringIO):
+    def __init__(self, filespec=None, *args, **kwargs):
+        self.filespec = filespec
+        self.files = []
+        self.currentfile = None
+        self.header = None
+        self.counter = 0
+        self.cycle_file()
+
+    def cycle_file(self):
+        if self.currentfile:
+            self.currentfile.close()
+        self.counter = 0
+        fname = '{}_split{}'.format(self.filespec, len(self.files))
+        self.currentfile = open(fname, 'w', encoding='utf-8')
+        self.files.append(fname)
+        if self.header:
+            self.currentfile.write('{}\n'.format(self.header))
+
+    def file_list(self):
+        self.currentfile.close()
+        # Check for an empty dump
+        if len(self.header) + 1 == self.counter:
+            os.remove(self.files[-1])
+            self.files = self.files[:-1]
+        # If we only have one file, remove the suffix
+        if len(self.files) == 1:
+            os.rename(self.files[0],self.files[0].replace('_split0',''))
+        return self.files
+
+    def write(self, s):
+        if not self.header:
+            self.header = s[0:s.index('\n')]
+        self.counter += self.currentfile.write(s)
+        if self.counter >= MAX_TABLE_SIZE:
+            self.cycle_file()
+
+
+def _copy_table(table, query, path):
+    file_path = os.path.join(path, table + '_table.csv')
+    file = FileSplitter(filespec=file_path)
+    with connection.cursor() as cursor:
+        cursor.copy_expert(query, file)
+    return file.file_list()
+
+
+@register('events_table', '1.2', format='csv', description=_('Automation task records'), expensive=True)
+def events_table(since, full_path, until, **kwargs):
     events_query = '''COPY (SELECT main_jobevent.id, 
                               main_jobevent.created,
+                              main_jobevent.modified,
                               main_jobevent.uuid,
                               main_jobevent.parent_uuid,
                               main_jobevent.event, 
                               main_jobevent.event_data::json->'task_action' AS task_action,
+                              (CASE WHEN event = 'playbook_on_stats' THEN event_data END) as playbook_on_stats,
                               main_jobevent.failed, 
                               main_jobevent.changed, 
                               main_jobevent.playbook, 
@@ -244,16 +299,27 @@ def copy_tables(since, full_path):
                               main_jobevent.job_id, 
                               main_jobevent.host_id, 
                               main_jobevent.host_name
+                              , CAST(main_jobevent.event_data::json->>'start' AS TIMESTAMP WITH TIME ZONE) AS start,
+                              CAST(main_jobevent.event_data::json->>'end' AS TIMESTAMP WITH TIME ZONE) AS end,
+                              main_jobevent.event_data::json->'duration' AS duration,
+                              main_jobevent.event_data::json->'res'->'warnings' AS warnings,
+                              main_jobevent.event_data::json->'res'->'deprecations' AS deprecations
                               FROM main_jobevent 
-                              WHERE main_jobevent.created > {}
-                              ORDER BY main_jobevent.id ASC) TO STDOUT WITH CSV HEADER'''.format(since.strftime("'%Y-%m-%d %H:%M:%S'"))
-    _copy_table(table='events', query=events_query, path=full_path)
+                              WHERE (main_jobevent.created > '{}' AND main_jobevent.created <= '{}')
+                              ORDER BY main_jobevent.id ASC) TO STDOUT WITH CSV HEADER
+                   '''.format(since.isoformat(),until.isoformat())
+    return _copy_table(table='events', query=events_query, path=full_path)
 
+
+@register('unified_jobs_table', '1.2', format='csv', description=_('Data on jobs run'), expensive=True)
+def unified_jobs_table(since, full_path, until, **kwargs):
     unified_job_query = '''COPY (SELECT main_unifiedjob.id,
                                  main_unifiedjob.polymorphic_ctype_id,
                                  django_content_type.model,
-                                 main_project.organization_id,
+                                 main_unifiedjob.organization_id,
                                  main_organization.name as organization_name,
+                                 main_job.inventory_id, 
+                                 main_inventory.name as inventory_name,
                                  main_unifiedjob.created,  
                                  main_unifiedjob.name,  
                                  main_unifiedjob.unified_job_template_id, 
@@ -268,17 +334,24 @@ def copy_tables(since, full_path):
                                  main_unifiedjob.finished, 
                                  main_unifiedjob.elapsed, 
                                  main_unifiedjob.job_explanation, 
-                                 main_unifiedjob.instance_group_id
+                                 main_unifiedjob.instance_group_id,
+                                 main_unifiedjob.installed_collections,
+                                 main_unifiedjob.ansible_version
                                  FROM main_unifiedjob
-                                 JOIN main_job ON main_unifiedjob.id = main_job.unifiedjob_ptr_id
                                  JOIN django_content_type ON main_unifiedjob.polymorphic_ctype_id = django_content_type.id
-                                 JOIN main_project ON main_project.unifiedjobtemplate_ptr_id = main_job.project_id
-                                 JOIN main_organization ON main_organization.id = main_project.organization_id
-                                 WHERE main_unifiedjob.created > {} 
-                                 AND main_unifiedjob.launch_type != 'sync'
-                                 ORDER BY main_unifiedjob.id ASC) TO STDOUT WITH CSV HEADER'''.format(since.strftime("'%Y-%m-%d %H:%M:%S'"))    
-    _copy_table(table='unified_jobs', query=unified_job_query, path=full_path)
+                                 LEFT JOIN main_job ON main_unifiedjob.id = main_job.unifiedjob_ptr_id
+                                 LEFT JOIN main_inventory ON main_job.inventory_id = main_inventory.id
+                                 LEFT JOIN main_organization ON main_organization.id = main_unifiedjob.organization_id
+                                 WHERE ((main_unifiedjob.created > '{0}' AND main_unifiedjob.created <= '{1}')
+                                       OR (main_unifiedjob.finished > '{0}' AND main_unifiedjob.finished <= '{1}'))
+                                       AND main_unifiedjob.launch_type != 'sync'
+                                 ORDER BY main_unifiedjob.id ASC) TO STDOUT WITH CSV HEADER
+                        '''.format(since.isoformat(),until.isoformat())
+    return _copy_table(table='unified_jobs', query=unified_job_query, path=full_path)
 
+
+@register('unified_job_template_table', '1.0', format='csv', description=_('Data on job templates'))
+def unified_job_template_table(since, full_path, **kwargs):
     unified_job_template_query = '''COPY (SELECT main_unifiedjobtemplate.id, 
                                  main_unifiedjobtemplate.polymorphic_ctype_id,
                                  django_content_type.model,
@@ -296,6 +369,73 @@ def copy_tables(since, full_path):
                                  main_unifiedjobtemplate.status 
                                  FROM main_unifiedjobtemplate, django_content_type
                                  WHERE main_unifiedjobtemplate.polymorphic_ctype_id = django_content_type.id
-                                 ORDER BY main_unifiedjobtemplate.id ASC) TO STDOUT WITH CSV HEADER'''.format(since.strftime("'%Y-%m-%d %H:%M:%S'"))    
-    _copy_table(table='unified_job_template', query=unified_job_template_query, path=full_path)
-    return
+                                 ORDER BY main_unifiedjobtemplate.id ASC) TO STDOUT WITH CSV HEADER'''  
+    return _copy_table(table='unified_job_template', query=unified_job_template_query, path=full_path)
+
+
+@register('workflow_job_node_table', '1.0', format='csv', description=_('Data on workflow runs'), expensive=True)
+def workflow_job_node_table(since, full_path, until, **kwargs):
+    workflow_job_node_query = '''COPY (SELECT main_workflowjobnode.id,
+                                 main_workflowjobnode.created,
+                                 main_workflowjobnode.modified, 
+                                 main_workflowjobnode.job_id, 
+                                 main_workflowjobnode.unified_job_template_id, 
+                                 main_workflowjobnode.workflow_job_id, 
+                                 main_workflowjobnode.inventory_id, 
+                                 success_nodes.nodes AS success_nodes,
+                                 failure_nodes.nodes AS failure_nodes,
+                                 always_nodes.nodes AS always_nodes,
+                                 main_workflowjobnode.do_not_run, 
+                                 main_workflowjobnode.all_parents_must_converge
+                                 FROM main_workflowjobnode
+                                 LEFT JOIN (
+                                     SELECT from_workflowjobnode_id, ARRAY_AGG(to_workflowjobnode_id) AS nodes
+                                     FROM main_workflowjobnode_success_nodes
+                                     GROUP BY from_workflowjobnode_id
+                                 ) success_nodes ON main_workflowjobnode.id = success_nodes.from_workflowjobnode_id
+                                 LEFT JOIN (
+                                     SELECT from_workflowjobnode_id, ARRAY_AGG(to_workflowjobnode_id) AS nodes
+                                     FROM main_workflowjobnode_failure_nodes
+                                     GROUP BY from_workflowjobnode_id
+                                 ) failure_nodes ON main_workflowjobnode.id = failure_nodes.from_workflowjobnode_id
+                                 LEFT JOIN (
+                                     SELECT from_workflowjobnode_id, ARRAY_AGG(to_workflowjobnode_id) AS nodes
+                                     FROM main_workflowjobnode_always_nodes
+                                     GROUP BY from_workflowjobnode_id
+                                 ) always_nodes ON main_workflowjobnode.id = always_nodes.from_workflowjobnode_id
+                                 WHERE (main_workflowjobnode.modified > '{}' AND main_workflowjobnode.modified <= '{}')
+                                 ORDER BY main_workflowjobnode.id ASC) TO STDOUT WITH CSV HEADER
+                              '''.format(since.isoformat(),until.isoformat())
+    return _copy_table(table='workflow_job_node', query=workflow_job_node_query, path=full_path)
+
+
+@register('workflow_job_template_node_table', '1.0', format='csv', description=_('Data on workflows'))
+def workflow_job_template_node_table(since, full_path, **kwargs):
+    workflow_job_template_node_query = '''COPY (SELECT main_workflowjobtemplatenode.id, 
+                                 main_workflowjobtemplatenode.created,
+                                 main_workflowjobtemplatenode.modified, 
+                                 main_workflowjobtemplatenode.unified_job_template_id, 
+                                 main_workflowjobtemplatenode.workflow_job_template_id, 
+                                 main_workflowjobtemplatenode.inventory_id, 
+                                 success_nodes.nodes AS success_nodes,
+                                 failure_nodes.nodes AS failure_nodes,
+                                 always_nodes.nodes AS always_nodes,
+                                 main_workflowjobtemplatenode.all_parents_must_converge
+                                 FROM main_workflowjobtemplatenode
+                                 LEFT JOIN (
+                                     SELECT from_workflowjobtemplatenode_id, ARRAY_AGG(to_workflowjobtemplatenode_id) AS nodes
+                                     FROM main_workflowjobtemplatenode_success_nodes
+                                     GROUP BY from_workflowjobtemplatenode_id
+                                 ) success_nodes ON main_workflowjobtemplatenode.id = success_nodes.from_workflowjobtemplatenode_id
+                                 LEFT JOIN (
+                                     SELECT from_workflowjobtemplatenode_id, ARRAY_AGG(to_workflowjobtemplatenode_id) AS nodes
+                                     FROM main_workflowjobtemplatenode_failure_nodes
+                                     GROUP BY from_workflowjobtemplatenode_id
+                                 ) failure_nodes ON main_workflowjobtemplatenode.id = failure_nodes.from_workflowjobtemplatenode_id
+                                 LEFT JOIN (
+                                     SELECT from_workflowjobtemplatenode_id, ARRAY_AGG(to_workflowjobtemplatenode_id) AS nodes
+                                     FROM main_workflowjobtemplatenode_always_nodes
+                                     GROUP BY from_workflowjobtemplatenode_id
+                                 ) always_nodes ON main_workflowjobtemplatenode.id = always_nodes.from_workflowjobtemplatenode_id
+                                 ORDER BY main_workflowjobtemplatenode.id ASC) TO STDOUT WITH CSV HEADER'''   
+    return _copy_table(table='workflow_job_template_node', query=workflow_job_template_node_query, path=full_path)

awx/main/analytics/core.py

@@ -14,20 +14,17 @@ from rest_framework.exceptions import PermissionDenied
 from awx.conf.license import get_license
 from awx.main.models import Job
 from awx.main.access import access_registry
-from awx.main.models.ha import TowerAnalyticsState
+from awx.main.utils import get_awx_http_client_headers, set_environ
 
-
-__all__ = ['register', 'gather', 'ship', 'table_version']
+__all__ = ['register', 'gather', 'ship']
 
 
 logger = logging.getLogger('awx.main.analytics')
 
-manifest = dict()
-
 
 def _valid_license():
     try:
-        if get_license(show_key=False).get('license_type', 'UNLICENSED') == 'open':
+        if get_license().get('license_type', 'UNLICENSED') == 'open':
             return False
         access_registry[Job](None).check_license()
     except PermissionDenied:
@@ -36,110 +33,194 @@ def _valid_license():
     return True
 
 
-def register(key, version):
+def all_collectors():
+    from awx.main.analytics import collectors
+
+    collector_dict = {}
+    module = collectors
+    for name, func in inspect.getmembers(module):
+        if inspect.isfunction(func) and hasattr(func, '__awx_analytics_key__'):
+            key = func.__awx_analytics_key__
+            desc = func.__awx_analytics_description__ or ''
+            version = func.__awx_analytics_version__
+            collector_dict[key] = { 'name': key, 'version': version, 'description': desc}
+    return collector_dict
+
+
+def expensive_collectors():
+    from awx.main.analytics import collectors
+
+    ret = []
+    module = collectors
+    for name, func in inspect.getmembers(module):
+        if inspect.isfunction(func) and hasattr(func, '__awx_analytics_key__') and func.__awx_expensive__:
+            ret.append(func.__awx_analytics_key__)
+    return ret
+
+
+def register(key, version, description=None, format='json', expensive=False):
     """
     A decorator used to register a function as a metric collector.
 
-    Decorated functions should return JSON-serializable objects.
+    Decorated functions should do the following based on format:
+    - json: return JSON-serializable objects.
+    - csv: write CSV data to a filename named 'key'
 
     @register('projects_by_scm_type', 1)
     def projects_by_scm_type():
-        return {'git': 5, 'svn': 1, 'hg': 0}
+        return {'git': 5, 'svn': 1}
     """
 
     def decorate(f):
         f.__awx_analytics_key__ = key
         f.__awx_analytics_version__ = version
+        f.__awx_analytics_description__ = description
+        f.__awx_analytics_type__ = format
+        f.__awx_expensive__ = expensive
         return f
 
     return decorate
 
 
-def table_version(file_name, version):
-
-    global manifest
-    manifest[file_name] = version
-
-    def decorate(f):
-        return f
-
-    return decorate
-
-
-def gather(dest=None, module=None, collection_type='scheduled'):
+def gather(dest=None, module=None, subset = None, since = None, until = now(), collection_type='scheduled'):
     """
     Gather all defined metrics and write them as JSON files in a .tgz
 
     :param dest:    the (optional) absolute path to write a compressed tarball
-    :pararm module: the module to search for registered analytic collector
+    :param module: the module to search for registered analytic collector
                     functions; defaults to awx.main.analytics.collectors
     """
+    def _write_manifest(destdir, manifest):
+        path = os.path.join(destdir, 'manifest.json')
+        with open(path, 'w', encoding='utf-8') as f:
+            try:
+                json.dump(manifest, f)
+            except Exception:
+                f.close()
+                os.remove(f.name)
+                logger.exception("Could not generate manifest.json")
 
-    run_now = now()
-    state = TowerAnalyticsState.get_solo()
-    last_run = state.last_run
-    logger.debug("Last analytics run was: {}".format(last_run))
-    
-    max_interval = now() - timedelta(weeks=4)
-    if last_run < max_interval or not last_run:
-        last_run = max_interval
+    last_run = since or settings.AUTOMATION_ANALYTICS_LAST_GATHER or (now() - timedelta(weeks=4))
+    logger.debug("Last analytics run was: {}".format(settings.AUTOMATION_ANALYTICS_LAST_GATHER))
 
     if _valid_license() is False:
         logger.exception("Invalid License provided, or No License Provided")
-        return "Error: Invalid License provided, or No License Provided"
+        return None
 
     if collection_type != 'dry-run' and not settings.INSIGHTS_TRACKING_STATE:
         logger.error("Automation Analytics not enabled. Use --dry-run to gather locally without sending.")
-        return
+        return None
 
-    if module is None:
+    collector_list = []
+    if module:
+        collector_module = module
+    else:
         from awx.main.analytics import collectors
-        module = collectors
-
+        collector_module = collectors
+    for name, func in inspect.getmembers(collector_module):
+        if (
+            inspect.isfunction(func) and
+            hasattr(func, '__awx_analytics_key__') and
+            (not subset or name in subset)
+        ):
+            collector_list.append((name, func))
 
+    manifest = dict()
     dest = dest or tempfile.mkdtemp(prefix='awx_analytics')
-    for name, func in inspect.getmembers(module):
-        if inspect.isfunction(func) and hasattr(func, '__awx_analytics_key__'):
+    gather_dir = os.path.join(dest, 'stage')
+    os.mkdir(gather_dir, 0o700)
+    num_splits = 1
+    for name, func in collector_list:
+        if func.__awx_analytics_type__ == 'json':
             key = func.__awx_analytics_key__
-            manifest['{}.json'.format(key)] = func.__awx_analytics_version__
-            path = '{}.json'.format(os.path.join(dest, key))
+            path = '{}.json'.format(os.path.join(gather_dir, key))
             with open(path, 'w', encoding='utf-8') as f:
                 try:
-                    if func.__name__ == 'query_info':
-                        json.dump(func(last_run, collection_type=collection_type), f)
-                    else:
-                        json.dump(func(last_run), f)
+                    json.dump(func(last_run, collection_type=collection_type, until=until), f)
+                    manifest['{}.json'.format(key)] = func.__awx_analytics_version__
                 except Exception:
                     logger.exception("Could not generate metric {}.json".format(key))
                     f.close()
                     os.remove(f.name)
-    
-    path = os.path.join(dest, 'manifest.json')
-    with open(path, 'w', encoding='utf-8') as f:
-        try:
-            json.dump(manifest, f)
-        except Exception:
-            logger.exception("Could not generate manifest.json")
-            f.close()
-            os.remove(f.name)
+        elif func.__awx_analytics_type__ == 'csv':
+            key = func.__awx_analytics_key__
+            try:
+                files = func(last_run, full_path=gather_dir, until=until)
+                if files:
+                    manifest['{}.csv'.format(key)] = func.__awx_analytics_version__
+                if len(files) > num_splits:
+                    num_splits = len(files)
+            except Exception:
+                logger.exception("Could not generate metric {}.csv".format(key))
 
+    if not manifest:
+        # No data was collected
+        logger.warning("No data from {} to {}".format(last_run, until))
+        shutil.rmtree(dest)
+        return None
+
+    # Always include config.json if we're using our collectors
+    if 'config.json' not in manifest.keys() and not module:
+        from awx.main.analytics import collectors
+        config = collectors.config
+        path = '{}.json'.format(os.path.join(gather_dir, config.__awx_analytics_key__))
+        with open(path, 'w', encoding='utf-8') as f:
+            try:
+                json.dump(collectors.config(last_run), f)
+                manifest['config.json'] = config.__awx_analytics_version__
+            except Exception:
+                logger.exception("Could not generate metric {}.json".format(key))
+                f.close()
+                os.remove(f.name)
+                shutil.rmtree(dest)
+                return None
+
+    stage_dirs = [gather_dir]
+    if num_splits > 1:
+        for i in range(0, num_splits):
+            split_path = os.path.join(dest, 'split{}'.format(i))
+            os.mkdir(split_path, 0o700)
+            filtered_manifest = {}
+            shutil.copy(os.path.join(gather_dir, 'config.json'), split_path)
+            filtered_manifest['config.json'] = manifest['config.json']
+            suffix = '_split{}'.format(i)
+            for file in os.listdir(gather_dir):
+                if file.endswith(suffix):
+                    old_file = os.path.join(gather_dir, file)
+                    new_filename = file.replace(suffix, '')
+                    new_file = os.path.join(split_path, new_filename)
+                    shutil.move(old_file, new_file)
+                    filtered_manifest[new_filename] = manifest[new_filename]
+            _write_manifest(split_path, filtered_manifest)
+            stage_dirs.append(split_path)
+
+    for item in list(manifest.keys()):
+        if not os.path.exists(os.path.join(gather_dir, item)):
+            manifest.pop(item)
+    _write_manifest(gather_dir, manifest)
+
+    tarfiles = []
     try:
-        collectors.copy_tables(since=last_run, full_path=dest)
+        for i in range(0, len(stage_dirs)):
+            stage_dir = stage_dirs[i]
+            # can't use isoformat() since it has colons, which GNU tar doesn't like
+            tarname = '_'.join([
+                settings.SYSTEM_UUID,
+                until.strftime('%Y-%m-%d-%H%M%S%z'),
+                str(i)
+            ])
+            tgz = shutil.make_archive(
+                os.path.join(os.path.dirname(dest), tarname),
+                'gztar',
+                stage_dir
+            )
+            tarfiles.append(tgz)
     except Exception:
-        logger.exception("Could not copy tables")
-        
-    # can't use isoformat() since it has colons, which GNU tar doesn't like
-    tarname = '_'.join([
-        settings.SYSTEM_UUID,
-        run_now.strftime('%Y-%m-%d-%H%M%S%z')
-    ])
-    tgz = shutil.make_archive(
-        os.path.join(os.path.dirname(dest), tarname),
-        'gztar',
-        dest
-    )
-    shutil.rmtree(dest)
-    return tgz
+        shutil.rmtree(stage_dir, ignore_errors = True)
+        logger.exception("Failed to write analytics archive file")
+    finally:
+        shutil.rmtree(dest, ignore_errors = True)
+    return tarfiles
 
 
 def ship(path):
@@ -149,6 +230,9 @@ def ship(path):
     if not path:
         logger.error('Automation Analytics TAR not found')
         return
+    if not os.path.exists(path):
+        logger.error('Automation Analytics TAR {} not found'.format(path))
+        return
     if "Error:" in str(path):
         return
     try:
@@ -165,18 +249,21 @@ def ship(path):
             return logger.error('REDHAT_PASSWORD is not set')
         with open(path, 'rb') as f:
             files = {'file': (os.path.basename(path), f, settings.INSIGHTS_AGENT_MIME)}
-            response = requests.post(url, 
-                                     files=files,
-                                     verify="/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem",
-                                     auth=(rh_user, rh_password),
-                                     timeout=(31, 31))
-            if response.status_code != 202:
+            s = requests.Session()
+            s.headers = get_awx_http_client_headers()
+            s.headers.pop('Content-Type')
+            with set_environ(**settings.AWX_TASK_ENV):
+                response = s.post(url,
+                                  files=files,
+                                  verify="/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem",
+                                  auth=(rh_user, rh_password),
+                                  headers=s.headers,
+                                  timeout=(31, 31))
+            # Accept 2XX status_codes
+            if response.status_code >= 300:
                 return logger.exception('Upload failed with status {}, {}'.format(response.status_code,
                                                                                   response.text))
-        run_now = now()
-        state = TowerAnalyticsState.get_solo()
-        state.last_run = run_now
-        state.save()
     finally:
         # cleanup tar.gz
-        os.remove(path)
+        if os.path.exists(path):
+            os.remove(path)

awx/main/analytics/metrics.py

@@ -12,7 +12,7 @@ from prometheus_client import (
 from awx.conf.license import get_license
 from awx.main.utils import (get_awx_version, get_ansible_version)
 from awx.main.analytics.collectors import (
-    counts, 
+    counts,
     instance_info,
     job_instance_counts,
     job_counts,
@@ -46,13 +46,15 @@ INSTANCE_MEMORY = Gauge('awx_instance_memory', 'RAM (Kb) on each node in a Tower
 INSTANCE_INFO = Info('awx_instance', 'Info about each node in a Tower system', ['hostname', 'instance_uuid',])
 INSTANCE_LAUNCH_TYPE = Gauge('awx_instance_launch_type_total', 'Type of Job launched', ['node', 'launch_type',])
 INSTANCE_STATUS = Gauge('awx_instance_status_total', 'Status of Job launched', ['node', 'status',])
+INSTANCE_CONSUMED_CAPACITY = Gauge('awx_instance_consumed_capacity', 'Consumed capacity of each node in a Tower system', ['hostname', 'instance_uuid',])
+INSTANCE_REMAINING_CAPACITY = Gauge('awx_instance_remaining_capacity', 'Remaining capacity of each node in a Tower system', ['hostname', 'instance_uuid',])
 
 LICENSE_INSTANCE_TOTAL = Gauge('awx_license_instance_total', 'Total number of managed hosts provided by your license')
 LICENSE_INSTANCE_FREE = Gauge('awx_license_instance_free', 'Number of remaining managed hosts provided by your license')
 
 
 def metrics():
-    license_info = get_license(show_key=False)
+    license_info = get_license()
     SYSTEM_INFO.info({
         'install_uuid': settings.INSTALL_UUID,
         'insights_analytics': str(settings.INSIGHTS_TRACKING_STATE),
@@ -66,7 +68,7 @@ def metrics():
         'external_logger_type': getattr(settings, 'LOG_AGGREGATOR_TYPE', 'None')
     })
 
-    LICENSE_INSTANCE_TOTAL.set(str(license_info.get('available_instances', 0)))
+    LICENSE_INSTANCE_TOTAL.set(str(license_info.get('instance_count', 0)))
     LICENSE_INSTANCE_FREE.set(str(license_info.get('free_instances', 0)))
 
     current_counts = counts(None)
@@ -104,6 +106,8 @@ def metrics():
         INSTANCE_CAPACITY.labels(hostname=hostname, instance_uuid=uuid).set(instance_data[uuid]['capacity'])
         INSTANCE_CPU.labels(hostname=hostname, instance_uuid=uuid).set(instance_data[uuid]['cpu'])
         INSTANCE_MEMORY.labels(hostname=hostname, instance_uuid=uuid).set(instance_data[uuid]['memory'])
+        INSTANCE_CONSUMED_CAPACITY.labels(hostname=hostname, instance_uuid=uuid).set(instance_data[uuid]['consumed_capacity'])
+        INSTANCE_REMAINING_CAPACITY.labels(hostname=hostname, instance_uuid=uuid).set(instance_data[uuid]['remaining_capacity'])
         INSTANCE_INFO.labels(hostname=hostname, instance_uuid=uuid).info({
             'enabled': str(instance_data[uuid]['enabled']),
             'last_isolated_check': getattr(instance_data[uuid], 'last_isolated_check', 'None'),

awx/main/conf.py

@@ -1,8 +1,5 @@
 # Python
-import json
 import logging
-import os
-from distutils.version import LooseVersion as Version
 
 # Django
 from django.utils.translation import ugettext_lazy as _
@@ -13,6 +10,8 @@ from rest_framework.fields import FloatField
 
 # Tower
 from awx.conf import fields, register, register_validate
+from awx.main.models import ExecutionEnvironment
+
 
 logger = logging.getLogger('awx.main.conf')
 
@@ -80,11 +79,11 @@ register(
 )
 
 register(
-    'PROXY_IP_WHITELIST',
+    'PROXY_IP_ALLOWED_LIST',
     field_class=fields.StringListField,
-    label=_('Proxy IP Whitelist'),
+    label=_('Proxy IP Allowed List'),
     help_text=_("If Tower is behind a reverse proxy/load balancer, use this setting "
-                "to whitelist the proxy IP addresses from which Tower should trust "
+                "to configure the proxy IP addresses from which Tower should trust "
                 "custom REMOTE_HOST_HEADERS header values. "
                 "If this setting is an empty list (the default), the headers specified by "
                 "REMOTE_HOST_HEADERS will be trusted unconditionally')"),
@@ -93,22 +92,10 @@ register(
 )
 
 
-def _load_default_license_from_file():
-    try:
-        license_file = os.environ.get('AWX_LICENSE_FILE', '/etc/tower/license')
-        if os.path.exists(license_file):
-            license_data = json.load(open(license_file))
-            logger.debug('Read license data from "%s".', license_file)
-            return license_data
-    except Exception:
-        logger.warning('Could not read license from "%s".', license_file, exc_info=True)
-    return {}
-
-
 register(
     'LICENSE',
     field_class=fields.DictField,
-    default=_load_default_license_from_file,
+    default=lambda: {},
     label=_('License'),
     help_text=_('The license controls which features and functionality are '
                 'enabled. Use /api/v2/config/ to update or change '
@@ -125,7 +112,7 @@ register(
     encrypted=False,
     read_only=False,
     label=_('Red Hat customer username'),
-    help_text=_('This username is used to retrieve license information and to send Automation Analytics'),  # noqa
+    help_text=_('This username is used to send data to Automation Analytics'),
     category=_('System'),
     category_slug='system',
 )
@@ -138,7 +125,33 @@ register(
     encrypted=True,
     read_only=False,
     label=_('Red Hat customer password'),
-    help_text=_('This password is used to retrieve license information and to send Automation Analytics'),  # noqa
+    help_text=_('This password is used to send data to Automation Analytics'),
+    category=_('System'),
+    category_slug='system',
+)
+
+register(
+    'SUBSCRIPTIONS_USERNAME',
+    field_class=fields.CharField,
+    default='',
+    allow_blank=True,
+    encrypted=False,
+    read_only=False,
+    label=_('Red Hat or Satellite username'),
+    help_text=_('This username is used to retrieve subscription and content information'),  # noqa
+    category=_('System'),
+    category_slug='system',
+)
+
+register(
+    'SUBSCRIPTIONS_PASSWORD',
+    field_class=fields.CharField,
+    default='',
+    allow_blank=True,
+    encrypted=True,
+    read_only=False,
+    label=_('Red Hat or Satellite password'),
+    help_text=_('This password is used to retrieve subscription and content information'),  # noqa
     category=_('System'),
     category_slug='system',
 )
@@ -149,7 +162,7 @@ register(
     default='https://example.com',
     schemes=('http', 'https'),
     allow_plain_hostname=True,  # Allow hostname only without TLD.
-    label=_('Automation Analytics upload URL.'),
+    label=_('Automation Analytics upload URL'),
     help_text=_('This setting is used to to configure data collection for the Automation Analytics dashboard'),
     category=_('System'),
     category_slug='system',
@@ -164,6 +177,18 @@ register(
     read_only=True,
 )
 
+register(
+    'DEFAULT_EXECUTION_ENVIRONMENT',
+    field_class=fields.PrimaryKeyRelatedField,
+    allow_null=True,
+    default=None,
+    queryset=ExecutionEnvironment.objects.all(),
+    label=_('Global default execution environment'),
+    help_text=_('.'),
+    category=_('System'),
+    category_slug='system',
+)
+
 register(
     'CUSTOM_VENV_PATHS',
     field_class=fields.StringListPathField,
@@ -241,23 +266,11 @@ register(
     field_class=fields.StringListField,
     required=False,
     label=_('Paths to expose to isolated jobs'),
-    help_text=_('Whitelist of paths that would otherwise be hidden to expose to isolated jobs. Enter one path per line.'),
+    help_text=_('List of paths that would otherwise be hidden to expose to isolated jobs. Enter one path per line.'),
     category=_('Jobs'),
     category_slug='jobs',
 )
 
-register(
-    'AWX_ISOLATED_VERBOSITY',
-    field_class=fields.IntegerField,
-    min_value=0,
-    max_value=5,
-    label=_('Verbosity level for isolated node management tasks'),
-    help_text=_('This can be raised to aid in debugging connection issues for isolated task execution'),
-    category=_('Jobs'),
-    category_slug='jobs',
-    default=0
-)
-
 register(
     'AWX_ISOLATED_CHECK_INTERVAL',
     field_class=fields.IntegerField,
@@ -266,6 +279,7 @@ register(
     help_text=_('The number of seconds to sleep between status checks for jobs running on isolated instances.'),
     category=_('Jobs'),
     category_slug='jobs',
+    unit=_('seconds'),
 )
 
 register(
@@ -277,6 +291,7 @@ register(
                 'This includes the time needed to copy source control files (playbooks) to the isolated instance.'),
     category=_('Jobs'),
     category_slug='jobs',
+    unit=_('seconds'),
 )
 
 register(
@@ -289,6 +304,7 @@ register(
                 'Value should be substantially greater than expected network latency.'),
     category=_('Jobs'),
     category_slug='jobs',
+    unit=_('seconds'),
 )
 
 register(
@@ -436,80 +452,25 @@ register(
 )
 
 register(
-    'PRIMARY_GALAXY_URL',
-    field_class=fields.URLField,
-    required=False,
-    allow_blank=True,
-    label=_('Primary Galaxy Server URL'),
+    'AWX_SHOW_PLAYBOOK_LINKS',
+    field_class=fields.BooleanField,
+    default=False,
+    label=_('Follow symlinks'),
     help_text=_(
-        'For organizations that run their own Galaxy service, this gives the option to specify a '
-        'host as the primary galaxy server. Requirements will be downloaded from the primary if the '
-        'specific role or collection is available there. If the content is not avilable in the primary, '
-        'or if this field is left blank, it will default to galaxy.ansible.com.'
+        'Follow symbolic links when scanning for playbooks. Be aware that setting this to True can lead '
+        'to infinite recursion if a link points to a parent directory of itself.'
     ),
     category=_('Jobs'),
-    category_slug='jobs'
+    category_slug='jobs',
 )
 
 register(
-    'PRIMARY_GALAXY_USERNAME',
-    field_class=fields.CharField,
-    required=False,
-    allow_blank=True,
-    label=_('Primary Galaxy Server Username'),
-    help_text=_('For using a galaxy server at higher precedence than the public Ansible Galaxy. '
-                'The username to use for basic authentication against the Galaxy instance, '
-                'this is mutually exclusive with PRIMARY_GALAXY_TOKEN.'),
-    category=_('Jobs'),
-    category_slug='jobs'
-)
-
-register(
-    'PRIMARY_GALAXY_PASSWORD',
-    field_class=fields.CharField,
-    encrypted=True,
-    required=False,
-    allow_blank=True,
-    label=_('Primary Galaxy Server Password'),
-    help_text=_('For using a galaxy server at higher precedence than the public Ansible Galaxy. '
-                'The password to use for basic authentication against the Galaxy instance, '
-                'this is mutually exclusive with PRIMARY_GALAXY_TOKEN.'),
-    category=_('Jobs'),
-    category_slug='jobs'
-)
-
-register(
-    'PRIMARY_GALAXY_TOKEN',
-    field_class=fields.CharField,
-    encrypted=True,
-    required=False,
-    allow_blank=True,
-    label=_('Primary Galaxy Server Token'),
-    help_text=_('For using a galaxy server at higher precedence than the public Ansible Galaxy. '
-                'The token to use for connecting with the Galaxy instance, '
-                'this is mutually exclusive with corresponding username and password settings.'),
-    category=_('Jobs'),
-    category_slug='jobs'
-)
-
-register(
-    'PRIMARY_GALAXY_AUTH_URL',
-    field_class=fields.CharField,
-    required=False,
-    allow_blank=True,
-    label=_('Primary Galaxy Authentication URL'),
-    help_text=_('For using a galaxy server at higher precedence than the public Ansible Galaxy. '
-                'The token_endpoint of a Keycloak server.'),
-    category=_('Jobs'),
-    category_slug='jobs'
-)
-
-register(
-    'PUBLIC_GALAXY_ENABLED',
+    'GALAXY_IGNORE_CERTS',
     field_class=fields.BooleanField,
-    default=True,
-    label=_('Allow Access to Public Galaxy'),
-    help_text=_('Allow or deny access to the public Ansible Galaxy during project updates.'),
+    default=False,
+    label=_('Ignore Ansible Galaxy SSL Certificate Verification'),
+    help_text=_('If set to true, certificate validation will not be done when '
+                'installing content from any Galaxy server.'),
     category=_('Jobs'),
     category_slug='jobs'
 )
@@ -565,6 +526,7 @@ register(
                 'timeout should be imposed. A timeout set on an individual job template will override this.'),
     category=_('Jobs'),
     category_slug='jobs',
+    unit=_('seconds'),
 )
 
 register(
@@ -577,6 +539,7 @@ register(
                 'timeout should be imposed. A timeout set on an individual inventory source will override this.'),
     category=_('Jobs'),
     category_slug='jobs',
+    unit=_('seconds'),
 )
 
 register(
@@ -589,6 +552,7 @@ register(
                 'timeout should be imposed. A timeout set on an individual project will override this.'),
     category=_('Jobs'),
     category_slug='jobs',
+    unit=_('seconds'),
 )
 
 register(
@@ -603,6 +567,19 @@ register(
                 'Use a value of 0 to indicate that no timeout should be imposed.'),
     category=_('Jobs'),
     category_slug='jobs',
+    unit=_('seconds'),
+)
+
+register(
+    'MAX_FORKS',
+    field_class=fields.IntegerField,
+    allow_null=False,
+    default=200,
+    label=_('Maximum number of forks per job'),
+    help_text=_('Saving a Job Template with more than this number of forks will result in an error. '
+                'When set to 0, no limit is applied.'),
+    category=_('Jobs'),
+    category_slug='jobs',
 )
 
 register(
@@ -644,7 +621,7 @@ register(
     allow_blank=True,
     default='',
     label=_('Logging Aggregator Username'),
-    help_text=_('Username for external log aggregator (if required).'),
+    help_text=_('Username for external log aggregator (if required; HTTP/s only).'),
     category=_('Logging'),
     category_slug='logging',
     required=False,
@@ -656,7 +633,7 @@ register(
     default='',
     encrypted=True,
     label=_('Logging Aggregator Password/Token'),
-    help_text=_('Password or authentication token for external log aggregator (if required).'),
+    help_text=_('Password or authentication token for external log aggregator (if required; HTTP/s only).'),
     category=_('Logging'),
     category_slug='logging',
     required=False,
@@ -728,6 +705,7 @@ register(
                 'aggregator protocols.'),
     category=_('Logging'),
     category_slug='logging',
+    unit=_('seconds'),
 )
 register(
     'LOG_AGGREGATOR_VERIFY_CERT',
@@ -755,24 +733,61 @@ register(
     category_slug='logging',
 )
 register(
-    'LOG_AGGREGATOR_AUDIT',
+    'LOG_AGGREGATOR_MAX_DISK_USAGE_GB',
+    field_class=fields.IntegerField,
+    default=1,
+    min_value=1,
+    label=_('Maximum disk persistance for external log aggregation (in GB)'),
+    help_text=_('Amount of data to store (in gigabytes) during an outage of '
+                'the external log aggregator (defaults to 1). '
+                'Equivalent to the rsyslogd queue.maxdiskspace setting.'),
+    category=_('Logging'),
+    category_slug='logging',
+)
+register(
+    'LOG_AGGREGATOR_MAX_DISK_USAGE_PATH',
+    field_class=fields.CharField,
+    default='/var/lib/awx',
+    label=_('File system location for rsyslogd disk persistence'),
+    help_text=_('Location to persist logs that should be retried after an outage '
+                'of the external log aggregator (defaults to /var/lib/awx). '
+                'Equivalent to the rsyslogd queue.spoolDirectory setting.'),
+    category=_('Logging'),
+    category_slug='logging',
+)
+register(
+    'LOG_AGGREGATOR_RSYSLOGD_DEBUG',
     field_class=fields.BooleanField,
-    allow_null=True,
     default=False,
-    label=_('Enabled external log aggregation auditing'),
-    help_text=_('When enabled, all external logs emitted by Tower will also be written to /var/log/tower/external.log.  This is an experimental setting intended to be used for debugging external log aggregation issues (and may be subject to change in the future).'),  # noqa
+    label=_('Enable rsyslogd debugging'),
+    help_text=_('Enabled high verbosity debugging for rsyslogd.  '
+                'Useful for debugging connection issues for external log aggregation.'),
     category=_('Logging'),
     category_slug='logging',
 )
 
 
+
 register(
-    'BROKER_DURABILITY',
-    field_class=fields.BooleanField,
-    label=_('Message Durability'),
-    help_text=_('When set (the default), underlying queues will be persisted to disk.  Disable this to enable higher message bus throughput.'),
+    'AUTOMATION_ANALYTICS_LAST_GATHER',
+    field_class=fields.DateTimeField,
+    label=_('Last gather date for Automation Analytics.'),
+    allow_null=True,
+    category=_('System'),
+    category_slug='system'
+)
+
+
+register(
+    'AUTOMATION_ANALYTICS_GATHER_INTERVAL',
+    field_class=fields.IntegerField,
+    label=_('Automation Analytics Gather Interval'),
+    help_text=_('Interval (in seconds) between data gathering.'),
+    default=14400,	# every 4 hours
+    min_value=1800,	# every 30 minutes
     category=_('System'),
     category_slug='system',
+    unit=_('seconds'),
 )
 
 
@@ -794,75 +809,4 @@ def logging_validate(serializer, attrs):
     return attrs
 
 
-def galaxy_validate(serializer, attrs):
-    """Ansible Galaxy config options have mutual exclusivity rules, these rules
-    are enforced here on serializer validation so that users will not be able
-    to save settings which obviously break all project updates.
-    """
-    prefix = 'PRIMARY_GALAXY_'
-
-    from awx.main.constants import GALAXY_SERVER_FIELDS
-    if not any('{}{}'.format(prefix, subfield.upper()) in attrs for subfield in GALAXY_SERVER_FIELDS):
-        return attrs
-
-    def _new_value(setting_name):
-        if setting_name in attrs:
-            return attrs[setting_name]
-        elif not serializer.instance:
-            return ''
-        return getattr(serializer.instance, setting_name, '')
-
-    galaxy_data = {}
-    for subfield in GALAXY_SERVER_FIELDS:
-        galaxy_data[subfield] = _new_value('{}{}'.format(prefix, subfield.upper()))
-    errors = {}
-    if not galaxy_data['url']:
-        for k, v in galaxy_data.items():
-            if v:
-                setting_name = '{}{}'.format(prefix, k.upper())
-                errors.setdefault(setting_name, [])
-                errors[setting_name].append(_(
-                    'Cannot provide field if PRIMARY_GALAXY_URL is not set.'
-                ))
-    for k in GALAXY_SERVER_FIELDS:
-        if galaxy_data[k]:
-            setting_name = '{}{}'.format(prefix, k.upper())
-            if (not serializer.instance) or (not getattr(serializer.instance, setting_name, '')):
-                # new auth is applied, so check if compatible with version
-                from awx.main.utils import get_ansible_version
-                current_version = get_ansible_version()
-                min_version = '2.9'
-                if Version(current_version) < Version(min_version):
-                    errors.setdefault(setting_name, [])
-                    errors[setting_name].append(_(
-                        'Galaxy server settings are not available until Ansible {min_version}, '
-                        'you are running {current_version}.'
-                    ).format(min_version=min_version, current_version=current_version))
-    if (galaxy_data['password'] or galaxy_data['username']) and (galaxy_data['token'] or galaxy_data['auth_url']):
-        for k in ('password', 'username', 'token', 'auth_url'):
-            setting_name = '{}{}'.format(prefix, k.upper())
-            if setting_name in attrs:
-                errors.setdefault(setting_name, [])
-                errors[setting_name].append(_(
-                    'Setting Galaxy token and authentication URL is mutually exclusive with username and password.'
-                ))
-    if bool(galaxy_data['username']) != bool(galaxy_data['password']):
-        msg = _('If authenticating via username and password, both must be provided.')
-        for k in ('username', 'password'):
-            setting_name = '{}{}'.format(prefix, k.upper())
-            errors.setdefault(setting_name, [])
-            errors[setting_name].append(msg)
-    if bool(galaxy_data['token']) != bool(galaxy_data['auth_url']):
-        msg = _('If authenticating via token, both token and authentication URL must be provided.')
-        for k in ('token', 'auth_url'):
-            setting_name = '{}{}'.format(prefix, k.upper())
-            errors.setdefault(setting_name, [])
-            errors[setting_name].append(msg)
-
-    if errors:
-        raise serializers.ValidationError(errors)
-    return attrs
-
-
 register_validate('logging', logging_validate)
-register_validate('jobs', galaxy_validate)

awx/main/constants.py

@@ -10,8 +10,7 @@ __all__ = [
     'ANSI_SGR_PATTERN', 'CAN_CANCEL', 'ACTIVE_STATES', 'STANDARD_INVENTORY_UPDATE_ENV'
 ]
 
-
-CLOUD_PROVIDERS = ('azure_rm', 'ec2', 'gce', 'vmware', 'openstack', 'rhv', 'satellite6', 'cloudforms', 'tower')
+CLOUD_PROVIDERS = ('azure_rm', 'ec2', 'gce', 'vmware', 'openstack', 'rhv', 'satellite6', 'tower')
 SCHEDULEABLE_PROVIDERS = CLOUD_PROVIDERS + ('custom', 'scm',)
 PRIVILEGE_ESCALATION_METHODS = [
     ('sudo', _('Sudo')), ('su', _('Su')), ('pbrun', _('Pbrun')), ('pfexec', _('Pfexec')),
@@ -32,17 +31,17 @@ STANDARD_INVENTORY_UPDATE_ENV = {
 CAN_CANCEL = ('new', 'pending', 'waiting', 'running')
 ACTIVE_STATES = CAN_CANCEL
 CENSOR_VALUE = '************'
-ENV_BLACKLIST = frozenset((
+ENV_BLOCKLIST = frozenset((
     'VIRTUAL_ENV', 'PATH', 'PYTHONPATH', 'PROOT_TMP_DIR', 'JOB_ID',
     'INVENTORY_ID', 'INVENTORY_SOURCE_ID', 'INVENTORY_UPDATE_ID',
     'AD_HOC_COMMAND_ID', 'REST_API_URL', 'REST_API_TOKEN', 'MAX_EVENT_RES',
     'CALLBACK_QUEUE', 'CALLBACK_CONNECTION', 'CACHE',
     'JOB_CALLBACK_DEBUG', 'INVENTORY_HOSTVARS',
-    'AWX_HOST', 'PROJECT_REVISION'
+    'AWX_HOST', 'PROJECT_REVISION', 'SUPERVISOR_WEB_CONFIG_PATH'
 ))
 
 # loggers that may be called in process of emitting a log
-LOGGER_BLACKLIST = (
+LOGGER_BLOCKLIST = (
     'awx.main.utils.handlers',
     'awx.main.utils.formatters',
     'awx.main.utils.filters',
@@ -51,7 +50,3 @@ LOGGER_BLACKLIST = (
     # loggers that may be called getting logging settings
     'awx.conf'
 )
-
-# these correspond to both AWX and Ansible settings to keep naming consistent
-# for instance, settings.PRIMARY_GALAXY_AUTH_URL vs env var ANSIBLE_GALAXY_SERVER_FOO_AUTH_URL
-GALAXY_SERVER_FIELDS = ('url', 'username', 'password', 'token', 'auth_url')

awx/main/consumers.py

@@ -1,97 +1,246 @@
 import json
 import logging
+import time
+import hmac
+import asyncio
 
-from channels import Group
-from channels.auth import channel_session_user_from_http, channel_session_user
-
-from django.utils.encoding import smart_str
-from django.http.cookie import parse_cookie
 from django.core.serializers.json import DjangoJSONEncoder
+from django.conf import settings
+from django.utils.encoding import force_bytes
+from django.contrib.auth.models import User
+
+from channels.generic.websocket import AsyncJsonWebsocketConsumer
+from channels.layers import get_channel_layer
+from channels.db import database_sync_to_async
 
 
 logger = logging.getLogger('awx.main.consumers')
 XRF_KEY = '_auth_user_xrf'
 
 
-def discard_groups(message):
-    if 'groups' in message.channel_session:
-        for group in message.channel_session['groups']:
-            Group(group).discard(message.reply_channel)
+class WebsocketSecretAuthHelper:
+    """
+    Middlewareish for websockets to verify node websocket broadcast interconnect.
+
+    Note: The "ish" is due to the channels routing interface. Routing occurs 
+    _after_ authentication; making it hard to apply this auth to _only_ a subset of
+    websocket endpoints.
+    """
+
+    @classmethod
+    def construct_secret(cls):
+        nonce_serialized = f"{int(time.time())}"
+        payload_dict = {
+            'secret': settings.BROADCAST_WEBSOCKET_SECRET,
+            'nonce': nonce_serialized
+        }
+        payload_serialized = json.dumps(payload_dict)
+
+        secret_serialized = hmac.new(force_bytes(settings.BROADCAST_WEBSOCKET_SECRET),
+                                     msg=force_bytes(payload_serialized),
+                                     digestmod='sha256').hexdigest()
+
+        return 'HMAC-SHA256 {}:{}'.format(nonce_serialized, secret_serialized)
 
 
-@channel_session_user_from_http
-def ws_connect(message):
-    headers = dict(message.content.get('headers', ''))
-    message.reply_channel.send({"accept": True})
-    message.content['method'] = 'FAKE'
-    if message.user.is_authenticated:
-        message.reply_channel.send(
-            {"text": json.dumps({"accept": True, "user": message.user.id})}
-        )
-        # store the valid CSRF token from the cookie so we can compare it later
-        # on ws_receive
-        cookie_token = parse_cookie(
-            smart_str(headers.get(b'cookie'))
-        ).get('csrftoken')
-        if cookie_token:
-            message.channel_session[XRF_KEY] = cookie_token
-    else:
-        logger.error("Request user is not authenticated to use websocket.")
-        message.reply_channel.send({"close": True})
-    return None
+    @classmethod
+    def verify_secret(cls, s, nonce_tolerance=300):
+        try:
+            (prefix, payload) = s.split(' ')
+            if prefix != 'HMAC-SHA256':
+                raise ValueError('Unsupported encryption algorithm')
+            (nonce_parsed, secret_parsed) = payload.split(':')
+        except Exception:
+            raise ValueError("Failed to parse secret")
+
+        try:
+            payload_expected = {
+                'secret': settings.BROADCAST_WEBSOCKET_SECRET,
+                'nonce': nonce_parsed,
+            }
+            payload_serialized = json.dumps(payload_expected)
+        except Exception:
+            raise ValueError("Failed to create hash to compare to secret.")
+
+        secret_serialized = hmac.new(force_bytes(settings.BROADCAST_WEBSOCKET_SECRET),
+                                     msg=force_bytes(payload_serialized),
+                                     digestmod='sha256').hexdigest()
+
+        if secret_serialized != secret_parsed:
+            raise ValueError("Invalid secret")
+
+        # Avoid timing attack and check the nonce after all the heavy lifting
+        now = int(time.time())
+        nonce_parsed = int(nonce_parsed)
+        nonce_diff = now - nonce_parsed
+        if abs(nonce_diff) > nonce_tolerance:
+            logger.warn(f"Potential replay attack or machine(s) time out of sync by {nonce_diff} seconds.")
+            raise ValueError(f"Potential replay attack or machine(s) time out of sync by {nonce_diff} seconds.")
+
+        return True
+
+    @classmethod
+    def is_authorized(cls, scope):
+        secret = ''
+        for k, v in scope['headers']:
+            if k.decode("utf-8") == 'secret':
+                secret = v.decode("utf-8")
+                break
+        WebsocketSecretAuthHelper.verify_secret(secret)
 
 
-@channel_session_user
-def ws_disconnect(message):
-    discard_groups(message)
+class BroadcastConsumer(AsyncJsonWebsocketConsumer):
+
+    async def connect(self):
+        try:
+            WebsocketSecretAuthHelper.is_authorized(self.scope)
+        except Exception:
+            logger.warn(f"client '{self.channel_name}' failed to authorize against the broadcast endpoint.")
+            await self.close()
+            return
+
+        await self.accept()
+        await self.channel_layer.group_add(settings.BROADCAST_WEBSOCKET_GROUP_NAME, self.channel_name)
+        logger.info(f"client '{self.channel_name}' joined the broadcast group.")
+
+    async def disconnect(self, code):
+        logger.info(f"client '{self.channel_name}' disconnected from the broadcast group.")
+        await self.channel_layer.group_discard(settings.BROADCAST_WEBSOCKET_GROUP_NAME, self.channel_name)
+
+    async def internal_message(self, event):
+        await self.send(event['text'])
 
 
-@channel_session_user
-def ws_receive(message):
-    from awx.main.access import consumer_access
-    user = message.user
-    raw_data = message.content['text']
-    data = json.loads(raw_data)
+class EventConsumer(AsyncJsonWebsocketConsumer):
+    async def connect(self):
+        user = self.scope['user']
+        if user and not user.is_anonymous:
+            await self.accept()
+            await self.send_json({"accept": True, "user": user.id})
+            # store the valid CSRF token from the cookie so we can compare it later
+            # on ws_receive
+            cookie_token = self.scope['cookies'].get('csrftoken')
+            if cookie_token:
+                self.scope['session'][XRF_KEY] = cookie_token
+        else:
+            logger.error("Request user is not authenticated to use websocket.")
+            # TODO: Carry over from channels 1 implementation
+            # We should never .accept() the client and close without sending a close message
+            await self.accept()
+            await self.send_json({"close": True})
+            await self.close()
 
-    xrftoken = data.get('xrftoken')
-    if (
-        not xrftoken or
-        XRF_KEY not in message.channel_session or
-        xrftoken != message.channel_session[XRF_KEY]
-    ):
-        logger.error(
-            "access denied to channel, XRF mismatch for {}".format(user.username)
-        )
-        message.reply_channel.send({
-            "text": json.dumps({"error": "access denied to channel"})
-        })
-        return
+    async def disconnect(self, code):
+        current_groups = set(self.scope['session'].pop('groups') if 'groups' in self.scope['session'] else [])
+        for group_name in current_groups:
+            await self.channel_layer.group_discard(
+                group_name,
+                self.channel_name,
+            )
 
-    if 'groups' in data:
-        discard_groups(message)
-        groups = data['groups']
-        current_groups = set(message.channel_session.pop('groups') if 'groups' in message.channel_session else [])
-        for group_name,v in groups.items():
-            if type(v) is list:
-                for oid in v:
-                    name = '{}-{}'.format(group_name, oid)
-                    access_cls = consumer_access(group_name)
-                    if access_cls is not None:
-                        user_access = access_cls(user)
-                        if not user_access.get_queryset().filter(pk=oid).exists():
-                            message.reply_channel.send({"text": json.dumps(
-                                {"error": "access denied to channel {0} for resource id {1}".format(group_name, oid)})})
-                            continue
-                    current_groups.add(name)
-                    Group(name).add(message.reply_channel)
-            else:
-                current_groups.add(group_name)
-                Group(group_name).add(message.reply_channel)
-        message.channel_session['groups'] = list(current_groups)
+    @database_sync_to_async
+    def user_can_see_object_id(self, user_access, oid):
+        # At this point user is a channels.auth.UserLazyObject object
+        # This causes problems with our generic role permissions checking.
+        # Specifically, type(user) != User
+        # Therefore, get the "real" User objects from the database before
+        # calling the access permission methods
+        user_access.user = User.objects.get(id=user_access.user.id)
+        res = user_access.get_queryset().filter(pk=oid).exists()
+        return res
+
+    async def receive_json(self, data):
+        from awx.main.access import consumer_access
+        user = self.scope['user']
+        xrftoken = data.get('xrftoken')
+        if (
+            not xrftoken or
+            XRF_KEY not in self.scope["session"] or
+            xrftoken != self.scope["session"][XRF_KEY]
+        ):
+            logger.error(f"access denied to channel, XRF mismatch for {user.username}")
+            await self.send_json({"error": "access denied to channel"})
+            return
+
+        if 'groups' in data:
+            groups = data['groups']
+            new_groups = set()
+            current_groups = set(self.scope['session'].pop('groups') if 'groups' in self.scope['session'] else [])
+            for group_name,v in groups.items():
+                if type(v) is list:
+                    for oid in v:
+                        name = '{}-{}'.format(group_name, oid)
+                        access_cls = consumer_access(group_name)
+                        if access_cls is not None:
+                            user_access = access_cls(user)
+                            if not await self.user_can_see_object_id(user_access, oid):
+                                await self.send_json({"error": "access denied to channel {0} for resource id {1}".format(group_name, oid)})
+                                continue
+                        new_groups.add(name)
+                else:
+                    await self.send_json({"error": "access denied to channel"})
+                    logger.error(f"groups must be a list, not {groups}")
+                    return
+
+            old_groups = current_groups - new_groups
+            for group_name in old_groups:
+                await self.channel_layer.group_discard(
+                    group_name,
+                    self.channel_name,
+                )
+
+            new_groups_exclusive = new_groups - current_groups
+            for group_name in new_groups_exclusive:
+                await self.channel_layer.group_add(
+                    group_name,
+                    self.channel_name
+                )
+            self.scope['session']['groups'] = new_groups
+            await self.send_json({
+                "groups_current": list(new_groups),
+                "groups_left": list(old_groups),
+                "groups_joined": list(new_groups_exclusive)
+            })
+
+    async def internal_message(self, event):
+        await self.send(event['text'])
+
+
+def run_sync(func):
+    event_loop = asyncio.new_event_loop()
+    event_loop.run_until_complete(func)
+    event_loop.close()
+
+
+def _dump_payload(payload):
+    try:
+        return json.dumps(payload, cls=DjangoJSONEncoder)
+    except ValueError:
+        logger.error("Invalid payload to emit")
+        return None
 
 
 def emit_channel_notification(group, payload):
-    try:
-        Group(group).send({"text": json.dumps(payload, cls=DjangoJSONEncoder)})
-    except ValueError:
-        logger.error("Invalid payload emitting channel {} on topic: {}".format(group, payload))
+    from awx.main.wsbroadcast import wrap_broadcast_msg # noqa
+
+    payload_dumped = _dump_payload(payload)
+    if payload_dumped is None:
+        return
+
+    channel_layer = get_channel_layer()
+
+    run_sync(channel_layer.group_send(
+        group,
+        {
+            "type": "internal.message",
+            "text": payload_dumped
+        },
+    ))
+
+    run_sync(channel_layer.group_send(
+        settings.BROADCAST_WEBSOCKET_GROUP_NAME,
+        {
+            "type": "internal.message",
+            "text": wrap_broadcast_msg(group, payload_dumped),
+        },
+    ))

awx/main/credential_plugins/aim.py

@@ -1,15 +1,10 @@
-from .plugin import CredentialPlugin
+from .plugin import CredentialPlugin, CertFiles, raise_for_status
 
 from urllib.parse import quote, urlencode, urljoin
 
 from django.utils.translation import ugettext_lazy as _
 import requests
 
-# AWX
-from awx.main.utils import (
-    create_temporary_fifo,
-)
-
 aim_inputs = {
     'fields': [{
         'id': 'url',
@@ -43,7 +38,7 @@ aim_inputs = {
         'id': 'object_query',
         'label': _('Object Query'),
         'type': 'string',
-        'help_text': _('Lookup query for the object. Ex: "Safe=TestSafe;Object=testAccountName123"'),
+        'help_text': _('Lookup query for the object. Ex: Safe=TestSafe;Object=testAccountName123'),
     }, {
         'id': 'object_query_format',
         'label': _('Object Query Format'),
@@ -81,22 +76,15 @@ def aim_backend(**kwargs):
     request_qs = '?' + urlencode(query_params, quote_via=quote)
     request_url = urljoin(url, '/'.join(['AIMWebService', 'api', 'Accounts']))
 
-    cert = None
-    if client_cert and client_key:
-        cert = (
-            create_temporary_fifo(client_cert.encode()),
-            create_temporary_fifo(client_key.encode())
+    with CertFiles(client_cert, client_key) as cert:
+        res = requests.get(
+            request_url + request_qs,
+            timeout=30,
+            cert=cert,
+            verify=verify,
+            allow_redirects=False,
         )
-    elif client_cert:
-        cert = create_temporary_fifo(client_cert.encode())
-
-    res = requests.get(
-        request_url + request_qs,
-        timeout=30,
-        cert=cert,
-        verify=verify,
-    )
-    res.raise_for_status()
+    raise_for_status(res)
     return res.json()['Content']
 
 

awx/main/credential_plugins/azure_kv.py

@@ -3,6 +3,16 @@ from .plugin import CredentialPlugin
 from django.utils.translation import ugettext_lazy as _
 from azure.keyvault import KeyVaultClient, KeyVaultAuthentication
 from azure.common.credentials import ServicePrincipalCredentials
+from msrestazure import azure_cloud
+
+
+# https://github.com/Azure/msrestazure-for-python/blob/master/msrestazure/azure_cloud.py
+clouds = [
+    vars(azure_cloud)[n]
+    for n in dir(azure_cloud)
+    if n.startswith("AZURE_") and n.endswith("_CLOUD")
+]
+default_cloud = vars(azure_cloud)["AZURE_PUBLIC_CLOUD"]
 
 
 azure_keyvault_inputs = {
@@ -24,6 +34,12 @@ azure_keyvault_inputs = {
         'id': 'tenant',
         'label': _('Tenant ID'),
         'type': 'string'
+    }, {
+        'id': 'cloud_name',
+        'label': _('Cloud Environment'),
+        'help_text': _('Specify which azure cloud environment to use.'),
+        'choices': list(set([default_cloud.name] + [c.name for c in clouds])),
+        'default': default_cloud.name
     }],
     'metadata': [{
         'id': 'secret_field',
@@ -42,6 +58,7 @@ azure_keyvault_inputs = {
 
 def azure_keyvault_backend(**kwargs):
     url = kwargs['url']
+    [cloud] = [c for c in clouds if c.name == kwargs.get('cloud_name', default_cloud.name)]
 
     def auth_callback(server, resource, scope):
         credentials = ServicePrincipalCredentials(
@@ -49,7 +66,7 @@ def azure_keyvault_backend(**kwargs):
             client_id = kwargs['client'],
             secret = kwargs['secret'],
             tenant = kwargs['tenant'],
-            resource = "https://vault.azure.net",
+            resource = f"https://{cloud.suffixes.keyvault_dns.split('.', 1).pop()}",
         )
         token = credentials.token
         return token['token_type'], token['access_token']

awx/main/credential_plugins/centrify_vault.py

@@ -0,0 +1,142 @@
+from .plugin import CredentialPlugin, raise_for_status
+from django.utils.translation import ugettext_lazy as _
+from urllib.parse import urljoin
+import requests
+pas_inputs = {
+    'fields': [{
+        'id': 'url',
+        'label': _('Centrify Tenant URL'),
+        'type': 'string',
+        'help_text': _('Centrify Tenant URL'),
+        'format': 'url',
+    }, {
+        'id':'client_id',
+        'label':_('Centrify API User'),
+        'type':'string',
+        'help_text': _('Centrify API User, having necessary permissions as mentioned in support doc'),
+
+    }, {
+        'id':'client_password',
+        'label':_('Centrify API Password'),
+        'type':'string',
+        'help_text': _('Password of Centrify API User with necessary permissions'),
+        'secret':True,
+    },{
+        'id':'oauth_application_id',
+        'label':_('OAuth2 Application ID'),
+        'type':'string',
+        'help_text': _('Application ID of the configured OAuth2 Client (defaults to \'awx\')'),
+        'default': 'awx',
+    },{
+        'id':'oauth_scope',
+        'label':_('OAuth2 Scope'),
+        'type':'string',
+        'help_text': _('Scope of the configured OAuth2 Client (defaults to \'awx\')'),
+        'default': 'awx',
+    }],
+    'metadata': [{
+        'id': 'account-name',
+        'label': _('Account Name'),
+        'type': 'string',
+        'help_text': _('Local system account or Domain account name enrolled in Centrify Vault. eg. (root or DOMAIN/Administrator)'),
+    },{
+        'id': 'system-name',
+        'label': _('System Name'),
+        'type': 'string',
+        'help_text': _('Machine Name enrolled with in Centrify Portal'),
+    }],
+    'required': ['url', 'account-name', 'system-name','client_id','client_password'],
+}
+
+
+# generate bearer token to authenticate with PAS portal, Input : Client ID, Client Secret
+def handle_auth(**kwargs):
+    post_data = {
+        "grant_type": "client_credentials", 
+        "scope": kwargs['oauth_scope']
+    }
+    response = requests.post(
+        kwargs['endpoint'],
+        data = post_data,
+        auth = (kwargs['client_id'],kwargs['client_password']), 
+        verify = True,
+        timeout = (5, 30)
+    )
+    raise_for_status(response)
+    try: 
+        return response.json()['access_token']
+    except KeyError:
+        raise RuntimeError('OAuth request to tenant was unsuccessful')
+
+
+# fetch the ID of system with RedRock query, Input : System Name, Account Name
+def get_ID(**kwargs):
+    endpoint = urljoin(kwargs['url'],'/Redrock/query')
+    name=" Name='{0}' and User='{1}'".format(kwargs['system_name'],kwargs['acc_name'])
+    query  = 'Select ID from VaultAccount where {0}'.format(name)
+    post_headers = {
+        "Authorization": "Bearer " + kwargs['access_token'], 
+        "X-CENTRIFY-NATIVE-CLIENT":"true"
+    }
+    response = requests.post(
+        endpoint,
+        json = {'Script': query},
+        headers = post_headers,
+        verify = True,
+        timeout = (5, 30)
+    )
+    raise_for_status(response)
+    try:
+        result_str = response.json()["Result"]["Results"]
+        return result_str[0]["Row"]["ID"]
+    except (IndexError, KeyError):
+        raise RuntimeError("Error Detected!! Check the Inputs")
+
+
+# CheckOut Password from Centrify Vault, Input : ID
+def get_passwd(**kwargs):
+    endpoint = urljoin(kwargs['url'],'/ServerManage/CheckoutPassword')
+    post_headers = {
+        "Authorization": "Bearer " + kwargs['access_token'], 
+        "X-CENTRIFY-NATIVE-CLIENT":"true"
+    }
+    response = requests.post(
+        endpoint,
+        json = {'ID': kwargs['acc_id']},
+        headers = post_headers,
+        verify = True,
+        timeout = (5, 30)
+    )
+    raise_for_status(response)
+    try:
+        return response.json()["Result"]["Password"]
+    except KeyError:
+        raise RuntimeError("Password Not Found")     
+
+
+def centrify_backend(**kwargs):
+    url = kwargs.get('url')
+    acc_name = kwargs.get('account-name')
+    system_name = kwargs.get('system-name')
+    client_id = kwargs.get('client_id')
+    client_password = kwargs.get('client_password')
+    app_id = kwargs.get('oauth_application_id', 'awx')
+    endpoint = urljoin(url, f'/oauth2/token/{app_id}')
+    endpoint = {
+        'endpoint': endpoint,
+        'client_id': client_id,
+        'client_password': client_password,
+        'oauth_scope': kwargs.get('oauth_scope', 'awx')
+    }
+    token = handle_auth(**endpoint)
+    get_id_args = {'system_name':system_name,'acc_name':acc_name,'url':url,'access_token':token}
+    acc_id = get_ID(**get_id_args)
+    get_pwd_args = {'url':url,'acc_id':acc_id,'access_token':token}
+    return get_passwd(**get_pwd_args)
+
+
+centrify_plugin = CredentialPlugin(
+    'Centrify Vault Credential Provider Lookup',
+    inputs=pas_inputs,
+    backend=centrify_backend
+)

awx/main/credential_plugins/conjur.py

@@ -1,16 +1,11 @@
-from .plugin import CredentialPlugin
+from .plugin import CredentialPlugin, CertFiles, raise_for_status
 
 import base64
-from urllib.parse import urljoin, quote_plus
+from urllib.parse import urljoin, quote
 
 from django.utils.translation import ugettext_lazy as _
 import requests
 
-# AWX
-from awx.main.utils import (
-    create_temporary_fifo,
-)
-
 
 conjur_inputs = {
     'fields': [{
@@ -55,32 +50,32 @@ conjur_inputs = {
 def conjur_backend(**kwargs):
     url = kwargs['url']
     api_key = kwargs['api_key']
-    account = quote_plus(kwargs['account'])
-    username = quote_plus(kwargs['username'])
-    secret_path = quote_plus(kwargs['secret_path'])
+    account = quote(kwargs['account'], safe='') 
+    username = quote(kwargs['username'], safe='')
+    secret_path = quote(kwargs['secret_path'], safe='')
     version = kwargs.get('secret_version')
     cacert = kwargs.get('cacert', None)
 
     auth_kwargs = {
         'headers': {'Content-Type': 'text/plain'},
-        'data': api_key
+        'data': api_key,
+        'allow_redirects': False,
     }
-    if cacert:
-        auth_kwargs['verify'] = create_temporary_fifo(cacert.encode())
 
-    # https://www.conjur.org/api.html#authentication-authenticate-post
-    resp = requests.post(
-        urljoin(url, '/'.join(['authn', account, username, 'authenticate'])),
-        **auth_kwargs
-    )
-    resp.raise_for_status()
+    with CertFiles(cacert) as cert:
+        # https://www.conjur.org/api.html#authentication-authenticate-post
+        auth_kwargs['verify'] = cert
+        resp = requests.post(
+            urljoin(url, '/'.join(['authn', account, username, 'authenticate'])),
+            **auth_kwargs
+        )
+    raise_for_status(resp)
     token = base64.b64encode(resp.content).decode('utf-8')
 
     lookup_kwargs = {
         'headers': {'Authorization': 'Token token="{}"'.format(token)},
+        'allow_redirects': False,
     }
-    if cacert:
-        lookup_kwargs['verify'] = create_temporary_fifo(cacert.encode())
 
     # https://www.conjur.org/api.html#secrets-retrieve-a-secret-get
     path = urljoin(url, '/'.join([
@@ -92,8 +87,10 @@ def conjur_backend(**kwargs):
     if version:
         path = '?'.join([path, version])
 
-    resp = requests.get(path, timeout=30, **lookup_kwargs)
-    resp.raise_for_status()
+    with CertFiles(cacert) as cert:
+        lookup_kwargs['verify'] = cert
+        resp = requests.get(path, timeout=30, **lookup_kwargs)
+    raise_for_status(resp)
     return resp.text
 
 

awx/main/credential_plugins/hashivault.py

@@ -3,16 +3,11 @@ import os
 import pathlib
 from urllib.parse import urljoin
 
-from .plugin import CredentialPlugin
+from .plugin import CredentialPlugin, CertFiles, raise_for_status
 
 import requests
 from django.utils.translation import ugettext_lazy as _
 
-# AWX
-from awx.main.utils import (
-    create_temporary_fifo,
-)
-
 base_inputs = {
     'fields': [{
         'id': 'url',
@@ -32,14 +27,47 @@ base_inputs = {
         'type': 'string',
         'multiline': True,
         'help_text': _('The CA certificate used to verify the SSL certificate of the Vault server')
-    }],
+    }, {
+        'id': 'role_id',
+        'label': _('AppRole role_id'),
+        'type': 'string',
+        'multiline': False,
+        'help_text': _('The Role ID for AppRole Authentication')
+    }, {
+        'id': 'secret_id',
+        'label': _('AppRole secret_id'),
+        'type': 'string',
+        'multiline': False,
+        'secret': True,
+        'help_text': _('The Secret ID for AppRole Authentication')
+    }, {
+        'id': 'namespace',
+        'label': _('Namespace name (Vault Enterprise only)'),
+        'type': 'string',
+        'multiline': False,
+        'help_text': _('Name of the namespace to use when authenticate and retrieve secrets')
+    }, {
+        'id': 'default_auth_path',
+        'label': _('Path to Approle Auth'),
+        'type': 'string',
+        'multiline': False,
+        'default': 'approle',
+        'help_text': _('The AppRole Authentication path to use if one isn\'t provided in the metadata when linking to an input field. Defaults to \'approle\'')
+    }
+    ],
     'metadata': [{
         'id': 'secret_path',
         'label': _('Path to Secret'),
         'type': 'string',
         'help_text': _('The path to the secret stored in the secret backend e.g, /some/secret/')
+    }, {
+        'id': 'auth_path',
+        'label': _('Path to Auth'),
+        'type': 'string',
+        'multiline': False,
+        'help_text': _('The path where the Authentication method is mounted e.g, approle')
     }],
-    'required': ['url', 'token', 'secret_path'],
+    'required': ['url', 'secret_path'],
 }
 
 hashi_kv_inputs = copy.deepcopy(base_inputs)
@@ -88,8 +116,47 @@ hashi_ssh_inputs['metadata'] = [{
 hashi_ssh_inputs['required'].extend(['public_key', 'role'])
 
 
+def handle_auth(**kwargs):
+    token = None
+
+    if kwargs.get('token'):
+        token = kwargs['token']
+    elif kwargs.get('role_id') and kwargs.get('secret_id'):
+        token = approle_auth(**kwargs)
+    else:
+        raise Exception('Either token or AppRole parameters must be set')
+
+    return token
+
+
+def approle_auth(**kwargs):
+    role_id = kwargs['role_id']
+    secret_id = kwargs['secret_id']
+    # we first try to use the 'auth_path' from the metadata
+    # if not found we try to fetch the 'default_auth_path' from inputs
+    auth_path = kwargs.get('auth_path') or kwargs['default_auth_path']
+
+    url = urljoin(kwargs['url'], 'v1')
+    cacert = kwargs.get('cacert', None)
+
+    request_kwargs = {'timeout': 30}
+    # AppRole Login
+    request_kwargs['json'] = {'role_id': role_id, 'secret_id': secret_id}
+    sess = requests.Session()
+    # Namespace support
+    if kwargs.get('namespace'):
+        sess.headers['X-Vault-Namespace'] = kwargs['namespace']
+    request_url = '/'.join([url, 'auth', auth_path, 'login']).rstrip('/')
+    with CertFiles(cacert) as cert:
+        request_kwargs['verify'] = cert
+        resp = sess.post(request_url, **request_kwargs)
+    resp.raise_for_status()
+    token = resp.json()['auth']['client_token']
+    return token
+
+
 def kv_backend(**kwargs):
-    token = kwargs['token']
+    token = handle_auth(**kwargs)
     url = kwargs['url']
     secret_path = kwargs['secret_path']
     secret_backend = kwargs.get('secret_backend', None)
@@ -97,14 +164,17 @@ def kv_backend(**kwargs):
     cacert = kwargs.get('cacert', None)
     api_version = kwargs['api_version']
 
-    request_kwargs = {'timeout': 30}
-    if cacert:
-        request_kwargs['verify'] = create_temporary_fifo(cacert.encode())
+    request_kwargs = {
+        'timeout': 30,
+        'allow_redirects': False,
+    }
 
     sess = requests.Session()
     sess.headers['Authorization'] = 'Bearer {}'.format(token)
-    # Compatability header for older installs of Hashicorp Vault
+    # Compatibility header for older installs of Hashicorp Vault
     sess.headers['X-Vault-Token'] = token
+    if kwargs.get('namespace'):
+        sess.headers['X-Vault-Namespace'] = kwargs['namespace']
 
     if api_version == 'v2':
         if kwargs.get('secret_version'):
@@ -126,8 +196,10 @@ def kv_backend(**kwargs):
             path_segments = [secret_path]
 
     request_url = urljoin(url, '/'.join(['v1'] + path_segments)).rstrip('/')
-    response = sess.get(request_url, **request_kwargs)
-    response.raise_for_status()
+    with CertFiles(cacert) as cert:
+        request_kwargs['verify'] = cert
+        response = sess.get(request_url, **request_kwargs)
+    raise_for_status(response)
 
     json = response.json()
     if api_version == 'v2':
@@ -144,15 +216,16 @@ def kv_backend(**kwargs):
 
 
 def ssh_backend(**kwargs):
-    token = kwargs['token']
+    token = handle_auth(**kwargs)
     url = urljoin(kwargs['url'], 'v1')
     secret_path = kwargs['secret_path']
     role = kwargs['role']
     cacert = kwargs.get('cacert', None)
 
-    request_kwargs = {'timeout': 30}
-    if cacert:
-        request_kwargs['verify'] = create_temporary_fifo(cacert.encode())
+    request_kwargs = {
+        'timeout': 30,
+        'allow_redirects': False,
+    }
 
     request_kwargs['json'] = {'public_key': kwargs['public_key']}
     if kwargs.get('valid_principals'):
@@ -160,13 +233,18 @@ def ssh_backend(**kwargs):
 
     sess = requests.Session()
     sess.headers['Authorization'] = 'Bearer {}'.format(token)
+    if kwargs.get('namespace'):
+        sess.headers['X-Vault-Namespace'] = kwargs['namespace']
     # Compatability header for older installs of Hashicorp Vault
     sess.headers['X-Vault-Token'] = token
     # https://www.vaultproject.io/api/secret/ssh/index.html#sign-ssh-key
     request_url = '/'.join([url, secret_path, 'sign', role]).rstrip('/')
-    resp = sess.post(request_url, **request_kwargs)
 
-    resp.raise_for_status()
+    with CertFiles(cacert) as cert:
+        request_kwargs['verify'] = cert
+        resp = sess.post(request_url, **request_kwargs)
+
+    raise_for_status(resp)
     return resp.json()['data']['signed_key']
 
 

awx/main/credential_plugins/plugin.py

@@ -1,3 +1,55 @@
+import os
+import tempfile
+
 from collections import namedtuple
 
+from requests.exceptions import HTTPError
+
 CredentialPlugin = namedtuple('CredentialPlugin', ['name', 'inputs', 'backend'])
+
+
+def raise_for_status(resp):
+    resp.raise_for_status()
+    if resp.status_code >= 300:
+        exc = HTTPError()
+        setattr(exc, 'response', resp)
+        raise exc
+
+
+class CertFiles():
+    """
+    A context manager used for writing a certificate and (optional) key
+    to $TMPDIR, and cleaning up afterwards.
+
+    This is particularly useful as a shared resource for credential plugins
+    that want to pull cert/key data out of the database and persist it
+    temporarily to the file system so that it can loaded into the openssl
+    certificate chain (generally, for HTTPS requests plugins make via the
+    Python requests library)
+
+    with CertFiles(cert_data, key_data) as cert:
+        # cert is string representing a path to the cert or pemfile
+        # temporarily written to disk
+        requests.post(..., cert=cert)
+    """
+
+    certfile = None
+
+    def __init__(self, cert, key=None):
+        self.cert = cert
+        self.key = key
+
+    def __enter__(self):
+        if not self.cert:
+            return None
+        self.certfile = tempfile.NamedTemporaryFile('wb', delete=False)
+        self.certfile.write(self.cert.encode())
+        if self.key:
+            self.certfile.write(b'\n')
+            self.certfile.write(self.key.encode())
+        self.certfile.flush()
+        return str(self.certfile.name)
+
+    def __exit__(self, *args):
+        if self.certfile and os.path.exists(self.certfile.name):
+            os.remove(self.certfile.name)

awx/main/db/profiled_pg/base.py

@@ -24,7 +24,7 @@ class RecordedQueryLog(object):
         try:
             self.threshold = cache.get('awx-profile-sql-threshold')
         except Exception:
-            # if we can't reach memcached, just assume profiling's off
+            # if we can't reach the cache, just assume profiling's off
             self.threshold = None
 
     def append(self, query):
@@ -64,7 +64,7 @@ class RecordedQueryLog(object):
             if not os.path.isdir(self.dest):
                 os.makedirs(self.dest)
             progname = ' '.join(sys.argv)
-            for match in ('uwsgi', 'dispatcher', 'callback_receiver', 'runworker'):
+            for match in ('uwsgi', 'dispatcher', 'callback_receiver', 'wsbroadcast'):
                 if match in progname:
                     progname = match
                     break
@@ -110,7 +110,7 @@ class RecordedQueryLog(object):
 class DatabaseWrapper(BaseDatabaseWrapper):
     """
     This is a special subclass of Django's postgres DB backend which - based on
-    the value of a special flag in memcached - captures slow queries and
+    the value of a special flag in cache - captures slow queries and
     writes profile and Python stack metadata to the disk.
     """
 
@@ -133,19 +133,19 @@ class DatabaseWrapper(BaseDatabaseWrapper):
         # is the same mechanism used by libraries like the django-debug-toolbar)
         #
         # in _this_ implementation, we represent it as a property which will
-        # check memcache for a special flag to be set (when the flag is set, it
+        # check the cache for a special flag to be set (when the flag is set, it
         # means we should start recording queries because somebody called
         # `awx-manage profile_sql`)
         #
         # it's worth noting that this property is wrapped w/ @memoize because
         # Django references this attribute _constantly_ (in particular, once
-        # per executed query); doing a memcached.get()  _at most_ once per
+        # per executed query); doing a cache.get()  _at most_ once per
         # second is a good enough window to detect when profiling is turned
         # on/off by a system administrator
         try:
             threshold = cache.get('awx-profile-sql-threshold')
         except Exception:
-            # if we can't reach memcached, just assume profiling's off
+            # if we can't reach the cache, just assume profiling's off
             threshold = None
         self.queries_log.threshold = threshold
         return threshold is not None

awx/main/dispatch/__init__.py

@@ -1,5 +1,62 @@
+import psycopg2
+import select
+
+from contextlib import contextmanager
+
 from django.conf import settings
 
 
+NOT_READY = ([], [], [])
+
+
 def get_local_queuename():
     return settings.CLUSTER_HOST_ID
+
+
+class PubSub(object):
+    def __init__(self, conn):
+        assert conn.autocommit, "Connection must be in autocommit mode."
+        self.conn = conn
+
+    def listen(self, channel):
+        with self.conn.cursor() as cur:
+            cur.execute('LISTEN "%s";' % channel)
+
+    def unlisten(self, channel):
+        with self.conn.cursor() as cur:
+            cur.execute('UNLISTEN "%s";' % channel)
+
+    def notify(self, channel, payload):
+        with self.conn.cursor() as cur:
+            cur.execute('SELECT pg_notify(%s, %s);', (channel, payload))
+
+    def events(self, select_timeout=5, yield_timeouts=False):
+        while True:
+            if select.select([self.conn], [], [], select_timeout) == NOT_READY:
+                if yield_timeouts:
+                    yield None
+            else:
+                self.conn.poll()
+                while self.conn.notifies:
+                    yield self.conn.notifies.pop(0)
+
+    def close(self):
+        self.conn.close()
+
+
+@contextmanager
+def pg_bus_conn():
+    conf = settings.DATABASES['default']
+    conn = psycopg2.connect(dbname=conf['NAME'],
+                            host=conf['HOST'],
+                            user=conf['USER'],
+                            password=conf['PASSWORD'],
+                            port=conf['PORT'],
+                            **conf.get("OPTIONS", {}))
+    # Django connection.cursor().connection doesn't have autocommit=True on
+    conn.set_session(autocommit=True)
+    pubsub = PubSub(conn)
+    yield pubsub
+    conn.close()
+
+

awx/main/dispatch/control.py

@@ -1,11 +1,13 @@
 import logging
-import socket
+import uuid
+import json
 
 from django.conf import settings
+import redis
 
 from awx.main.dispatch import get_local_queuename
-from awx.main.dispatch.kombu import Connection
-from kombu import Queue, Exchange, Producer, Consumer
+
+from . import pg_bus_conn
 
 logger = logging.getLogger('awx.main.dispatch')
 
@@ -20,40 +22,43 @@ class Control(object):
             raise RuntimeError('{} must be in {}'.format(service, self.services))
         self.service = service
         self.queuename = host or get_local_queuename()
-        self.queue = Queue(self.queuename, Exchange(self.queuename), routing_key=self.queuename)
-
-    def publish(self, msg, conn, **kwargs):
-        producer = Producer(
-            exchange=self.queue.exchange,
-            channel=conn,
-            routing_key=self.queuename
-        )
-        producer.publish(msg, expiration=5, **kwargs)
 
     def status(self, *args, **kwargs):
-        return self.control_with_reply('status', *args, **kwargs)
+        r = redis.Redis.from_url(settings.BROKER_URL)
+        if self.service == 'dispatcher':
+            stats = r.get(f'awx_{self.service}_statistics') or b''
+            return stats.decode('utf-8')
+        else:
+            workers = []
+            for key in r.keys('awx_callback_receiver_statistics_*'):
+                workers.append(r.get(key).decode('utf-8'))
+            return '\n'.join(workers)
 
     def running(self, *args, **kwargs):
         return self.control_with_reply('running', *args, **kwargs)
 
+    @classmethod
+    def generate_reply_queue_name(cls):
+        return f"reply_to_{str(uuid.uuid4()).replace('-','_')}"
+
     def control_with_reply(self, command, timeout=5):
         logger.warn('checking {} {} for {}'.format(self.service, command, self.queuename))
-        reply_queue = Queue(name="amq.rabbitmq.reply-to")
+        reply_queue = Control.generate_reply_queue_name()
         self.result = None
-        with Connection(settings.BROKER_URL) as conn:
-            with Consumer(conn, reply_queue, callbacks=[self.process_message], no_ack=True):
-                self.publish({'control': command}, conn, reply_to='amq.rabbitmq.reply-to')
-                try:
-                    conn.drain_events(timeout=timeout)
-                except socket.timeout:
-                    logger.error('{} did not reply within {}s'.format(self.service, timeout))
-                    raise
-        return self.result
+
+        with pg_bus_conn() as conn:
+            conn.listen(reply_queue)
+            conn.notify(self.queuename,
+                        json.dumps({'control': command, 'reply_to': reply_queue}))
+
+            for reply in conn.events(select_timeout=timeout, yield_timeouts=True):
+                if reply is None:
+                    logger.error(f'{self.service} did not reply within {timeout}s')
+                    raise RuntimeError(f"{self.service} did not reply within {timeout}s")
+                break
+
+        return json.loads(reply.payload)
 
     def control(self, msg, **kwargs):
-        with Connection(settings.BROKER_URL) as conn:
-            self.publish(msg, conn)
-
-    def process_message(self, body, message):
-        self.result = body
-        message.ack()
+        with pg_bus_conn() as conn:
+            conn.notify(self.queuename, json.dumps(msg))

awx/main/dispatch/kombu.py

@@ -1,42 +0,0 @@
-from amqp.exceptions import PreconditionFailed
-from django.conf import settings
-from kombu.connection import Connection as KombuConnection
-from kombu.transport import pyamqp
-
-import logging
-
-logger = logging.getLogger('awx.main.dispatch')
-
-
-__all__ = ['Connection']
-
-
-class Connection(KombuConnection):
-
-    def __init__(self, *args, **kwargs):
-        super(Connection, self).__init__(*args, **kwargs)
-        class _Channel(pyamqp.Channel):
-
-            def queue_declare(self, queue, *args, **kwargs):
-                kwargs['durable'] = settings.BROKER_DURABILITY
-                try:
-                    return super(_Channel, self).queue_declare(queue, *args, **kwargs)
-                except PreconditionFailed as e:
-                    if "inequivalent arg 'durable'" in getattr(e, 'reply_text', None):
-                        logger.error(
-                            'queue {} durability is not {}, deleting and recreating'.format(
-
-                                queue,
-                                kwargs['durable']
-                            )
-                        )
-                        self.queue_delete(queue)
-                    return super(_Channel, self).queue_declare(queue, *args, **kwargs)
-
-        class _Connection(pyamqp.Connection):
-            Channel = _Channel
-
-        class _Transport(pyamqp.Transport):
-            Connection = _Connection
-
-        self.transport_cls = _Transport

awx/main/dispatch/periodic.py

@@ -0,0 +1,58 @@
+import logging
+import os
+import time
+from multiprocessing import Process
+
+from django.conf import settings
+from django.db import connections
+from schedule import Scheduler
+from django_guid.middleware import GuidMiddleware
+
+from awx.main.dispatch.worker import TaskWorker
+
+logger = logging.getLogger('awx.main.dispatch.periodic')
+
+
+class Scheduler(Scheduler):
+
+    def run_continuously(self):
+        idle_seconds = max(
+            1,
+            min(self.jobs).period.total_seconds() / 2
+        )
+
+        def run():
+            ppid = os.getppid()
+            logger.warn('periodic beat started')
+            while True:
+                if os.getppid() != ppid:
+                    # if the parent PID changes, this process has been orphaned
+                    # via e.g., segfault or sigkill, we should exit too
+                    pid = os.getpid()
+                    logger.warn(f'periodic beat exiting gracefully pid:{pid}')
+                    raise SystemExit()
+                try:
+                    for conn in connections.all():
+                        # If the database connection has a hiccup, re-establish a new
+                        # connection
+                        conn.close_if_unusable_or_obsolete()
+                    GuidMiddleware.set_guid(GuidMiddleware._generate_guid())
+                    self.run_pending()
+                except Exception:
+                    logger.exception(
+                        'encountered an error while scheduling periodic tasks'
+                    )
+                time.sleep(idle_seconds)
+
+        process = Process(target=run)
+        process.daemon = True
+        process.start()
+
+
+def run_continuously():
+    scheduler = Scheduler()
+    for task in settings.CELERYBEAT_SCHEDULE.values():
+        apply_async = TaskWorker.resolve_callable(task['task']).apply_async
+        total_seconds = task['schedule'].total_seconds()
+        scheduler.every(total_seconds).seconds.do(apply_async)
+    scheduler.run_continuously()

awx/main/dispatch/pool.py

@@ -1,8 +1,11 @@
 import logging
 import os
 import random
+import signal
 import sys
+import time
 import traceback
+from datetime import datetime
 from uuid import uuid4
 
 import collections
@@ -13,6 +16,7 @@ from queue import Full as QueueFull, Empty as QueueEmpty
 from django.conf import settings
 from django.db import connection as django_connection, connections
 from django.core.cache import cache as django_cache
+from django_guid.middleware import GuidMiddleware
 from jinja2 import Template
 import psutil
 
@@ -25,6 +29,12 @@ else:
     logger = logging.getLogger('awx.main.dispatch')
 
 
+class NoOpResultQueue(object):
+
+    def put(self, item):
+        pass
+
+
 class PoolWorker(object):
     '''
     Used to track a worker child process and its pending and finished messages.
@@ -54,11 +64,13 @@ class PoolWorker(object):
     It is "idle" when self.managed_tasks is empty.
     '''
 
-    def __init__(self, queue_size, target, args):
+    track_managed_tasks = False
+
+    def __init__(self, queue_size, target, args, **kwargs):
         self.messages_sent = 0
         self.messages_finished = 0
         self.managed_tasks = collections.OrderedDict()
-        self.finished = MPQueue(queue_size)
+        self.finished = MPQueue(queue_size) if self.track_managed_tasks else NoOpResultQueue()
         self.queue = MPQueue(queue_size)
         self.process = Process(target=target, args=(self.queue, self.finished) + args)
         self.process.daemon = True
@@ -72,10 +84,8 @@ class PoolWorker(object):
             if not body.get('uuid'):
                 body['uuid'] = str(uuid4())
             uuid = body['uuid']
-        logger.debug('delivered {} to worker[{}] qsize {}'.format(
-            uuid, self.pid, self.qsize
-        ))
-        self.managed_tasks[uuid] = body
+        if self.track_managed_tasks:
+            self.managed_tasks[uuid] = body
         self.queue.put(body, block=True, timeout=5)
         self.messages_sent += 1
         self.calculate_managed_tasks()
@@ -112,6 +122,8 @@ class PoolWorker(object):
         return str(self.process.exitcode)
 
     def calculate_managed_tasks(self):
+        if not self.track_managed_tasks:
+            return
         # look to see if any tasks were finished
         finished = []
         for _ in range(self.finished.qsize()):
@@ -132,10 +144,12 @@ class PoolWorker(object):
                 # when this occurs, it's _fine_ to ignore this KeyError because
                 # the purpose of self.managed_tasks is to just track internal
                 # state of which events are *currently* being processed.
-                pass
+                logger.warn('Event UUID {} appears to be have been duplicated.'.format(uuid))
 
     @property
     def current_task(self):
+        if not self.track_managed_tasks:
+            return None
         self.calculate_managed_tasks()
         # the task at [0] is the one that's running right now (or is about to
         # be running)
@@ -146,6 +160,8 @@ class PoolWorker(object):
 
     @property
     def orphaned_tasks(self):
+        if not self.track_managed_tasks:
+            return []
         orphaned = []
         if not self.alive:
             # if this process had a running task that never finished,
@@ -180,6 +196,11 @@ class PoolWorker(object):
         return not self.busy
 
 
+class StatefulPoolWorker(PoolWorker):
+
+    track_managed_tasks = True
+
+
 class WorkerPool(object):
     '''
     Creates a pool of forked PoolWorkers.
@@ -201,6 +222,7 @@ class WorkerPool(object):
     )
     '''
 
+    pool_cls = PoolWorker
     debug_meta = ''
 
     def __init__(self, min_workers=None, queue_size=None):
@@ -223,10 +245,10 @@ class WorkerPool(object):
         idx = len(self.workers)
         # It's important to close these because we're _about_ to fork, and we
         # don't want the forked processes to inherit the open sockets
-        # for the DB and memcached connections (that way lies race conditions)
+        # for the DB and cache connections (that way lies race conditions)
         django_connection.close()
         django_cache.close()
-        worker = PoolWorker(self.queue_size, self.target, (idx,) + self.target_args)
+        worker = self.pool_cls(self.queue_size, self.target, (idx,) + self.target_args)
         self.workers.append(worker)
         try:
             worker.start()
@@ -237,17 +259,17 @@ class WorkerPool(object):
         return idx, worker
 
     def debug(self, *args, **kwargs):
-        self.cleanup()
         tmpl = Template(
+            'Recorded at: {{ dt }} \n'
             '{{ pool.name }}[pid:{{ pool.pid }}] workers total={{ workers|length }} {{ meta }} \n'
             '{% for w in workers %}'
             '.  worker[pid:{{ w.pid }}]{% if not w.alive %} GONE exit={{ w.exitcode }}{% endif %}'
             ' sent={{ w.messages_sent }}'
-            ' finished={{ w.messages_finished }}'
+            '{% if w.messages_finished %} finished={{ w.messages_finished }}{% endif %}'
             ' qsize={{ w.managed_tasks|length }}'
             ' rss={{ w.mb }}MB'
             '{% for task in w.managed_tasks.values() %}'
-            '\n     - {% if loop.index0 == 0 %}running {% else %}queued {% endif %}'
+            '\n     - {% if loop.index0 == 0 %}running {% if "age" in task %}for: {{ "%.1f" % task["age"] }}s {% endif %}{% else %}queued {% endif %}'
             '{{ task["uuid"] }} '
             '{% if "task" in task %}'
             '{{ task["task"].rsplit(".", 1)[-1] }}'
@@ -261,7 +283,11 @@ class WorkerPool(object):
             '\n'
             '{% endfor %}'
         )
-        return tmpl.render(pool=self, workers=self.workers, meta=self.debug_meta)
+        now = datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S UTC')
+        return tmpl.render(
+            pool=self, workers=self.workers, meta=self.debug_meta,
+            dt=now
+        )
 
     def write(self, preferred_queue, body):
         queue_order = sorted(range(len(self.workers)), key=lambda x: -1 if x==preferred_queue else x)
@@ -277,7 +303,7 @@ class WorkerPool(object):
                 logger.warn("could not write to queue %s" % preferred_queue)
                 logger.warn("detail: {}".format(tb))
             write_attempt_order.append(preferred_queue)
-        logger.warn("could not write payload to any queue, attempted order: {}".format(write_attempt_order))
+        logger.error("could not write payload to any queue, attempted order: {}".format(write_attempt_order))
         return None
 
     def stop(self, signum):
@@ -294,6 +320,8 @@ class AutoscalePool(WorkerPool):
     down based on demand
     '''
 
+    pool_cls = StatefulPoolWorker
+
     def __init__(self, *args, **kwargs):
         self.max_workers = kwargs.pop('max_workers', None)
         super(AutoscalePool, self).__init__(*args, **kwargs)
@@ -310,6 +338,10 @@ class AutoscalePool(WorkerPool):
         # max workers can't be less than min_workers
         self.max_workers = max(self.min_workers, self.max_workers)
 
+    def debug(self, *args, **kwargs):
+        self.cleanup()
+        return super(AutoscalePool, self).debug(*args, **kwargs)
+
     @property
     def should_grow(self):
         if len(self.workers) < self.min_workers:
@@ -368,6 +400,26 @@ class AutoscalePool(WorkerPool):
                 logger.warn('scaling down worker pid:{}'.format(w.pid))
                 w.quit()
                 self.workers.remove(w)
+            if w.alive:
+                # if we discover a task manager invocation that's been running
+                # too long, reap it (because otherwise it'll just hold the postgres
+                # advisory lock forever); the goal of this code is to discover
+                # deadlocks or other serious issues in the task manager that cause
+                # the task manager to never do more work
+                current_task = w.current_task
+                if current_task and isinstance(current_task, dict):
+                    if current_task.get('task', '').endswith('tasks.run_task_manager'):
+                        if 'started' not in current_task:
+                            w.managed_tasks[
+                                current_task['uuid']
+                            ]['started'] = time.time()
+                        age = time.time() - current_task['started']
+                        w.managed_tasks[current_task['uuid']]['age'] = age
+                        if age > (60 * 5):
+                            logger.error(
+                                f'run_task_manager has held the advisory lock for >5m, sending SIGTERM to {w.pid}'
+                            )  # noqa
+                            os.kill(w.pid, signal.SIGTERM)
 
         for m in orphaned:
             # if all the workers are dead, spawn at least one
@@ -394,6 +446,8 @@ class AutoscalePool(WorkerPool):
             return super(AutoscalePool, self).up()
 
     def write(self, preferred_queue, body):
+        if 'guid' in body:
+            GuidMiddleware.set_guid(body['guid'])
         try:
             # when the cluster heartbeat occurs, clean up internally
             if isinstance(body, dict) and 'cluster_node_heartbeat' in body['task']:

awx/main/dispatch/publish.py

@@ -1,12 +1,13 @@
 import inspect
 import logging
 import sys
+import json
 from uuid import uuid4
 
 from django.conf import settings
-from kombu import Exchange, Producer
+from django_guid.middleware import GuidMiddleware
 
-from awx.main.dispatch.kombu import Connection
+from . import pg_bus_conn
 
 logger = logging.getLogger('awx.main.dispatch')
 
@@ -39,24 +40,22 @@ class task:
     add.apply_async([1, 1])
     Adder.apply_async([1, 1])
 
-    # Tasks can also define a specific target queue or exchange type:
+    # Tasks can also define a specific target queue or use the special fan-out queue tower_broadcast:
 
     @task(queue='slow-tasks')
     def snooze():
         time.sleep(10)
 
-    @task(queue='tower_broadcast', exchange_type='fanout')
+    @task(queue='tower_broadcast')
     def announce():
         print("Run this everywhere!")
     """
 
-    def __init__(self, queue=None, exchange_type=None):
+    def __init__(self, queue=None):
         self.queue = queue
-        self.exchange_type = exchange_type
 
     def __call__(self, fn=None):
         queue = self.queue
-        exchange_type = self.exchange_type
 
         class PublisherMixin(object):
 
@@ -73,34 +72,27 @@ class task:
                 kwargs = kwargs or {}
                 queue = (
                     queue or
-                    getattr(cls.queue, 'im_func', cls.queue) or
-                    settings.CELERY_DEFAULT_QUEUE
+                    getattr(cls.queue, 'im_func', cls.queue)
                 )
+                if not queue:
+                    msg = f'{cls.name}: Queue value required and may not be None'
+                    logger.error(msg)
+                    raise ValueError(msg)
                 obj = {
                     'uuid': task_id,
                     'args': args,
                     'kwargs': kwargs,
                     'task': cls.name
                 }
+                guid = GuidMiddleware.get_guid()
+                if guid:
+                    obj['guid'] = guid
                 obj.update(**kw)
                 if callable(queue):
                     queue = queue()
                 if not settings.IS_TESTING(sys.argv):
-                    with Connection(settings.BROKER_URL) as conn:
-                        exchange = Exchange(queue, type=exchange_type or 'direct')
-                        producer = Producer(conn)
-                        logger.debug('publish {}({}, queue={})'.format(
-                            cls.name,
-                            task_id,
-                            queue
-                        ))
-                        producer.publish(obj,
-                                         serializer='json',
-                                         compression='bzip2',
-                                         exchange=exchange,
-                                         declare=[exchange],
-                                         delivery_mode="persistent",
-                                         routing_key=queue)
+                    with pg_bus_conn() as conn:
+                        conn.notify(queue, json.dumps(obj))
                 return (obj, queue)
 
         # If the object we're wrapping *is* a class (e.g., RunJob), return

awx/main/dispatch/worker/__init__.py

@@ -1,3 +1,3 @@
-from .base import AWXConsumer, BaseWorker  # noqa
+from .base import AWXConsumerRedis, AWXConsumerPG, BaseWorker  # noqa
 from .callback import CallbackBrokerWorker  # noqa
 from .task import TaskWorker  # noqa

awx/main/dispatch/worker/base.py

@@ -5,14 +5,18 @@ import os
 import logging
 import signal
 import sys
+import redis
+import json
+import psycopg2
+import time
 from uuid import UUID
 from queue import Empty as QueueEmpty
 
 from django import db
-from kombu import Producer
-from kombu.mixins import ConsumerMixin
+from django.conf import settings
 
 from awx.main.dispatch.pool import WorkerPool
+from awx.main.dispatch import pg_bus_conn
 
 if 'run_callback_receiver' in sys.argv:
     logger = logging.getLogger('awx.main.commands.run_callback_receiver')
@@ -31,16 +35,21 @@ class WorkerSignalHandler:
 
     def __init__(self):
         self.kill_now = False
+        signal.signal(signal.SIGTERM, signal.SIG_DFL)
         signal.signal(signal.SIGINT, self.exit_gracefully)
 
     def exit_gracefully(self, *args, **kwargs):
         self.kill_now = True
 
 
-class AWXConsumer(ConsumerMixin):
+class AWXConsumerBase(object):
 
-    def __init__(self, name, connection, worker, queues=[], pool=None):
-        self.connection = connection
+    last_stats = time.time()
+
+    def __init__(self, name, worker, queues=[], pool=None):
+        self.should_stop = False
+
+        self.name = name
         self.total_messages = 0
         self.queues = queues
         self.worker = worker
@@ -48,26 +57,17 @@ class AWXConsumer(ConsumerMixin):
         if pool is None:
             self.pool = WorkerPool()
         self.pool.init_workers(self.worker.work_loop)
-
-    def get_consumers(self, Consumer, channel):
-        logger.debug(self.listening_on)
-        return [Consumer(queues=self.queues, accept=['json'],
-                         callbacks=[self.process_task])]
+        self.redis = redis.Redis.from_url(settings.BROKER_URL)
 
     @property
     def listening_on(self):
-        return 'listening on {}'.format([
-            '{} [{}]'.format(q.name, q.exchange.type) for q in self.queues
-        ])
+        return f'listening on {self.queues}'
 
-    def control(self, body, message):
+    def control(self, body):
         logger.warn(body)
         control = body.get('control')
         if control in ('status', 'running'):
-            producer = Producer(
-                channel=self.connection,
-                routing_key=message.properties['reply_to']
-            )
+            reply_queue = body['reply_to']
             if control == 'status':
                 msg = '\n'.join([self.listening_on, self.pool.debug()])
             elif control == 'running':
@@ -75,20 +75,21 @@ class AWXConsumer(ConsumerMixin):
                 for worker in self.pool.workers:
                     worker.calculate_managed_tasks()
                     msg.extend(worker.managed_tasks.keys())
-            producer.publish(msg)
+
+            with pg_bus_conn() as conn:
+                conn.notify(reply_queue, json.dumps(msg))
         elif control == 'reload':
             for worker in self.pool.workers:
                 worker.quit()
         else:
             logger.error('unrecognized control message: {}'.format(control))
-        message.ack()
 
-    def process_task(self, body, message):
+    def process_task(self, body):
         if 'control' in body:
             try:
-                return self.control(body, message)
+                return self.control(body)
             except Exception:
-                logger.exception("Exception handling control message:")
+                logger.exception(f"Exception handling control message: {body}")
                 return
         if len(self.pool):
             if "uuid" in body and body['uuid']:
@@ -102,23 +103,69 @@ class AWXConsumer(ConsumerMixin):
             queue = 0
         self.pool.write(queue, body)
         self.total_messages += 1
-        message.ack()
+        self.record_statistics()
+
+    def record_statistics(self):
+        if time.time() - self.last_stats > 1:  # buffer stat recording to once per second
+            try:
+                self.redis.set(f'awx_{self.name}_statistics', self.pool.debug())
+                self.last_stats = time.time()
+            except Exception:
+                logger.exception(f"encountered an error communicating with redis to store {self.name} statistics")
+                self.last_stats = time.time()
 
     def run(self, *args, **kwargs):
         signal.signal(signal.SIGINT, self.stop)
         signal.signal(signal.SIGTERM, self.stop)
-        self.worker.on_start()
-        super(AWXConsumer, self).run(*args, **kwargs)
+
+        # Child should implement other things here
 
     def stop(self, signum, frame):
-        self.should_stop = True  # this makes the kombu mixin stop consuming
+        self.should_stop = True
         logger.warn('received {}, stopping'.format(signame(signum)))
         self.worker.on_stop()
         raise SystemExit()
 
 
+class AWXConsumerRedis(AWXConsumerBase):
+    def run(self, *args, **kwargs):
+        super(AWXConsumerRedis, self).run(*args, **kwargs)
+        self.worker.on_start()
+
+        while True:
+            logger.debug(f'{os.getpid()} is alive')
+            time.sleep(60)
+
+
+class AWXConsumerPG(AWXConsumerBase):
+    def run(self, *args, **kwargs):
+        super(AWXConsumerPG, self).run(*args, **kwargs)
+
+        logger.warn(f"Running worker {self.name} listening to queues {self.queues}")
+        init = False
+
+        while True:
+            try:
+                with pg_bus_conn() as conn:
+                    for queue in self.queues:
+                        conn.listen(queue)
+                    if init is False:
+                        self.worker.on_start()
+                        init = True
+                    for e in conn.events():
+                        self.process_task(json.loads(e.payload))
+                    if self.should_stop:
+                        return
+            except psycopg2.InterfaceError:
+                logger.warn("Stale Postgres message bus connection, reconnecting")
+                continue
+
+
 class BaseWorker(object):
 
+    def read(self, queue):
+        return queue.get(block=True, timeout=1)
+
     def work_loop(self, queue, finished, idx, *args):
         ppid = os.getppid()
         signal_handler = WorkerSignalHandler()
@@ -128,7 +175,7 @@ class BaseWorker(object):
             if os.getppid() != ppid:
                 break
             try:
-                body = queue.get(block=True, timeout=1)
+                body = self.read(queue)
                 if body == 'QUIT':
                     break
             except QueueEmpty:
@@ -145,7 +192,6 @@ class BaseWorker(object):
             finally:
                 if 'uuid' in body:
                     uuid = body['uuid']
-                    logger.debug('task {} is finished'.format(uuid))
                     finished.put(uuid)
         logger.warn('worker exiting gracefully pid:{}'.format(os.getpid()))
 

awx/main/dispatch/worker/callback.py

@@ -1,14 +1,27 @@
+import json
 import logging
+import os
+import signal
 import time
 import traceback
 
 from django.conf import settings
+from django.utils.timezone import now as tz_now
 from django.db import DatabaseError, OperationalError, connection as django_connection
 from django.db.utils import InterfaceError, InternalError
+from django_guid.middleware import GuidMiddleware
+
+import psutil
+
+import redis
 
 from awx.main.consumers import emit_channel_notification
 from awx.main.models import (JobEvent, AdHocCommandEvent, ProjectUpdateEvent,
-                             InventoryUpdateEvent, SystemJobEvent, UnifiedJob)
+                             InventoryUpdateEvent, SystemJobEvent, UnifiedJob,
+                             Job)
+from awx.main.tasks import handle_success_and_failure_notifications
+from awx.main.models.events import emit_event_detail
+from awx.main.utils.profiling import AWXProfiler
 
 from .base import BaseWorker
 
@@ -25,90 +38,162 @@ class CallbackBrokerWorker(BaseWorker):
     '''
 
     MAX_RETRIES = 2
+    last_stats = time.time()
+    last_flush = time.time()
+    total = 0
+    last_event = ''
+    prof = None
+
+    def __init__(self):
+        self.buff = {}
+        self.pid = os.getpid()
+        self.redis = redis.Redis.from_url(settings.BROKER_URL)
+        self.prof = AWXProfiler("CallbackBrokerWorker")
+        for key in self.redis.keys('awx_callback_receiver_statistics_*'):
+            self.redis.delete(key)
+
+    def read(self, queue):
+        try:
+            res = self.redis.blpop(settings.CALLBACK_QUEUE, timeout=1)
+            if res is None:
+                return {'event': 'FLUSH'}
+            self.total += 1
+            return json.loads(res[1])
+        except redis.exceptions.RedisError:
+            logger.exception("encountered an error communicating with redis")
+            time.sleep(1)
+        except (json.JSONDecodeError, KeyError):
+            logger.exception("failed to decode JSON message from redis")
+        finally:
+            self.record_statistics()
+        return {'event': 'FLUSH'}
+
+    def record_statistics(self):
+        # buffer stat recording to once per (by default) 5s
+        if time.time() - self.last_stats > settings.JOB_EVENT_STATISTICS_INTERVAL:
+            try:
+                self.redis.set(f'awx_callback_receiver_statistics_{self.pid}', self.debug())
+                self.last_stats = time.time()
+            except Exception:
+                logger.exception("encountered an error communicating with redis")
+                self.last_stats = time.time()
+
+    def debug(self):
+        return f'.  worker[pid:{self.pid}] sent={self.total} rss={self.mb}MB {self.last_event}'
+
+    @property
+    def mb(self):
+        return '{:0.3f}'.format(
+            psutil.Process(self.pid).memory_info().rss / 1024.0 / 1024.0
+        )
+
+    def toggle_profiling(self, *args):
+        if not self.prof.is_started():
+            self.prof.start()
+            logger.error('profiling is enabled')
+        else:
+            filepath = self.prof.stop()
+            logger.error(f'profiling is disabled, wrote {filepath}')
+
+    def work_loop(self, *args, **kw):
+        if settings.AWX_CALLBACK_PROFILE:
+            signal.signal(signal.SIGUSR1, self.toggle_profiling)
+        return super(CallbackBrokerWorker, self).work_loop(*args, **kw)
+
+    def flush(self, force=False):
+        now = tz_now()
+        if (
+            force or
+            (time.time() - self.last_flush) > settings.JOB_EVENT_BUFFER_SECONDS or
+            any([len(events) >= 1000 for events in self.buff.values()])
+        ):
+            for cls, events in self.buff.items():
+                logger.debug(f'{cls.__name__}.objects.bulk_create({len(events)})')
+                for e in events:
+                    if not e.created:
+                        e.created = now
+                    e.modified = now
+                try:
+                    cls.objects.bulk_create(events)
+                except Exception:
+                    # if an exception occurs, we should re-attempt to save the
+                    # events one-by-one, because something in the list is
+                    # broken/stale
+                    for e in events:
+                        try:
+                            e.save()
+                        except Exception:
+                            logger.exception('Database Error Saving Job Event')
+                for e in events:
+                    emit_event_detail(e)
+            self.buff = {}
+            self.last_flush = time.time()
 
     def perform_work(self, body):
         try:
-            event_map = {
-                'job_id': JobEvent,
-                'ad_hoc_command_id': AdHocCommandEvent,
-                'project_update_id': ProjectUpdateEvent,
-                'inventory_update_id': InventoryUpdateEvent,
-                'system_job_id': SystemJobEvent,
-            }
+            flush = body.get('event') == 'FLUSH'
+            if flush:
+                self.last_event = ''
+            if not flush:
+                event_map = {
+                    'job_id': JobEvent,
+                    'ad_hoc_command_id': AdHocCommandEvent,
+                    'project_update_id': ProjectUpdateEvent,
+                    'inventory_update_id': InventoryUpdateEvent,
+                    'system_job_id': SystemJobEvent,
+                }
 
-            if not any([key in body for key in event_map]):
-                raise Exception('Payload does not have a job identifier')
-
-            def _save_event_data():
+                job_identifier = 'unknown job'
                 for key, cls in event_map.items():
                     if key in body:
-                        cls.create_from_data(**body)
+                        job_identifier = body[key]
+                        break
 
-            job_identifier = 'unknown job'
-            job_key = 'unknown'
-            for key in event_map.keys():
-                if key in body:
-                    job_identifier = body[key]
-                    job_key = key
-                    break
+                self.last_event = f'\n\t- {cls.__name__} for #{job_identifier} ({body.get("event", "")} {body.get("uuid", "")})'  # noqa
 
-            if settings.DEBUG:
-                from pygments import highlight
-                from pygments.lexers import PythonLexer
-                from pygments.formatters import Terminal256Formatter
-                from pprint import pformat
                 if body.get('event') == 'EOF':
-                    event_thing = 'EOF event'
-                else:
-                    event_thing = 'event {}'.format(body.get('counter', 'unknown'))
-                logger.info('Callback worker received {} for {} {}'.format(
-                    event_thing, job_key[:-len('_id')], job_identifier
-                ))
-                logger.debug('Body: {}'.format(
-                    highlight(pformat(body, width=160), PythonLexer(), Terminal256Formatter(style='friendly'))
-                )[:1024 * 4])
+                    try:
+                        if 'guid' in body:
+                            GuidMiddleware.set_guid(body['guid'])
+                        final_counter = body.get('final_counter', 0)
+                        logger.info('Event processing is finished for Job {}, sending notifications'.format(job_identifier))
+                        # EOF events are sent when stdout for the running task is
+                        # closed. don't actually persist them to the database; we
+                        # just use them to report `summary` websocket events as an
+                        # approximation for when a job is "done"
+                        emit_channel_notification(
+                            'jobs-summary',
+                            dict(group_name='jobs', unified_job_id=job_identifier, final_counter=final_counter)
+                        )
+                        # Additionally, when we've processed all events, we should
+                        # have all the data we need to send out success/failure
+                        # notification templates
+                        uj = UnifiedJob.objects.get(pk=job_identifier)
 
-            if body.get('event') == 'EOF':
-                try:
-                    final_counter = body.get('final_counter', 0)
-                    logger.info('Event processing is finished for Job {}, sending notifications'.format(job_identifier))
-                    # EOF events are sent when stdout for the running task is
-                    # closed. don't actually persist them to the database; we
-                    # just use them to report `summary` websocket events as an
-                    # approximation for when a job is "done"
-                    emit_channel_notification(
-                        'jobs-summary',
-                        dict(group_name='jobs', unified_job_id=job_identifier, final_counter=final_counter)
-                    )
-                    # Additionally, when we've processed all events, we should
-                    # have all the data we need to send out success/failure
-                    # notification templates
-                    uj = UnifiedJob.objects.get(pk=job_identifier)
-                    if hasattr(uj, 'send_notification_templates'):
-                        retries = 0
-                        while retries < 5:
-                            if uj.finished:
-                                uj.send_notification_templates('succeeded' if uj.status == 'successful' else 'failed')
-                                break
-                            else:
-                                # wait a few seconds to avoid a race where the
-                                # events are persisted _before_ the UJ.status
-                                # changes from running -> successful
-                                retries += 1
-                                time.sleep(1)
-                                uj = UnifiedJob.objects.get(pk=job_identifier)
-                except Exception:
-                    logger.exception('Worker failed to emit notifications: Job {}'.format(job_identifier))
-                return
+                        if isinstance(uj, Job):
+                            # *actual playbooks* send their success/failure
+                            # notifications in response to the playbook_on_stats
+                            # event handling code in main.models.events
+                            pass
+                        elif hasattr(uj, 'send_notification_templates'):
+                            handle_success_and_failure_notifications.apply_async([uj.id])
+                    except Exception:
+                        logger.exception('Worker failed to emit notifications: Job {}'.format(job_identifier))
+                    finally:
+                        GuidMiddleware.set_guid('')
+                    return
+
+                event = cls.create_from_data(**body)
+                self.buff.setdefault(cls, []).append(event)
 
             retries = 0
             while retries <= self.MAX_RETRIES:
                 try:
-                    _save_event_data()
+                    self.flush(force=flush)
                     break
                 except (OperationalError, InterfaceError, InternalError):
                     if retries >= self.MAX_RETRIES:
-                        logger.exception('Worker could not re-establish database connectivity, giving up on event for Job {}'.format(job_identifier))
+                        logger.exception('Worker could not re-establish database connectivity, giving up on one or more events.')
                         return
                     delay = 60 * retries
                     logger.exception('Database Error Saving Job Event, retry #{i} in {delay} seconds:'.format(
@@ -119,7 +204,7 @@ class CallbackBrokerWorker(BaseWorker):
                     time.sleep(delay)
                     retries += 1
                 except DatabaseError:
-                    logger.exception('Database Error Saving Job Event for Job {}'.format(job_identifier))
+                    logger.exception('Database Error Saving Job Event')
                     break
         except Exception as exc:
             tb = traceback.format_exc()

awx/main/dispatch/worker/task.py

@@ -6,6 +6,9 @@ import traceback
 
 from kubernetes.config import kube_config
 
+from django.conf import settings
+from django_guid.middleware import GuidMiddleware
+
 from awx.main.tasks import dispatch_startup, inform_cluster_of_shutdown
 
 from .base import BaseWorker
@@ -52,6 +55,8 @@ class TaskWorker(BaseWorker):
         uuid = body.get('uuid', '<unknown>')
         args = body.get('args', [])
         kwargs = body.get('kwargs', {})
+        if 'guid' in body:
+            GuidMiddleware.set_guid(body.pop('guid'))
         _call = TaskWorker.resolve_callable(task)
         if inspect.isclass(_call):
             # the callable is a class, e.g., RunJob; instantiate and
@@ -81,6 +86,7 @@ class TaskWorker(BaseWorker):
             'task': u'awx.main.tasks.RunProjectUpdate'
         }
         '''
+        settings.__clean_on_fork__()
         result = None
         try:
             result = self.run_callable(body)

awx/main/exceptions.py

@@ -30,3 +30,10 @@ class _AwxTaskError():
 
 
 AwxTaskError = _AwxTaskError()
+
+
+class PostRunError(Exception):
+    def __init__(self, msg, status='failed', tb=''):
+        self.status = status
+        self.tb = tb
+        super(PostRunError, self).__init__(msg)

awx/main/fields.py

@@ -7,8 +7,8 @@ import json
 import re
 import urllib.parse
 
-from jinja2 import Environment, StrictUndefined
-from jinja2.exceptions import UndefinedError, TemplateSyntaxError
+from jinja2 import sandbox, StrictUndefined
+from jinja2.exceptions import UndefinedError, TemplateSyntaxError, SecurityError
 
 # Django
 from django.contrib.postgres.fields import JSONField as upstream_JSONBField
@@ -50,13 +50,14 @@ from awx.main.models.rbac import (
     batch_role_ancestor_rebuilding, Role,
     ROLE_SINGLETON_SYSTEM_ADMINISTRATOR, ROLE_SINGLETON_SYSTEM_AUDITOR
 )
-from awx.main.constants import ENV_BLACKLIST
+from awx.main.constants import ENV_BLOCKLIST
 from awx.main import utils
 
 
 __all__ = ['AutoOneToOneField', 'ImplicitRoleField', 'JSONField',
            'SmartFilterField', 'OrderedManyToManyField',
-           'update_role_parentage_for_instance', 'is_implicit_parent']
+           'update_role_parentage_for_instance',
+           'is_implicit_parent']
 
 
 # Provide a (better) custom error message for enum jsonschema validation
@@ -140,8 +141,9 @@ def resolve_role_field(obj, field):
         return []
 
     if len(field_components) == 1:
-        role_cls = str(utils.get_current_apps().get_model('main', 'Role'))
-        if not str(type(obj)) == role_cls:
+        # use extremely generous duck typing to accomidate all possible forms
+        # of the model that may be used during various migrations
+        if obj._meta.model_name != 'role' or obj._meta.app_label != 'main':
             raise Exception(smart_text('{} refers to a {}, not a Role'.format(field, type(obj))))
         ret.append(obj.id)
     else:
@@ -197,18 +199,27 @@ def update_role_parentage_for_instance(instance):
     updates the parents listing for all the roles
     of a given instance if they have changed
     '''
+    parents_removed = set()
+    parents_added = set()
     for implicit_role_field in getattr(instance.__class__, '__implicit_role_fields'):
         cur_role = getattr(instance, implicit_role_field.name)
         original_parents = set(json.loads(cur_role.implicit_parents))
         new_parents = implicit_role_field._resolve_parent_roles(instance)
-        cur_role.parents.remove(*list(original_parents - new_parents))
-        cur_role.parents.add(*list(new_parents - original_parents))
+        removals = original_parents - new_parents
+        if removals:
+            cur_role.parents.remove(*list(removals))
+            parents_removed.add(cur_role.pk)
+        additions = new_parents - original_parents
+        if additions:
+            cur_role.parents.add(*list(additions))
+            parents_added.add(cur_role.pk)
         new_parents_list = list(new_parents)
         new_parents_list.sort()
         new_parents_json = json.dumps(new_parents_list)
         if cur_role.implicit_parents != new_parents_json:
             cur_role.implicit_parents = new_parents_json
-            cur_role.save()
+            cur_role.save(update_fields=['implicit_parents'])
+    return (parents_added, parents_removed)
 
 
 class ImplicitRoleDescriptor(ForwardManyToOneDescriptor):
@@ -256,20 +267,18 @@ class ImplicitRoleField(models.ForeignKey):
             field_names = [field_names]
 
         for field_name in field_names:
-            # Handle the OR syntax for role parents
-            if type(field_name) == tuple:
-                continue
-
-            if type(field_name) == bytes:
-                field_name = field_name.decode('utf-8')
 
             if field_name.startswith('singleton:'):
                 continue
 
             field_name, sep, field_attr = field_name.partition('.')
-            field = getattr(cls, field_name)
+            # Non existent fields will occur if ever a parent model is
+            # moved inside a migration, needed for job_template_organization_field
+            # migration in particular
+            # consistency is assured by unit test awx.main.tests.functional
+            field = getattr(cls, field_name, None)
 
-            if type(field) is ReverseManyToOneDescriptor or \
+            if field and type(field) is ReverseManyToOneDescriptor or \
                type(field) is ManyToManyDescriptor:
 
                 if '.' in field_attr:
@@ -628,6 +637,14 @@ class CredentialInputField(JSONSchemaField):
             else:
                 decrypted_values[k] = v
 
+        # don't allow secrets with $encrypted$ on new object creation
+        if not model_instance.pk:
+            for field in model_instance.credential_type.secret_fields:
+                if value.get(field) == '$encrypted$':
+                    raise serializers.ValidationError({
+                        self.name: [f'$encrypted$ is a reserved keyword, and cannot be used for {field}.']
+                    })
+
         super(JSONSchemaField, self).validate(decrypted_values, model_instance)
         errors = {}
         for error in Draft4Validator(
@@ -861,9 +878,9 @@ class CredentialTypeInjectorField(JSONSchemaField):
                   'use is not allowed in credentials.').format(env_var),
                 code='invalid', params={'value': env_var},
             )
-        if env_var in ENV_BLACKLIST:
+        if env_var in ENV_BLOCKLIST:
             raise django_exceptions.ValidationError(
-                _('Environment variable {} is blacklisted from use in credentials.').format(env_var),
+                _('Environment variable {} is not allowed to be used in credentials.').format(env_var),
                 code='invalid', params={'value': env_var},
             )
 
@@ -923,7 +940,7 @@ class CredentialTypeInjectorField(JSONSchemaField):
                     self.validate_env_var_allowed(key)
             for key, tmpl in injector.items():
                 try:
-                    Environment(
+                    sandbox.ImmutableSandboxedEnvironment(
                         undefined=StrictUndefined
                     ).from_string(tmpl).render(valid_namespace)
                 except UndefinedError as e:
@@ -933,6 +950,10 @@ class CredentialTypeInjectorField(JSONSchemaField):
                         code='invalid',
                         params={'value': value},
                     )
+                except SecurityError as e:
+                    raise django_exceptions.ValidationError(
+                        _('Encountered unsafe code execution: {}').format(e)
+                    )
                 except TemplateSyntaxError as e:
                     raise django_exceptions.ValidationError(
                         _('Syntax error rendering template for {sub_key} inside of {type} ({error_msg})').format(

awx/main/isolated/manager.py

@@ -6,7 +6,7 @@ import stat
 import tempfile
 import time
 import logging
-import yaml
+import datetime
 
 from django.conf import settings
 import ansible_runner
@@ -15,7 +15,6 @@ import awx
 from awx.main.utils import (
     get_system_task_capacity
 )
-from awx.main.queue import CallbackQueueDispatcher
 
 logger = logging.getLogger('awx.isolated.manager')
 playbook_logger = logging.getLogger('awx.isolated.manager.playbooks')
@@ -32,40 +31,25 @@ def set_pythonpath(venv_libdir, env):
 
 class IsolatedManager(object):
 
-    def __init__(self, cancelled_callback=None, check_callback=None, pod_manager=None):
+    def __init__(self, event_handler, canceled_callback=None, check_callback=None):
         """
-        :param cancelled_callback:  a callable - which returns `True` or `False`
+        :param event_handler: a callable used to persist event data from isolated nodes
+        :param canceled_callback:  a callable - which returns `True` or `False`
                                     - signifying if the job has been prematurely
-                                      cancelled
+                                      canceled
         """
-        self.cancelled_callback = cancelled_callback
+        self.event_handler = event_handler
+        self.canceled_callback = canceled_callback
         self.check_callback = check_callback
-        self.idle_timeout = max(60, 2 * settings.AWX_ISOLATED_CONNECTION_TIMEOUT)
         self.started_at = None
         self.captured_command_artifact = False
         self.instance = None
-        self.pod_manager = pod_manager
 
     def build_inventory(self, hosts):
-        if self.instance and self.instance.is_containerized:
-            inventory = {'all': {'hosts': {}}}
-            fd, path = tempfile.mkstemp(
-                prefix='.kubeconfig', dir=self.private_data_dir
-            )
-            with open(path, 'wb') as temp:
-                temp.write(yaml.dump(self.pod_manager.kube_config).encode())
-                temp.flush()
-                os.chmod(temp.name, stat.S_IRUSR | stat.S_IWUSR | stat.S_IXUSR)
-            for host in hosts:
-                inventory['all']['hosts'][host] = {
-                    "ansible_connection": "kubectl",
-                    "ansible_kubectl_config": path,
-                }
-        else:
-            inventory = '\n'.join([
-                '{} ansible_ssh_user={}'.format(host, settings.AWX_ISOLATED_USERNAME)
-                for host in hosts
-            ])
+        inventory = '\n'.join([
+            '{} ansible_ssh_user={}'.format(host, settings.AWX_ISOLATED_USERNAME)
+            for host in hosts
+        ])
 
         return inventory
 
@@ -74,6 +58,7 @@ class IsolatedManager(object):
         env['ANSIBLE_RETRY_FILES_ENABLED'] = 'False'
         env['ANSIBLE_HOST_KEY_CHECKING'] = str(settings.AWX_ISOLATED_HOST_KEY_CHECKING)
         env['ANSIBLE_LIBRARY'] = os.path.join(os.path.dirname(awx.__file__), 'plugins', 'isolated')
+        env['ANSIBLE_COLLECTIONS_PATHS'] = settings.AWX_ANSIBLE_COLLECTIONS_PATHS
         set_pythonpath(os.path.join(settings.ANSIBLE_VENV_PATH, 'lib'), env)
 
         def finished_callback(runner_obj):
@@ -106,11 +91,9 @@ class IsolatedManager(object):
             'envvars': env,
             'finished_callback': finished_callback,
             'verbosity': verbosity,
-            'cancel_callback': self.cancelled_callback,
+            'cancel_callback': self.canceled_callback,
             'settings': {
-                'idle_timeout': self.idle_timeout,
                 'job_timeout': settings.AWX_ISOLATED_LAUNCH_TIMEOUT,
-                'pexpect_timeout': getattr(settings, 'PEXPECT_TIMEOUT', 5),
                 'suppress_ansible_output': True,
             },
         }
@@ -118,14 +101,19 @@ class IsolatedManager(object):
     def path_to(self, *args):
         return os.path.join(self.private_data_dir, *args)
 
-    def run_management_playbook(self, playbook, private_data_dir, **kw):
+    def run_management_playbook(self, playbook, private_data_dir, idle_timeout=None, **kw):
         iso_dir = tempfile.mkdtemp(
             prefix=playbook,
             dir=private_data_dir
         )
         params = self.runner_params.copy()
+        params.get('envvars', dict())['ANSIBLE_CALLBACK_WHITELIST'] = 'profile_tasks'
         params['playbook'] = playbook
         params['private_data_dir'] = iso_dir
+        if idle_timeout:
+            params['settings']['idle_timeout'] = idle_timeout
+        else:
+            params['settings'].pop('idle_timeout', None)
         params.update(**kw)
         if all([
             getattr(settings, 'AWX_ISOLATED_KEY_GENERATION', False) is True,
@@ -146,7 +134,6 @@ class IsolatedManager(object):
             # don't rsync source control metadata (it can be huge!)
             '- /project/.git',
             '- /project/.svn',
-            '- /project/.hg',
             # don't rsync job events that are in the process of being written
             '- /artifacts/job_events/*-partial.json.tmp',
             # don't rsync the ssh_key FIFO
@@ -166,7 +153,8 @@ class IsolatedManager(object):
         extravars = {
             'src': self.private_data_dir,
             'dest': settings.AWX_PROOT_BASE_PATH,
-            'ident': self.ident
+            'ident': self.ident,
+            'job_id': self.instance.id,
         }
         if playbook:
             extravars['playbook'] = playbook
@@ -177,6 +165,7 @@ class IsolatedManager(object):
         logger.debug('Starting job {} on isolated host with `run_isolated.yml` playbook.'.format(self.instance.id))
         runner_obj = self.run_management_playbook('run_isolated.yml',
                                                   self.private_data_dir,
+                                                  idle_timeout=max(60, 2 * settings.AWX_ISOLATED_CONNECTION_TIMEOUT),
                                                   extravars=extravars)
 
         if runner_obj.status == 'failed':
@@ -201,26 +190,32 @@ class IsolatedManager(object):
         :param interval: an interval (in seconds) to wait between status polls
         """
         interval = interval if interval is not None else settings.AWX_ISOLATED_CHECK_INTERVAL
-        extravars = {'src': self.private_data_dir}
+        extravars = {
+            'src': self.private_data_dir,
+            'job_id': self.instance.id
+        }
         status = 'failed'
         rc = None
         last_check = time.time()
-        dispatcher = CallbackQueueDispatcher()
 
         while status == 'failed':
-            canceled = self.cancelled_callback() if self.cancelled_callback else False
+            canceled = self.canceled_callback() if self.canceled_callback else False
             if not canceled and time.time() - last_check < interval:
-                # If the job isn't cancelled, but we haven't waited `interval` seconds, wait longer
+                # If the job isn't canceled, but we haven't waited `interval` seconds, wait longer
                 time.sleep(1)
                 continue
 
             if canceled:
-                logger.warning('Isolated job {} was manually cancelled.'.format(self.instance.id))
+                logger.warning('Isolated job {} was manually canceled.'.format(self.instance.id))
 
             logger.debug('Checking on isolated job {} with `check_isolated.yml`.'.format(self.instance.id))
+            time_start = datetime.datetime.now()
             runner_obj = self.run_management_playbook('check_isolated.yml',
                                                       self.private_data_dir,
                                                       extravars=extravars)
+            time_end = datetime.datetime.now()
+            time_diff = time_end - time_start
+            logger.debug('Finished checking on isolated job {} with `check_isolated.yml` took {} seconds.'.format(self.instance.id, time_diff.total_seconds()))
             status, rc = runner_obj.status, runner_obj.rc
 
             if self.check_callback is not None and not self.captured_command_artifact:
@@ -235,7 +230,7 @@ class IsolatedManager(object):
                     except json.decoder.JSONDecodeError:  # Just in case it's not fully here yet.
                         pass
 
-            self.consume_events(dispatcher)
+            self.consume_events()
 
             last_check = time.time()
 
@@ -263,19 +258,11 @@ class IsolatedManager(object):
 
         # consume events one last time just to be sure we didn't miss anything
         # in the final sync
-        self.consume_events(dispatcher)
-
-        # emit an EOF event
-        event_data = {
-            'event': 'EOF',
-            'final_counter': len(self.handled_events)
-        }
-        event_data.setdefault(self.event_data_key, self.instance.id)
-        dispatcher.dispatch(event_data)
+        self.consume_events()
 
         return status, rc
 
-    def consume_events(self, dispatcher):
+    def consume_events(self):
         # discover new events and ingest them
         events_path = self.path_to('artifacts', self.ident, 'job_events')
 
@@ -285,7 +272,7 @@ class IsolatedManager(object):
         if os.path.exists(events_path):
             for event in set(os.listdir(events_path)) - self.handled_events:
                 path = os.path.join(events_path, event)
-                if os.path.exists(path):
+                if os.path.exists(path) and os.path.isfile(path):
                     try:
                         event_data = json.load(
                             open(os.path.join(events_path, event), 'r')
@@ -299,16 +286,10 @@ class IsolatedManager(object):
                         # practice
                         # in this scenario, just ignore this event and try it
                         # again on the next sync
-                        pass
-                    event_data.setdefault(self.event_data_key, self.instance.id)
-                    dispatcher.dispatch(event_data)
+                        continue
+                    self.event_handler(event_data)
                     self.handled_events.add(event)
 
-                    # handle artifacts
-                    if event_data.get('event_data', {}).get('artifact_data', {}):
-                        self.instance.artifacts = event_data['event_data']['artifact_data']
-                        self.instance.save(update_fields=['artifacts'])
-
 
     def cleanup(self):
         extravars = {
@@ -367,39 +348,37 @@ class IsolatedManager(object):
                 private_data_dir
             )
 
-            if runner_obj.status == 'successful':
-                for instance in instance_qs:
-                    task_result = {}
-                    try:
-                        task_result = runner_obj.get_fact_cache(instance.hostname)
-                    except Exception:
-                        logger.exception('Failed to read status from isolated instances')
-                    if 'awx_capacity_cpu' in task_result and 'awx_capacity_mem' in task_result:
-                        task_result = {
-                            'cpu': task_result['awx_cpu'],
-                            'mem': task_result['awx_mem'],
-                            'capacity_cpu': task_result['awx_capacity_cpu'],
-                            'capacity_mem': task_result['awx_capacity_mem'],
-                            'version': task_result['awx_capacity_version']
-                        }
-                        IsolatedManager.update_capacity(instance, task_result)
-                        logger.debug('Isolated instance {} successful heartbeat'.format(instance.hostname))
-                    elif instance.capacity == 0:
-                        logger.debug('Isolated instance {} previously marked as lost, could not re-join.'.format(
-                            instance.hostname))
-                    else:
-                        logger.warning('Could not update status of isolated instance {}'.format(instance.hostname))
-                        if instance.is_lost(isolated=True):
-                            instance.capacity = 0
-                            instance.save(update_fields=['capacity'])
-                            logger.error('Isolated instance {} last checked in at {}, marked as lost.'.format(
-                                instance.hostname, instance.modified))
+            for instance in instance_qs:
+                task_result = {}
+                try:
+                    task_result = runner_obj.get_fact_cache(instance.hostname)
+                except Exception:
+                    logger.exception('Failed to read status from isolated instances')
+                if 'awx_capacity_cpu' in task_result and 'awx_capacity_mem' in task_result:
+                    task_result = {
+                        'cpu': task_result['awx_cpu'],
+                        'mem': task_result['awx_mem'],
+                        'capacity_cpu': task_result['awx_capacity_cpu'],
+                        'capacity_mem': task_result['awx_capacity_mem'],
+                        'version': task_result['awx_capacity_version']
+                    }
+                    IsolatedManager.update_capacity(instance, task_result)
+                    logger.debug('Isolated instance {} successful heartbeat'.format(instance.hostname))
+                elif instance.capacity == 0:
+                    logger.debug('Isolated instance {} previously marked as lost, could not re-join.'.format(
+                        instance.hostname))
+                else:
+                    logger.warning('Could not update status of isolated instance {}'.format(instance.hostname))
+                    if instance.is_lost(isolated=True):
+                        instance.capacity = 0
+                        instance.save(update_fields=['capacity'])
+                        logger.error('Isolated instance {} last checked in at {}, marked as lost.'.format(
+                            instance.hostname, instance.modified))
         finally:
             if os.path.exists(private_data_dir):
                 shutil.rmtree(private_data_dir)
 
-    def run(self, instance, private_data_dir, playbook, module, module_args,
-            event_data_key, ident=None):
+    def run(self, instance, private_data_dir, playbook, module, module_args, ident=None):
         """
         Run a job on an isolated host.
 
@@ -410,14 +389,12 @@ class IsolatedManager(object):
         :param playbook:         the playbook to run
         :param module:           the module to run
         :param module_args:      the module args to use
-        :param event_data_key:   e.g., job_id, inventory_id, ...
 
         For a completed job run, this function returns (status, rc),
         representing the status and return code of the isolated
         `ansible-playbook` run.
         """
         self.ident = ident
-        self.event_data_key = event_data_key
         self.instance = instance
         self.private_data_dir = private_data_dir
         self.runner_params = self.build_runner_params(
@@ -428,9 +405,4 @@ class IsolatedManager(object):
         status, rc = self.dispatch(playbook, module, module_args)
         if status == 'successful':
             status, rc = self.check()
-        else:
-            # emit an EOF event
-            event_data = {'event': 'EOF', 'final_counter': 0}
-            event_data.setdefault(self.event_data_key, self.instance.id)
-            CallbackQueueDispatcher().dispatch(event_data)
         return status, rc

awx/main/management/commands/bottleneck.py

@@ -0,0 +1,96 @@
+from django.core.management.base import BaseCommand
+from django.db import connection
+
+from awx.main.models import JobTemplate
+
+
+class Command(BaseCommand):
+    help = "Find the slowest tasks and hosts for a Job Template's most recent runs."
+
+    def add_arguments(self, parser):
+        parser.add_argument('--template', dest='jt', type=int,
+                            help='ID of the Job Template to profile')
+        parser.add_argument('--threshold', dest='threshold', type=float, default=30,
+                            help='Only show tasks that took at least this many seconds (defaults to 30)')
+        parser.add_argument('--history', dest='history', type=float, default=25,
+                            help='The number of historic jobs to look at')
+        parser.add_argument('--ignore', action='append', help='ignore a specific action (e.g., --ignore git)')
+
+    def handle(self, *args, **options):
+        jt = options['jt']
+        threshold = options['threshold']
+        history = options['history']
+        ignore = options['ignore']
+
+        print('## ' + JobTemplate.objects.get(pk=jt).name + f' (last {history} runs)\n')
+        with connection.cursor() as cursor:
+            cursor.execute(
+                f'''
+                SELECT 
+                    b.id, b.job_id, b.host_name, b.created - a.created delta,
+                    b.task task,
+                    b.event_data::json->'task_action' task_action,
+                    b.event_data::json->'task_path' task_path
+                FROM main_jobevent a JOIN main_jobevent b
+                ON b.parent_uuid = a.parent_uuid  AND a.host_name = b.host_name
+                WHERE
+                    a.event = 'runner_on_start' AND
+                    b.event != 'runner_on_start' AND
+                    b.event != 'runner_on_skipped' AND
+                    b.failed = false AND
+                    a.job_id IN (
+                        SELECT unifiedjob_ptr_id FROM main_job
+                        WHERE job_template_id={jt}
+                        ORDER BY unifiedjob_ptr_id DESC
+                        LIMIT {history}
+                    )
+                ORDER BY delta DESC;
+                '''
+            )
+            slowest_events = cursor.fetchall()
+
+        def format_td(x):
+            return str(x).split('.')[0]
+
+        fastest = dict()
+        for event in slowest_events:
+            _id, job_id, host, duration, task, action, playbook = event
+            playbook = playbook.rsplit('/')[-1]
+            if ignore and action in ignore:
+                continue
+            if host:
+                fastest[(action, playbook)] = (_id, host, format_td(duration))
+
+        host_counts = dict()
+        warned = set()
+        print(f'slowest tasks (--threshold={threshold})\n---')
+
+        for event in slowest_events:
+            _id, job_id, host, duration, task, action, playbook = event
+            if ignore and action in ignore:
+                continue
+            if duration.total_seconds() < threshold:
+                break
+            playbook = playbook.rsplit('/')[-1]
+            human_duration = format_td(duration)
+
+            fastest_summary = ''
+            fastest_match = fastest.get((action, playbook))
+            if fastest_match[2] != human_duration and (host, action, playbook) not in warned:
+                warned.add((host, action, playbook))
+                fastest_summary = ' ' + self.style.WARNING(f'{fastest_match[1]} ran this in {fastest_match[2]}s at /api/v2/job_events/{fastest_match[0]}/')
+
+            url = f'/api/v2/jobs/{job_id}/'
+            print(' -- '.join([url, host, human_duration, action, task, playbook]) + fastest_summary)
+            host_counts.setdefault(host, [])
+            host_counts[host].append(duration)
+
+        host_counts = sorted(host_counts.items(), key=lambda item: [e.total_seconds() for e in item[1]], reverse=True)
+
+        print('\nslowest hosts\n---')
+        for h, matches in host_counts:
+            total = len(matches)
+            total_seconds = sum([e.total_seconds() for e in matches])
+            print(f'{h} had {total} tasks that ran longer than {threshold} second(s) for a total of {total_seconds}')
+
+        print('')

awx/main/management/commands/callback_stats.py

@@ -0,0 +1,40 @@
+import time
+import sys
+
+from django.db import connection
+from django.core.management.base import BaseCommand
+
+
+class Command(BaseCommand):
+
+    def handle(self, *args, **options):
+        with connection.cursor() as cursor:
+            start = {}
+            for relation in (
+                'main_jobevent', 'main_inventoryupdateevent',
+                'main_projectupdateevent', 'main_adhoccommandevent'
+            ):
+                cursor.execute(f"SELECT MAX(id) FROM {relation};")
+                start[relation] = cursor.fetchone()[0] or 0
+            clear = False
+            while True:
+                lines = []
+                for relation in (
+                    'main_jobevent', 'main_inventoryupdateevent',
+                    'main_projectupdateevent', 'main_adhoccommandevent'
+                ):
+                    lines.append(relation)
+                    minimum = start[relation]
+                    cursor.execute(
+                        f"SELECT MAX(id) - MIN(id) FROM {relation} WHERE id > {minimum} AND modified > now() - '1 minute'::interval;"
+                    )
+                    events = cursor.fetchone()[0] or 0
+                    lines.append(f'↳  last minute {events}')
+                    lines.append('')
+                if clear:
+                    for i in range(12):
+                        sys.stdout.write('\x1b[1A\x1b[2K')
+                for line in lines:
+                    print(line)
+                clear = True
+                time.sleep(.25)

awx/main/management/commands/check_license.py

@@ -18,7 +18,5 @@ class Command(BaseCommand):
         super(Command, self).__init__()
         license = get_licenser().validate()
         if options.get('data'):
-            if license.get('license_key', '') != 'UNLICENSED':
-                license['license_key'] = '********'
             return json.dumps(license)
         return license.get('license_type', 'none')

awx/main/management/commands/check_migrations.py

@@ -8,5 +8,7 @@ class Command(MakeMigrations):
     def execute(self, *args, **options):
         settings = connections['default'].settings_dict.copy()
         settings['ENGINE'] = 'sqlite3'
+        if 'application_name' in settings['OPTIONS']:
+            del settings['OPTIONS']['application_name']
         connections['default'] = DatabaseWrapper(settings)
         return MakeMigrations().execute(*args, **options)

awx/main/management/commands/cleanup_jobs.py

@@ -16,13 +16,12 @@ from awx.main.models import (
     Job, AdHocCommand, ProjectUpdate, InventoryUpdate,
     SystemJob, WorkflowJob, Notification
 )
-from awx.main.signals import ( # noqa
-    emit_update_inventory_on_created_or_deleted,
-    emit_update_inventory_computed_fields,
+from awx.main.signals import (
     disable_activity_stream,
     disable_computed_fields
 )
-from django.db.models.signals import post_save, post_delete, m2m_changed # noqa
+
+from awx.main.utils.deletion import AWXCollector, pre_delete
 
 
 class Command(BaseCommand):
@@ -60,27 +59,37 @@ class Command(BaseCommand):
                             action='store_true', dest='only_workflow_jobs',
                             help='Remove workflow jobs')
 
-    def cleanup_jobs(self):
-        #jobs_qs = Job.objects.exclude(status__in=('pending', 'running'))
-        #jobs_qs = jobs_qs.filter(created__lte=self.cutoff)
-        skipped, deleted = 0, 0
-        jobs = Job.objects.filter(created__lt=self.cutoff)
-        for job in jobs.iterator():
-            job_display = '"%s" (%d host summaries, %d events)' % \
-                          (str(job),
-                           job.job_host_summaries.count(), job.job_events.count())
-            if job.status in ('pending', 'waiting', 'running'):
-                action_text = 'would skip' if self.dry_run else 'skipping'
-                self.logger.debug('%s %s job %s', action_text, job.status, job_display)
-                skipped += 1
-            else:
-                action_text = 'would delete' if self.dry_run else 'deleting'
-                self.logger.info('%s %s', action_text, job_display)
-                if not self.dry_run:
-                    job.delete()
-                deleted += 1
 
-        skipped += Job.objects.filter(created__gte=self.cutoff).count()
+    def cleanup_jobs(self):
+        skipped, deleted = 0, 0
+
+        batch_size = 1000000
+
+        while True:
+            # get queryset for available jobs to remove
+            qs = Job.objects.filter(created__lt=self.cutoff).exclude(status__in=['pending', 'waiting', 'running'])
+            # get pk list for the first N (batch_size) objects
+            pk_list = qs[0:batch_size].values_list('pk')
+            # You cannot delete queries with sql LIMIT set, so we must
+            # create a new query from this pk_list
+            qs_batch = Job.objects.filter(pk__in=pk_list)
+            just_deleted = 0
+            if not self.dry_run:
+                del_query = pre_delete(qs_batch)
+                collector = AWXCollector(del_query.db)
+                collector.collect(del_query)
+                _, models_deleted = collector.delete()
+                if models_deleted:
+                    just_deleted = models_deleted['main.Job']
+                deleted += just_deleted
+            else:
+                just_deleted = 0 # break from loop, this is dry run
+                deleted = qs.count()
+
+            if just_deleted == 0:
+                break
+
+        skipped += (Job.objects.filter(created__gte=self.cutoff) | Job.objects.filter(status__in=['pending', 'waiting', 'running'])).count()
         return skipped, deleted
 
     def cleanup_ad_hoc_commands(self):

awx/main/management/commands/create_preload_data.py

@@ -2,22 +2,22 @@
 # All Rights Reserved
 
 from django.core.management.base import BaseCommand
+from django.conf import settings
 from crum import impersonate
-from awx.main.models import User, Organization, Project, Inventory, CredentialType, Credential, Host, JobTemplate
+from awx.main.models import (
+    User, Organization, Project, Inventory, CredentialType,
+    Credential, Host, JobTemplate, ExecutionEnvironment
+)
 from awx.main.signals import disable_computed_fields
 
 
 class Command(BaseCommand):
     """Create preloaded data, intended for new installs
     """
-    help = 'Creates a preload tower data iff there is none.'
+    help = 'Creates a preload tower data if there is none.'
 
     def handle(self, *args, **kwargs):
-        # Sanity check: Is there already an organization in the system?
-        if Organization.objects.count():
-            print('An organization is already in the system, exiting.')
-            print('(changed: False)')
-            return
+        changed = False
 
         # Create a default organization as the first superuser found.
         try:
@@ -26,34 +26,62 @@ class Command(BaseCommand):
             superuser = None
         with impersonate(superuser):
             with disable_computed_fields():
-                o = Organization.objects.create(name='Default')
-                p = Project(name='Demo Project',
-                            scm_type='git',
-                            scm_url='https://github.com/ansible/ansible-tower-samples',
-                            scm_update_on_launch=True,
-                            scm_update_cache_timeout=0,
-                            organization=o)
-                p.save(skip_update=True)
-                ssh_type = CredentialType.objects.filter(namespace='ssh').first()
-                c = Credential.objects.create(credential_type=ssh_type,
-                                              name='Demo Credential',
-                                              inputs={
-                                                  'username': superuser.username
-                                              },
-                                              created_by=superuser)
-                c.admin_role.members.add(superuser)
-                i = Inventory.objects.create(name='Demo Inventory',
-                                             organization=o,
-                                             created_by=superuser)
-                Host.objects.create(name='localhost',
-                                    inventory=i,
-                                    variables="ansible_connection: local\nansible_python_interpreter: '{{ ansible_playbook_python }}'",
-                                    created_by=superuser)
-                jt = JobTemplate.objects.create(name='Demo Job Template',
-                                                playbook='hello_world.yml',
-                                                project=p,
-                                                inventory=i)
-                jt.credentials.add(c)
-        print('Default organization added.')
-        print('Demo Credential, Inventory, and Job Template added.')
-        print('(changed: True)')
+                if not Organization.objects.exists():
+                    o = Organization.objects.create(name='Default')
+
+                    p = Project(name='Demo Project',
+                                scm_type='git',
+                                scm_url='https://github.com/ansible/ansible-tower-samples',
+                                scm_update_on_launch=True,
+                                scm_update_cache_timeout=0,
+                                organization=o)
+                    p.save(skip_update=True)
+
+                    ssh_type = CredentialType.objects.filter(namespace='ssh').first()
+                    c = Credential.objects.create(credential_type=ssh_type,
+                                                  name='Demo Credential',
+                                                  inputs={
+                                                      'username': superuser.username
+                                                  },
+                                                  created_by=superuser)
+
+                    c.admin_role.members.add(superuser)
+
+                    public_galaxy_credential = Credential(name='Ansible Galaxy',
+                                                          managed_by_tower=True,
+                                                          credential_type=CredentialType.objects.get(kind='galaxy'),
+                                                          inputs={'url': 'https://galaxy.ansible.com/'})
+                    public_galaxy_credential.save()
+                    o.galaxy_credentials.add(public_galaxy_credential)
+
+                    i = Inventory.objects.create(name='Demo Inventory',
+                                                 organization=o,
+                                                 created_by=superuser)
+
+                    Host.objects.create(name='localhost',
+                                        inventory=i,
+                                        variables="ansible_connection: local\nansible_python_interpreter: '{{ ansible_playbook_python }}'",
+                                        created_by=superuser)
+
+                    jt = JobTemplate.objects.create(name='Demo Job Template',
+                                                    playbook='hello_world.yml',
+                                                    project=p,
+                                                    inventory=i)
+                    jt.credentials.add(c)
+
+                    print('Default organization added.')
+                    print('Demo Credential, Inventory, and Job Template added.')
+                    changed = True
+
+                default_ee = settings.AWX_EXECUTION_ENVIRONMENT_DEFAULT_IMAGE
+                ee, created = ExecutionEnvironment.objects.get_or_create(name='Default EE', defaults={'image': default_ee,
+                                                                                                      'managed_by_tower': True})
+
+                if created:
+                    changed = True
+                    print('Default Execution Environment registered.')
+
+        if changed:
+            print('(changed: True)')
+        else:
+            print('(changed: False)')

awx/main/management/commands/deprovision_instance.py

@@ -1,8 +1,6 @@
 # Copyright (c) 2016 Ansible, Inc.
 # All Rights Reserved
 
-import subprocess
-
 from django.db import transaction
 from django.core.management.base import BaseCommand, CommandError
 
@@ -33,18 +31,9 @@ class Command(BaseCommand):
         with advisory_lock('instance_registration_%s' % hostname):
             instance = Instance.objects.filter(hostname=hostname)
             if instance.exists():
-                isolated = instance.first().is_isolated()
                 instance.delete()
                 print("Instance Removed")
-                if isolated:
-                    print('Successfully deprovisioned {}'.format(hostname))
-                else:
-                    result = subprocess.Popen("rabbitmqctl forget_cluster_node rabbitmq@{}".format(hostname), shell=True).wait()
-                    if result != 0:
-                        print("Node deprovisioning may have failed when attempting to "
-                              "remove the RabbitMQ instance {} from the cluster".format(hostname))
-                    else:
-                        print('Successfully deprovisioned {}'.format(hostname))
+                print('Successfully deprovisioned {}'.format(hostname))
                 print('(changed: True)')
             else:
                 print('No instance found matching name {}'.format(hostname))