Merge pull request #8723 from ryanpetrello/bump-16

Bump version to 16.0.0

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul

.gitignore

@@ -34,8 +34,6 @@ awx/ui_next/coverage/
 awx/ui_next/build
 awx/ui_next/.env.local
 rsyslog.pid
-/tower-license
-/tower-license/**
 tools/prometheus/data
 tools/docker-compose/Dockerfile
 
@@ -147,3 +145,4 @@ use_dev_supervisor.txt
 .idea/*
 *.unison.tmp
 *.#
+/tools/docker-compose/overrides/

CHANGELOG.md

@@ -2,12 +2,65 @@
 
 This is a list of high-level changes for each release of AWX. A full list of commits can be found at `https://github.com/ansible/awx/releases/tag/<version>`.
 
+## 16.0.0 (December 10, 2020)
+- AWX now ships with a reimagined user interface.  **Please read this before upgrading:** https://groups.google.com/g/awx-project/c/KuT5Ao92HWo
+- Removed support for syncing inventory from Red Hat CloudForms - https://github.com/ansible/awx/commit/0b701b3b2
+- Removed support for Mercurial-based project updates - https://github.com/ansible/awx/issues/7932
+- Upgraded NodeJS to actively maintained LTS 14.15.1 - https://github.com/ansible/awx/pull/8766
+- Added Git-LFS to the default image build - https://github.com/ansible/awx/pull/8700
+- Added the ability to specify `metadata.labels` in the podspec for container groups - https://github.com/ansible/awx/issues/8486
+- Added support for Kubernetes pod annotations - https://github.com/ansible/awx/pull/8434
+- Added the ability to label the web container in local Docker installs - https://github.com/ansible/awx/pull/8449
+- Added additional metadata (as an extra var) to playbook runs to report the SCM branch name - https://github.com/ansible/awx/pull/8433
+- Fixed a bug that caused k8s installations to fail due to an incorrect Helm repo - https://github.com/ansible/awx/issues/8715
+- Fixed a bug that prevented certain Workflow Approval resources from being deleted - https://github.com/ansible/awx/pull/8612
+- Fixed a bug that prevented the deletion of inventories stuck in "pending deletion" state - https://github.com/ansible/awx/issues/8525
+- Fixed a display bug in webhook notifications with certain unicode characters - https://github.com/ansible/awx/issues/7400
+- Improved support for exporting dependent objects (Inventory Hosts and Groups) in the `awx export` CLI tool - https://github.com/ansible/awx/commit/607bc0788
+
+## 15.0.1 (October 20, 2020)
+- Added several optimizations to improve performance for a variety of high-load simultaneous job launch use cases https://github.com/ansible/awx/pull/8403
+- Added the ability to source roles and collections from requirements.yaml files (not just requirements.yml) - https://github.com/ansible/awx/issues/4540
+- awx.awx collection modules now provide a clearer error message for incompatible versions of awxkit - https://github.com/ansible/awx/issues/8127
+- Fixed a bug in notification messages that contain certain unicode characters - https://github.com/ansible/awx/issues/7400
+- Fixed a bug that prevents the deletion of Workflow Approval records - https://github.com/ansible/awx/issues/8305
+- Fixed a bug that broke the selection of webhook credentials - https://github.com/ansible/awx/issues/7892
+- Fixed a bug which can cause confusing behavior for social auth logins across distinct browser tabs - https://github.com/ansible/awx/issues/8154
+- Fixed several bugs in the output of Workflow Job Templates using the `awx export` tool - https://github.com/ansible/awx/issues/7798 https://github.com/ansible/awx/pull/7847
+- Fixed a race condition that can lead to missing hosts when running parallel inventory syncs - https://github.com/ansible/awx/issues/5571
+- Fixed an HTTP 500 error when certain LDAP group parameters aren't properly set - https://github.com/ansible/awx/issues/7622
+- Updated a few dependencies in response to several CVEs:
+    * CVE-2020-7720
+    * CVE-2020-7743
+    * CVE-2020-7676
+
+## 15.0.0 (September 30, 2020)
+- Added improved support for fetching Ansible collections from private Galaxy content sources (such as https://github.com/ansible/galaxy_ng) - https://github.com/ansible/awx/issues/7813
+  **Note:** as part of this change, new Organizations created in the AWX API will _no longer_ automatically synchronize roles and collections from galaxy.ansible.com by default.  More details on this change can be found at:  https://github.com/ansible/awx/issues/8341#issuecomment-707310633
+- AWX now utilizes a version of certifi that auto-discovers certificates in the system certificate store - https://github.com/ansible/awx/pull/8242
+- Added support for arbitrary custom inventory plugin configuration: https://github.com/ansible/awx/issues/5150
+- Added an optional setting to disable the auto-creation of organizations and teams on successful SAML login. - https://github.com/ansible/awx/pull/8069
+- Added a number of optimizations to AWX's callback receiver to improve the speed of stdout processing for simultaneous playbooks runs - https://github.com/ansible/awx/pull/8193 https://github.com/ansible/awx/pull/8191
+- Added the ability to use `!include` and `!import` constructors when constructing YAML for use with the AWX CLI - https://github.com/ansible/awx/issues/8135
+- Fixed a bug that prevented certain users from being able to edit approval nodes in Workflows - https://github.com/ansible/awx/pull/8253
+- Fixed a bug that broke password prompting for credentials in certain cases - https://github.com/ansible/awx/issues/8202
+- Fixed a bug which can cause PostgreSQL deadlocks when running many parallel playbooks against large shared inventories - https://github.com/ansible/awx/issues/8145
+- Fixed a bug which can cause delays in AWX's task manager when large numbers of simultaneous jobs are scheduled - https://github.com/ansible/awx/issues/7655
+- Fixed a bug which can cause certain scheduled jobs - those that run every X minute(s) or hour(s) - to fail to run at the proper time - https://github.com/ansible/awx/issues/8071
+- Fixed a performance issue for playbooks that store large amounts of data using the `set_stats` module - https://github.com/ansible/awx/issues/8006
+- Fixed a bug related to AWX's handling of the auth_path argument for the HashiVault KeyValue credential plugin - https://github.com/ansible/awx/pull/7991
+- Fixed a bug that broke support for Remote Archive SCM Type project syncs on platforms that utilize Python2 - https://github.com/ansible/awx/pull/8057
+- Updated to the latest version of Django Rest Framework to address CVE-2020-25626
+- Updated to the latest version of Django to address CVE-2020-24583 and CVE-2020-24584
+- Updated to the latest verson of channels_redis to address a bug that slowly causes Daphne processes to leak memory over time - https://github.com/django/channels_redis/issues/212
+
 ## 14.1.0 (Aug 25, 2020)
 - AWX images can now be built on ARM64 - https://github.com/ansible/awx/pull/7607
 - Added the Remote Archive SCM Type to support using immutable artifacts and releases (such as tarballs and zip files) as projects - https://github.com/ansible/awx/issues/7954
 - Deprecated official support for Mercurial-based project updates - https://github.com/ansible/awx/issues/7932
 - Added resource import/export support to the official AWX collection - https://github.com/ansible/awx/issues/7329
 - Added the ability to import YAML-based resources (instead of just JSON) when using the AWX CLI - https://github.com/ansible/awx/pull/7808
+- Users upgrading from older versions of AWX may encounter an issue that causes their postgres container to restart in a loop (https://github.com/ansible/awx/issues/7854) - if you encounter this, bring your containers down and then back up (e.g., `docker-compose down && docker-compose up -d`) after upgrading to 14.1.0.
 - Updated the AWX CLI to export labels associated with Workflow Job Templates - https://github.com/ansible/awx/pull/7847
 - Updated to the latest python-ldap to address a bug - https://github.com/ansible/awx/issues/7868
 - Upgraded git-python to fix a bug that caused workflows to sometimes fail - https://github.com/ansible/awx/issues/6119
@@ -51,7 +104,7 @@ This is a list of high-level changes for each release of AWX. A full list of com
 - Fixed a bug that caused rsyslogd's configuration file to have world-readable file permissions, potentially leaking secrets (CVE-2020-10782)
 
 ## 12.0.0 (Jun 9, 2020)
-- Removed memcached as a dependency of AWX (https://github.com/ansible/awx/pull/7240) 
+- Removed memcached as a dependency of AWX (https://github.com/ansible/awx/pull/7240)
 - Moved to a single container image build instead of separate awx_web and awx_task images. The container image is just `awx` (https://github.com/ansible/awx/pull/7228)
 - Official AWX container image builds now use a two-stage container build process that notably reduces the size of our published images (https://github.com/ansible/awx/pull/7017)
 - Removed support for HipChat notifications ([EoL announcement](https://www.atlassian.com/partnerships/slack/faq#faq-98b17ca3-247f-423b-9a78-70a91681eff0)); all previously-created HipChat notification templates will be deleted due to this removal.

CONTRIBUTING.md

@@ -80,7 +80,7 @@ For Linux platforms, refer to the following from Docker:
 If you're not using Docker for Mac, or Docker for Windows, you may need, or choose to, install the Docker compose Python module separately, in which case you'll need to run the following:
 
 ```bash
-(host)$ pip install docker-compose
+(host)$ pip3 install docker-compose
 ```
 
 #### Frontend Development

INSTALL.md

@@ -78,10 +78,12 @@ Before you can run a deployment, you'll need the following installed in your loc
 - [docker](https://pypi.org/project/docker/) Python module
     + This is incompatible with `docker-py`. If you have previously installed `docker-py`, please uninstall it.
     + We use this module instead of `docker-py` because it is what the `docker-compose` Python module requires.
+- [community.general.docker_image collection](https://docs.ansible.com/ansible/latest/collections/community/general/docker_image_module.html)
+    + This is only required if you are using Ansible >= 2.10
 - [GNU Make](https://www.gnu.org/software/make/)
 - [Git](https://git-scm.com/) Requires Version 1.8.4+
 - Python 3.6+
-- [Node 10.x LTS version](https://nodejs.org/en/download/)
+- [Node 14.x LTS version](https://nodejs.org/en/download/)
     + This is only required if you're [building your own container images](#official-vs-building-images) with `use_container_for_build=false`
 - [NPM 6.x LTS](https://docs.npmjs.com/)
     + This is only required if you're [building your own container images](#official-vs-building-images) with `use_container_for_build=false`
@@ -662,6 +664,7 @@ The preferred way to install the AWX CLI is through pip directly from PyPI:
 
 To build the docs, spin up a real AWX server, `pip3 install sphinx sphinxcontrib-autoprogram`, and run:
 
+    ~ cd awxkit/awxkit/cli/docs
     ~ TOWER_HOST=https://awx.example.org TOWER_USERNAME=example TOWER_PASSWORD=secret make clean html
     ~ cd build/html/ && python -m http.server
     Serving HTTP on 0.0.0.0 port 8000 (http://0.0.0.0:8000/) ..

MANIFEST.in

@@ -4,8 +4,6 @@ recursive-include awx *.mo
 recursive-include awx/static *
 recursive-include awx/templates *.html
 recursive-include awx/api/templates *.md *.html
-recursive-include awx/ui/templates *.html
-recursive-include awx/ui/static *
 recursive-include awx/ui_next/build *.html
 recursive-include awx/ui_next/build *
 recursive-include awx/playbooks *.yml

Makefile

@@ -56,11 +56,6 @@ WHEEL_COMMAND ?= bdist_wheel
 SDIST_TAR_FILE ?= $(SDIST_TAR_NAME).tar.gz
 WHEEL_FILE ?= $(WHEEL_NAME)-py2-none-any.whl
 
-# UI flag files
-UI_DEPS_FLAG_FILE = awx/ui/.deps_built
-UI_RELEASE_DEPS_FLAG_FILE = awx/ui/.release_deps_built
-UI_RELEASE_FLAG_FILE = awx/ui/.release_built
-
 I18N_FLAG_FILE = .i18n_built
 
 .PHONY: awx-link clean clean-tmp clean-venv requirements requirements_dev \
@@ -70,22 +65,6 @@ I18N_FLAG_FILE = .i18n_built
 	ui-docker-machine ui-docker ui-release ui-devel \
 	ui-test ui-deps ui-test-ci VERSION
 
-# remove ui build artifacts
-clean-ui: clean-languages
-	rm -rf awx/ui/static/
-	rm -rf awx/ui/node_modules/
-	rm -rf awx/ui/test/unit/reports/
-	rm -rf awx/ui/test/spec/reports/
-	rm -rf awx/ui/test/e2e/reports/
-	rm -rf awx/ui/client/languages/
-	rm -rf awx/ui_next/node_modules/
-	rm -rf node_modules
-	rm -rf awx/ui_next/coverage/
-	rm -rf awx/ui_next/build/locales/_build/
-	rm -f $(UI_DEPS_FLAG_FILE)
-	rm -f $(UI_RELEASE_DEPS_FLAG_FILE)
-	rm -f $(UI_RELEASE_FLAG_FILE)
-
 clean-tmp:
 	rm -rf tmp/
 
@@ -214,7 +193,11 @@ requirements_awx_dev:
 
 requirements_collections:
 	mkdir -p $(COLLECTION_BASE)
-	ansible-galaxy collection install -r requirements/collections_requirements.yml -p $(COLLECTION_BASE)
+	n=0; \
+	until [ "$$n" -ge 5 ]; do \
+	    ansible-galaxy collection install -r requirements/collections_requirements.yml -p $(COLLECTION_BASE) && break; \
+	    n=$$((n+1)); \
+	done
 
 requirements: requirements_ansible requirements_awx requirements_collections
 
@@ -476,110 +459,23 @@ else
 	@echo No PO files
 endif
 
-# generate UI .pot
-pot: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run pot
-
-# generate django .pot .po
-LANG = "en-us"
-messages:
-	@if [ "$(VENV_BASE)" ]; then \
-		. $(VENV_BASE)/awx/bin/activate; \
-	fi; \
-	$(PYTHON) manage.py makemessages -l $(LANG) --keep-pot
-
-# generate l10n .json .mo
-languages: $(I18N_FLAG_FILE)
-
-$(I18N_FLAG_FILE): $(UI_RELEASE_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run languages
-	$(PYTHON) tools/scripts/compilemessages.py
-	touch $(I18N_FLAG_FILE)
-
-# End l10n TASKS
-# --------------------------------------
-
-# UI RELEASE TASKS
-# --------------------------------------
-ui-release: $(UI_RELEASE_FLAG_FILE)
-
-$(UI_RELEASE_FLAG_FILE): $(I18N_FLAG_FILE) $(UI_RELEASE_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run build-release
-	touch $(UI_RELEASE_FLAG_FILE)
-
-$(UI_RELEASE_DEPS_FLAG_FILE):
-	PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=1 $(NPM_BIN) --unsafe-perm --prefix awx/ui ci --no-save awx/ui
-	touch $(UI_RELEASE_DEPS_FLAG_FILE)
-
-# END UI RELEASE TASKS
-# --------------------------------------
 
 # UI TASKS
 # --------------------------------------
-ui-deps: $(UI_DEPS_FLAG_FILE)
-
-$(UI_DEPS_FLAG_FILE):
-	@if [ -f ${UI_RELEASE_DEPS_FLAG_FILE} ]; then \
-		rm -rf awx/ui/node_modules; \
-		rm -f ${UI_RELEASE_DEPS_FLAG_FILE}; \
-	fi; \
-	$(NPM_BIN) --unsafe-perm --prefix awx/ui ci --no-save awx/ui
-	touch $(UI_DEPS_FLAG_FILE)
-
-ui-docker-machine: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run ui-docker-machine -- $(MAKEFLAGS)
-
-# Native docker. Builds UI and raises BrowserSync & filesystem polling.
-ui-docker: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run ui-docker -- $(MAKEFLAGS)
-
-# Builds UI with development UI without raising browser-sync or filesystem polling.
-ui-devel: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run build-devel -- $(MAKEFLAGS)
-
-ui-test: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run test
-
-ui-lint: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) run --prefix awx/ui jshint
-	$(NPM_BIN) run --prefix awx/ui lint
-
-# A standard go-to target for API developers to use building the frontend
-ui: clean-ui ui-devel
-
-ui-test-ci: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) --prefix awx/ui run test:ci
-	$(NPM_BIN) --prefix awx/ui run unit
-
-jshint: $(UI_DEPS_FLAG_FILE)
-	$(NPM_BIN) run --prefix awx/ui jshint
-	$(NPM_BIN) run --prefix awx/ui lint
-
-ui-zuul-lint-and-test:
-	CHROMIUM_BIN=$(CHROMIUM_BIN) ./awx/ui/build/zuul_download_chromium.sh
-	CHROMIUM_BIN=$(CHROMIUM_BIN) PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=1 $(NPM_BIN) --unsafe-perm --prefix awx/ui ci --no-save awx/ui
-	CHROMIUM_BIN=$(CHROMIUM_BIN) $(NPM_BIN) run --prefix awx/ui jshint
-	CHROMIUM_BIN=$(CHROMIUM_BIN) $(NPM_BIN) run --prefix awx/ui lint
-	CHROME_BIN=$(CHROMIUM_BIN) $(NPM_BIN) --prefix awx/ui run test:ci
-	CHROME_BIN=$(CHROMIUM_BIN) $(NPM_BIN) --prefix awx/ui run unit
-
-# END UI TASKS
-# --------------------------------------
-
-# UI NEXT TASKS
-# --------------------------------------
-
 awx/ui_next/node_modules:
 	$(NPM_BIN) --prefix awx/ui_next install
 
-ui-release-next:
-	mkdir -p awx/ui_next/build/static
-	touch awx/ui_next/build/static/.placeholder
+clean-ui:
+	rm -rf node_modules
+	rm -rf awx/ui_next/node_modules
+	rm -rf awx/ui_next/build
 
-ui-devel-next: awx/ui_next/node_modules
+ui-release: ui-devel
+ui-devel: awx/ui_next/node_modules
 	$(NPM_BIN) --prefix awx/ui_next run extract-strings
 	$(NPM_BIN) --prefix awx/ui_next run compile-strings
 	$(NPM_BIN) --prefix awx/ui_next run build
+	git checkout awx/ui_next/src/locales
 	mkdir -p awx/public/static/css
 	mkdir -p awx/public/static/js
 	mkdir -p awx/public/static/media
@@ -587,19 +483,12 @@ ui-devel-next: awx/ui_next/node_modules
 	cp -r awx/ui_next/build/static/js/* awx/public/static/js
 	cp -r awx/ui_next/build/static/media/* awx/public/static/media
 
-clean-ui-next:
-	rm -rf node_modules
-	rm -rf awx/ui_next/node_modules
-	rm -rf awx/ui_next/build
-
-ui-next-zuul-lint-and-test:
+ui-zuul-lint-and-test:
 	$(NPM_BIN) --prefix awx/ui_next install
 	$(NPM_BIN) run --prefix awx/ui_next lint
 	$(NPM_BIN) run --prefix awx/ui_next prettier-check
 	$(NPM_BIN) run --prefix awx/ui_next test
 
-# END UI NEXT TASKS
-# --------------------------------------
 
 # Build a pip-installable package into dist/ with a timestamped version number.
 dev_build:
@@ -609,10 +498,10 @@ dev_build:
 release_build:
 	$(PYTHON) setup.py release_build
 
-dist/$(SDIST_TAR_FILE): ui-release ui-release-next VERSION
+dist/$(SDIST_TAR_FILE): ui-release VERSION
 	$(PYTHON) setup.py $(SDIST_COMMAND)
 
-dist/$(WHEEL_FILE): ui-release ui-release-next
+dist/$(WHEEL_FILE): ui-release
 	$(PYTHON) setup.py $(WHEEL_COMMAND)
 
 sdist: dist/$(SDIST_TAR_FILE)
@@ -646,9 +535,11 @@ awx/projects:
 docker-compose-isolated: awx/projects
 	CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml -f tools/docker-isolated-override.yml up
 
+COMPOSE_UP_OPTS ?=
+
 # Docker Compose Development environment
 docker-compose: docker-auth awx/projects
-	CURRENT_UID=$(shell id -u) OS="$(shell docker info | grep 'Operating System')" TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml up --no-recreate awx
+	CURRENT_UID=$(shell id -u) OS="$(shell docker info | grep 'Operating System')" TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose.yml $(COMPOSE_UP_OPTS) up --no-recreate awx
 
 docker-compose-cluster: docker-auth awx/projects
 	CURRENT_UID=$(shell id -u) TAG=$(COMPOSE_TAG) DEV_DOCKER_TAG_BASE=$(DEV_DOCKER_TAG_BASE) docker-compose -f tools/docker-compose-cluster.yml up

VERSION

@@ -1 +1 @@
-14.1.0
+16.0.0

awx/api/conf.py

@@ -16,6 +16,7 @@ register(
     help_text=_('Number of seconds that a user is inactive before they will need to login again.'),
     category=_('Authentication'),
     category_slug='authentication',
+    unit=_('seconds'),
 )
 register(
     'SESSIONS_PER_USER',
@@ -49,6 +50,7 @@ register(
                 'in the number of seconds.'),
     category=_('Authentication'),
     category_slug='authentication',
+    unit=_('seconds'),
 )
 register(
     'ALLOW_OAUTH2_FOR_EXTERNAL_USERS',

awx/api/generics.py

@@ -47,8 +47,6 @@ from awx.main.utils import (
     get_object_or_400,
     decrypt_field,
     get_awx_version,
-    get_licenser,
-    StubLicense
 )
 from awx.main.utils.db import get_all_field_names
 from awx.main.views import ApiErrorView
@@ -189,7 +187,8 @@ class APIView(views.APIView):
         '''
         Log warning for 400 requests.  Add header with elapsed time.
         '''
-
+        from awx.main.utils import get_licenser
+        from awx.main.utils.licensing import OpenLicense
         #
         # If the URL was rewritten, and we get a 404, we should entirely
         # replace the view in the request context with an ApiErrorView()
@@ -225,7 +224,8 @@ class APIView(views.APIView):
         response = super(APIView, self).finalize_response(request, response, *args, **kwargs)
         time_started = getattr(self, 'time_started', None)
         response['X-API-Product-Version'] = get_awx_version()
-        response['X-API-Product-Name'] = 'AWX' if isinstance(get_licenser(), StubLicense) else 'Red Hat Ansible Tower'
+        response['X-API-Product-Name'] = 'AWX' if isinstance(get_licenser(), OpenLicense) else 'Red Hat Ansible Tower'
+        
         response['X-API-Node'] = settings.CLUSTER_HOST_ID
         if time_started:
             time_elapsed = time.time() - self.time_started

awx/api/metadata.py

@@ -23,7 +23,7 @@ from rest_framework.request import clone_request
 # AWX
 from awx.api.fields import ChoiceNullField
 from awx.main.fields import JSONField, ImplicitRoleField
-from awx.main.models import InventorySource, NotificationTemplate
+from awx.main.models import NotificationTemplate
 from awx.main.scheduler.kubernetes import PodManager
 
 
@@ -39,7 +39,7 @@ class Metadata(metadata.SimpleMetadata):
             'min_length', 'max_length',
             'min_value', 'max_value',
             'category', 'category_slug',
-            'defined_in_file'
+            'defined_in_file', 'unit',
         ]
 
         for attr in text_attrs:
@@ -115,19 +115,6 @@ class Metadata(metadata.SimpleMetadata):
         if getattr(field, 'write_only', False):
             field_info['write_only'] = True
 
-        # Special handling of inventory source_region choices that vary based on
-        # selected inventory source.
-        if field.field_name == 'source_regions':
-            for cp in ('azure_rm', 'ec2', 'gce'):
-                get_regions = getattr(InventorySource, 'get_%s_region_choices' % cp)
-                field_info['%s_region_choices' % cp] = get_regions()
-
-        # Special handling of group_by choices for EC2.
-        if field.field_name == 'group_by':
-            for cp in ('ec2',):
-                get_group_by_choices = getattr(InventorySource, 'get_%s_group_by_choices' % cp)
-                field_info['%s_group_by_choices' % cp] = get_group_by_choices()
-
         # Special handling of notification configuration where the required properties
         # are conditional on the type selected.
         if field.field_name == 'notification_configuration':

awx/api/serializers.py

@@ -453,7 +453,7 @@ class BaseSerializer(serializers.ModelSerializer, metaclass=BaseSerializerMetacl
                 if 'capability_map' not in self.context:
                     if hasattr(self, 'polymorphic_base'):
                         model = self.polymorphic_base.Meta.model
-                        prefetch_list = self.polymorphic_base._capabilities_prefetch
+                        prefetch_list = self.polymorphic_base.capabilities_prefetch
                     else:
                         model = self.Meta.model
                         prefetch_list = self.capabilities_prefetch
@@ -640,12 +640,9 @@ class EmptySerializer(serializers.Serializer):
 
 
 class UnifiedJobTemplateSerializer(BaseSerializer):
-    # As a base serializer, the capabilities prefetch is not used directly
-    _capabilities_prefetch = [
-        'admin', 'execute',
-        {'copy': ['jobtemplate.project.use', 'jobtemplate.inventory.use',
-                  'organization.workflow_admin']}
-    ]
+    # As a base serializer, the capabilities prefetch is not used directly,
+    # instead they are derived from the Workflow Job Template Serializer and the Job Template Serializer, respectively.
+    capabilities_prefetch = []
 
     class Meta:
         model = UnifiedJobTemplate
@@ -695,7 +692,7 @@ class UnifiedJobTemplateSerializer(BaseSerializer):
                 serializer.polymorphic_base = self
                 # capabilities prefetch is only valid for these models
                 if isinstance(obj, (JobTemplate, WorkflowJobTemplate)):
-                    serializer.capabilities_prefetch = self._capabilities_prefetch
+                    serializer.capabilities_prefetch = serializer_class.capabilities_prefetch
                 else:
                     serializer.capabilities_prefetch = None
             return serializer.to_representation(obj)
@@ -1269,6 +1266,7 @@ class OrganizationSerializer(BaseSerializer):
             object_roles = self.reverse('api:organization_object_roles_list', kwargs={'pk': obj.pk}),
             access_list = self.reverse('api:organization_access_list', kwargs={'pk': obj.pk}),
             instance_groups = self.reverse('api:organization_instance_groups_list', kwargs={'pk': obj.pk}),
+            galaxy_credentials = self.reverse('api:organization_galaxy_credentials_list', kwargs={'pk': obj.pk}),
         ))
         return res
 
@@ -1332,6 +1330,8 @@ class ProjectOptionsSerializer(BaseSerializer):
             scm_type = attrs.get('scm_type', u'') or u''
         if self.instance and not scm_type:
             valid_local_paths.append(self.instance.local_path)
+        if self.instance and scm_type and "local_path" in attrs and self.instance.local_path != attrs['local_path']:
+            errors['local_path'] = _(f'Cannot change local_path for {scm_type}-based projects')
         if scm_type:
             attrs.pop('local_path', None)
         if 'local_path' in attrs and attrs['local_path'] not in valid_local_paths:
@@ -1702,7 +1702,10 @@ class HostSerializer(BaseSerializerWithVariables):
             'type': j.job.job_type_name,
             'status': j.job.status,
             'finished': j.job.finished,
-        } for j in obj.job_host_summaries.select_related('job__job_template').order_by('-created')[:5]])
+        } for j in obj.job_host_summaries.select_related('job__job_template').order_by('-created').defer(
+            'job__extra_vars',
+            'job__artifacts',
+        )[:5]])
         return d
 
     def _get_host_port_from_name(self, name):
@@ -1745,7 +1748,7 @@ class HostSerializer(BaseSerializerWithVariables):
             attrs['variables'] = json.dumps(vars_dict)
         if Group.objects.filter(name=name, inventory=inventory).exists():
             raise serializers.ValidationError(_('A Group with that name already exists.'))
-            
+
         return super(HostSerializer, self).validate(attrs)
 
     def to_representation(self, obj):
@@ -1934,7 +1937,7 @@ class InventorySourceOptionsSerializer(BaseSerializer):
 
     class Meta:
         fields = ('*', 'source', 'source_path', 'source_script', 'source_vars', 'credential',
-                  'source_regions', 'instance_filters', 'group_by', 'overwrite', 'overwrite_vars',
+                  'enabled_var', 'enabled_value', 'host_filter', 'overwrite', 'overwrite_vars',
                   'custom_virtualenv', 'timeout', 'verbosity')
 
     def get_related(self, obj):
@@ -1954,7 +1957,7 @@ class InventorySourceOptionsSerializer(BaseSerializer):
         return ret
 
     def validate(self, attrs):
-        # TODO: Validate source, validate source_regions
+        # TODO: Validate source
         errors = {}
 
         source = attrs.get('source', self.instance and self.instance.source or '')
@@ -2533,10 +2536,11 @@ class CredentialTypeSerializer(BaseSerializer):
 class CredentialSerializer(BaseSerializer):
     show_capabilities = ['edit', 'delete', 'copy', 'use']
     capabilities_prefetch = ['admin', 'use']
+    managed_by_tower = serializers.ReadOnlyField()
 
     class Meta:
         model = Credential
-        fields = ('*', 'organization', 'credential_type', 'inputs', 'kind', 'cloud', 'kubernetes')
+        fields = ('*', 'organization', 'credential_type', 'managed_by_tower', 'inputs', 'kind', 'cloud', 'kubernetes')
         extra_kwargs = {
             'credential_type': {
                 'label': _('Credential Type'),
@@ -2600,6 +2604,13 @@ class CredentialSerializer(BaseSerializer):
 
         return summary_dict
 
+    def validate(self, attrs):
+        if self.instance and self.instance.managed_by_tower:
+            raise PermissionDenied(
+                detail=_("Modifications not allowed for managed credentials")
+            )
+        return super(CredentialSerializer, self).validate(attrs)
+
     def get_validation_exclusions(self, obj=None):
         ret = super(CredentialSerializer, self).get_validation_exclusions(obj)
         for field in ('credential_type', 'inputs'):
@@ -2607,6 +2618,17 @@ class CredentialSerializer(BaseSerializer):
                 ret.remove(field)
         return ret
 
+    def validate_organization(self, org):
+        if (
+            self.instance and
+            self.instance.credential_type.kind == 'galaxy' and
+            org is None
+        ):
+            raise serializers.ValidationError(_(
+                "Galaxy credentials must be owned by an Organization."
+            ))
+        return org
+
     def validate_credential_type(self, credential_type):
         if self.instance and credential_type.pk != self.instance.credential_type.pk:
             for related_objects in (
@@ -2671,6 +2693,15 @@ class CredentialSerializerCreate(CredentialSerializer):
         if attrs.get('team'):
             attrs['organization'] = attrs['team'].organization
 
+        if (
+            'credential_type' in attrs and
+            attrs['credential_type'].kind == 'galaxy' and
+            list(owner_fields) != ['organization']
+        ):
+            raise serializers.ValidationError({"organization": _(
+                "Galaxy credentials must be owned by an Organization."
+            )})
+
         return super(CredentialSerializerCreate, self).validate(attrs)
 
     def create(self, validated_data):
@@ -3406,6 +3437,12 @@ class WorkflowJobTemplateSerializer(JobTemplateMixin, LabelsListMixin, UnifiedJo
             res['organization'] = self.reverse('api:organization_detail',   kwargs={'pk': obj.organization.pk})
         if obj.webhook_credential_id:
             res['webhook_credential'] = self.reverse('api:credential_detail', kwargs={'pk': obj.webhook_credential_id})
+        if obj.inventory_id:
+            res['inventory'] = self.reverse(
+                'api:inventory_detail', kwargs={
+                    'pk': obj.inventory_id
+                }
+            )
         return res
 
     def validate_extra_vars(self, value):
@@ -3908,12 +3945,12 @@ class ProjectUpdateEventSerializer(JobEventSerializer):
         return UriCleaner.remove_sensitive(obj.stdout)
 
     def get_event_data(self, obj):
-        # the project update playbook uses the git, hg, or svn modules
+        # the project update playbook uses the git or svn modules
         # to clone repositories, and those modules are prone to printing
         # raw SCM URLs in their stdout (which *could* contain passwords)
         # attempt to detect and filter HTTP basic auth passwords in the stdout
         # of these types of events
-        if obj.event_data.get('task_action') in ('git', 'hg', 'svn'):
+        if obj.event_data.get('task_action') in ('git', 'svn'):
             try:
                 return json.loads(
                     UriCleaner.remove_sensitive(
@@ -4125,7 +4162,10 @@ class JobLaunchSerializer(BaseSerializer):
         # verify that credentials (either provided or existing) don't
         # require launch-time passwords that have not been provided
         if 'credentials' in accepted:
-            launch_credentials = accepted['credentials']
+            launch_credentials = Credential.unique_dict(
+                list(template_credentials.all()) +
+                list(accepted['credentials'])
+            ).values()
         else:
             launch_credentials = template_credentials
         passwords = attrs.get('credential_passwords', {})  # get from original attrs

awx/api/templates/api/_schedule_detail.md

@@ -4,7 +4,6 @@ The following lists the expected format and details of our rrules:
 * DTSTART is expected to be in UTC
 * INTERVAL is required
 * SECONDLY is not supported
-* TZID is not supported
 * RRULE must precede the rule statements
 * BYDAY is supported but not BYDAY with a numerical prefix
 * BYYEARDAY and BYWEEKNO are not supported

awx/api/templates/api/dashboard_jobs_graph_view.md

@@ -8,7 +8,7 @@ The `period` of the data can be adjusted with:
 
     ?period=month
 
-Where `month` can be replaced with `week`, or `day`.  `month` is the default.
+Where `month` can be replaced with `week`, `two_weeks`, or `day`.  `month` is the default.
 
 The type of job can be filtered with:
 

awx/api/urls/organization.py

@@ -21,6 +21,7 @@ from awx.api.views import (
     OrganizationNotificationTemplatesSuccessList,
     OrganizationNotificationTemplatesApprovalList,
     OrganizationInstanceGroupsList,
+    OrganizationGalaxyCredentialsList,
     OrganizationObjectRolesList,
     OrganizationAccessList,
     OrganizationApplicationList,
@@ -49,6 +50,7 @@ urls = [
     url(r'^(?P<pk>[0-9]+)/notification_templates_approvals/$', OrganizationNotificationTemplatesApprovalList.as_view(),
         name='organization_notification_templates_approvals_list'),
     url(r'^(?P<pk>[0-9]+)/instance_groups/$', OrganizationInstanceGroupsList.as_view(), name='organization_instance_groups_list'),
+    url(r'^(?P<pk>[0-9]+)/galaxy_credentials/$', OrganizationGalaxyCredentialsList.as_view(), name='organization_galaxy_credentials_list'),
     url(r'^(?P<pk>[0-9]+)/object_roles/$', OrganizationObjectRolesList.as_view(), name='organization_object_roles_list'),
     url(r'^(?P<pk>[0-9]+)/access_list/$', OrganizationAccessList.as_view(), name='organization_access_list'),
     url(r'^(?P<pk>[0-9]+)/applications/$', OrganizationApplicationList.as_view(), name='organization_applications_list'),

awx/api/urls/urls.py

@@ -15,6 +15,7 @@ from awx.api.views import (
     ApiV2PingView,
     ApiV2ConfigView,
     ApiV2SubscriptionView,
+    ApiV2AttachView,
     AuthView,
     UserMeList,
     DashboardView,
@@ -94,6 +95,7 @@ v2_urls = [
     url(r'^ping/$', ApiV2PingView.as_view(), name='api_v2_ping_view'),
     url(r'^config/$', ApiV2ConfigView.as_view(), name='api_v2_config_view'),
     url(r'^config/subscriptions/$', ApiV2SubscriptionView.as_view(), name='api_v2_subscription_view'),
+    url(r'^config/attach/$', ApiV2AttachView.as_view(), name='api_v2_attach_view'),
     url(r'^auth/$', AuthView.as_view()),
     url(r'^me/$', UserMeList.as_view(), name='user_me_list'),
     url(r'^dashboard/$', DashboardView.as_view(), name='dashboard_view'),

awx/api/views/__init__.py

@@ -124,6 +124,7 @@ from awx.api.views.organization import ( # noqa
     OrganizationNotificationTemplatesSuccessList,
     OrganizationNotificationTemplatesApprovalList,
     OrganizationInstanceGroupsList,
+    OrganizationGalaxyCredentialsList,
     OrganizationAccessList,
     OrganizationObjectRolesList,
 )
@@ -152,6 +153,7 @@ from awx.api.views.root import ( # noqa
     ApiV2PingView,
     ApiV2ConfigView,
     ApiV2SubscriptionView,
+    ApiV2AttachView,
 )
 from awx.api.views.webhooks import ( # noqa
     WebhookKeyView,
@@ -240,8 +242,6 @@ class DashboardView(APIView):
         git_failed_projects = git_projects.filter(last_job_failed=True)
         svn_projects = user_projects.filter(scm_type='svn')
         svn_failed_projects = svn_projects.filter(last_job_failed=True)
-        hg_projects = user_projects.filter(scm_type='hg')
-        hg_failed_projects = hg_projects.filter(last_job_failed=True)
         archive_projects = user_projects.filter(scm_type='archive')
         archive_failed_projects = archive_projects.filter(last_job_failed=True)
         data['scm_types'] = {}
@@ -255,11 +255,6 @@ class DashboardView(APIView):
                                     'failures_url': reverse('api:project_list', request=request) + "?scm_type=svn&last_job_failed=True",
                                     'total': svn_projects.count(),
                                     'failed': svn_failed_projects.count()}
-        data['scm_types']['hg'] = {'url': reverse('api:project_list', request=request) + "?scm_type=hg",
-                                   'label': 'Mercurial',
-                                   'failures_url': reverse('api:project_list', request=request) + "?scm_type=hg&last_job_failed=True",
-                                   'total': hg_projects.count(),
-                                   'failed': hg_failed_projects.count()}
         data['scm_types']['archive'] = {'url': reverse('api:project_list', request=request) + "?scm_type=archive",
                                         'label': 'Remote Archive',
                                         'failures_url': reverse('api:project_list', request=request) + "?scm_type=archive&last_job_failed=True",
@@ -315,6 +310,9 @@ class DashboardJobsGraphView(APIView):
         if period == 'month':
             end_date = start_date - dateutil.relativedelta.relativedelta(months=1)
             interval = 'days'
+        elif period == 'two_weeks':
+            end_date = start_date - dateutil.relativedelta.relativedelta(weeks=2)
+            interval = 'days'
         elif period == 'week':
             end_date = start_date - dateutil.relativedelta.relativedelta(weeks=1)
             interval = 'days'
@@ -1355,6 +1353,13 @@ class CredentialDetail(RetrieveUpdateDestroyAPIView):
     model = models.Credential
     serializer_class = serializers.CredentialSerializer
 
+    def destroy(self, request, *args, **kwargs):
+        instance = self.get_object()
+        if instance.managed_by_tower:
+            raise PermissionDenied(detail=_("Deletion not allowed for managed credentials"))
+        return super(CredentialDetail, self).destroy(request, *args, **kwargs)
+
+
 
 class CredentialActivityStreamList(SubListAPIView):
 
@@ -3035,7 +3040,7 @@ class WorkflowJobTemplateNodeCreateApproval(RetrieveAPIView):
             approval_template,
             context=self.get_serializer_context()
         ).data
-        return Response(data, status=status.HTTP_200_OK)
+        return Response(data, status=status.HTTP_201_CREATED)
 
     def check_permissions(self, request):
         obj = self.get_object().workflow_job_template
@@ -4245,7 +4250,9 @@ class NotificationTemplateDetail(RetrieveUpdateDestroyAPIView):
         obj = self.get_object()
         if not request.user.can_access(self.model, 'delete', obj):
             return Response(status=status.HTTP_404_NOT_FOUND)
-        if obj.notifications.filter(status='pending').exists():
+
+        hours_old = now() - dateutil.relativedelta.relativedelta(hours=8)
+        if obj.notifications.filter(status='pending', created__gt=hours_old).exists():
             return Response({"error": _("Delete not allowed while there are pending notifications")},
                             status=status.HTTP_405_METHOD_NOT_ALLOWED)
         return super(NotificationTemplateDetail, self).delete(request, *args, **kwargs)

awx/api/views/metrics.py

@@ -22,7 +22,7 @@ from awx.api.generics import (
 )
 
 
-logger = logging.getLogger('awx.main.analytics')
+logger = logging.getLogger('awx.analytics')
 
 
 class MetricsView(APIView):

awx/api/views/organization.py

@@ -7,6 +7,7 @@ import logging
 # Django
 from django.db.models import Count
 from django.contrib.contenttypes.models import ContentType
+from django.utils.translation import ugettext_lazy as _
 
 # AWX
 from awx.main.models import (
@@ -20,7 +21,8 @@ from awx.main.models import (
     Role,
     User,
     Team,
-    InstanceGroup
+    InstanceGroup,
+    Credential
 )
 from awx.api.generics import (
     ListCreateAPIView,
@@ -42,7 +44,8 @@ from awx.api.serializers import (
     RoleSerializer,
     NotificationTemplateSerializer,
     InstanceGroupSerializer,
-    ProjectSerializer, JobTemplateSerializer, WorkflowJobTemplateSerializer
+    ProjectSerializer, JobTemplateSerializer, WorkflowJobTemplateSerializer,
+    CredentialSerializer
 )
 from awx.api.views.mixin import (
     RelatedJobsPreventDeleteMixin,
@@ -214,6 +217,20 @@ class OrganizationInstanceGroupsList(SubListAttachDetachAPIView):
     relationship = 'instance_groups'
 
 
+class OrganizationGalaxyCredentialsList(SubListAttachDetachAPIView):
+
+    model = Credential
+    serializer_class = CredentialSerializer
+    parent_model = Organization
+    relationship = 'galaxy_credentials'
+
+    def is_valid_relation(self, parent, sub, created=False):
+        if sub.kind != 'galaxy_api_token':
+            return {'msg': _(
+                f"Credential must be a Galaxy credential, not {sub.credential_type.name}."
+            )}
+
+
 class OrganizationAccessList(ResourceAccessList):
 
     model = User # needs to be User for AccessLists's

awx/api/views/root.py

@@ -1,9 +1,10 @@
 # Copyright (c) 2018 Ansible, Inc.
 # All Rights Reserved.
 
+import base64
+import json
 import logging
 import operator
-import json
 from collections import OrderedDict
 
 from django.conf import settings
@@ -21,6 +22,7 @@ import requests
 
 from awx.api.generics import APIView
 from awx.conf.registry import settings_registry
+from awx.main.analytics import all_collectors
 from awx.main.ha import is_ha_environment
 from awx.main.utils import (
     get_awx_version,
@@ -28,8 +30,8 @@ from awx.main.utils import (
     get_custom_venv_choices,
     to_python_boolean,
 )
+from awx.main.utils.licensing import validate_entitlement_manifest
 from awx.api.versioning import reverse, drf_reverse
-from awx.conf.license import get_license
 from awx.main.constants import PRIVILEGE_ESCALATION_METHODS
 from awx.main.models import (
     Project,
@@ -177,7 +179,7 @@ class ApiV2PingView(APIView):
 class ApiV2SubscriptionView(APIView):
 
     permission_classes = (IsAuthenticated,)
-    name = _('Configuration')
+    name = _('Subscriptions')
     swagger_topic = 'System Configuration'
 
     def check_permissions(self, request):
@@ -188,18 +190,18 @@ class ApiV2SubscriptionView(APIView):
     def post(self, request):
         from awx.main.utils.common import get_licenser
         data = request.data.copy()
-        if data.get('rh_password') == '$encrypted$':
-            data['rh_password'] = settings.REDHAT_PASSWORD
+        if data.get('subscriptions_password') == '$encrypted$':
+            data['subscriptions_password'] = settings.SUBSCRIPTIONS_PASSWORD
         try:
-            user, pw = data.get('rh_username'), data.get('rh_password')
+            user, pw = data.get('subscriptions_username'), data.get('subscriptions_password')
             with set_environ(**settings.AWX_TASK_ENV):
                 validated = get_licenser().validate_rh(user, pw)
             if user:
-                settings.REDHAT_USERNAME = data['rh_username']
+                settings.SUBSCRIPTIONS_USERNAME = data['subscriptions_username']
             if pw:
-                settings.REDHAT_PASSWORD = data['rh_password']
+                settings.SUBSCRIPTIONS_PASSWORD = data['subscriptions_password']
         except Exception as exc:
-            msg = _("Invalid License")
+            msg = _("Invalid Subscription")
             if (
                 isinstance(exc, requests.exceptions.HTTPError) and
                 getattr(getattr(exc, 'response', None), 'status_code', None) == 401
@@ -212,13 +214,63 @@ class ApiV2SubscriptionView(APIView):
             elif isinstance(exc, (ValueError, OSError)) and exc.args:
                 msg = exc.args[0]
             else:
-                logger.exception(smart_text(u"Invalid license submitted."),
+                logger.exception(smart_text(u"Invalid subscription submitted."),
                                  extra=dict(actor=request.user.username))
             return Response({"error": msg}, status=status.HTTP_400_BAD_REQUEST)
 
         return Response(validated)
 
 
+class ApiV2AttachView(APIView):
+
+    permission_classes = (IsAuthenticated,)
+    name = _('Attach Subscription')
+    swagger_topic = 'System Configuration'
+
+    def check_permissions(self, request):
+        super(ApiV2AttachView, self).check_permissions(request)
+        if not request.user.is_superuser and request.method.lower() not in {'options', 'head'}:
+            self.permission_denied(request)  # Raises PermissionDenied exception.
+
+    def post(self, request):
+        data = request.data.copy()
+        pool_id = data.get('pool_id', None)
+        if not pool_id:
+            return Response({"error": _("No subscription pool ID provided.")}, status=status.HTTP_400_BAD_REQUEST)
+        user = getattr(settings, 'SUBSCRIPTIONS_USERNAME', None)
+        pw = getattr(settings, 'SUBSCRIPTIONS_PASSWORD', None)
+        if pool_id and user and pw:
+            from awx.main.utils.common import get_licenser
+            data = request.data.copy()
+            try:
+                with set_environ(**settings.AWX_TASK_ENV):
+                    validated = get_licenser().validate_rh(user, pw)
+            except Exception as exc:
+                msg = _("Invalid Subscription")
+                if (
+                    isinstance(exc, requests.exceptions.HTTPError) and
+                    getattr(getattr(exc, 'response', None), 'status_code', None) == 401
+                ):
+                    msg = _("The provided credentials are invalid (HTTP 401).")
+                elif isinstance(exc, requests.exceptions.ProxyError):
+                    msg = _("Unable to connect to proxy server.")
+                elif isinstance(exc, requests.exceptions.ConnectionError):
+                    msg = _("Could not connect to subscription service.")
+                elif isinstance(exc, (ValueError, OSError)) and exc.args:
+                    msg = exc.args[0]
+                else:
+                    logger.exception(smart_text(u"Invalid subscription submitted."),
+                                     extra=dict(actor=request.user.username))
+                return Response({"error": msg}, status=status.HTTP_400_BAD_REQUEST)
+        for sub in validated:
+            if sub['pool_id'] == pool_id:
+                sub['valid_key'] = True
+                settings.LICENSE = sub
+                return Response(sub)
+
+        return Response({"error": _("Error processing subscription metadata.")}, status=status.HTTP_400_BAD_REQUEST)
+
+
 class ApiV2ConfigView(APIView):
 
     permission_classes = (IsAuthenticated,)
@@ -233,15 +285,11 @@ class ApiV2ConfigView(APIView):
     def get(self, request, format=None):
         '''Return various sitewide configuration settings'''
 
-        if request.user.is_superuser or request.user.is_system_auditor:
-            license_data = get_license(show_key=True)
-        else:
-            license_data = get_license(show_key=False)
+        from awx.main.utils.common import get_licenser
+        license_data = get_licenser().validate()
+
         if not license_data.get('valid_key', False):
             license_data = {}
-        if license_data and 'features' in license_data and 'activity_streams' in license_data['features']:
-            # FIXME: Make the final setting value dependent on the feature?
-            license_data['features']['activity_streams'] &= settings.ACTIVITY_STREAM_ENABLED
 
         pendo_state = settings.PENDO_TRACKING_STATE if settings.PENDO_TRACKING_STATE in ('off', 'anonymous', 'detailed') else 'off'
 
@@ -252,6 +300,7 @@ class ApiV2ConfigView(APIView):
             ansible_version=get_ansible_version(),
             eula=render_to_string("eula.md") if license_data.get('license_type', 'UNLICENSED') != 'open' else '',
             analytics_status=pendo_state,
+            analytics_collectors=all_collectors(),
             become_methods=PRIVILEGE_ESCALATION_METHODS,
         )
 
@@ -279,9 +328,10 @@ class ApiV2ConfigView(APIView):
 
         return Response(data)
 
+
     def post(self, request):
         if not isinstance(request.data, dict):
-            return Response({"error": _("Invalid license data")}, status=status.HTTP_400_BAD_REQUEST)
+            return Response({"error": _("Invalid subscription data")}, status=status.HTTP_400_BAD_REQUEST)
         if "eula_accepted" not in request.data:
             return Response({"error": _("Missing 'eula_accepted' property")}, status=status.HTTP_400_BAD_REQUEST)
         try:
@@ -298,25 +348,47 @@ class ApiV2ConfigView(APIView):
             logger.info(smart_text(u"Invalid JSON submitted for license."),
                         extra=dict(actor=request.user.username))
             return Response({"error": _("Invalid JSON")}, status=status.HTTP_400_BAD_REQUEST)
-        try:
-            from awx.main.utils.common import get_licenser
-            license_data = json.loads(data_actual)
-            license_data_validated = get_licenser(**license_data).validate()
-        except Exception:
-            logger.warning(smart_text(u"Invalid license submitted."),
-                           extra=dict(actor=request.user.username))
-            return Response({"error": _("Invalid License")}, status=status.HTTP_400_BAD_REQUEST)
+
+        from awx.main.utils.common import get_licenser
+        license_data = json.loads(data_actual)
+        if 'license_key' in license_data:
+            return Response({"error": _('Legacy license submitted. A subscription manifest is now required.')}, status=status.HTTP_400_BAD_REQUEST)
+        if 'manifest' in license_data:
+            try:
+                json_actual = json.loads(base64.b64decode(license_data['manifest']))
+                if 'license_key' in json_actual:
+                    return Response(
+                        {"error": _('Legacy license submitted. A subscription manifest is now required.')},
+                        status=status.HTTP_400_BAD_REQUEST
+                    )
+            except Exception:
+                pass
+            try:
+                license_data = validate_entitlement_manifest(license_data['manifest'])
+            except ValueError as e:
+                return Response({"error": str(e)}, status=status.HTTP_400_BAD_REQUEST)
+            except Exception:
+                logger.exception('Invalid manifest submitted. {}')
+                return Response({"error": _('Invalid manifest submitted.')}, status=status.HTTP_400_BAD_REQUEST)
+
+            try:
+                license_data_validated = get_licenser().license_from_manifest(license_data)
+            except Exception:
+                logger.warning(smart_text(u"Invalid subscription submitted."),
+                               extra=dict(actor=request.user.username))
+                return Response({"error": _("Invalid License")}, status=status.HTTP_400_BAD_REQUEST)
+        else:
+            license_data_validated = get_licenser().validate()
 
         # If the license is valid, write it to the database.
         if license_data_validated['valid_key']:
-            settings.LICENSE = license_data
             if not settings_registry.is_setting_read_only('TOWER_URL_BASE'):
                 settings.TOWER_URL_BASE = "{}://{}".format(request.scheme, request.get_host())
             return Response(license_data_validated)
 
-        logger.warning(smart_text(u"Invalid license submitted."),
+        logger.warning(smart_text(u"Invalid subscription submitted."),
                        extra=dict(actor=request.user.username))
-        return Response({"error": _("Invalid license")}, status=status.HTTP_400_BAD_REQUEST)
+        return Response({"error": _("Invalid subscription")}, status=status.HTTP_400_BAD_REQUEST)
 
     def delete(self, request):
         try:

awx/asgi.py

@@ -25,10 +25,12 @@ if MODE == 'production':
     try:
         fd = open("/var/lib/awx/.tower_version", "r")
         if fd.read().strip() != tower_version:
-            raise Exception()
-    except Exception:
+            raise ValueError()
+    except FileNotFoundError:
+        pass
+    except ValueError as e:
         logger.error("Missing or incorrect metadata for Tower version.  Ensure Tower was installed using the setup playbook.")
-        raise Exception("Missing or incorrect metadata for Tower version.  Ensure Tower was installed using the setup playbook.")
+        raise Exception("Missing or incorrect metadata for Tower version.  Ensure Tower was installed using the setup playbook.") from e
 
 
 os.environ.setdefault("DJANGO_SETTINGS_MODULE", "awx.settings")

awx/conf/license.py

@@ -1,18 +1,14 @@
 # Copyright (c) 2016 Ansible, Inc.
 # All Rights Reserved.
 
-
 __all__ = ['get_license']
 
 
 def _get_validated_license_data():
-    from awx.main.utils.common import get_licenser
+    from awx.main.utils import get_licenser
     return get_licenser().validate()
 
 
-def get_license(show_key=False):
+def get_license():
     """Return a dictionary representing the active license on this Tower instance."""
-    license_data = _get_validated_license_data()
-    if not show_key:
-        license_data.pop('license_key', None)
-    return license_data
+    return _get_validated_license_data()

awx/conf/migrations/0008_subscriptions.py

@@ -0,0 +1,26 @@
+# Generated by Django 2.2.11 on 2020-08-04 15:19
+
+import logging
+
+from django.db import migrations
+
+from awx.conf.migrations._subscriptions import clear_old_license, prefill_rh_credentials
+
+logger = logging.getLogger('awx.conf.migrations')
+
+
+def _noop(apps, schema_editor):
+    pass
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('conf', '0007_v380_rename_more_settings'),
+    ]
+
+
+    operations = [
+        migrations.RunPython(clear_old_license, _noop),
+        migrations.RunPython(prefill_rh_credentials, _noop)
+    ]

awx/conf/migrations/_subscriptions.py

@@ -0,0 +1,34 @@
+# -*- coding: utf-8 -*-
+import logging
+from django.utils.timezone import now
+from awx.main.utils.encryption import decrypt_field, encrypt_field
+
+logger = logging.getLogger('awx.conf.settings')
+
+__all__ = ['clear_old_license', 'prefill_rh_credentials']
+    
+
+def clear_old_license(apps, schema_editor):
+    Setting = apps.get_model('conf', 'Setting')
+    Setting.objects.filter(key='LICENSE').delete()
+
+
+def _migrate_setting(apps, old_key, new_key, encrypted=False):
+    Setting = apps.get_model('conf', 'Setting')
+    if not Setting.objects.filter(key=old_key).exists():
+        return
+    new_setting = Setting.objects.create(key=new_key,
+                                         created=now(),
+                                         modified=now()
+                                         )
+    if encrypted:
+        new_setting.value = decrypt_field(Setting.objects.filter(key=old_key).first(), 'value')
+        new_setting.value = encrypt_field(new_setting, 'value')
+    else:
+        new_setting.value = getattr(Setting.objects.filter(key=old_key).first(), 'value')
+    new_setting.save()
+
+
+def prefill_rh_credentials(apps, schema_editor):
+    _migrate_setting(apps, 'REDHAT_USERNAME', 'SUBSCRIPTIONS_USERNAME', encrypted=False)
+    _migrate_setting(apps, 'REDHAT_PASSWORD', 'SUBSCRIPTIONS_PASSWORD', encrypted=True)

awx/conf/models.py

@@ -78,14 +78,6 @@ class Setting(CreatedModifiedModel):
     def get_cache_id_key(self, key):
         return '{}_ID'.format(key)
 
-    def display_value(self):
-        if self.key == 'LICENSE' and 'license_key' in self.value:
-            # don't log the license key in activity stream
-            value = self.value.copy()
-            value['license_key'] = '********'
-            return value
-        return self.value
-
 
 import awx.conf.signals  # noqa
 

awx/conf/registry.py

@@ -129,12 +129,14 @@ class SettingsRegistry(object):
         placeholder = field_kwargs.pop('placeholder', empty)
         encrypted = bool(field_kwargs.pop('encrypted', False))
         defined_in_file = bool(field_kwargs.pop('defined_in_file', False))
+        unit = field_kwargs.pop('unit', None)
         if getattr(field_kwargs.get('child', None), 'source', None) is not None:
             field_kwargs['child'].source = None
         field_instance = field_class(**field_kwargs)
         field_instance.category_slug = category_slug
         field_instance.category = category
         field_instance.depends_on = depends_on
+        field_instance.unit = unit
         if placeholder is not empty:
             field_instance.placeholder = placeholder
         field_instance.defined_in_file = defined_in_file

awx/conf/settings.py

@@ -17,6 +17,8 @@ from django.utils.functional import cached_property
 # Django REST Framework
 from rest_framework.fields import empty, SkipField
 
+import cachetools
+
 # Tower
 from awx.main.utils import encrypt_field, decrypt_field
 from awx.conf import settings_registry
@@ -28,6 +30,8 @@ from awx.conf.migrations._reencrypt import decrypt_field as old_decrypt_field
 
 logger = logging.getLogger('awx.conf.settings')
 
+SETTING_MEMORY_TTL = 5 if 'callback_receiver' in ' '.join(sys.argv) else 0
+
 # Store a special value to indicate when a setting is not set in the database.
 SETTING_CACHE_NOTSET = '___notset___'
 
@@ -406,6 +410,7 @@ class SettingsWrapper(UserSettingsHolder):
     def SETTINGS_MODULE(self):
         return self._get_default('SETTINGS_MODULE')
 
+    @cachetools.cached(cache=cachetools.TTLCache(maxsize=2048, ttl=SETTING_MEMORY_TTL))
     def __getattr__(self, name):
         value = empty
         if name in self.all_supported_settings:

awx/main/access.py

@@ -333,14 +333,14 @@ class BaseAccess(object):
             report_violation(_("License has expired."))
 
         free_instances = validation_info.get('free_instances', 0)
-        available_instances = validation_info.get('available_instances', 0)
+        instance_count = validation_info.get('instance_count', 0)
 
         if add_host_name:
             host_exists = Host.objects.filter(name=add_host_name).exists()
             if not host_exists and free_instances == 0:
-                report_violation(_("License count of %s instances has been reached.") % available_instances)
+                report_violation(_("License count of %s instances has been reached.") % instance_count)
             elif not host_exists and free_instances < 0:
-                report_violation(_("License count of %s instances has been exceeded.") % available_instances)
+                report_violation(_("License count of %s instances has been exceeded.") % instance_count)
         elif not add_host_name and free_instances < 0:
             report_violation(_("Host count exceeds available instances."))
 
@@ -1103,11 +1103,6 @@ class CredentialTypeAccess(BaseAccess):
     def can_use(self, obj):
         return True
 
-    def get_method_capability(self, method, obj, parent_obj):
-        if obj.managed_by_tower:
-            return False
-        return super(CredentialTypeAccess, self).get_method_capability(method, obj, parent_obj)
-
     def filtered_queryset(self):
         return self.model.objects.all()
 
@@ -1182,6 +1177,8 @@ class CredentialAccess(BaseAccess):
     def get_user_capabilities(self, obj, **kwargs):
         user_capabilities = super(CredentialAccess, self).get_user_capabilities(obj, **kwargs)
         user_capabilities['use'] = self.can_use(obj)
+        if getattr(obj, 'managed_by_tower', False) is True:
+            user_capabilities['edit'] = user_capabilities['delete'] = False
         return user_capabilities
 
 
@@ -2753,6 +2750,9 @@ class WorkflowApprovalTemplateAccess(BaseAccess):
         else:
             return (self.check_related('workflow_approval_template', UnifiedJobTemplate, role_field='admin_role'))
 
+    def can_change(self, obj, data):
+        return self.user.can_access(WorkflowJobTemplate, 'change', obj.workflow_job_template, data={})
+
     def can_start(self, obj, validate_license=False):
         # for copying WFJTs that contain approval nodes
         if self.user.is_superuser:

awx/main/analytics/__init__.py

@@ -1 +1 @@
-from .core import register, gather, ship, table_version  # noqa
+from .core import all_collectors, expensive_collectors, register, gather, ship  # noqa

awx/main/analytics/broadcast_websocket.py

@@ -20,7 +20,7 @@ from django.conf import settings
 BROADCAST_WEBSOCKET_REDIS_KEY_NAME = 'broadcast_websocket_stats'
 
 
-logger = logging.getLogger('awx.main.analytics.broadcast_websocket')
+logger = logging.getLogger('awx.analytics.broadcast_websocket')
 
 
 def dt_to_seconds(dt):

awx/main/analytics/collectors.py

@@ -1,3 +1,4 @@
+import io
 import os
 import os.path
 import platform
@@ -6,13 +7,14 @@ from django.db import connection
 from django.db.models import Count
 from django.conf import settings
 from django.utils.timezone import now
+from django.utils.translation import ugettext_lazy as _
 
 from awx.conf.license import get_license
 from awx.main.utils import (get_awx_version, get_ansible_version,
                             get_custom_venv_choices, camelcase_to_underscore)
 from awx.main import models
 from django.contrib.sessions.models import Session
-from awx.main.analytics import register, table_version
+from awx.main.analytics import register
 
 '''
 This module is used to define metrics collected by awx.main.analytics.gather()
@@ -31,9 +33,9 @@ data _since_ the last report date - i.e., new data in the last 24 hours)
 '''
 
 
-@register('config', '1.1')
-def config(since):
-    license_info = get_license(show_key=False)
+@register('config', '1.2', description=_('General platform configuration.'))
+def config(since, **kwargs):
+    license_info = get_license()
     install_type = 'traditional'
     if os.environ.get('container') == 'oci':
         install_type = 'openshift'
@@ -63,8 +65,8 @@ def config(since):
     }
 
 
-@register('counts', '1.0')
-def counts(since):
+@register('counts', '1.0', description=_('Counts of objects such as organizations, inventories, and projects'))
+def counts(since, **kwargs):
     counts = {}
     for cls in (models.Organization, models.Team, models.User,
                 models.Inventory, models.Credential, models.Project,
@@ -98,8 +100,8 @@ def counts(since):
     return counts
 
     
-@register('org_counts', '1.0')
-def org_counts(since):
+@register('org_counts', '1.0', description=_('Counts of users and teams by organization'))
+def org_counts(since, **kwargs):
     counts = {}
     for org in models.Organization.objects.annotate(num_users=Count('member_role__members', distinct=True), 
                                                     num_teams=Count('teams', distinct=True)).values('name', 'id', 'num_users', 'num_teams'):
@@ -110,8 +112,8 @@ def org_counts(since):
     return counts
     
     
-@register('cred_type_counts', '1.0')
-def cred_type_counts(since):
+@register('cred_type_counts', '1.0', description=_('Counts of credentials by credential type'))
+def cred_type_counts(since, **kwargs):
     counts = {}
     for cred_type in models.CredentialType.objects.annotate(num_credentials=Count(
                                                             'credentials', distinct=True)).values('name', 'id', 'managed_by_tower', 'num_credentials'):  
@@ -122,8 +124,8 @@ def cred_type_counts(since):
     return counts
     
     
-@register('inventory_counts', '1.2')
-def inventory_counts(since):
+@register('inventory_counts', '1.2', description=_('Inventories, their inventory sources, and host counts'))
+def inventory_counts(since, **kwargs):
     counts = {}
     for inv in models.Inventory.objects.filter(kind='').annotate(num_sources=Count('inventory_sources', distinct=True), 
                                                                  num_hosts=Count('hosts', distinct=True)).only('id', 'name', 'kind'):
@@ -147,8 +149,8 @@ def inventory_counts(since):
     return counts
 
 
-@register('projects_by_scm_type', '1.0')
-def projects_by_scm_type(since):
+@register('projects_by_scm_type', '1.0', description=_('Counts of projects by source control type'))
+def projects_by_scm_type(since, **kwargs):
     counts = dict(
         (t[0] or 'manual', 0)
         for t in models.Project.SCM_TYPE_CHOICES
@@ -166,8 +168,8 @@ def _get_isolated_datetime(last_check):
     return last_check
 
 
-@register('instance_info', '1.0')
-def instance_info(since, include_hostnames=False):
+@register('instance_info', '1.0', description=_('Cluster topology and capacity'))
+def instance_info(since, include_hostnames=False, **kwargs):
     info = {}
     instances = models.Instance.objects.values_list('hostname').values(
         'uuid', 'version', 'capacity', 'cpu', 'memory', 'managed_by_policy', 'hostname', 'last_isolated_check', 'enabled')
@@ -192,8 +194,7 @@ def instance_info(since, include_hostnames=False):
     return info
 
 
-@register('job_counts', '1.0')
-def job_counts(since):
+def job_counts(since, **kwargs):
     counts = {}
     counts['total_jobs'] = models.UnifiedJob.objects.exclude(launch_type='sync').count()
     counts['status'] = dict(models.UnifiedJob.objects.exclude(launch_type='sync').values_list('status').annotate(Count('status')).order_by())
@@ -202,8 +203,7 @@ def job_counts(since):
     return counts
     
     
-@register('job_instance_counts', '1.0')
-def job_instance_counts(since):
+def job_instance_counts(since, **kwargs):
     counts = {}
     job_types = models.UnifiedJob.objects.exclude(launch_type='sync').values_list(
         'execution_node', 'launch_type').annotate(job_launch_type=Count('launch_type')).order_by()
@@ -217,36 +217,79 @@ def job_instance_counts(since):
     return counts
 
 
-@register('query_info', '1.0')
-def query_info(since, collection_type):
+@register('query_info', '1.0', description=_('Metadata about the analytics collected'))
+def query_info(since, collection_type, until, **kwargs):
     query_info = {}
     query_info['last_run'] = str(since)
-    query_info['current_time'] = str(now())
+    query_info['current_time'] = str(until)
     query_info['collection_type'] = collection_type
     return query_info
 
 
-# Copies Job Events from db to a .csv to be shipped
-@table_version('events_table.csv', '1.1')
-@table_version('unified_jobs_table.csv', '1.0')
-@table_version('unified_job_template_table.csv', '1.0')
-@table_version('workflow_job_node_table.csv', '1.0')
-@table_version('workflow_job_template_node_table.csv', '1.0')
-def copy_tables(since, full_path, subset=None):
-    def _copy_table(table, query, path):
-        file_path = os.path.join(path, table + '_table.csv')
-        file = open(file_path, 'w', encoding='utf-8')
-        with connection.cursor() as cursor:
-            cursor.copy_expert(query, file)
-            file.close()
-        return file_path
+'''
+The event table can be *very* large, and we have a 100MB upload limit.
 
+Split large table dumps at dump time into a series of files.
+'''
+MAX_TABLE_SIZE = 200 * 1048576
+
+
+class FileSplitter(io.StringIO):
+    def __init__(self, filespec=None, *args, **kwargs):
+        self.filespec = filespec
+        self.files = []
+        self.currentfile = None
+        self.header = None
+        self.counter = 0
+        self.cycle_file()
+
+    def cycle_file(self):
+        if self.currentfile:
+            self.currentfile.close()
+        self.counter = 0
+        fname = '{}_split{}'.format(self.filespec, len(self.files))
+        self.currentfile = open(fname, 'w', encoding='utf-8')
+        self.files.append(fname)
+        if self.header:
+            self.currentfile.write('{}\n'.format(self.header))
+
+    def file_list(self):
+        self.currentfile.close()
+        # Check for an empty dump
+        if len(self.header) + 1 == self.counter:
+            os.remove(self.files[-1])
+            self.files = self.files[:-1]
+        # If we only have one file, remove the suffix
+        if len(self.files) == 1:
+            os.rename(self.files[0],self.files[0].replace('_split0',''))
+        return self.files
+
+    def write(self, s):
+        if not self.header:
+            self.header = s[0:s.index('\n')]
+        self.counter += self.currentfile.write(s)
+        if self.counter >= MAX_TABLE_SIZE:
+            self.cycle_file()
+
+
+def _copy_table(table, query, path):
+    file_path = os.path.join(path, table + '_table.csv')
+    file = FileSplitter(filespec=file_path)
+    with connection.cursor() as cursor:
+        cursor.copy_expert(query, file)
+    return file.file_list()
+
+
+@register('events_table', '1.2', format='csv', description=_('Automation task records'), expensive=True)
+def events_table(since, full_path, until, **kwargs):
     events_query = '''COPY (SELECT main_jobevent.id, 
                               main_jobevent.created,
+                              main_jobevent.modified,
                               main_jobevent.uuid,
                               main_jobevent.parent_uuid,
                               main_jobevent.event, 
                               main_jobevent.event_data::json->'task_action' AS task_action,
+                              (CASE WHEN event = 'playbook_on_stats' THEN event_data END) as playbook_on_stats,
                               main_jobevent.failed, 
                               main_jobevent.changed, 
                               main_jobevent.playbook, 
@@ -262,16 +305,21 @@ def copy_tables(since, full_path, subset=None):
                               main_jobevent.event_data::json->'res'->'warnings' AS warnings,
                               main_jobevent.event_data::json->'res'->'deprecations' AS deprecations
                               FROM main_jobevent 
-                              WHERE main_jobevent.created > {}
-                              ORDER BY main_jobevent.id ASC) TO STDOUT WITH CSV HEADER'''.format(since.strftime("'%Y-%m-%d %H:%M:%S'"))
-    if not subset or 'events' in subset:
-        _copy_table(table='events', query=events_query, path=full_path)
+                              WHERE (main_jobevent.created > '{}' AND main_jobevent.created <= '{}')
+                              ORDER BY main_jobevent.id ASC) TO STDOUT WITH CSV HEADER
+                   '''.format(since.isoformat(),until.isoformat())
+    return _copy_table(table='events', query=events_query, path=full_path)
 
+
+@register('unified_jobs_table', '1.1', format='csv', description=_('Data on jobs run'), expensive=True)
+def unified_jobs_table(since, full_path, until, **kwargs):
     unified_job_query = '''COPY (SELECT main_unifiedjob.id,
                                  main_unifiedjob.polymorphic_ctype_id,
                                  django_content_type.model,
                                  main_unifiedjob.organization_id,
                                  main_organization.name as organization_name,
+                                 main_job.inventory_id, 
+                                 main_inventory.name as inventory_name,
                                  main_unifiedjob.created,  
                                  main_unifiedjob.name,  
                                  main_unifiedjob.unified_job_template_id, 
@@ -289,13 +337,19 @@ def copy_tables(since, full_path, subset=None):
                                  main_unifiedjob.instance_group_id
                                  FROM main_unifiedjob
                                  JOIN django_content_type ON main_unifiedjob.polymorphic_ctype_id = django_content_type.id
+                                 LEFT JOIN main_job ON main_unifiedjob.id = main_job.unifiedjob_ptr_id
+                                 LEFT JOIN main_inventory ON main_job.inventory_id = main_inventory.id
                                  LEFT JOIN main_organization ON main_organization.id = main_unifiedjob.organization_id
-                                 WHERE (main_unifiedjob.created > {0} OR main_unifiedjob.finished > {0})
+                                 WHERE ((main_unifiedjob.created > '{0}' AND main_unifiedjob.created <= '{1}')
+                                       OR (main_unifiedjob.finished > '{0}' AND main_unifiedjob.finished <= '{1}'))
                                        AND main_unifiedjob.launch_type != 'sync'
-                                 ORDER BY main_unifiedjob.id ASC) TO STDOUT WITH CSV HEADER'''.format(since.strftime("'%Y-%m-%d %H:%M:%S'"))    
-    if not subset or 'unified_jobs' in subset:
-        _copy_table(table='unified_jobs', query=unified_job_query, path=full_path)
+                                 ORDER BY main_unifiedjob.id ASC) TO STDOUT WITH CSV HEADER
+                        '''.format(since.isoformat(),until.isoformat())
+    return _copy_table(table='unified_jobs', query=unified_job_query, path=full_path)
 
+
+@register('unified_job_template_table', '1.0', format='csv', description=_('Data on job templates'))
+def unified_job_template_table(since, full_path, **kwargs):
     unified_job_template_query = '''COPY (SELECT main_unifiedjobtemplate.id, 
                                  main_unifiedjobtemplate.polymorphic_ctype_id,
                                  django_content_type.model,
@@ -314,9 +368,11 @@ def copy_tables(since, full_path, subset=None):
                                  FROM main_unifiedjobtemplate, django_content_type
                                  WHERE main_unifiedjobtemplate.polymorphic_ctype_id = django_content_type.id
                                  ORDER BY main_unifiedjobtemplate.id ASC) TO STDOUT WITH CSV HEADER'''  
-    if not subset or 'unified_job_template' in subset:
-        _copy_table(table='unified_job_template', query=unified_job_template_query, path=full_path)
+    return _copy_table(table='unified_job_template', query=unified_job_template_query, path=full_path)
 
+
+@register('workflow_job_node_table', '1.0', format='csv', description=_('Data on workflow runs'), expensive=True)
+def workflow_job_node_table(since, full_path, until, **kwargs):
     workflow_job_node_query = '''COPY (SELECT main_workflowjobnode.id,
                                  main_workflowjobnode.created,
                                  main_workflowjobnode.modified, 
@@ -345,11 +401,14 @@ def copy_tables(since, full_path, subset=None):
                                      FROM main_workflowjobnode_always_nodes
                                      GROUP BY from_workflowjobnode_id
                                  ) always_nodes ON main_workflowjobnode.id = always_nodes.from_workflowjobnode_id
-                                 WHERE main_workflowjobnode.modified > {}
-                                 ORDER BY main_workflowjobnode.id ASC) TO STDOUT WITH CSV HEADER'''.format(since.strftime("'%Y-%m-%d %H:%M:%S'"))    
-    if not subset or 'workflow_job_node' in subset:
-        _copy_table(table='workflow_job_node', query=workflow_job_node_query, path=full_path)
+                                 WHERE (main_workflowjobnode.modified > '{}' AND main_workflowjobnode.modified <= '{}')
+                                 ORDER BY main_workflowjobnode.id ASC) TO STDOUT WITH CSV HEADER
+                              '''.format(since.isoformat(),until.isoformat())
+    return _copy_table(table='workflow_job_node', query=workflow_job_node_query, path=full_path)
 
+
+@register('workflow_job_template_node_table', '1.0', format='csv', description=_('Data on workflows'))
+def workflow_job_template_node_table(since, full_path, **kwargs):
     workflow_job_template_node_query = '''COPY (SELECT main_workflowjobtemplatenode.id, 
                                  main_workflowjobtemplatenode.created,
                                  main_workflowjobtemplatenode.modified, 
@@ -377,7 +436,4 @@ def copy_tables(since, full_path, subset=None):
                                      GROUP BY from_workflowjobtemplatenode_id
                                  ) always_nodes ON main_workflowjobtemplatenode.id = always_nodes.from_workflowjobtemplatenode_id
                                  ORDER BY main_workflowjobtemplatenode.id ASC) TO STDOUT WITH CSV HEADER'''   
-    if not subset or 'workflow_job_template_node' in subset:
-        _copy_table(table='workflow_job_template_node', query=workflow_job_template_node_query, path=full_path)
-
-    return
+    return _copy_table(table='workflow_job_template_node', query=workflow_job_template_node_query, path=full_path)

awx/main/analytics/core.py

@@ -14,21 +14,17 @@ from rest_framework.exceptions import PermissionDenied
 from awx.conf.license import get_license
 from awx.main.models import Job
 from awx.main.access import access_registry
-from awx.main.models.ha import TowerAnalyticsState
 from awx.main.utils import get_awx_http_client_headers, set_environ
 
-
-__all__ = ['register', 'gather', 'ship', 'table_version']
+__all__ = ['register', 'gather', 'ship']
 
 
 logger = logging.getLogger('awx.main.analytics')
 
-manifest = dict()
-
 
 def _valid_license():
     try:
-        if get_license(show_key=False).get('license_type', 'UNLICENSED') == 'open':
+        if get_license().get('license_type', 'UNLICENSED') == 'open':
             return False
         access_registry[Job](None).check_license()
     except PermissionDenied:
@@ -37,114 +33,194 @@ def _valid_license():
     return True
 
 
-def register(key, version):
+def all_collectors():
+    from awx.main.analytics import collectors
+
+    collector_dict = {}
+    module = collectors
+    for name, func in inspect.getmembers(module):
+        if inspect.isfunction(func) and hasattr(func, '__awx_analytics_key__'):
+            key = func.__awx_analytics_key__
+            desc = func.__awx_analytics_description__ or ''
+            version = func.__awx_analytics_version__
+            collector_dict[key] = { 'name': key, 'version': version, 'description': desc}
+    return collector_dict
+
+
+def expensive_collectors():
+    from awx.main.analytics import collectors
+
+    ret = []
+    module = collectors
+    for name, func in inspect.getmembers(module):
+        if inspect.isfunction(func) and hasattr(func, '__awx_analytics_key__') and func.__awx_expensive__:
+            ret.append(func.__awx_analytics_key__)
+    return ret
+
+
+def register(key, version, description=None, format='json', expensive=False):
     """
     A decorator used to register a function as a metric collector.
 
-    Decorated functions should return JSON-serializable objects.
+    Decorated functions should do the following based on format:
+    - json: return JSON-serializable objects.
+    - csv: write CSV data to a filename named 'key'
 
     @register('projects_by_scm_type', 1)
     def projects_by_scm_type():
-        return {'git': 5, 'svn': 1, 'hg': 0}
+        return {'git': 5, 'svn': 1}
     """
 
     def decorate(f):
         f.__awx_analytics_key__ = key
         f.__awx_analytics_version__ = version
+        f.__awx_analytics_description__ = description
+        f.__awx_analytics_type__ = format
+        f.__awx_expensive__ = expensive
         return f
 
     return decorate
 
 
-def table_version(file_name, version):
-
-    global manifest
-    manifest[file_name] = version
-
-    def decorate(f):
-        return f
-
-    return decorate
-
-
-def gather(dest=None, module=None, collection_type='scheduled'):
+def gather(dest=None, module=None, subset = None, since = None, until = now(), collection_type='scheduled'):
     """
     Gather all defined metrics and write them as JSON files in a .tgz
 
     :param dest:    the (optional) absolute path to write a compressed tarball
-    :pararm module: the module to search for registered analytic collector
+    :param module: the module to search for registered analytic collector
                     functions; defaults to awx.main.analytics.collectors
     """
+    def _write_manifest(destdir, manifest):
+        path = os.path.join(destdir, 'manifest.json')
+        with open(path, 'w', encoding='utf-8') as f:
+            try:
+                json.dump(manifest, f)
+            except Exception:
+                f.close()
+                os.remove(f.name)
+                logger.exception("Could not generate manifest.json")
 
-    run_now = now()
-    state = TowerAnalyticsState.get_solo()
-    last_run = state.last_run
-    logger.debug("Last analytics run was: {}".format(last_run))
-    
-    max_interval = now() - timedelta(weeks=4)
-    if last_run < max_interval or not last_run:
-        last_run = max_interval
+    last_run = since or settings.AUTOMATION_ANALYTICS_LAST_GATHER or (now() - timedelta(weeks=4))
+    logger.debug("Last analytics run was: {}".format(settings.AUTOMATION_ANALYTICS_LAST_GATHER))
 
     if _valid_license() is False:
         logger.exception("Invalid License provided, or No License Provided")
-        return "Error: Invalid License provided, or No License Provided"
+        return None
 
     if collection_type != 'dry-run' and not settings.INSIGHTS_TRACKING_STATE:
         logger.error("Automation Analytics not enabled. Use --dry-run to gather locally without sending.")
-        return
+        return None
 
-    if module is None:
+    collector_list = []
+    if module:
+        collector_module = module
+    else:
         from awx.main.analytics import collectors
-        module = collectors
-
+        collector_module = collectors
+    for name, func in inspect.getmembers(collector_module):
+        if (
+            inspect.isfunction(func) and
+            hasattr(func, '__awx_analytics_key__') and
+            (not subset or name in subset)
+        ):
+            collector_list.append((name, func))
 
+    manifest = dict()
     dest = dest or tempfile.mkdtemp(prefix='awx_analytics')
-    for name, func in inspect.getmembers(module):
-        if inspect.isfunction(func) and hasattr(func, '__awx_analytics_key__'):
+    gather_dir = os.path.join(dest, 'stage')
+    os.mkdir(gather_dir, 0o700)
+    num_splits = 1
+    for name, func in collector_list:
+        if func.__awx_analytics_type__ == 'json':
             key = func.__awx_analytics_key__
-            manifest['{}.json'.format(key)] = func.__awx_analytics_version__
-            path = '{}.json'.format(os.path.join(dest, key))
+            path = '{}.json'.format(os.path.join(gather_dir, key))
             with open(path, 'w', encoding='utf-8') as f:
                 try:
-                    if func.__name__ == 'query_info':
-                        json.dump(func(last_run, collection_type=collection_type), f)
-                    else:
-                        json.dump(func(last_run), f)
+                    json.dump(func(last_run, collection_type=collection_type, until=until), f)
+                    manifest['{}.json'.format(key)] = func.__awx_analytics_version__
                 except Exception:
                     logger.exception("Could not generate metric {}.json".format(key))
                     f.close()
                     os.remove(f.name)
-    
-    path = os.path.join(dest, 'manifest.json')
-    with open(path, 'w', encoding='utf-8') as f:
-        try:
-            json.dump(manifest, f)
-        except Exception:
-            logger.exception("Could not generate manifest.json")
-            f.close()
-            os.remove(f.name)
+        elif func.__awx_analytics_type__ == 'csv':
+            key = func.__awx_analytics_key__
+            try:
+                files = func(last_run, full_path=gather_dir, until=until)
+                if files:
+                    manifest['{}.csv'.format(key)] = func.__awx_analytics_version__
+                if len(files) > num_splits:
+                    num_splits = len(files)
+            except Exception:
+                logger.exception("Could not generate metric {}.csv".format(key))
 
-    try:
-        collectors.copy_tables(since=last_run, full_path=dest)
-    except Exception:
-        logger.exception("Could not copy tables")
-        
-    # can't use isoformat() since it has colons, which GNU tar doesn't like
-    tarname = '_'.join([
-        settings.SYSTEM_UUID,
-        run_now.strftime('%Y-%m-%d-%H%M%S%z')
-    ])
-    try:
-        tgz = shutil.make_archive(
-            os.path.join(os.path.dirname(dest), tarname),
-            'gztar',
-            dest
-        )
-        return tgz
-    except Exception:
-        logger.exception("Failed to write analytics archive file")
-    finally: 
+    if not manifest:
+        # No data was collected
+        logger.warning("No data from {} to {}".format(last_run, until))
         shutil.rmtree(dest)
+        return None
+
+    # Always include config.json if we're using our collectors
+    if 'config.json' not in manifest.keys() and not module:
+        from awx.main.analytics import collectors
+        config = collectors.config
+        path = '{}.json'.format(os.path.join(gather_dir, config.__awx_analytics_key__))
+        with open(path, 'w', encoding='utf-8') as f:
+            try:
+                json.dump(collectors.config(last_run), f)
+                manifest['config.json'] = config.__awx_analytics_version__
+            except Exception:
+                logger.exception("Could not generate metric {}.json".format(key))
+                f.close()
+                os.remove(f.name)
+                shutil.rmtree(dest)
+                return None
+
+    stage_dirs = [gather_dir]
+    if num_splits > 1:
+        for i in range(0, num_splits):
+            split_path = os.path.join(dest, 'split{}'.format(i))
+            os.mkdir(split_path, 0o700)
+            filtered_manifest = {}
+            shutil.copy(os.path.join(gather_dir, 'config.json'), split_path)
+            filtered_manifest['config.json'] = manifest['config.json']
+            suffix = '_split{}'.format(i)
+            for file in os.listdir(gather_dir):
+                if file.endswith(suffix):
+                    old_file = os.path.join(gather_dir, file)
+                    new_filename = file.replace(suffix, '')
+                    new_file = os.path.join(split_path, new_filename)
+                    shutil.move(old_file, new_file)
+                    filtered_manifest[new_filename] = manifest[new_filename]
+            _write_manifest(split_path, filtered_manifest)
+            stage_dirs.append(split_path)
+
+    for item in list(manifest.keys()):
+        if not os.path.exists(os.path.join(gather_dir, item)):
+            manifest.pop(item)
+    _write_manifest(gather_dir, manifest)
+
+    tarfiles = []
+    try:
+        for i in range(0, len(stage_dirs)):
+            stage_dir = stage_dirs[i]
+            # can't use isoformat() since it has colons, which GNU tar doesn't like
+            tarname = '_'.join([
+                settings.SYSTEM_UUID,
+                until.strftime('%Y-%m-%d-%H%M%S%z'),
+                str(i)
+            ])
+            tgz = shutil.make_archive(
+                os.path.join(os.path.dirname(dest), tarname),
+                'gztar',
+                stage_dir
+            )
+            tarfiles.append(tgz)
+    except Exception:
+        shutil.rmtree(stage_dir, ignore_errors = True)
+        logger.exception("Failed to write analytics archive file")
+    finally:
+        shutil.rmtree(dest, ignore_errors = True)
+    return tarfiles
 
 
 def ship(path):
@@ -154,6 +230,9 @@ def ship(path):
     if not path:
         logger.error('Automation Analytics TAR not found')
         return
+    if not os.path.exists(path):
+        logger.error('Automation Analytics TAR {} not found'.format(path))
+        return
     if "Error:" in str(path):
         return
     try:
@@ -180,13 +259,11 @@ def ship(path):
                                   auth=(rh_user, rh_password),
                                   headers=s.headers,
                                   timeout=(31, 31))
-            if response.status_code != 202:
+            # Accept 2XX status_codes
+            if response.status_code >= 300:
                 return logger.exception('Upload failed with status {}, {}'.format(response.status_code,
                                                                                   response.text))
-        run_now = now()
-        state = TowerAnalyticsState.get_solo()
-        state.last_run = run_now
-        state.save()
     finally:
         # cleanup tar.gz
-        os.remove(path)
+        if os.path.exists(path):
+            os.remove(path)

awx/main/analytics/metrics.py

@@ -12,7 +12,7 @@ from prometheus_client import (
 from awx.conf.license import get_license
 from awx.main.utils import (get_awx_version, get_ansible_version)
 from awx.main.analytics.collectors import (
-    counts, 
+    counts,
     instance_info,
     job_instance_counts,
     job_counts,
@@ -54,7 +54,7 @@ LICENSE_INSTANCE_FREE = Gauge('awx_license_instance_free', 'Number of remaining
 
 
 def metrics():
-    license_info = get_license(show_key=False)
+    license_info = get_license()
     SYSTEM_INFO.info({
         'install_uuid': settings.INSTALL_UUID,
         'insights_analytics': str(settings.INSIGHTS_TRACKING_STATE),
@@ -68,7 +68,7 @@ def metrics():
         'external_logger_type': getattr(settings, 'LOG_AGGREGATOR_TYPE', 'None')
     })
 
-    LICENSE_INSTANCE_TOTAL.set(str(license_info.get('available_instances', 0)))
+    LICENSE_INSTANCE_TOTAL.set(str(license_info.get('instance_count', 0)))
     LICENSE_INSTANCE_FREE.set(str(license_info.get('free_instances', 0)))
 
     current_counts = counts(None)

awx/main/conf.py

@@ -1,8 +1,5 @@
 # Python
-import json
 import logging
-import os
-from distutils.version import LooseVersion as Version
 
 # Django
 from django.utils.translation import ugettext_lazy as _
@@ -14,6 +11,7 @@ from rest_framework.fields import FloatField
 # Tower
 from awx.conf import fields, register, register_validate
 
+
 logger = logging.getLogger('awx.main.conf')
 
 register(
@@ -93,22 +91,10 @@ register(
 )
 
 
-def _load_default_license_from_file():
-    try:
-        license_file = os.environ.get('AWX_LICENSE_FILE', '/etc/tower/license')
-        if os.path.exists(license_file):
-            license_data = json.load(open(license_file))
-            logger.debug('Read license data from "%s".', license_file)
-            return license_data
-    except Exception:
-        logger.warning('Could not read license from "%s".', license_file, exc_info=True)
-    return {}
-
-
 register(
     'LICENSE',
     field_class=fields.DictField,
-    default=_load_default_license_from_file,
+    default=lambda: {},
     label=_('License'),
     help_text=_('The license controls which features and functionality are '
                 'enabled. Use /api/v2/config/ to update or change '
@@ -125,7 +111,7 @@ register(
     encrypted=False,
     read_only=False,
     label=_('Red Hat customer username'),
-    help_text=_('This username is used to retrieve license information and to send Automation Analytics'),  # noqa
+    help_text=_('This username is used to send data to Automation Analytics'),
     category=_('System'),
     category_slug='system',
 )
@@ -138,7 +124,33 @@ register(
     encrypted=True,
     read_only=False,
     label=_('Red Hat customer password'),
-    help_text=_('This password is used to retrieve license information and to send Automation Analytics'),  # noqa
+    help_text=_('This password is used to send data to Automation Analytics'),
+    category=_('System'),
+    category_slug='system',
+)
+
+register(
+    'SUBSCRIPTIONS_USERNAME',
+    field_class=fields.CharField,
+    default='',
+    allow_blank=True,
+    encrypted=False,
+    read_only=False,
+    label=_('Red Hat or Satellite username'),
+    help_text=_('This username is used to retrieve subscription and content information'),  # noqa
+    category=_('System'),
+    category_slug='system',
+)
+
+register(
+    'SUBSCRIPTIONS_PASSWORD',
+    field_class=fields.CharField,
+    default='',
+    allow_blank=True,
+    encrypted=True,
+    read_only=False,
+    label=_('Red Hat or Satellite password'),
+    help_text=_('This password is used to retrieve subscription and content information'),  # noqa
     category=_('System'),
     category_slug='system',
 )
@@ -149,7 +161,7 @@ register(
     default='https://example.com',
     schemes=('http', 'https'),
     allow_plain_hostname=True,  # Allow hostname only without TLD.
-    label=_('Automation Analytics upload URL.'),
+    label=_('Automation Analytics upload URL'),
     help_text=_('This setting is used to to configure data collection for the Automation Analytics dashboard'),
     category=_('System'),
     category_slug='system',
@@ -254,6 +266,7 @@ register(
     help_text=_('The number of seconds to sleep between status checks for jobs running on isolated instances.'),
     category=_('Jobs'),
     category_slug='jobs',
+    unit=_('seconds'),
 )
 
 register(
@@ -265,6 +278,7 @@ register(
                 'This includes the time needed to copy source control files (playbooks) to the isolated instance.'),
     category=_('Jobs'),
     category_slug='jobs',
+    unit=_('seconds'),
 )
 
 register(
@@ -277,6 +291,7 @@ register(
                 'Value should be substantially greater than expected network latency.'),
     category=_('Jobs'),
     category_slug='jobs',
+    unit=_('seconds'),
 )
 
 register(
@@ -436,93 +451,12 @@ register(
     category_slug='jobs',
 )
 
-register(
-    'PRIMARY_GALAXY_URL',
-    field_class=fields.URLField,
-    required=False,
-    allow_blank=True,
-    label=_('Primary Galaxy Server URL'),
-    help_text=_(
-        'For organizations that run their own Galaxy service, this gives the option to specify a '
-        'host as the primary galaxy server. Requirements will be downloaded from the primary if the '
-        'specific role or collection is available there. If the content is not avilable in the primary, '
-        'or if this field is left blank, it will default to galaxy.ansible.com.'
-    ),
-    category=_('Jobs'),
-    category_slug='jobs'
-)
-
-register(
-    'PRIMARY_GALAXY_USERNAME',
-    field_class=fields.CharField,
-    required=False,
-    allow_blank=True,
-    label=_('Primary Galaxy Server Username'),
-    help_text=_('(This setting is deprecated and will be removed in a future release) '
-                'For using a galaxy server at higher precedence than the public Ansible Galaxy. '
-                'The username to use for basic authentication against the Galaxy instance, '
-                'this is mutually exclusive with PRIMARY_GALAXY_TOKEN.'),
-    category=_('Jobs'),
-    category_slug='jobs'
-)
-
-register(
-    'PRIMARY_GALAXY_PASSWORD',
-    field_class=fields.CharField,
-    encrypted=True,
-    required=False,
-    allow_blank=True,
-    label=_('Primary Galaxy Server Password'),
-    help_text=_('(This setting is deprecated and will be removed in a future release) '
-                'For using a galaxy server at higher precedence than the public Ansible Galaxy. '
-                'The password to use for basic authentication against the Galaxy instance, '
-                'this is mutually exclusive with PRIMARY_GALAXY_TOKEN.'),
-    category=_('Jobs'),
-    category_slug='jobs'
-)
-
-register(
-    'PRIMARY_GALAXY_TOKEN',
-    field_class=fields.CharField,
-    encrypted=True,
-    required=False,
-    allow_blank=True,
-    label=_('Primary Galaxy Server Token'),
-    help_text=_('For using a galaxy server at higher precedence than the public Ansible Galaxy. '
-                'The token to use for connecting with the Galaxy instance, '
-                'this is mutually exclusive with corresponding username and password settings.'),
-    category=_('Jobs'),
-    category_slug='jobs'
-)
-
-register(
-    'PRIMARY_GALAXY_AUTH_URL',
-    field_class=fields.CharField,
-    required=False,
-    allow_blank=True,
-    label=_('Primary Galaxy Authentication URL'),
-    help_text=_('For using a galaxy server at higher precedence than the public Ansible Galaxy. '
-                'The token_endpoint of a Keycloak server.'),
-    category=_('Jobs'),
-    category_slug='jobs'
-)
-
-register(
-    'PUBLIC_GALAXY_ENABLED',
-    field_class=fields.BooleanField,
-    default=True,
-    label=_('Allow Access to Public Galaxy'),
-    help_text=_('Allow or deny access to the public Ansible Galaxy during project updates.'),
-    category=_('Jobs'),
-    category_slug='jobs'
-)
-
 register(
     'GALAXY_IGNORE_CERTS',
     field_class=fields.BooleanField,
     default=False,
     label=_('Ignore Ansible Galaxy SSL Certificate Verification'),
-    help_text=_('If set to true, certificate validation will not be done when'
+    help_text=_('If set to true, certificate validation will not be done when '
                 'installing content from any Galaxy server.'),
     category=_('Jobs'),
     category_slug='jobs'
@@ -579,6 +513,7 @@ register(
                 'timeout should be imposed. A timeout set on an individual job template will override this.'),
     category=_('Jobs'),
     category_slug='jobs',
+    unit=_('seconds'),
 )
 
 register(
@@ -591,6 +526,7 @@ register(
                 'timeout should be imposed. A timeout set on an individual inventory source will override this.'),
     category=_('Jobs'),
     category_slug='jobs',
+    unit=_('seconds'),
 )
 
 register(
@@ -603,6 +539,7 @@ register(
                 'timeout should be imposed. A timeout set on an individual project will override this.'),
     category=_('Jobs'),
     category_slug='jobs',
+    unit=_('seconds'),
 )
 
 register(
@@ -617,6 +554,7 @@ register(
                 'Use a value of 0 to indicate that no timeout should be imposed.'),
     category=_('Jobs'),
     category_slug='jobs',
+    unit=_('seconds'),
 )
 
 register(
@@ -624,7 +562,7 @@ register(
     field_class=fields.IntegerField,
     allow_null=False,
     default=200,
-    label=_('Maximum number of forks per job.'),
+    label=_('Maximum number of forks per job'),
     help_text=_('Saving a Job Template with more than this number of forks will result in an error. '
                 'When set to 0, no limit is applied.'),
     category=_('Jobs'),
@@ -754,6 +692,7 @@ register(
                 'aggregator protocols.'),
     category=_('Logging'),
     category_slug='logging',
+    unit=_('seconds'),
 )
 register(
     'LOG_AGGREGATOR_VERIFY_CERT',
@@ -834,7 +773,8 @@ register(
     default=14400,	# every 4 hours
     min_value=1800,	# every 30 minutes
     category=_('System'),
-    category_slug='system'
+    category_slug='system',
+    unit=_('seconds'),
 )
 
 
@@ -856,84 +796,4 @@ def logging_validate(serializer, attrs):
     return attrs
 
 
-def galaxy_validate(serializer, attrs):
-    """Ansible Galaxy config options have mutual exclusivity rules, these rules
-    are enforced here on serializer validation so that users will not be able
-    to save settings which obviously break all project updates.
-    """
-    prefix = 'PRIMARY_GALAXY_'
-    errors = {}
-
-    def _new_value(setting_name):
-        if setting_name in attrs:
-            return attrs[setting_name]
-        elif not serializer.instance:
-            return ''
-        return getattr(serializer.instance, setting_name, '')
-
-    if not _new_value('PRIMARY_GALAXY_URL'):
-        if _new_value('PUBLIC_GALAXY_ENABLED') is False:
-            msg = _('A URL for Primary Galaxy must be defined before disabling public Galaxy.')
-            # put error in both keys because UI has trouble with errors in toggles
-            for key in ('PRIMARY_GALAXY_URL', 'PUBLIC_GALAXY_ENABLED'):
-                errors.setdefault(key, [])
-                errors[key].append(msg)
-            raise serializers.ValidationError(errors)
-
-    from awx.main.constants import GALAXY_SERVER_FIELDS
-    if not any('{}{}'.format(prefix, subfield.upper()) in attrs for subfield in GALAXY_SERVER_FIELDS):
-        return attrs
-
-    galaxy_data = {}
-    for subfield in GALAXY_SERVER_FIELDS:
-        galaxy_data[subfield] = _new_value('{}{}'.format(prefix, subfield.upper()))
-    if not galaxy_data['url']:
-        for k, v in galaxy_data.items():
-            if v:
-                setting_name = '{}{}'.format(prefix, k.upper())
-                errors.setdefault(setting_name, [])
-                errors[setting_name].append(_(
-                    'Cannot provide field if PRIMARY_GALAXY_URL is not set.'
-                ))
-    for k in GALAXY_SERVER_FIELDS:
-        if galaxy_data[k]:
-            setting_name = '{}{}'.format(prefix, k.upper())
-            if (not serializer.instance) or (not getattr(serializer.instance, setting_name, '')):
-                # new auth is applied, so check if compatible with version
-                from awx.main.utils import get_ansible_version
-                current_version = get_ansible_version()
-                min_version = '2.9'
-                if Version(current_version) < Version(min_version):
-                    errors.setdefault(setting_name, [])
-                    errors[setting_name].append(_(
-                        'Galaxy server settings are not available until Ansible {min_version}, '
-                        'you are running {current_version}.'
-                    ).format(min_version=min_version, current_version=current_version))
-    if (galaxy_data['password'] or galaxy_data['username']) and (galaxy_data['token'] or galaxy_data['auth_url']):
-        for k in ('password', 'username', 'token', 'auth_url'):
-            setting_name = '{}{}'.format(prefix, k.upper())
-            if setting_name in attrs:
-                errors.setdefault(setting_name, [])
-                errors[setting_name].append(_(
-                    'Setting Galaxy token and authentication URL is mutually exclusive with username and password.'
-                ))
-    if bool(galaxy_data['username']) != bool(galaxy_data['password']):
-        msg = _('If authenticating via username and password, both must be provided.')
-        for k in ('username', 'password'):
-            setting_name = '{}{}'.format(prefix, k.upper())
-            errors.setdefault(setting_name, [])
-            errors[setting_name].append(msg)
-    if bool(galaxy_data['token']) != bool(galaxy_data['auth_url']):
-        msg = _('If authenticating via token, both token and authentication URL must be provided.')
-        for k in ('token', 'auth_url'):
-            setting_name = '{}{}'.format(prefix, k.upper())
-            errors.setdefault(setting_name, [])
-            errors[setting_name].append(msg)
-
-    if errors:
-        raise serializers.ValidationError(errors)
-    return attrs
-
-
 register_validate('logging', logging_validate)
-register_validate('jobs', galaxy_validate)

awx/main/constants.py

@@ -50,7 +50,3 @@ LOGGER_BLOCKLIST = (
     # loggers that may be called getting logging settings
     'awx.conf'
 )
-
-# these correspond to both AWX and Ansible settings to keep naming consistent
-# for instance, settings.PRIMARY_GALAXY_AUTH_URL vs env var ANSIBLE_GALAXY_SERVER_FOO_AUTH_URL
-GALAXY_SERVER_FIELDS = ('url', 'username', 'password', 'token', 'auth_url')

awx/main/consumers.py

@@ -1,5 +1,3 @@
-import collections
-import functools
 import json
 import logging
 import time
@@ -14,40 +12,12 @@ from django.contrib.auth.models import User
 from channels.generic.websocket import AsyncJsonWebsocketConsumer
 from channels.layers import get_channel_layer
 from channels.db import database_sync_to_async
-from channels_redis.core import RedisChannelLayer
 
 
 logger = logging.getLogger('awx.main.consumers')
 XRF_KEY = '_auth_user_xrf'
 
 
-class BoundedQueue(asyncio.Queue):
-
-    def put_nowait(self, item):
-        if self.full():
-            # dispose the oldest item
-            # if we actually get into this code block, it likely means that
-            # this specific consumer has stopped reading
-            # unfortunately, channels_redis will just happily continue to
-            # queue messages specific to their channel until the heat death
-            # of the sun: https://github.com/django/channels_redis/issues/212
-            # this isn't a huge deal for browser clients that disconnect,
-            # but it *does* cause a problem for our global broadcast topic
-            # that's used to broadcast messages to peers in a cluster
-            # if we get into this code block, it's better to drop messages
-            # than to continue to malloc() forever
-            self.get_nowait()
-        return super(BoundedQueue, self).put_nowait(item)
-
-
-class ExpiringRedisChannelLayer(RedisChannelLayer):
-    def __init__(self, *args, **kw):
-        super(ExpiringRedisChannelLayer, self).__init__(*args, **kw)
-        self.receive_buffer = collections.defaultdict(
-            functools.partial(BoundedQueue, self.capacity)
-        )
-
-
 class WebsocketSecretAuthHelper:
     """
     Middlewareish for websockets to verify node websocket broadcast interconnect.

awx/main/credential_plugins/hashivault.py

@@ -40,6 +40,13 @@ base_inputs = {
         'multiline': False,
         'secret': True,
         'help_text': _('The Secret ID for AppRole Authentication')
+    }, {
+        'id': 'default_auth_path',
+        'label': _('Path to Approle Auth'),
+        'type': 'string',
+        'multiline': False,
+        'default': 'approle',
+        'help_text': _('The AppRole Authentication path to use if one isn\'t provided in the metadata when linking to an input field. Defaults to \'approle\'')
     }
     ],
     'metadata': [{
@@ -47,10 +54,11 @@ base_inputs = {
         'label': _('Path to Secret'),
         'type': 'string',
         'help_text': _('The path to the secret stored in the secret backend e.g, /some/secret/')
-    },{
+    }, {
         'id': 'auth_path',
         'label': _('Path to Auth'),
         'type': 'string',
+        'multiline': False,
         'help_text': _('The path where the Authentication method is mounted e.g, approle')
     }],
     'required': ['url', 'secret_path'],
@@ -118,7 +126,9 @@ def handle_auth(**kwargs):
 def approle_auth(**kwargs):
     role_id = kwargs['role_id']
     secret_id = kwargs['secret_id']
-    auth_path = kwargs.get('auth_path') or 'approle'
+    # we first try to use the 'auth_path' from the metadata
+    # if not found we try to fetch the 'default_auth_path' from inputs
+    auth_path = kwargs.get('auth_path') or kwargs['default_auth_path']
 
     url = urljoin(kwargs['url'], 'v1')
     cacert = kwargs.get('cacert', None)
@@ -152,7 +162,7 @@ def kv_backend(**kwargs):
 
     sess = requests.Session()
     sess.headers['Authorization'] = 'Bearer {}'.format(token)
-    # Compatability header for older installs of Hashicorp Vault
+    # Compatibility header for older installs of Hashicorp Vault
     sess.headers['X-Vault-Token'] = token
 
     if api_version == 'v2':

awx/main/dispatch/control.py

@@ -2,6 +2,9 @@ import logging
 import uuid
 import json
 
+from django.conf import settings
+import redis
+
 from awx.main.dispatch import get_local_queuename
 
 from . import pg_bus_conn
@@ -21,7 +24,15 @@ class Control(object):
         self.queuename = host or get_local_queuename()
 
     def status(self, *args, **kwargs):
-        return self.control_with_reply('status', *args, **kwargs)
+        r = redis.Redis.from_url(settings.BROKER_URL)
+        if self.service == 'dispatcher':
+            stats = r.get(f'awx_{self.service}_statistics') or b''
+            return stats.decode('utf-8')
+        else:
+            workers = []
+            for key in r.keys('awx_callback_receiver_statistics_*'):
+                workers.append(r.get(key).decode('utf-8'))
+            return '\n'.join(workers)
 
     def running(self, *args, **kwargs):
         return self.control_with_reply('running', *args, **kwargs)

awx/main/dispatch/pool.py

@@ -5,6 +5,7 @@ import signal
 import sys
 import time
 import traceback
+from datetime import datetime
 from uuid import uuid4
 
 import collections
@@ -27,6 +28,12 @@ else:
     logger = logging.getLogger('awx.main.dispatch')
 
 
+class NoOpResultQueue(object):
+
+    def put(self, item):
+        pass
+
+
 class PoolWorker(object):
     '''
     Used to track a worker child process and its pending and finished messages.
@@ -56,11 +63,13 @@ class PoolWorker(object):
     It is "idle" when self.managed_tasks is empty.
     '''
 
-    def __init__(self, queue_size, target, args):
+    track_managed_tasks = False
+
+    def __init__(self, queue_size, target, args, **kwargs):
         self.messages_sent = 0
         self.messages_finished = 0
         self.managed_tasks = collections.OrderedDict()
-        self.finished = MPQueue(queue_size)
+        self.finished = MPQueue(queue_size) if self.track_managed_tasks else NoOpResultQueue()
         self.queue = MPQueue(queue_size)
         self.process = Process(target=target, args=(self.queue, self.finished) + args)
         self.process.daemon = True
@@ -74,7 +83,8 @@ class PoolWorker(object):
             if not body.get('uuid'):
                 body['uuid'] = str(uuid4())
             uuid = body['uuid']
-        self.managed_tasks[uuid] = body
+        if self.track_managed_tasks:
+            self.managed_tasks[uuid] = body
         self.queue.put(body, block=True, timeout=5)
         self.messages_sent += 1
         self.calculate_managed_tasks()
@@ -111,6 +121,8 @@ class PoolWorker(object):
         return str(self.process.exitcode)
 
     def calculate_managed_tasks(self):
+        if not self.track_managed_tasks:
+            return
         # look to see if any tasks were finished
         finished = []
         for _ in range(self.finished.qsize()):
@@ -135,6 +147,8 @@ class PoolWorker(object):
 
     @property
     def current_task(self):
+        if not self.track_managed_tasks:
+            return None
         self.calculate_managed_tasks()
         # the task at [0] is the one that's running right now (or is about to
         # be running)
@@ -145,6 +159,8 @@ class PoolWorker(object):
 
     @property
     def orphaned_tasks(self):
+        if not self.track_managed_tasks:
+            return []
         orphaned = []
         if not self.alive:
             # if this process had a running task that never finished,
@@ -179,6 +195,11 @@ class PoolWorker(object):
         return not self.busy
 
 
+class StatefulPoolWorker(PoolWorker):
+
+    track_managed_tasks = True
+
+
 class WorkerPool(object):
     '''
     Creates a pool of forked PoolWorkers.
@@ -200,6 +221,7 @@ class WorkerPool(object):
     )
     '''
 
+    pool_cls = PoolWorker
     debug_meta = ''
 
     def __init__(self, min_workers=None, queue_size=None):
@@ -225,7 +247,7 @@ class WorkerPool(object):
         # for the DB and cache connections (that way lies race conditions)
         django_connection.close()
         django_cache.close()
-        worker = PoolWorker(self.queue_size, self.target, (idx,) + self.target_args)
+        worker = self.pool_cls(self.queue_size, self.target, (idx,) + self.target_args)
         self.workers.append(worker)
         try:
             worker.start()
@@ -236,13 +258,13 @@ class WorkerPool(object):
         return idx, worker
 
     def debug(self, *args, **kwargs):
-        self.cleanup()
         tmpl = Template(
+            'Recorded at: {{ dt }} \n'
             '{{ pool.name }}[pid:{{ pool.pid }}] workers total={{ workers|length }} {{ meta }} \n'
             '{% for w in workers %}'
             '.  worker[pid:{{ w.pid }}]{% if not w.alive %} GONE exit={{ w.exitcode }}{% endif %}'
             ' sent={{ w.messages_sent }}'
-            ' finished={{ w.messages_finished }}'
+            '{% if w.messages_finished %} finished={{ w.messages_finished }}{% endif %}'
             ' qsize={{ w.managed_tasks|length }}'
             ' rss={{ w.mb }}MB'
             '{% for task in w.managed_tasks.values() %}'
@@ -260,7 +282,11 @@ class WorkerPool(object):
             '\n'
             '{% endfor %}'
         )
-        return tmpl.render(pool=self, workers=self.workers, meta=self.debug_meta)
+        now = datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S UTC')
+        return tmpl.render(
+            pool=self, workers=self.workers, meta=self.debug_meta,
+            dt=now
+        )
 
     def write(self, preferred_queue, body):
         queue_order = sorted(range(len(self.workers)), key=lambda x: -1 if x==preferred_queue else x)
@@ -293,6 +319,8 @@ class AutoscalePool(WorkerPool):
     down based on demand
     '''
 
+    pool_cls = StatefulPoolWorker
+
     def __init__(self, *args, **kwargs):
         self.max_workers = kwargs.pop('max_workers', None)
         super(AutoscalePool, self).__init__(*args, **kwargs)
@@ -309,6 +337,10 @@ class AutoscalePool(WorkerPool):
         # max workers can't be less than min_workers
         self.max_workers = max(self.min_workers, self.max_workers)
 
+    def debug(self, *args, **kwargs):
+        self.cleanup()
+        return super(AutoscalePool, self).debug(*args, **kwargs)
+
     @property
     def should_grow(self):
         if len(self.workers) < self.min_workers:

awx/main/dispatch/worker/base.py

@@ -43,6 +43,9 @@ class WorkerSignalHandler:
 
 
 class AWXConsumerBase(object):
+
+    last_stats = time.time()
+
     def __init__(self, name, worker, queues=[], pool=None):
         self.should_stop = False
 
@@ -54,6 +57,7 @@ class AWXConsumerBase(object):
         if pool is None:
             self.pool = WorkerPool()
         self.pool.init_workers(self.worker.work_loop)
+        self.redis = redis.Redis.from_url(settings.BROKER_URL)
 
     @property
     def listening_on(self):
@@ -99,6 +103,16 @@ class AWXConsumerBase(object):
             queue = 0
         self.pool.write(queue, body)
         self.total_messages += 1
+        self.record_statistics()
+
+    def record_statistics(self):
+        if time.time() - self.last_stats > 1:  # buffer stat recording to once per second
+            try:
+                self.redis.set(f'awx_{self.name}_statistics', self.pool.debug())
+                self.last_stats = time.time()
+            except Exception:
+                logger.exception(f"encountered an error communicating with redis to store {self.name} statistics")
+                self.last_stats = time.time()
 
     def run(self, *args, **kwargs):
         signal.signal(signal.SIGINT, self.stop)
@@ -118,23 +132,9 @@ class AWXConsumerRedis(AWXConsumerBase):
         super(AWXConsumerRedis, self).run(*args, **kwargs)
         self.worker.on_start()
 
-        time_to_sleep = 1
         while True:
-            queue = redis.Redis.from_url(settings.BROKER_URL)
-            while True:
-                try:
-                    res = queue.blpop(self.queues)
-                    time_to_sleep = 1
-                    res = json.loads(res[1])
-                    self.process_task(res)
-                except redis.exceptions.RedisError:
-                    time_to_sleep = min(time_to_sleep * 2, 30)
-                    logger.exception(f"encountered an error communicating with redis. Reconnect attempt in {time_to_sleep} seconds")
-                    time.sleep(time_to_sleep)
-                except (json.JSONDecodeError, KeyError):
-                    logger.exception("failed to decode JSON message from redis")
-                if self.should_stop:
-                    return
+            logger.debug(f'{os.getpid()} is alive')
+            time.sleep(60)
 
 
 class AWXConsumerPG(AWXConsumerBase):

awx/main/dispatch/worker/callback.py

@@ -1,17 +1,18 @@
-import cProfile
+import json
 import logging
 import os
-import pstats
 import signal
-import tempfile
 import time
 import traceback
-from queue import Empty as QueueEmpty
 
 from django.conf import settings
 from django.utils.timezone import now as tz_now
 from django.db import DatabaseError, OperationalError, connection as django_connection
-from django.db.utils import InterfaceError, InternalError, IntegrityError
+from django.db.utils import InterfaceError, InternalError
+
+import psutil
+
+import redis
 
 from awx.main.consumers import emit_channel_notification
 from awx.main.models import (JobEvent, AdHocCommandEvent, ProjectUpdateEvent,
@@ -19,15 +20,12 @@ from awx.main.models import (JobEvent, AdHocCommandEvent, ProjectUpdateEvent,
                              Job)
 from awx.main.tasks import handle_success_and_failure_notifications
 from awx.main.models.events import emit_event_detail
+from awx.main.utils.profiling import AWXProfiler
 
 from .base import BaseWorker
 
 logger = logging.getLogger('awx.main.commands.run_callback_receiver')
 
-# the number of seconds to buffer events in memory before flushing
-# using JobEvent.objects.bulk_create()
-BUFFER_SECONDS = .1
-
 
 class CallbackBrokerWorker(BaseWorker):
     '''
@@ -39,31 +37,61 @@ class CallbackBrokerWorker(BaseWorker):
     '''
 
     MAX_RETRIES = 2
+    last_stats = time.time()
+    total = 0
+    last_event = ''
     prof = None
 
     def __init__(self):
         self.buff = {}
+        self.pid = os.getpid()
+        self.redis = redis.Redis.from_url(settings.BROKER_URL)
+        self.prof = AWXProfiler("CallbackBrokerWorker")
+        for key in self.redis.keys('awx_callback_receiver_statistics_*'):
+            self.redis.delete(key)
 
     def read(self, queue):
         try:
-            return queue.get(block=True, timeout=BUFFER_SECONDS)
-        except QueueEmpty:
-            return {'event': 'FLUSH'}
+            res = self.redis.blpop(settings.CALLBACK_QUEUE, timeout=settings.JOB_EVENT_BUFFER_SECONDS)
+            if res is None:
+                return {'event': 'FLUSH'}
+            self.total += 1
+            return json.loads(res[1])
+        except redis.exceptions.RedisError:
+            logger.exception("encountered an error communicating with redis")
+            time.sleep(1)
+        except (json.JSONDecodeError, KeyError):
+            logger.exception("failed to decode JSON message from redis")
+        finally:
+            self.record_statistics()
+        return {'event': 'FLUSH'}
+
+    def record_statistics(self):
+        # buffer stat recording to once per (by default) 5s
+        if time.time() - self.last_stats > settings.JOB_EVENT_STATISTICS_INTERVAL:
+            try:
+                self.redis.set(f'awx_callback_receiver_statistics_{self.pid}', self.debug())
+                self.last_stats = time.time()
+            except Exception:
+                logger.exception("encountered an error communicating with redis")
+                self.last_stats = time.time()
+
+    def debug(self):
+        return f'.  worker[pid:{self.pid}] sent={self.total} rss={self.mb}MB {self.last_event}'
+
+    @property
+    def mb(self):
+        return '{:0.3f}'.format(
+            psutil.Process(self.pid).memory_info().rss / 1024.0 / 1024.0
+        )
 
     def toggle_profiling(self, *args):
-        if self.prof:
-            self.prof.disable()
-            filename = f'callback-{os.getpid()}.pstats'
-            filepath = os.path.join(tempfile.gettempdir(), filename)
-            with open(filepath, 'w') as f:
-                pstats.Stats(self.prof, stream=f).sort_stats('cumulative').print_stats()
-            pstats.Stats(self.prof).dump_stats(filepath + '.raw')
-            self.prof = False
-            logger.error(f'profiling is disabled, wrote {filepath}')
-        else:
-            self.prof = cProfile.Profile()
-            self.prof.enable()
+        if not self.prof.is_started():
+            self.prof.start()
             logger.error('profiling is enabled')
+        else:
+            filepath = self.prof.stop()
+            logger.error(f'profiling is disabled, wrote {filepath}')
 
     def work_loop(self, *args, **kw):
         if settings.AWX_CALLBACK_PROFILE:
@@ -84,20 +112,12 @@ class CallbackBrokerWorker(BaseWorker):
                     e.modified = now
                 try:
                     cls.objects.bulk_create(events)
-                except Exception as exc:
+                except Exception:
                     # if an exception occurs, we should re-attempt to save the
                     # events one-by-one, because something in the list is
-                    # broken/stale (e.g., an IntegrityError on a specific event)
+                    # broken/stale
                     for e in events:
                         try:
-                            if (
-                                isinstance(exc, IntegrityError) and
-                                getattr(e, 'host_id', '')
-                            ):
-                                # this is one potential IntegrityError we can
-                                # work around - if the host disappears before
-                                # the event can be processed
-                                e.host_id = None
                             e.save()
                         except Exception:
                             logger.exception('Database Error Saving Job Event')
@@ -108,6 +128,8 @@ class CallbackBrokerWorker(BaseWorker):
     def perform_work(self, body):
         try:
             flush = body.get('event') == 'FLUSH'
+            if flush:
+                self.last_event = ''
             if not flush:
                 event_map = {
                     'job_id': JobEvent,
@@ -123,6 +145,8 @@ class CallbackBrokerWorker(BaseWorker):
                         job_identifier = body[key]
                         break
 
+                self.last_event = f'\n\t- {cls.__name__} for #{job_identifier} ({body.get("event", "")} {body.get("uuid", "")})'  # noqa
+
                 if body.get('event') == 'EOF':
                     try:
                         final_counter = body.get('final_counter', 0)

awx/main/exceptions.py

@@ -30,3 +30,10 @@ class _AwxTaskError():
 
 
 AwxTaskError = _AwxTaskError()
+
+
+class PostRunError(Exception):
+    def __init__(self, msg, status='failed', tb=''):
+        self.status = status
+        self.tb = tb
+        super(PostRunError, self).__init__(msg)

awx/main/isolated/manager.py

@@ -149,7 +149,6 @@ class IsolatedManager(object):
             # don't rsync source control metadata (it can be huge!)
             '- /project/.git',
             '- /project/.svn',
-            '- /project/.hg',
             # don't rsync job events that are in the process of being written
             '- /artifacts/job_events/*-partial.json.tmp',
             # don't rsync the ssh_key FIFO

awx/main/management/commands/check_license.py

@@ -18,7 +18,5 @@ class Command(BaseCommand):
         super(Command, self).__init__()
         license = get_licenser().validate()
         if options.get('data'):
-            if license.get('license_key', '') != 'UNLICENSED':
-                license['license_key'] = '********'
             return json.dumps(license)
         return license.get('license_type', 'none')

awx/main/management/commands/check_migrations.py

@@ -8,5 +8,7 @@ class Command(MakeMigrations):
     def execute(self, *args, **options):
         settings = connections['default'].settings_dict.copy()
         settings['ENGINE'] = 'sqlite3'
+        if 'application_name' in settings['OPTIONS']:
+            del settings['OPTIONS']['application_name']
         connections['default'] = DatabaseWrapper(settings)
         return MakeMigrations().execute(*args, **options)

awx/main/management/commands/create_preload_data.py

@@ -42,6 +42,16 @@ class Command(BaseCommand):
                                               },
                                               created_by=superuser)
                 c.admin_role.members.add(superuser)
+                public_galaxy_credential = Credential(
+                    name='Ansible Galaxy',
+                    managed_by_tower=True,
+                    credential_type=CredentialType.objects.get(kind='galaxy'),
+                    inputs = {
+                        'url': 'https://galaxy.ansible.com/'
+                    }
+                )
+                public_galaxy_credential.save()
+                o.galaxy_credentials.add(public_galaxy_credential)
                 i = Inventory.objects.create(name='Demo Inventory',
                                              organization=o,
                                              created_by=superuser)

awx/main/management/commands/gather_analytics.py

@@ -1,6 +1,9 @@
 import logging
+
 from awx.main.analytics import gather, ship
+from dateutil import parser
 from django.core.management.base import BaseCommand
+from django.utils.timezone import now
 
 
 class Command(BaseCommand):
@@ -15,6 +18,10 @@ class Command(BaseCommand):
                             help='Gather analytics without shipping. Works even if analytics are disabled in settings.')
         parser.add_argument('--ship', dest='ship', action='store_true',
                             help='Enable to ship metrics to the Red Hat Cloud')
+        parser.add_argument('--since', dest='since', action='store',
+                            help='Start date for collection')
+        parser.add_argument('--until', dest='until', action='store',
+                            help='End date for collection')
 
     def init_logging(self):
         self.logger = logging.getLogger('awx.main.analytics')
@@ -28,11 +35,28 @@ class Command(BaseCommand):
         self.init_logging()
         opt_ship = options.get('ship')
         opt_dry_run = options.get('dry-run')
+        opt_since = options.get('since') or None
+        opt_until = options.get('until') or None
+
+        if opt_since:
+            since = parser.parse(opt_since)
+        else:
+            since = None
+        if opt_until:
+            until = parser.parse(opt_until)
+        else:
+            until = now()
+
         if opt_ship and opt_dry_run:
             self.logger.error('Both --ship and --dry-run cannot be processed at the same time.')
             return
-        tgz = gather(collection_type='manual' if not opt_dry_run else 'dry-run')
-        if tgz:
-            self.logger.debug(tgz)
+        tgzfiles = gather(collection_type='manual' if not opt_dry_run else 'dry-run', since = since, until = until)
+        if tgzfiles:
+            for tgz in tgzfiles:
+                self.logger.info(tgz)
+        else:
+            self.logger.error('No analytics collected')
         if opt_ship:
-            ship(tgz)
+            if tgzfiles:
+                for tgz in tgzfiles:
+                    ship(tgz)

awx/main/management/commands/graph_jobs.py

@@ -0,0 +1,117 @@
+# Python
+import asciichartpy as chart
+import collections
+import time
+import sys
+
+# Django
+from django.db.models import Count
+from django.core.management.base import BaseCommand
+
+# AWX
+from awx.main.models import (
+    Job,
+    Instance
+)
+
+
+DEFAULT_WIDTH = 100
+DEFAULT_HEIGHT = 30
+
+
+def chart_color_lookup(color_str):
+    return getattr(chart, color_str)
+
+
+def clear_screen():
+    print(chr(27) + "[2J")
+
+
+class JobStatus():
+    def __init__(self, status, color, width):
+        self.status = status
+        self.color = color
+        self.color_code = chart_color_lookup(color)
+        self.x = collections.deque(maxlen=width)
+        self.y = collections.deque(maxlen=width)
+
+    def tick(self, x, y):
+        self.x.append(x)
+        self.y.append(y)
+
+
+class JobStatusController:
+    RESET = chart_color_lookup('reset')
+
+    def __init__(self, width):
+        self.plots = [
+            JobStatus('pending', 'red', width),
+            JobStatus('waiting', 'blue', width),
+            JobStatus('running', 'green', width)
+        ]
+        self.ts_start = int(time.time())
+
+    def tick(self):
+        ts = int(time.time()) - self.ts_start
+        q = Job.objects.filter(status__in=['pending','waiting','running']).values_list('status').order_by().annotate(Count('status'))
+        status_count = dict(pending=0, waiting=0, running=0)
+        for status, count in q:
+            status_count[status] = count
+
+        for p in self.plots:
+            p.tick(ts, status_count[p.status])
+
+    def series(self):
+        return [list(p.y) for p in self.plots]
+
+    def generate_status(self):
+        line = ""
+        lines = []
+        for p in self.plots:
+            lines.append(f'{p.color_code}{p.status} {p.y[-1]}{self.RESET}')
+
+        line += ", ".join(lines) + '\n'
+
+        width = 5
+        time_running = int(time.time()) - self.ts_start
+        instances = Instance.objects.all().order_by('hostname')
+        line += "Capacity:  " + ", ".join([f"{instance.capacity:{width}}" for instance in instances]) + '\n'
+        line += "Remaining: " + ", ".join([f"{instance.remaining_capacity:{width}}" for instance in instances]) + '\n'
+        line += f"Seconds running: {time_running}" + '\n'
+
+        return line
+
+
+class Command(BaseCommand):
+    help = "Plot pending, waiting, running jobs over time on the terminal"
+
+    def add_arguments(self, parser):
+        parser.add_argument('--refresh', dest='refresh', type=float, default=1.0,
+                            help='Time between refreshes of the graph and data in seconds (defaults to 1.0)')
+        parser.add_argument('--width', dest='width', type=int, default=DEFAULT_WIDTH,
+                            help=f'Width of the graph (defaults to {DEFAULT_WIDTH})')
+        parser.add_argument('--height', dest='height', type=int, default=DEFAULT_HEIGHT,
+                            help=f'Height of the graph (defaults to {DEFAULT_HEIGHT})')
+
+    def handle(self, *args, **options):
+        refresh_seconds = options['refresh']
+        width = options['width']
+        height = options['height']
+
+        jctl = JobStatusController(width)
+
+        conf = {
+            'colors': [chart_color_lookup(p.color) for p in jctl.plots],
+            'height': height,
+        }
+
+        while True:
+            jctl.tick()
+
+            draw = chart.plot(jctl.series(), conf)
+            status_line = jctl.generate_status()
+            clear_screen()
+            print(draw)
+            sys.stdout.write(status_line)
+            time.sleep(refresh_seconds)
+

awx/main/management/commands/inventory_import.py

@@ -12,7 +12,6 @@ import sys
 import time
 import traceback
 import shutil
-from distutils.version import LooseVersion as Version
 
 # Django
 from django.conf import settings
@@ -20,6 +19,9 @@ from django.core.management.base import BaseCommand, CommandError
 from django.db import connection, transaction
 from django.utils.encoding import smart_text
 
+# DRF error class to distinguish license exceptions
+from rest_framework.exceptions import PermissionDenied
+
 # AWX inventory imports
 from awx.main.models.inventory import (
     Inventory,
@@ -32,14 +34,14 @@ from awx.main.utils.safe_yaml import sanitize_jinja
 
 # other AWX imports
 from awx.main.models.rbac import batch_role_ancestor_rebuilding
+# TODO: remove proot utils once we move to running inv. updates in containers
 from awx.main.utils import (
-    ignore_inventory_computed_fields,
     check_proot_installed,
     wrap_args_with_proot,
     build_proot_temp_dir,
+    ignore_inventory_computed_fields,
     get_licenser
 )
-from awx.main.utils.common import _get_ansible_version
 from awx.main.signals import disable_activity_stream
 from awx.main.constants import STANDARD_INVENTORY_UPDATE_ENV
 from awx.main.utils.pglock import advisory_lock
@@ -55,11 +57,11 @@ No license.
 See http://www.ansible.com/renew for license information.'''
 
 LICENSE_MESSAGE = '''\
-Number of licensed instances exceeded, would bring available instances to %(new_count)d, system is licensed for %(available_instances)d.
+Number of licensed instances exceeded, would bring available instances to %(new_count)d, system is licensed for %(instance_count)d.
 See http://www.ansible.com/renew for license extension information.'''
 
 DEMO_LICENSE_MESSAGE = '''\
-Demo mode free license count exceeded, would bring available instances to %(new_count)d, demo mode allows %(available_instances)d.
+Demo mode free license count exceeded, would bring available instances to %(new_count)d, demo mode allows %(instance_count)d.
 See http://www.ansible.com/renew for licensing information.'''
 
 
@@ -77,13 +79,11 @@ class AnsibleInventoryLoader(object):
         /usr/bin/ansible/ansible-inventory -i hosts --list
     '''
 
-    def __init__(self, source, is_custom=False, venv_path=None, verbosity=0):
+    def __init__(self, source, venv_path=None, verbosity=0):
         self.source = source
-        self.source_dir = functioning_dir(self.source)
-        self.is_custom = is_custom
-        self.tmp_private_dir = None
-        self.method = 'ansible-inventory'
         self.verbosity = verbosity
+        # TODO: remove once proot has been removed
+        self.tmp_private_dir = None
         if venv_path:
             self.venv_path = venv_path
         else:
@@ -136,40 +136,31 @@ class AnsibleInventoryLoader(object):
         # inside of /venv/ansible, so we override the specified interpreter
         # https://github.com/ansible/ansible/issues/50714
         bargs = ['python', ansible_inventory_path, '-i', self.source]
-        ansible_version = _get_ansible_version(ansible_inventory_path[:-len('-inventory')])
-        if ansible_version != 'unknown':
-            this_version = Version(ansible_version)
-            if this_version >= Version('2.5'):
-                bargs.extend(['--playbook-dir', self.source_dir])
-            if this_version >= Version('2.8'):
-                if self.verbosity:
-                    # INFO: -vvv, DEBUG: -vvvvv, for inventory, any more than 3 makes little difference
-                    bargs.append('-{}'.format('v' * min(5, self.verbosity * 2 + 1)))
+        bargs.extend(['--playbook-dir', functioning_dir(self.source)])
+        if self.verbosity:
+            # INFO: -vvv, DEBUG: -vvvvv, for inventory, any more than 3 makes little difference
+            bargs.append('-{}'.format('v' * min(5, self.verbosity * 2 + 1)))
         logger.debug('Using base command: {}'.format(' '.join(bargs)))
         return bargs
 
+    # TODO: Remove this once we move to running ansible-inventory in containers
+    # and don't need proot for process isolation anymore
     def get_proot_args(self, cmd, env):
         cwd = os.getcwd()
         if not check_proot_installed():
             raise RuntimeError("proot is not installed but is configured for use")
 
         kwargs = {}
-        if self.is_custom:
-            # use source's tmp dir for proot, task manager will delete folder
-            logger.debug("Using provided directory '{}' for isolation.".format(self.source_dir))
-            kwargs['proot_temp_dir'] = self.source_dir
-            cwd = self.source_dir
-        else:
-            # we cannot safely store tmp data in source dir or trust script contents
-            if env['AWX_PRIVATE_DATA_DIR']:
-                # If this is non-blank, file credentials are being used and we need access
-                private_data_dir = functioning_dir(env['AWX_PRIVATE_DATA_DIR'])
-                logger.debug("Using private credential data in '{}'.".format(private_data_dir))
-                kwargs['private_data_dir'] = private_data_dir
-            self.tmp_private_dir = build_proot_temp_dir()
-            logger.debug("Using fresh temporary directory '{}' for isolation.".format(self.tmp_private_dir))
-            kwargs['proot_temp_dir'] = self.tmp_private_dir
-            kwargs['proot_show_paths'] = [functioning_dir(self.source), settings.AWX_ANSIBLE_COLLECTIONS_PATHS]
+        # we cannot safely store tmp data in source dir or trust script contents
+        if env['AWX_PRIVATE_DATA_DIR']:
+            # If this is non-blank, file credentials are being used and we need access
+            private_data_dir = functioning_dir(env['AWX_PRIVATE_DATA_DIR'])
+            logger.debug("Using private credential data in '{}'.".format(private_data_dir))
+            kwargs['private_data_dir'] = private_data_dir
+        self.tmp_private_dir = build_proot_temp_dir()
+        logger.debug("Using fresh temporary directory '{}' for isolation.".format(self.tmp_private_dir))
+        kwargs['proot_temp_dir'] = self.tmp_private_dir
+        kwargs['proot_show_paths'] = [functioning_dir(self.source), settings.AWX_ANSIBLE_COLLECTIONS_PATHS]
         logger.debug("Running from `{}` working directory.".format(cwd))
 
         if self.venv_path != settings.ANSIBLE_VENV_PATH:
@@ -177,12 +168,14 @@ class AnsibleInventoryLoader(object):
 
         return wrap_args_with_proot(cmd, cwd, **kwargs)
 
+
     def command_to_json(self, cmd):
         data = {}
         stdout, stderr = '', ''
         env = self.build_env()
 
-        if ((self.is_custom or 'AWX_PRIVATE_DATA_DIR' in env) and
+        # TODO: remove proot args once inv. updates run in containers
+        if (('AWX_PRIVATE_DATA_DIR' in env) and
                 getattr(settings, 'AWX_PROOT_ENABLED', False)):
             cmd = self.get_proot_args(cmd, env)
 
@@ -191,11 +184,13 @@ class AnsibleInventoryLoader(object):
         stdout = smart_text(stdout)
         stderr = smart_text(stderr)
 
+        # TODO: can be removed when proot is removed
         if self.tmp_private_dir:
             shutil.rmtree(self.tmp_private_dir, True)
+
         if proc.returncode != 0:
             raise RuntimeError('%s failed (rc=%d) with stdout:\n%s\nstderr:\n%s' % (
-                self.method, proc.returncode, stdout, stderr))
+                'ansible-inventory', proc.returncode, stdout, stderr))
 
         for line in stderr.splitlines():
             logger.error(line)
@@ -238,9 +233,9 @@ class Command(BaseCommand):
                             action='store_true', default=False,
                             help='overwrite (rather than merge) variables')
         parser.add_argument('--keep-vars', dest='keep_vars', action='store_true', default=False,
-                            help='use database variables if set')
+                            help='DEPRECATED legacy option, has no effect')
         parser.add_argument('--custom', dest='custom', action='store_true', default=False,
-                            help='this is a custom inventory script')
+                            help='DEPRECATED indicates a custom inventory script, no longer used')
         parser.add_argument('--source', dest='source', type=str, default=None,
                             metavar='s', help='inventory directory, file, or script to load')
         parser.add_argument('--enabled-var', dest='enabled_var', type=str,
@@ -266,10 +261,10 @@ class Command(BaseCommand):
                             'specifies the unique, immutable instance ID, may be '
                             'specified as "foo.bar" to traverse nested dicts.')
 
-    def set_logging_level(self):
+    def set_logging_level(self, verbosity):
         log_levels = dict(enumerate([logging.WARNING, logging.INFO,
                                      logging.DEBUG, 0]))
-        logger.setLevel(log_levels.get(self.verbosity, 0))
+        logger.setLevel(log_levels.get(verbosity, 0))
 
     def _get_instance_id(self, variables, default=''):
         '''
@@ -329,7 +324,8 @@ class Command(BaseCommand):
         else:
             raise NotImplementedError('Value of enabled {} not understood.'.format(enabled))
 
-    def get_source_absolute_path(self, source):
+    @staticmethod
+    def get_source_absolute_path(source):
         if not os.path.exists(source):
             raise IOError('Source does not exist: %s' % source)
         source = os.path.join(os.getcwd(), os.path.dirname(source),
@@ -337,61 +333,6 @@ class Command(BaseCommand):
         source = os.path.normpath(os.path.abspath(source))
         return source
 
-    def load_inventory_from_database(self):
-        '''
-        Load inventory and related objects from the database.
-        '''
-        # Load inventory object based on name or ID.
-        if self.inventory_id:
-            q = dict(id=self.inventory_id)
-        else:
-            q = dict(name=self.inventory_name)
-        try:
-            self.inventory = Inventory.objects.get(**q)
-        except Inventory.DoesNotExist:
-            raise CommandError('Inventory with %s = %s cannot be found' % list(q.items())[0])
-        except Inventory.MultipleObjectsReturned:
-            raise CommandError('Inventory with %s = %s returned multiple results' % list(q.items())[0])
-        logger.info('Updating inventory %d: %s' % (self.inventory.pk,
-                                                   self.inventory.name))
-
-        # Load inventory source if specified via environment variable (when
-        # inventory_import is called from an InventoryUpdate task).
-        inventory_source_id = os.getenv('INVENTORY_SOURCE_ID', None)
-        inventory_update_id = os.getenv('INVENTORY_UPDATE_ID', None)
-        if inventory_source_id:
-            try:
-                self.inventory_source = InventorySource.objects.get(pk=inventory_source_id,
-                                                                    inventory=self.inventory)
-            except InventorySource.DoesNotExist:
-                raise CommandError('Inventory source with id=%s not found' %
-                                   inventory_source_id)
-            try:
-                self.inventory_update = InventoryUpdate.objects.get(pk=inventory_update_id)
-            except InventoryUpdate.DoesNotExist:
-                raise CommandError('Inventory update with id=%s not found' %
-                                   inventory_update_id)
-        # Otherwise, create a new inventory source to capture this invocation
-        # via command line.
-        else:
-            with ignore_inventory_computed_fields():
-                self.inventory_source, created = InventorySource.objects.get_or_create(
-                    inventory=self.inventory,
-                    source='file',
-                    source_path=os.path.abspath(self.source),
-                    overwrite=self.overwrite,
-                    overwrite_vars=self.overwrite_vars,
-                )
-                self.inventory_update = self.inventory_source.create_inventory_update(
-                    _eager_fields=dict(
-                        job_args=json.dumps(sys.argv),
-                        job_env=dict(os.environ.items()),
-                        job_cwd=os.getcwd())
-                )
-
-        # FIXME: Wait or raise error if inventory is being updated by another
-        # source.
-
     def _batch_add_m2m(self, related_manager, *objs, **kwargs):
         key = (related_manager.instance.pk, related_manager.through._meta.db_table)
         flush = bool(kwargs.get('flush', False))
@@ -881,42 +822,41 @@ class Command(BaseCommand):
         Load inventory from in-memory groups to the database, overwriting or
         merging as appropriate.
         '''
-        with advisory_lock('inventory_{}_update'.format(self.inventory.id)):
-            # FIXME: Attribute changes to superuser?
-            # Perform __in queries in batches (mainly for unit tests using SQLite).
-            self._batch_size = 500
-            self._build_db_instance_id_map()
-            self._build_mem_instance_id_map()
-            if self.overwrite:
-                self._delete_hosts()
-                self._delete_groups()
-                self._delete_group_children_and_hosts()
-            self._update_inventory()
-            self._create_update_groups()
-            self._create_update_hosts()
-            self._create_update_group_children()
-            self._create_update_group_hosts()
+        # FIXME: Attribute changes to superuser?
+        # Perform __in queries in batches (mainly for unit tests using SQLite).
+        self._batch_size = 500
+        self._build_db_instance_id_map()
+        self._build_mem_instance_id_map()
+        if self.overwrite:
+            self._delete_hosts()
+            self._delete_groups()
+            self._delete_group_children_and_hosts()
+        self._update_inventory()
+        self._create_update_groups()
+        self._create_update_hosts()
+        self._create_update_group_children()
+        self._create_update_group_hosts()
 
     def remote_tower_license_compare(self, local_license_type):
         # this requires https://github.com/ansible/ansible/pull/52747
         source_vars = self.all_group.variables
         remote_license_type = source_vars.get('tower_metadata', {}).get('license_type', None)
         if remote_license_type is None:
-            raise CommandError('Unexpected Error: Tower inventory plugin missing needed metadata!')
+            raise PermissionDenied('Unexpected Error: Tower inventory plugin missing needed metadata!')
         if local_license_type != remote_license_type:
-            raise CommandError('Tower server licenses must match: source: {} local: {}'.format(
+            raise PermissionDenied('Tower server licenses must match: source: {} local: {}'.format(
                 remote_license_type, local_license_type
             ))
 
     def check_license(self):
         license_info = get_licenser().validate()
         local_license_type = license_info.get('license_type', 'UNLICENSED')
-        if license_info.get('license_key', 'UNLICENSED') == 'UNLICENSED':
+        if local_license_type == 'UNLICENSED':
             logger.error(LICENSE_NON_EXISTANT_MESSAGE)
-            raise CommandError('No license found!')
+            raise PermissionDenied('No license found!')
         elif local_license_type == 'open':
             return
-        available_instances = license_info.get('available_instances', 0)
+        instance_count = license_info.get('instance_count', 0)
         free_instances = license_info.get('free_instances', 0)
         time_remaining = license_info.get('time_remaining', 0)
         hard_error = license_info.get('trial', False) is True or license_info['instance_count'] == 10
@@ -924,24 +864,24 @@ class Command(BaseCommand):
         if time_remaining <= 0:
             if hard_error:
                 logger.error(LICENSE_EXPIRED_MESSAGE)
-                raise CommandError("License has expired!")
+                raise PermissionDenied("License has expired!")
             else:
                 logger.warning(LICENSE_EXPIRED_MESSAGE)
         # special check for tower-type inventory sources
         # but only if running the plugin
         TOWER_SOURCE_FILES = ['tower.yml', 'tower.yaml']
-        if self.inventory_source.source == 'tower' and any(f in self.source for f in TOWER_SOURCE_FILES):
+        if self.inventory_source.source == 'tower' and any(f in self.inventory_source.source_path for f in TOWER_SOURCE_FILES):
             # only if this is the 2nd call to license check, we cannot compare before running plugin
             if hasattr(self, 'all_group'):
                 self.remote_tower_license_compare(local_license_type)
         if free_instances < 0:
             d = {
                 'new_count': new_count,
-                'available_instances': available_instances,
+                'instance_count': instance_count,
             }
             if hard_error:
                 logger.error(LICENSE_MESSAGE % d)
-                raise CommandError('License count exceeded!')
+                raise PermissionDenied('License count exceeded!')
             else:
                 logger.warning(LICENSE_MESSAGE % d)
 
@@ -956,7 +896,7 @@ class Command(BaseCommand):
 
         active_count = Host.objects.org_active_count(org.id)
         if active_count > org.max_hosts:
-            raise CommandError('Host limit for organization exceeded!')
+            raise PermissionDenied('Host limit for organization exceeded!')
 
     def mark_license_failure(self, save=True):
         self.inventory_update.license_error = True
@@ -967,16 +907,103 @@ class Command(BaseCommand):
         self.inventory_update.save(update_fields=['org_host_limit_error'])
 
     def handle(self, *args, **options):
-        self.verbosity = int(options.get('verbosity', 1))
-        self.set_logging_level()
-        self.inventory_name = options.get('inventory_name', None)
-        self.inventory_id = options.get('inventory_id', None)
-        venv_path = options.get('venv', None)
+        # Load inventory and related objects from database.
+        inventory_name = options.get('inventory_name', None)
+        inventory_id = options.get('inventory_id', None)
+        if inventory_name and inventory_id:
+            raise CommandError('--inventory-name and --inventory-id are mutually exclusive')
+        elif not inventory_name and not inventory_id:
+            raise CommandError('--inventory-name or --inventory-id is required')
+
+        with advisory_lock('inventory_{}_import'.format(inventory_id)):
+            # Obtain rest of the options needed to run update
+            raw_source = options.get('source', None)
+            if not raw_source:
+                raise CommandError('--source is required')
+            verbosity = int(options.get('verbosity', 1))
+            self.set_logging_level(verbosity)
+            venv_path = options.get('venv', None)
+
+            # Load inventory object based on name or ID.
+            if inventory_id:
+                q = dict(id=inventory_id)
+            else:
+                q = dict(name=inventory_name)
+            try:
+                inventory = Inventory.objects.get(**q)
+            except Inventory.DoesNotExist:
+                raise CommandError('Inventory with %s = %s cannot be found' % list(q.items())[0])
+            except Inventory.MultipleObjectsReturned:
+                raise CommandError('Inventory with %s = %s returned multiple results' % list(q.items())[0])
+            logger.info('Updating inventory %d: %s' % (inventory.pk, inventory.name))
+
+
+            # Create ad-hoc inventory source and inventory update objects
+            with ignore_inventory_computed_fields():
+                source = Command.get_source_absolute_path(raw_source)
+
+                inventory_source, created = InventorySource.objects.get_or_create(
+                    inventory=inventory,
+                    source='file',
+                    source_path=os.path.abspath(source),
+                    overwrite=bool(options.get('overwrite', False)),
+                    overwrite_vars=bool(options.get('overwrite_vars', False)),
+                )
+                inventory_update = inventory_source.create_inventory_update(
+                    _eager_fields=dict(
+                        job_args=json.dumps(sys.argv),
+                        job_env=dict(os.environ.items()),
+                        job_cwd=os.getcwd())
+                )
+
+            data = AnsibleInventoryLoader(
+                source=source, venv_path=venv_path, verbosity=verbosity
+            ).load()
+
+            logger.debug('Finished loading from source: %s', source)
+
+            status, tb, exc = 'error', '', None
+            try:
+                self.perform_update(options, data, inventory_update)
+                status = 'successful'
+            except Exception as e:
+                exc = e
+                if isinstance(e, KeyboardInterrupt):
+                    status = 'canceled'
+                else:
+                    tb = traceback.format_exc()
+
+            with ignore_inventory_computed_fields():
+                inventory_update = InventoryUpdate.objects.get(pk=inventory_update.pk)
+                inventory_update.result_traceback = tb
+                inventory_update.status = status
+                inventory_update.save(update_fields=['status', 'result_traceback'])
+                inventory_source.status = status
+                inventory_source.save(update_fields=['status'])
+
+        if exc:
+            logger.error(str(exc))
+
+        if exc:
+            if isinstance(exc, CommandError):
+                sys.exit(1)
+            raise exc
+
+    def perform_update(self, options, data, inventory_update):
+        """Shared method for both awx-manage CLI updates and inventory updates
+        from the tasks system.
+
+        This saves the inventory data to the database, calling load_into_database
+        but also wraps that method in a host of options processing
+        """
+        # outside of normal options, these are needed as part of programatic interface
+        self.inventory = inventory_update.inventory
+        self.inventory_source = inventory_update.inventory_source
+        self.inventory_update = inventory_update
+
+        # the update options, could be parser object or dict
         self.overwrite = bool(options.get('overwrite', False))
         self.overwrite_vars = bool(options.get('overwrite_vars', False))
-        self.keep_vars = bool(options.get('keep_vars', False))
-        self.is_custom = bool(options.get('custom', False))
-        self.source = options.get('source', None)
         self.enabled_var = options.get('enabled_var', None)
         self.enabled_value = options.get('enabled_value', None)
         self.group_filter = options.get('group_filter', None) or r'^.+$'
@@ -984,17 +1011,6 @@ class Command(BaseCommand):
         self.exclude_empty_groups = bool(options.get('exclude_empty_groups', False))
         self.instance_id_var = options.get('instance_id_var', None)
 
-        self.invoked_from_dispatcher = False if os.getenv('INVENTORY_SOURCE_ID', None) is None else True
-
-        # Load inventory and related objects from database.
-        if self.inventory_name and self.inventory_id:
-            raise CommandError('--inventory-name and --inventory-id are mutually exclusive')
-        elif not self.inventory_name and not self.inventory_id:
-            raise CommandError('--inventory-name or --inventory-id is required')
-        if (self.overwrite or self.overwrite_vars) and self.keep_vars:
-            raise CommandError('--overwrite/--overwrite-vars and --keep-vars are mutually exclusive')
-        if not self.source:
-            raise CommandError('--source is required')
         try:
             self.group_filter_re = re.compile(self.group_filter)
         except re.error:
@@ -1005,47 +1021,43 @@ class Command(BaseCommand):
             raise CommandError('invalid regular expression for --host-filter')
 
         begin = time.time()
-        self.load_inventory_from_database()
 
-        try:
-            self.check_license()
-        except CommandError as e:
-            self.mark_license_failure(save=True)
-            raise e
+        # Since perform_update can be invoked either through the awx-manage CLI
+        # or from the task system, we need to create a new lock at this level
+        # (even though inventory_import.Command.handle -- which calls
+        # perform_update -- has its own lock, inventory_ID_import)
+        with advisory_lock('inventory_{}_perform_update'.format(self.inventory.id)):
 
-        try:
-            # Check the per-org host limits
-            self.check_org_host_limit()
-        except CommandError as e:
-            self.mark_org_limits_failure(save=True)
-            raise e
+            try:
+                self.check_license()
+            except PermissionDenied as e:
+                self.mark_license_failure(save=True)
+                raise e
+
+            try:
+                # Check the per-org host limits
+                self.check_org_host_limit()
+            except PermissionDenied as e:
+                self.mark_org_limits_failure(save=True)
+                raise e
 
-        status, tb, exc = 'error', '', None
-        try:
             if settings.SQL_DEBUG:
                 queries_before = len(connection.queries)
 
             # Update inventory update for this command line invocation.
             with ignore_inventory_computed_fields():
+                # TODO: move this to before perform_update
                 iu = self.inventory_update
                 if iu.status != 'running':
                     with transaction.atomic():
                         self.inventory_update.status = 'running'
                         self.inventory_update.save()
 
-            source = self.get_source_absolute_path(self.source)
-
-            data = AnsibleInventoryLoader(source=source, is_custom=self.is_custom,
-                                          venv_path=venv_path, verbosity=self.verbosity).load()
-
-            logger.debug('Finished loading from source: %s', source)
             logger.info('Processing JSON output...')
             inventory = MemInventory(
                 group_filter_re=self.group_filter_re, host_filter_re=self.host_filter_re)
             inventory = dict_to_mem_data(data, inventory=inventory)
 
-            del data  # forget dict from import, could be large
-
             logger.info('Loaded %d groups, %d hosts', len(inventory.all_group.all_groups),
                         len(inventory.all_group.all_hosts))
 
@@ -1085,9 +1097,10 @@ class Command(BaseCommand):
                             if settings.SQL_DEBUG:
                                 queries_before2 = len(connection.queries)
                             self.inventory.update_computed_fields()
-                            if settings.SQL_DEBUG:
-                                logger.warning('update computed fields took %d queries',
-                                               len(connection.queries) - queries_before2)
+                        if settings.SQL_DEBUG:
+                            logger.warning('update computed fields took %d queries',
+                                           len(connection.queries) - queries_before2)
+
                         # Check if the license is valid.
                         # If the license is not valid, a CommandError will be thrown,
                         # and inventory update will be marked as invalid.
@@ -1098,11 +1111,11 @@ class Command(BaseCommand):
                         # Check the per-org host limits
                         license_fail = False
                         self.check_org_host_limit()
-                except CommandError as e:
+                except PermissionDenied as e:
                     if license_fail:
-                        self.mark_license_failure()
+                        self.mark_license_failure(save=True)
                     else:
-                        self.mark_org_limits_failure()
+                        self.mark_org_limits_failure(save=True)
                     raise e
 
                 if settings.SQL_DEBUG:
@@ -1111,7 +1124,6 @@ class Command(BaseCommand):
                 else:
                     logger.info('Inventory import completed for %s in %0.1fs',
                                 self.inventory_source.name, time.time() - begin)
-                status = 'successful'
 
             # If we're in debug mode, then log the queries and time
             # used to do the operation.
@@ -1121,29 +1133,3 @@ class Command(BaseCommand):
                 logger.warning('Inventory import required %d queries '
                                'taking %0.3fs', len(queries_this_import),
                                sqltime)
-        except Exception as e:
-            if isinstance(e, KeyboardInterrupt):
-                status = 'canceled'
-                exc = e
-            elif isinstance(e, CommandError):
-                exc = e
-            else:
-                tb = traceback.format_exc()
-                exc = e
-
-        if not self.invoked_from_dispatcher:
-            with ignore_inventory_computed_fields():
-                self.inventory_update = InventoryUpdate.objects.get(pk=self.inventory_update.pk)
-                self.inventory_update.result_traceback = tb
-                self.inventory_update.status = status
-                self.inventory_update.save(update_fields=['status', 'result_traceback'])
-                self.inventory_source.status = status
-                self.inventory_source.save(update_fields=['status'])
-
-            if exc:
-                logger.error(str(exc))
-
-        if exc:
-            if isinstance(exc, CommandError):
-                sys.exit(1)
-            raise exc

awx/main/management/commands/profile_sql.py

@@ -19,7 +19,9 @@ class Command(BaseCommand):
         profile_sql.delay(
             threshold=options['threshold'], minutes=options['minutes']
         )
-        print(f"Logging initiated with a threshold of {options['threshold']} second(s) and a duration of"
-              f" {options['minutes']} minute(s), any queries that meet criteria can"
-              f" be found in /var/log/tower/profile/."
-              )
+        if options['threshold'] > 0:
+            print(f"SQL profiling initiated with a threshold of {options['threshold']} second(s) and a"
+                  f" duration of {options['minutes']} minute(s), any queries that meet criteria can"
+                  f" be found in /var/log/tower/profile/.")
+        else:
+            print("SQL profiling disabled.")

awx/main/management/commands/provision_instance.py

@@ -13,7 +13,7 @@ from django.core.management.base import BaseCommand, CommandError
 class Command(BaseCommand):
     """
     Internal tower command.
-    Regsiter this instance with the database for HA tracking.
+    Register this instance with the database for HA tracking.
     """
 
     help = (

awx/main/management/commands/run_callback_receiver.py

@@ -4,6 +4,7 @@
 from django.conf import settings
 from django.core.management.base import BaseCommand
 
+from awx.main.dispatch.control import Control
 from awx.main.dispatch.worker import AWXConsumerRedis, CallbackBrokerWorker
 
 
@@ -15,7 +16,14 @@ class Command(BaseCommand):
     '''
     help = 'Launch the job callback receiver'
 
+    def add_arguments(self, parser):
+        parser.add_argument('--status', dest='status', action='store_true',
+                            help='print the internal state of any running dispatchers')
+
     def handle(self, *arg, **options):
+        if options.get('status'):
+            print(Control('callback_receiver').status())
+            return
         consumer = None
         try:
             consumer = AWXConsumerRedis(

awx/main/managers.py

@@ -48,7 +48,13 @@ class HostManager(models.Manager):
         """When the parent instance of the host query set has a `kind=smart` and a `host_filter`
         set. Use the `host_filter` to generate the queryset for the hosts.
         """
-        qs = super(HostManager, self).get_queryset()
+        qs = super(HostManager, self).get_queryset().defer(
+            'last_job__extra_vars',
+            'last_job_host_summary__job__extra_vars',
+            'last_job__artifacts',
+            'last_job_host_summary__job__artifacts',
+        )
+
         if (hasattr(self, 'instance') and
            hasattr(self.instance, 'host_filter') and
            hasattr(self.instance, 'kind')):

awx/main/middleware.py

@@ -1,13 +1,9 @@
 # Copyright (c) 2015 Ansible, Inc.
 # All Rights Reserved.
 
-import uuid
 import logging
 import threading
 import time
-import cProfile
-import pstats
-import os
 import urllib.parse
 
 from django.conf import settings
@@ -22,6 +18,7 @@ from django.urls import reverse, resolve
 
 from awx.main.utils.named_url_graph import generate_graph, GraphNode
 from awx.conf import fields, register
+from awx.main.utils.profiling import AWXProfiler
 
 
 logger = logging.getLogger('awx.main.middleware')
@@ -32,11 +29,14 @@ class TimingMiddleware(threading.local, MiddlewareMixin):
 
     dest = '/var/log/tower/profile'
 
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.prof = AWXProfiler("TimingMiddleware")
+
     def process_request(self, request):
         self.start_time = time.time()
         if settings.AWX_REQUEST_PROFILE:
-            self.prof = cProfile.Profile()
-            self.prof.enable()
+            self.prof.start()
 
     def process_response(self, request, response):
         if not hasattr(self, 'start_time'):  # some tools may not invoke process_request
@@ -44,33 +44,10 @@ class TimingMiddleware(threading.local, MiddlewareMixin):
         total_time = time.time() - self.start_time
         response['X-API-Total-Time'] = '%0.3fs' % total_time
         if settings.AWX_REQUEST_PROFILE:
-            self.prof.disable()
-            cprofile_file = self.save_profile_file(request)
-            response['cprofile_file'] = cprofile_file
+            response['X-API-Profile-File'] = self.prof.stop()
         perf_logger.info('api response times', extra=dict(python_objects=dict(request=request, response=response)))
         return response
 
-    def save_profile_file(self, request):
-        if not os.path.isdir(self.dest):
-            os.makedirs(self.dest)
-        filename = '%.3fs-%s.pstats' % (pstats.Stats(self.prof).total_tt, uuid.uuid4())
-        filepath = os.path.join(self.dest, filename)
-        with open(filepath, 'w') as f:
-            f.write('%s %s\n' % (request.method, request.get_full_path()))
-            pstats.Stats(self.prof, stream=f).sort_stats('cumulative').print_stats()
-
-        if settings.AWX_REQUEST_PROFILE_WITH_DOT:
-            from gprof2dot import main as generate_dot
-            raw = os.path.join(self.dest, filename) + '.raw'
-            pstats.Stats(self.prof).dump_stats(raw)
-            generate_dot([
-                '-n', '2.5', '-f', 'pstats', '-o',
-                os.path.join( self.dest, filename).replace('.pstats', '.dot'),
-                raw
-            ])
-            os.remove(raw)
-        return filepath
-
 
 class SessionTimeoutMiddleware(MiddlewareMixin):
     """
@@ -204,4 +181,4 @@ class MigrationRanCheckMiddleware(MiddlewareMixin):
         plan = executor.migration_plan(executor.loader.graph.leaf_nodes())
         if bool(plan) and \
                 getattr(resolve(request.path), 'url_name', '') != 'migrations_notran':
-            return redirect(reverse("ui:migrations_notran"))
+            return redirect(reverse("ui_next:migrations_notran"))

awx/main/migrations/0117_v400_remove_cloudforms_inventory.py

@@ -1,11 +1,7 @@
 # Generated by Django 2.2.11 on 2020-05-01 13:25
 
 from django.db import migrations, models
-from awx.main.migrations._inventory_source import create_scm_script_substitute
-
-
-def convert_cloudforms_to_scm(apps, schema_editor):
-    create_scm_script_substitute(apps, 'cloudforms')
+from awx.main.migrations._inventory_source import delete_cloudforms_inv_source
 
 
 class Migration(migrations.Migration):
@@ -15,7 +11,7 @@ class Migration(migrations.Migration):
     ]
 
     operations = [
-        migrations.RunPython(convert_cloudforms_to_scm),
+        migrations.RunPython(delete_cloudforms_inv_source),
         migrations.AlterField(
             model_name='inventorysource',
             name='source',

awx/main/migrations/0119_inventory_plugins.py

@@ -0,0 +1,104 @@
+# Generated by Django 2.2.11 on 2020-07-20 19:56
+
+import logging
+import yaml
+
+from django.db import migrations, models
+
+from awx.main.models.base import VarsDictProperty
+
+from ._inventory_source_vars import FrozenInjectors
+
+
+logger = logging.getLogger('awx.main.migrations')
+
+
+def _get_inventory_sources(InventorySource):
+    return InventorySource.objects.filter(source__in=['ec2', 'gce', 'azure_rm', 'vmware', 'satellite6', 'openstack', 'rhv', 'tower'])
+
+
+def inventory_source_vars_forward(apps, schema_editor):
+    InventorySource = apps.get_model("main", "InventorySource")
+    '''
+    The Django app registry does not keep track of model inheritance. The
+    source_vars_dict property comes from InventorySourceOptions via inheritance.
+    This adds that property. Luckily, other properteries and functionality from
+    InventorySourceOptions is not needed by the injector logic.
+    '''
+    setattr(InventorySource, 'source_vars_dict', VarsDictProperty('source_vars'))
+    source_vars_backup = dict()
+
+    for inv_source_obj in _get_inventory_sources(InventorySource):
+
+        if inv_source_obj.source in FrozenInjectors:
+            source_vars_backup[inv_source_obj.id] = dict(inv_source_obj.source_vars_dict)
+
+            injector = FrozenInjectors[inv_source_obj.source]()
+            new_inv_source_vars = injector.inventory_as_dict(inv_source_obj, None)
+            inv_source_obj.source_vars = yaml.dump(new_inv_source_vars)
+            inv_source_obj.save()
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('main', '0118_add_remote_archive_scm_type'),
+    ]
+
+    operations = [
+        migrations.RunPython(inventory_source_vars_forward),
+        migrations.RemoveField(
+            model_name='inventorysource',
+            name='group_by',
+        ),
+        migrations.RemoveField(
+            model_name='inventoryupdate',
+            name='group_by',
+        ),
+        migrations.RemoveField(
+            model_name='inventorysource',
+            name='instance_filters',
+        ),
+        migrations.RemoveField(
+            model_name='inventoryupdate',
+            name='instance_filters',
+        ),
+        migrations.RemoveField(
+            model_name='inventorysource',
+            name='source_regions',
+        ),
+        migrations.RemoveField(
+            model_name='inventoryupdate',
+            name='source_regions',
+        ),
+        migrations.AddField(
+            model_name='inventorysource',
+            name='enabled_value',
+            field=models.TextField(blank=True, default='', help_text='Only used when enabled_var is set. Value when the host is considered enabled. For example if enabled_var="status.power_state"and enabled_value="powered_on" with host variables:{   "status": {     "power_state": "powered_on",     "created": "2020-08-04T18:13:04+00:00",     "healthy": true    },    "name": "foobar",    "ip_address": "192.168.2.1"}The host would be marked enabled. If power_state where any value other than powered_on then the host would be disabled when imported into Tower. If the key is not found then the host will be enabled'),
+        ),
+        migrations.AddField(
+            model_name='inventorysource',
+            name='enabled_var',
+            field=models.TextField(blank=True, default='', help_text='Retrieve the enabled state from the given dict of host variables. The enabled variable may be specified as "foo.bar", in which case the lookup will traverse into nested dicts, equivalent to: from_dict.get("foo", {}).get("bar", default)'),
+        ),
+        migrations.AddField(
+            model_name='inventorysource',
+            name='host_filter',
+            field=models.TextField(blank=True, default='', help_text='Regex where only matching hosts will be imported into Tower.'),
+        ),
+        migrations.AddField(
+            model_name='inventoryupdate',
+            name='enabled_value',
+            field=models.TextField(blank=True, default='', help_text='Only used when enabled_var is set. Value when the host is considered enabled. For example if enabled_var="status.power_state"and enabled_value="powered_on" with host variables:{   "status": {     "power_state": "powered_on",     "created": "2020-08-04T18:13:04+00:00",     "healthy": true    },    "name": "foobar",    "ip_address": "192.168.2.1"}The host would be marked enabled. If power_state where any value other than powered_on then the host would be disabled when imported into Tower. If the key is not found then the host will be enabled'),
+        ),
+        migrations.AddField(
+            model_name='inventoryupdate',
+            name='enabled_var',
+            field=models.TextField(blank=True, default='', help_text='Retrieve the enabled state from the given dict of host variables. The enabled variable may be specified as "foo.bar", in which case the lookup will traverse into nested dicts, equivalent to: from_dict.get("foo", {}).get("bar", default)'),
+        ),
+        migrations.AddField(
+            model_name='inventoryupdate',
+            name='host_filter',
+            field=models.TextField(blank=True, default='', help_text='Regex where only matching hosts will be imported into Tower.'),
+        ),
+    ]

awx/main/migrations/0120_galaxy_credentials.py

@@ -0,0 +1,51 @@
+# Generated by Django 2.2.11 on 2020-08-04 15:19
+
+import logging
+
+import awx.main.fields
+from awx.main.utils.encryption import encrypt_field, decrypt_field
+
+from django.db import migrations, models
+from django.utils.timezone import now
+import django.db.models.deletion
+
+from awx.main.migrations import _galaxy as galaxy
+from awx.main.models import CredentialType as ModernCredentialType
+from awx.main.utils.common import set_current_apps
+
+logger = logging.getLogger('awx.main.migrations')
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('main', '0119_inventory_plugins'),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name='credentialtype',
+            name='kind',
+            field=models.CharField(choices=[('ssh', 'Machine'), ('vault', 'Vault'), ('net', 'Network'), ('scm', 'Source Control'), ('cloud', 'Cloud'), ('token', 'Personal Access Token'), ('insights', 'Insights'), ('external', 'External'), ('kubernetes', 'Kubernetes'), ('galaxy', 'Galaxy/Automation Hub')], max_length=32),
+        ),
+        migrations.CreateModel(
+            name='OrganizationGalaxyCredentialMembership',
+            fields=[
+                ('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('position', models.PositiveIntegerField(db_index=True, default=None, null=True)),
+                ('credential', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='main.Credential')),
+                ('organization', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='main.Organization')),
+            ],
+        ),
+        migrations.AddField(
+            model_name='organization',
+            name='galaxy_credentials',
+            field=awx.main.fields.OrderedManyToManyField(blank=True, related_name='organization_galaxy_credentials', through='main.OrganizationGalaxyCredentialMembership', to='main.Credential'),
+        ),
+        migrations.AddField(
+            model_name='credential',
+            name='managed_by_tower',
+            field=models.BooleanField(default=False, editable=False),
+        ),
+        migrations.RunPython(galaxy.migrate_galaxy_settings)
+    ]

awx/main/migrations/0121_delete_toweranalyticsstate.py

@@ -0,0 +1,16 @@
+# Generated by Django 2.2.11 on 2020-07-24 17:41
+
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('main', '0120_galaxy_credentials'),
+    ]
+
+    operations = [
+        migrations.DeleteModel(
+            name='TowerAnalyticsState',
+        ),
+    ]

awx/main/migrations/0122_really_remove_cloudforms_inventory.py

@@ -0,0 +1,13 @@
+from django.db import migrations
+from awx.main.migrations._inventory_source import delete_cloudforms_inv_source
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('main', '0121_delete_toweranalyticsstate'),
+    ]
+
+    operations = [
+        migrations.RunPython(delete_cloudforms_inv_source),
+    ]

awx/main/migrations/0123_drop_hg_support.py

@@ -0,0 +1,23 @@
+from django.db import migrations, models
+from awx.main.migrations._hg_removal import delete_hg_scm
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('main', '0122_really_remove_cloudforms_inventory'),
+    ]
+
+    operations = [
+        migrations.RunPython(delete_hg_scm),
+        migrations.AlterField(
+            model_name='project',
+            name='scm_type',
+            field=models.CharField(blank=True, choices=[('', 'Manual'), ('git', 'Git'), ('svn', 'Subversion'), ('insights', 'Red Hat Insights'), ('archive', 'Remote Archive')], default='', help_text='Specifies the source control system used to store the project.', max_length=8, verbose_name='SCM Type'),
+        ),
+        migrations.AlterField(
+            model_name='projectupdate',
+            name='scm_type',
+            field=models.CharField(blank=True, choices=[('', 'Manual'), ('git', 'Git'), ('svn', 'Subversion'), ('insights', 'Red Hat Insights'), ('archive', 'Remote Archive')], default='', help_text='Specifies the source control system used to store the project.', max_length=8, verbose_name='SCM Type'),
+        ),
+    ]

awx/main/migrations/_galaxy.py

@@ -0,0 +1,125 @@
+# Generated by Django 2.2.11 on 2020-08-04 15:19
+
+import logging
+
+from awx.main.utils.encryption import encrypt_field, decrypt_field
+
+from django.conf import settings
+from django.utils.timezone import now
+
+from awx.main.models import CredentialType as ModernCredentialType
+from awx.main.utils.common import set_current_apps
+
+logger = logging.getLogger('awx.main.migrations')
+
+
+def migrate_galaxy_settings(apps, schema_editor):
+    Organization = apps.get_model('main', 'Organization')
+    if Organization.objects.count() == 0:
+        # nothing to migrate
+        return
+    set_current_apps(apps)
+    ModernCredentialType.setup_tower_managed_defaults()
+    CredentialType = apps.get_model('main', 'CredentialType')
+    Credential = apps.get_model('main', 'Credential')
+    Setting = apps.get_model('conf', 'Setting')
+
+    galaxy_type = CredentialType.objects.get(kind='galaxy')
+    private_galaxy_url = Setting.objects.filter(key='PRIMARY_GALAXY_URL').first()
+
+    # by default, prior versions of AWX/Tower automatically pulled content
+    # from galaxy.ansible.com
+    public_galaxy_enabled = True
+    public_galaxy_setting = Setting.objects.filter(key='PUBLIC_GALAXY_ENABLED').first()
+    if public_galaxy_setting and public_galaxy_setting.value is False:
+        # ...UNLESS this behavior was explicitly disabled via this setting
+        public_galaxy_enabled = False
+
+    public_galaxy_credential = Credential(
+        created=now(),
+        modified=now(),
+        name='Ansible Galaxy',
+        managed_by_tower=True,
+        credential_type=galaxy_type,
+        inputs = {
+            'url': 'https://galaxy.ansible.com/'
+        }
+    )
+    public_galaxy_credential.save()
+
+    for org in Organization.objects.all():
+        if private_galaxy_url and private_galaxy_url.value:
+            # If a setting exists for a private Galaxy URL, make a credential for it
+            username = Setting.objects.filter(key='PRIMARY_GALAXY_USERNAME').first()
+            password = Setting.objects.filter(key='PRIMARY_GALAXY_PASSWORD').first()
+            if (username and username.value) or (password and password.value):
+                logger.error(
+                    f'Specifying HTTP basic auth for the Ansible Galaxy API '
+                    f'({private_galaxy_url.value}) is no longer supported. '
+                    'Please provide an API token instead after your upgrade '
+                    'has completed',
+                )
+            inputs = {
+                'url': private_galaxy_url.value
+            }
+            token = Setting.objects.filter(key='PRIMARY_GALAXY_TOKEN').first()
+            if token and token.value:
+                inputs['token'] = decrypt_field(token, 'value')
+            auth_url = Setting.objects.filter(key='PRIMARY_GALAXY_AUTH_URL').first()
+            if auth_url and auth_url.value:
+                inputs['auth_url'] = auth_url.value
+            name = f'Private Galaxy ({private_galaxy_url.value})'
+            if 'cloud.redhat.com' in inputs['url']:
+                name = f'Ansible Automation Hub ({private_galaxy_url.value})'
+            cred = Credential(
+                created=now(),
+                modified=now(),
+                name=name,
+                organization=org,
+                credential_type=galaxy_type,
+                inputs=inputs
+            )
+            cred.save()
+            if token and token.value:
+                # encrypt based on the primary key from the prior save
+                cred.inputs['token'] = encrypt_field(cred, 'token')
+                cred.save()
+            org.galaxy_credentials.add(cred)
+
+        fallback_servers = getattr(settings, 'FALLBACK_GALAXY_SERVERS', [])
+        for fallback in fallback_servers:
+            url = fallback.get('url', None)
+            auth_url = fallback.get('auth_url', None)
+            username = fallback.get('username', None)
+            password = fallback.get('password', None)
+            token = fallback.get('token', None)
+            if username or password:
+                logger.error(
+                    f'Specifying HTTP basic auth for the Ansible Galaxy API '
+                    f'({url}) is no longer supported. '
+                    'Please provide an API token instead after your upgrade '
+                    'has completed',
+                )
+            inputs = {'url': url}
+            if token:
+                inputs['token'] = token
+            if auth_url:
+                inputs['auth_url'] = auth_url
+            cred = Credential(
+                created=now(),
+                modified=now(),
+                name=f'Ansible Galaxy ({url})',
+                organization=org,
+                credential_type=galaxy_type,
+                inputs=inputs
+            )
+            cred.save()
+            if token:
+                # encrypt based on the primary key from the prior save
+                cred.inputs['token'] = encrypt_field(cred, 'token')
+                cred.save()
+            org.galaxy_credentials.add(cred)
+
+        if public_galaxy_enabled:
+            # If public Galaxy was enabled, associate it to the org
+            org.galaxy_credentials.add(public_galaxy_credential)

awx/main/migrations/_hg_removal.py

@@ -0,0 +1,19 @@
+import logging
+
+from awx.main.utils.common import set_current_apps
+
+logger = logging.getLogger('awx.main.migrations')
+
+
+def delete_hg_scm(apps, schema_editor):
+    set_current_apps(apps)
+    Project = apps.get_model('main', 'Project')
+    ProjectUpdate = apps.get_model('main', 'ProjectUpdate')
+
+    ProjectUpdate.objects.filter(project__scm_type='hg').update(scm_type='')
+    update_ct = Project.objects.filter(scm_type='hg').update(scm_type='')
+
+    if update_ct:
+        logger.warn('Changed {} mercurial projects to manual, deprecation period ended'.format(
+            update_ct
+        ))

awx/main/migrations/_inventory_source.py

@@ -5,6 +5,7 @@ from uuid import uuid4
 from django.utils.encoding import smart_text
 from django.utils.timezone import now
 
+from awx.main.utils.common import set_current_apps
 from awx.main.utils.common import parse_yaml_or_json
 
 logger = logging.getLogger('awx.main.migrations')
@@ -91,43 +92,14 @@ def back_out_new_instance_id(apps, source, new_id):
         ))
 
 
-def create_scm_script_substitute(apps, source):
-    """Only applies for cloudforms in practice, but written generally.
-    Given a source type, this will replace all inventory sources of that type
-    with SCM inventory sources that source the script from Ansible core
-    """
-    # the revision in the Ansible 2.9 stable branch this project will start out as
-    # it can still be updated manually later (but staying within 2.9 branch), if desired
-    ansible_rev = '6f83b9aff42331e15c55a171de0a8b001208c18c'
+def delete_cloudforms_inv_source(apps, schema_editor):
+    set_current_apps(apps)
     InventorySource = apps.get_model('main', 'InventorySource')
-    ContentType = apps.get_model('contenttypes', 'ContentType')
-    Project = apps.get_model('main', 'Project')
-    if not InventorySource.objects.filter(source=source).exists():
-        logger.debug('No sources of type {} to migrate'.format(source))
-        return
-    proj_name = 'Replacement project for {} type sources - {}'.format(source, uuid4())
-    right_now = now()
-    project = Project.objects.create(
-        name=proj_name,
-        created=right_now,
-        modified=right_now,
-        description='Created by migration',
-        polymorphic_ctype=ContentType.objects.get(model='project'),
-        # project-specific fields
-        scm_type='git',
-        scm_url='https://github.com/ansible/ansible.git',
-        scm_branch='stable-2.9',
-        scm_revision=ansible_rev
-    )
-    ct = 0
-    for inv_src in InventorySource.objects.filter(source=source).iterator():
-        inv_src.source = 'scm'
-        inv_src.source_project = project
-        inv_src.source_path = 'contrib/inventory/{}.py'.format(source)
-        inv_src.scm_last_revision = ansible_rev
-        inv_src.save(update_fields=['source', 'source_project', 'source_path', 'scm_last_revision'])
-        logger.debug('Changed inventory source {} to scm type'.format(inv_src.pk))
-        ct += 1
+    InventoryUpdate = apps.get_model('main', 'InventoryUpdate')
+    CredentialType = apps.get_model('main', 'CredentialType')
+    InventoryUpdate.objects.filter(inventory_source__source='cloudforms').delete()
+    InventorySource.objects.filter(source='cloudforms').delete()
+    ct = CredentialType.objects.filter(namespace='cloudforms').first()
     if ct:
-        logger.info('Changed total of {} inventory sources from {} type to scm'.format(ct, source))
-
+        ct.credentials.all().delete()
+        ct.delete()

awx/main/migrations/_inventory_source_vars.py

@@ -0,0 +1,757 @@
+import json
+import re
+import logging
+
+from django.utils.translation import ugettext_lazy as _
+from django.utils.encoding import iri_to_uri
+
+
+FrozenInjectors = dict()
+logger = logging.getLogger('awx.main.migrations')
+
+
+class PluginFileInjector(object):
+    plugin_name = None  # Ansible core name used to reference plugin
+    # every source should have collection, these are for the collection name
+    namespace = None
+    collection = None
+
+    def inventory_as_dict(self, inventory_source, private_data_dir):
+        """Default implementation of inventory plugin file contents.
+        There are some valid cases when all parameters can be obtained from
+        the environment variables, example "plugin: linode" is valid
+        ideally, however, some options should be filled from the inventory source data
+        """
+        if self.plugin_name is None:
+            raise NotImplementedError('At minimum the plugin name is needed for inventory plugin use.')
+        proper_name = f'{self.namespace}.{self.collection}.{self.plugin_name}'
+        return {'plugin': proper_name}
+
+
+class azure_rm(PluginFileInjector):
+    plugin_name = 'azure_rm'
+    namespace = 'azure'
+    collection = 'azcollection'
+
+    def inventory_as_dict(self, inventory_source, private_data_dir):
+        ret = super(azure_rm, self).inventory_as_dict(inventory_source, private_data_dir)
+
+        source_vars = inventory_source.source_vars_dict
+
+        ret['fail_on_template_errors'] = False
+
+        group_by_hostvar = {
+            'location': {'prefix': '', 'separator': '', 'key': 'location'},
+            'tag': {'prefix': '', 'separator': '', 'key': 'tags.keys() | list if tags else []'},
+            # Introduced with https://github.com/ansible/ansible/pull/53046
+            'security_group': {'prefix': '', 'separator': '', 'key': 'security_group'},
+            'resource_group': {'prefix': '', 'separator': '', 'key': 'resource_group'},
+            # Note, os_family was not documented correctly in script, but defaulted to grouping by it
+            'os_family': {'prefix': '', 'separator': '', 'key': 'os_disk.operating_system_type'}
+        }
+        # by default group by everything
+        # always respect user setting, if they gave it
+        group_by = [
+            grouping_name for grouping_name in group_by_hostvar
+            if source_vars.get('group_by_{}'.format(grouping_name), True)
+        ]
+        ret['keyed_groups'] = [group_by_hostvar[grouping_name] for grouping_name in group_by]
+        if 'tag' in group_by:
+            # Nasty syntax to reproduce "key_value" group names in addition to "key"
+            ret['keyed_groups'].append({
+                'prefix': '', 'separator': '',
+                'key': r'dict(tags.keys() | map("regex_replace", "^(.*)$", "\1_") | list | zip(tags.values() | list)) if tags else []'
+            })
+
+        # Compatibility content
+        # TODO: add proper support for instance_filters non-specific to compatibility
+        # TODO: add proper support for group_by non-specific to compatibility
+        # Dashes were not configurable in azure_rm.py script, we do not want unicode, so always use this
+        ret['use_contrib_script_compatible_sanitization'] = True
+        # use same host names as script
+        ret['plain_host_names'] = True
+        # By default the script did not filter hosts
+        ret['default_host_filters'] = []
+        # User-given host filters
+        user_filters = []
+        old_filterables = [
+            ('resource_groups', 'resource_group'),
+            ('tags', 'tags')
+            # locations / location would be an entry
+            # but this would conflict with source_regions
+        ]
+        for key, loc in old_filterables:
+            value = source_vars.get(key, None)
+            if value and isinstance(value, str):
+                # tags can be list of key:value pairs
+                #  e.g. 'Creator:jmarshall, peanutbutter:jelly'
+                # or tags can be a list of keys
+                #  e.g. 'Creator, peanutbutter'
+                if key == "tags":
+                    # grab each key value pair
+                    for kvpair in value.split(','):
+                        # split into key and value
+                        kv = kvpair.split(':')
+                        # filter out any host that does not have key
+                        # in their tags.keys() variable
+                        user_filters.append('"{}" not in tags.keys()'.format(kv[0].strip()))
+                        # if a value is provided, check that the key:value pair matches
+                        if len(kv) > 1:
+                            user_filters.append('tags["{}"] != "{}"'.format(kv[0].strip(), kv[1].strip()))
+                else:
+                    user_filters.append('{} not in {}'.format(
+                        loc, value.split(',')
+                    ))
+        if user_filters:
+            ret.setdefault('exclude_host_filters', [])
+            ret['exclude_host_filters'].extend(user_filters)
+
+        ret['conditional_groups'] = {'azure': True}
+        ret['hostvar_expressions'] = {
+            'provisioning_state': 'provisioning_state | title',
+            'computer_name': 'name',
+            'type': 'resource_type',
+            'private_ip': 'private_ipv4_addresses[0] if private_ipv4_addresses else None',
+            'public_ip': 'public_ipv4_addresses[0] if public_ipv4_addresses else None',
+            'public_ip_name': 'public_ip_name if public_ip_name is defined else None',
+            'public_ip_id': 'public_ip_id if public_ip_id is defined else None',
+            'tags': 'tags if tags else None'
+        }
+        # Special functionality from script
+        if source_vars.get('use_private_ip', False):
+            ret['hostvar_expressions']['ansible_host'] = 'private_ipv4_addresses[0]'
+        # end compatibility content
+
+        if inventory_source.source_regions and 'all' not in inventory_source.source_regions:
+            # initialize a list for this section in inventory file
+            ret.setdefault('exclude_host_filters', [])
+            # make a python list of the regions we will use
+            python_regions = [x.strip() for x in inventory_source.source_regions.split(',')]
+            # convert that list in memory to python syntax in a string
+            # now put that in jinja2 syntax operating on hostvar key "location"
+            # and put that as an entry in the exclusions list
+            ret['exclude_host_filters'].append("location not in {}".format(repr(python_regions)))
+        return ret
+
+
+class ec2(PluginFileInjector):
+    plugin_name = 'aws_ec2'
+    namespace = 'amazon'
+    collection = 'aws'
+
+
+    def _get_ec2_group_by_choices(self):
+        return [
+            ('ami_id', _('Image ID')),
+            ('availability_zone', _('Availability Zone')),
+            ('aws_account', _('Account')),
+            ('instance_id', _('Instance ID')),
+            ('instance_state', _('Instance State')),
+            ('platform', _('Platform')),
+            ('instance_type', _('Instance Type')),
+            ('key_pair', _('Key Name')),
+            ('region', _('Region')),
+            ('security_group', _('Security Group')),
+            ('tag_keys', _('Tags')),
+            ('tag_none', _('Tag None')),
+            ('vpc_id', _('VPC ID')),
+        ]
+
+    def _compat_compose_vars(self):
+        return {
+            # vars that change
+            'ec2_block_devices': (
+                "dict(block_device_mappings | map(attribute='device_name') | list | zip(block_device_mappings "
+                "| map(attribute='ebs.volume_id') | list))"
+            ),
+            'ec2_dns_name': 'public_dns_name',
+            'ec2_group_name': 'placement.group_name',
+            'ec2_instance_profile': 'iam_instance_profile | default("")',
+            'ec2_ip_address': 'public_ip_address',
+            'ec2_kernel': 'kernel_id | default("")',
+            'ec2_monitored':  "monitoring.state in ['enabled', 'pending']",
+            'ec2_monitoring_state': 'monitoring.state',
+            'ec2_placement': 'placement.availability_zone',
+            'ec2_ramdisk': 'ramdisk_id | default("")',
+            'ec2_reason': 'state_transition_reason',
+            'ec2_security_group_ids': "security_groups | map(attribute='group_id') | list |  join(',')",
+            'ec2_security_group_names': "security_groups | map(attribute='group_name') | list |  join(',')",
+            'ec2_tag_Name': 'tags.Name',
+            'ec2_state': 'state.name',
+            'ec2_state_code': 'state.code',
+            'ec2_state_reason': 'state_reason.message if state_reason is defined else ""',
+            'ec2_sourceDestCheck': 'source_dest_check | default(false) | lower | string',  # snake_case syntax intended
+            'ec2_account_id': 'owner_id',
+            # vars that just need ec2_ prefix
+            'ec2_ami_launch_index': 'ami_launch_index | string',
+            'ec2_architecture': 'architecture',
+            'ec2_client_token': 'client_token',
+            'ec2_ebs_optimized': 'ebs_optimized',
+            'ec2_hypervisor': 'hypervisor',
+            'ec2_image_id': 'image_id',
+            'ec2_instance_type': 'instance_type',
+            'ec2_key_name': 'key_name',
+            'ec2_launch_time': r'launch_time | regex_replace(" ", "T") | regex_replace("(\+)(\d\d):(\d)(\d)$", ".\g<2>\g<3>Z")',
+            'ec2_platform': 'platform | default("")',
+            'ec2_private_dns_name': 'private_dns_name',
+            'ec2_private_ip_address': 'private_ip_address',
+            'ec2_public_dns_name': 'public_dns_name',
+            'ec2_region': 'placement.region',
+            'ec2_root_device_name': 'root_device_name',
+            'ec2_root_device_type': 'root_device_type',
+            # many items need blank defaults because the script tended to keep a common schema
+            'ec2_spot_instance_request_id': 'spot_instance_request_id | default("")',
+            'ec2_subnet_id': 'subnet_id | default("")',
+            'ec2_virtualization_type': 'virtualization_type',
+            'ec2_vpc_id': 'vpc_id | default("")',
+            # same as ec2_ip_address, the script provided this
+            'ansible_host': 'public_ip_address',
+            # new with https://github.com/ansible/ansible/pull/53645
+            'ec2_eventsSet': 'events | default("")',
+            'ec2_persistent': 'persistent | default(false)',
+            'ec2_requester_id': 'requester_id | default("")'
+        }
+
+    def inventory_as_dict(self, inventory_source, private_data_dir):
+        ret = super(ec2, self).inventory_as_dict(inventory_source, private_data_dir)
+
+        keyed_groups = []
+        group_by_hostvar = {
+            'ami_id': {'prefix': '', 'separator': '', 'key': 'image_id', 'parent_group': 'images'},
+            # 2 entries for zones for same groups to establish 2 parentage trees
+            'availability_zone': {'prefix': '', 'separator': '', 'key': 'placement.availability_zone', 'parent_group': 'zones'},
+            'aws_account': {'prefix': '', 'separator': '', 'key': 'ec2_account_id', 'parent_group': 'accounts'},  # composed var
+            'instance_id': {'prefix': '', 'separator': '', 'key': 'instance_id', 'parent_group': 'instances'},  # normally turned off
+            'instance_state': {'prefix': 'instance_state', 'key': 'ec2_state', 'parent_group': 'instance_states'},  # composed var
+            # ec2_platform is a composed var, but group names do not match up to hostvar exactly
+            'platform': {'prefix': 'platform', 'key': 'platform | default("undefined")', 'parent_group': 'platforms'},
+            'instance_type': {'prefix': 'type', 'key': 'instance_type', 'parent_group': 'types'},
+            'key_pair': {'prefix': 'key', 'key': 'key_name', 'parent_group': 'keys'},
+            'region': {'prefix': '', 'separator': '', 'key': 'placement.region', 'parent_group': 'regions'},
+            # Security requires some ninja jinja2 syntax, credit to s-hertel
+            'security_group': {'prefix': 'security_group', 'key': 'security_groups | map(attribute="group_name")', 'parent_group': 'security_groups'},
+            # tags cannot be parented in exactly the same way as the script due to
+            # https://github.com/ansible/ansible/pull/53812
+            'tag_keys': [
+                {'prefix': 'tag', 'key': 'tags', 'parent_group': 'tags'},
+                {'prefix': 'tag', 'key': 'tags.keys()', 'parent_group': 'tags'}
+            ],
+            # 'tag_none': None,  # grouping by no tags isn't a different thing with plugin
+            # naming is redundant, like vpc_id_vpc_8c412cea, but intended
+            'vpc_id': {'prefix': 'vpc_id', 'key': 'vpc_id', 'parent_group': 'vpcs'},
+        }
+        # -- same-ish as script here --
+        group_by = [x.strip().lower() for x in inventory_source.group_by.split(',') if x.strip()]
+        for choice in self._get_ec2_group_by_choices():
+            value = bool((group_by and choice[0] in group_by) or (not group_by and choice[0] != 'instance_id'))
+            # -- end sameness to script --
+            if value:
+                this_keyed_group = group_by_hostvar.get(choice[0], None)
+                # If a keyed group syntax does not exist, there is nothing we can do to get this group
+                if this_keyed_group is not None:
+                    if isinstance(this_keyed_group, list):
+                        keyed_groups.extend(this_keyed_group)
+                    else:
+                        keyed_groups.append(this_keyed_group)
+        # special case, this parentage is only added if both zones and regions are present
+        if not group_by or ('region' in group_by and 'availability_zone' in group_by):
+            keyed_groups.append({'prefix': '', 'separator': '', 'key': 'placement.availability_zone', 'parent_group': '{{ placement.region }}'})
+
+        source_vars = inventory_source.source_vars_dict
+        # This is a setting from the script, hopefully no one used it
+        # if true, it replaces dashes, but not in region / loc names
+        replace_dash = bool(source_vars.get('replace_dash_in_groups', True))
+        # Compatibility content
+        legacy_regex = {
+            True: r"[^A-Za-z0-9\_]",
+            False: r"[^A-Za-z0-9\_\-]"  # do not replace dash, dash is allowed
+        }[replace_dash]
+        list_replacer = 'map("regex_replace", "{rx}", "_") | list'.format(rx=legacy_regex)
+        # this option, a plugin option, will allow dashes, but not unicode
+        # when set to False, unicode will be allowed, but it was not allowed by script
+        # thus, we always have to use this option, and always use our custom regex
+        ret['use_contrib_script_compatible_sanitization'] = True
+        for grouping_data in keyed_groups:
+            if grouping_data['key'] in ('placement.region', 'placement.availability_zone'):
+                # us-east-2 is always us-east-2 according to ec2.py
+                # no sanitization in region-ish groups for the script standards, ever ever
+                continue
+            if grouping_data['key'] == 'tags':
+                # dict jinja2 transformation
+                grouping_data['key'] = 'dict(tags.keys() | {replacer} | zip(tags.values() | {replacer}))'.format(
+                    replacer=list_replacer
+                )
+            elif grouping_data['key'] == 'tags.keys()' or grouping_data['prefix'] == 'security_group':
+                # list jinja2 transformation
+                grouping_data['key'] += ' | {replacer}'.format(replacer=list_replacer)
+            else:
+                # string transformation
+                grouping_data['key'] += ' | regex_replace("{rx}", "_")'.format(rx=legacy_regex)
+        # end compatibility content
+
+        if source_vars.get('iam_role_arn', None):
+            ret['iam_role_arn'] = source_vars['iam_role_arn']
+
+        # This was an allowed ec2.ini option, also plugin option, so pass through
+        if source_vars.get('boto_profile', None):
+            ret['boto_profile'] = source_vars['boto_profile']
+
+        elif not replace_dash:
+            # Using the plugin, but still want dashes allowed
+            ret['use_contrib_script_compatible_sanitization'] = True
+
+        if source_vars.get('nested_groups') is False:
+            for this_keyed_group in keyed_groups:
+                this_keyed_group.pop('parent_group', None)
+
+        if keyed_groups:
+            ret['keyed_groups'] = keyed_groups
+
+        # Instance ID not part of compat vars, because of settings.EC2_INSTANCE_ID_VAR
+        compose_dict = {'ec2_id': 'instance_id'}
+        inst_filters = {}
+
+        # Compatibility content
+        compose_dict.update(self._compat_compose_vars())
+        # plugin provides "aws_ec2", but not this which the script gave
+        ret['groups'] = {'ec2': True}
+        if source_vars.get('hostname_variable') is not None:
+            hnames = []
+            for expr in source_vars.get('hostname_variable').split(','):
+                if expr == 'public_dns_name':
+                    hnames.append('dns-name')
+                elif not expr.startswith('tag:') and '_' in expr:
+                    hnames.append(expr.replace('_', '-'))
+                else:
+                    hnames.append(expr)
+            ret['hostnames'] = hnames
+        else:
+            # public_ip as hostname is non-default plugin behavior, script behavior
+            ret['hostnames'] = [
+                'network-interface.addresses.association.public-ip',
+                'dns-name',
+                'private-dns-name'
+            ]
+        # The script returned only running state by default, the plugin does not
+        # https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html#options
+        # options: pending | running | shutting-down | terminated | stopping | stopped
+        inst_filters['instance-state-name'] = ['running']
+        # end compatibility content
+
+        if source_vars.get('destination_variable') or source_vars.get('vpc_destination_variable'):
+            for fd in ('destination_variable', 'vpc_destination_variable'):
+                if source_vars.get(fd):
+                    compose_dict['ansible_host'] = source_vars.get(fd)
+                    break
+
+        if compose_dict:
+            ret['compose'] = compose_dict
+
+        if inventory_source.instance_filters:
+            # logic used to live in ec2.py, now it belongs to us. Yay more code?
+            filter_sets = [f for f in inventory_source.instance_filters.split(',') if f]
+
+            for instance_filter in filter_sets:
+                # AND logic not supported, unclear how to...
+                instance_filter = instance_filter.strip()
+                if not instance_filter or '=' not in instance_filter:
+                    continue
+                filter_key, filter_value = [x.strip() for x in instance_filter.split('=', 1)]
+                if not filter_key:
+                    continue
+                inst_filters[filter_key] = filter_value
+
+        if inst_filters:
+            ret['filters'] = inst_filters
+
+        if inventory_source.source_regions and 'all' not in inventory_source.source_regions:
+            ret['regions'] = inventory_source.source_regions.split(',')
+
+        return ret
+
+
+class gce(PluginFileInjector):
+    plugin_name = 'gcp_compute'
+    namespace = 'google'
+    collection = 'cloud'
+
+    def _compat_compose_vars(self):
+        # missing: gce_image, gce_uuid
+        # https://github.com/ansible/ansible/issues/51884
+        return {
+            'gce_description': 'description if description else None',
+            'gce_machine_type': 'machineType',
+            'gce_name': 'name',
+            'gce_network': 'networkInterfaces[0].network.name',
+            'gce_private_ip': 'networkInterfaces[0].networkIP',
+            'gce_public_ip': 'networkInterfaces[0].accessConfigs[0].natIP | default(None)',
+            'gce_status': 'status',
+            'gce_subnetwork': 'networkInterfaces[0].subnetwork.name',
+            'gce_tags': 'tags.get("items", [])',
+            'gce_zone': 'zone',
+            'gce_metadata': 'metadata.get("items", []) | items2dict(key_name="key", value_name="value")',
+            # NOTE: image hostvar is enabled via retrieve_image_info option
+            'gce_image': 'image',
+            # We need this as long as hostnames is non-default, otherwise hosts
+            # will not be addressed correctly, was returned in script
+            'ansible_ssh_host': 'networkInterfaces[0].accessConfigs[0].natIP | default(networkInterfaces[0].networkIP)'
+        }
+
+    def inventory_as_dict(self, inventory_source, private_data_dir):
+        ret = super(gce, self).inventory_as_dict(inventory_source, private_data_dir)
+
+        # auth related items
+        ret['auth_kind'] = "serviceaccount"
+
+        filters = []
+        # TODO: implement gce group_by options
+        # gce never processed the group_by field, if it had, we would selectively
+        # apply those options here, but it did not, so all groups are added here
+        keyed_groups = [
+            # the jinja2 syntax is duplicated with compose
+            # https://github.com/ansible/ansible/issues/51883
+            {'prefix': 'network', 'key': 'gce_subnetwork'},  # composed var
+            {'prefix': '', 'separator': '', 'key': 'gce_private_ip'},  # composed var
+            {'prefix': '', 'separator': '', 'key': 'gce_public_ip'},  # composed var
+            {'prefix': '', 'separator': '', 'key': 'machineType'},
+            {'prefix': '', 'separator': '', 'key': 'zone'},
+            {'prefix': 'tag', 'key': 'gce_tags'},  # composed var
+            {'prefix': 'status', 'key': 'status | lower'},
+            # NOTE: image hostvar is enabled via retrieve_image_info option
+            {'prefix': '', 'separator': '', 'key': 'image'},
+        ]
+        # This will be used as the gce instance_id, must be universal, non-compat
+        compose_dict = {'gce_id': 'id'}
+
+        # Compatibility content
+        # TODO: proper group_by and instance_filters support, irrelevant of compat mode
+        # The gce.py script never sanitized any names in any way
+        ret['use_contrib_script_compatible_sanitization'] = True
+        # Perform extra API query to get the image hostvar
+        ret['retrieve_image_info'] = True
+        # Add in old hostvars aliases
+        compose_dict.update(self._compat_compose_vars())
+        # Non-default names to match script
+        ret['hostnames'] = ['name', 'public_ip', 'private_ip']
+        # end compatibility content
+
+        if keyed_groups:
+            ret['keyed_groups'] = keyed_groups
+        if filters:
+            ret['filters'] = filters
+        if compose_dict:
+            ret['compose'] = compose_dict
+        if inventory_source.source_regions and 'all' not in inventory_source.source_regions:
+            ret['zones'] = inventory_source.source_regions.split(',')
+        return ret
+
+
+class vmware(PluginFileInjector):
+    plugin_name = 'vmware_vm_inventory'
+    namespace = 'community'
+    collection = 'vmware'
+
+    def inventory_as_dict(self, inventory_source, private_data_dir):
+        ret = super(vmware, self).inventory_as_dict(inventory_source, private_data_dir)
+        ret['strict'] = False
+        # Documentation of props, see
+        # https://github.com/ansible/ansible/blob/devel/docs/docsite/rst/scenario_guides/vmware_scenarios/vmware_inventory_vm_attributes.rst
+        UPPERCASE_PROPS = [
+            "availableField",
+            "configIssue",
+            "configStatus",
+            "customValue",  # optional
+            "datastore",
+            "effectiveRole",
+            "guestHeartbeatStatus",  # optional
+            "layout",  # optional
+            "layoutEx",  # optional
+            "name",
+            "network",
+            "overallStatus",
+            "parentVApp",  # optional
+            "permission",
+            "recentTask",
+            "resourcePool",
+            "rootSnapshot",
+            "snapshot",  # optional
+            "triggeredAlarmState",
+            "value"
+        ]
+        NESTED_PROPS = [
+            "capability",
+            "config",
+            "guest",
+            "runtime",
+            "storage",
+            "summary",  # repeat of other properties
+        ]
+        ret['properties'] = UPPERCASE_PROPS + NESTED_PROPS
+        ret['compose'] = {'ansible_host': 'guest.ipAddress'}  # default value
+        ret['compose']['ansible_ssh_host'] = ret['compose']['ansible_host']
+        # the ansible_uuid was unique every host, every import, from the script
+        ret['compose']['ansible_uuid'] = '99999999 | random | to_uuid'
+        for prop in UPPERCASE_PROPS:
+            if prop == prop.lower():
+                continue
+            ret['compose'][prop.lower()] = prop
+        ret['with_nested_properties'] = True
+        # ret['property_name_format'] = 'lower_case'  # only dacrystal/topic/vmware-inventory-plugin-property-format
+
+        # process custom options
+        vmware_opts = dict(inventory_source.source_vars_dict.items())
+        if inventory_source.instance_filters:
+            vmware_opts.setdefault('host_filters', inventory_source.instance_filters)
+        if inventory_source.group_by:
+            vmware_opts.setdefault('groupby_patterns', inventory_source.group_by)
+
+        alias_pattern = vmware_opts.get('alias_pattern')
+        if alias_pattern:
+            ret.setdefault('hostnames', [])
+            for alias in alias_pattern.split(','):  # make best effort
+                striped_alias = alias.replace('{', '').replace('}', '').strip()  # make best effort
+                if not striped_alias:
+                    continue
+                ret['hostnames'].append(striped_alias)
+
+        host_pattern = vmware_opts.get('host_pattern')  # not working in script
+        if host_pattern:
+            stripped_hp = host_pattern.replace('{', '').replace('}', '').strip()  # make best effort
+            ret['compose']['ansible_host'] = stripped_hp
+            ret['compose']['ansible_ssh_host'] = stripped_hp
+
+        host_filters = vmware_opts.get('host_filters')
+        if host_filters:
+            ret.setdefault('filters', [])
+            for hf in host_filters.split(','):
+                striped_hf = hf.replace('{', '').replace('}', '').strip()  # make best effort
+                if not striped_hf:
+                    continue
+                ret['filters'].append(striped_hf)
+        else:
+            # default behavior filters by power state
+            ret['filters'] = ['runtime.powerState == "poweredOn"']
+
+        groupby_patterns = vmware_opts.get('groupby_patterns')
+        ret.setdefault('keyed_groups', [])
+        if groupby_patterns:
+            for pattern in groupby_patterns.split(','):
+                stripped_pattern = pattern.replace('{', '').replace('}', '').strip()  # make best effort
+                ret['keyed_groups'].append({
+                    'prefix': '', 'separator': '',
+                    'key': stripped_pattern
+                })
+        else:
+            # default groups from script
+            for entry in ('config.guestId', '"templates" if config.template else "guests"'):
+                ret['keyed_groups'].append({
+                    'prefix': '', 'separator': '',
+                    'key': entry
+                })
+
+        return ret
+
+
+class openstack(PluginFileInjector):
+    plugin_name = 'openstack'
+    namespace = 'openstack'
+    collection = 'cloud'
+
+    def inventory_as_dict(self, inventory_source, private_data_dir):
+        def use_host_name_for_name(a_bool_maybe):
+            if not isinstance(a_bool_maybe, bool):
+                # Could be specified by user via "host" or "uuid"
+                return a_bool_maybe
+            elif a_bool_maybe:
+                return 'name'  # plugin default
+            else:
+                return 'uuid'
+
+        ret = super(openstack, self).inventory_as_dict(inventory_source, private_data_dir)
+        ret['fail_on_errors'] = True
+        ret['expand_hostvars'] = True
+        ret['inventory_hostname'] = use_host_name_for_name(False)
+        # Note: mucking with defaults will break import integrity
+        # For the plugin, we need to use the same defaults as the old script
+        # or else imports will conflict. To find script defaults you have
+        # to read source code of the script.
+        #
+        # Script Defaults                           Plugin Defaults
+        # 'use_hostnames': False,                   'name' (True)
+        # 'expand_hostvars': True,                  'no' (False)
+        # 'fail_on_errors': True,                   'no' (False)
+        #
+        # These are, yet again, different from ansible_variables in script logic
+        # but those are applied inconsistently
+        source_vars = inventory_source.source_vars_dict
+        for var_name in ['expand_hostvars', 'fail_on_errors']:
+            if var_name in source_vars:
+                ret[var_name] = source_vars[var_name]
+        if 'use_hostnames' in source_vars:
+            ret['inventory_hostname'] = use_host_name_for_name(source_vars['use_hostnames'])
+        return ret
+
+
+class rhv(PluginFileInjector):
+    """ovirt uses the custom credential templating, and that is all
+    """
+    plugin_name = 'ovirt'
+    initial_version = '2.9'
+    namespace = 'ovirt'
+    collection = 'ovirt'
+
+    def inventory_as_dict(self, inventory_source, private_data_dir):
+        ret = super(rhv, self).inventory_as_dict(inventory_source, private_data_dir)
+        ret['ovirt_insecure'] = False  # Default changed from script
+        # TODO: process strict option upstream
+        ret['compose'] = {
+            'ansible_host': '(devices.values() | list)[0][0] if devices else None'
+        }
+        ret['keyed_groups'] = []
+        for key in ('cluster', 'status'):
+            ret['keyed_groups'].append({'prefix': key, 'separator': '_', 'key': key})
+        ret['keyed_groups'].append({'prefix': 'tag', 'separator': '_', 'key': 'tags'})
+        ret['ovirt_hostname_preference'] = ['name', 'fqdn']
+        source_vars = inventory_source.source_vars_dict
+        for key, value in source_vars.items():
+            if key == 'plugin':
+                continue
+            ret[key] = value
+        return ret
+
+
+class satellite6(PluginFileInjector):
+    plugin_name = 'foreman'
+    namespace = 'theforeman'
+    collection = 'foreman'
+
+    def inventory_as_dict(self, inventory_source, private_data_dir):
+        ret = super(satellite6, self).inventory_as_dict(inventory_source, private_data_dir)
+        ret['validate_certs'] = False
+
+        group_patterns = '[]'
+        group_prefix = 'foreman_'
+        want_hostcollections = False
+        want_ansible_ssh_host = False
+        want_facts = True
+
+        foreman_opts = inventory_source.source_vars_dict.copy()
+        for k, v in foreman_opts.items():
+            if k == 'satellite6_group_patterns' and isinstance(v, str):
+                group_patterns = v
+            elif k == 'satellite6_group_prefix' and isinstance(v, str):
+                group_prefix = v
+            elif k == 'satellite6_want_hostcollections' and isinstance(v, bool):
+                want_hostcollections = v
+            elif k == 'satellite6_want_ansible_ssh_host' and isinstance(v, bool):
+                want_ansible_ssh_host = v
+            elif k == 'satellite6_want_facts' and isinstance(v, bool):
+                want_facts = v
+            # add backwards support for ssl_verify
+            # plugin uses new option, validate_certs, instead
+            elif k == 'ssl_verify' and isinstance(v, bool):
+                ret['validate_certs'] = v
+            else:
+                ret[k] = str(v)
+
+        # Compatibility content
+        group_by_hostvar = {
+            "environment":           {"prefix": "{}environment_".format(group_prefix),
+                                      "separator": "",
+                                      "key": "foreman['environment_name'] | lower | regex_replace(' ', '') | "
+                                             "regex_replace('[^A-Za-z0-9_]', '_') | regex_replace('none', '')"},
+            "location":              {"prefix": "{}location_".format(group_prefix),
+                                      "separator": "",
+                                      "key": "foreman['location_name'] | lower | regex_replace(' ', '') | regex_replace('[^A-Za-z0-9_]', '_')"},
+            "organization":          {"prefix": "{}organization_".format(group_prefix),
+                                      "separator": "",
+                                      "key": "foreman['organization_name'] | lower | regex_replace(' ', '') | regex_replace('[^A-Za-z0-9_]', '_')"},
+            "lifecycle_environment": {"prefix": "{}lifecycle_environment_".format(group_prefix),
+                                      "separator": "",
+                                      "key": "foreman['content_facet_attributes']['lifecycle_environment_name'] | "
+                                             "lower | regex_replace(' ', '') | regex_replace('[^A-Za-z0-9_]', '_')"},
+            "content_view":          {"prefix": "{}content_view_".format(group_prefix),
+                                      "separator": "",
+                                      "key": "foreman['content_facet_attributes']['content_view_name'] | "
+                                             "lower | regex_replace(' ', '') | regex_replace('[^A-Za-z0-9_]', '_')"}
+        }
+
+        ret['legacy_hostvars'] = True  # convert hostvar structure to the form used by the script
+        ret['want_params'] = True
+        ret['group_prefix'] = group_prefix
+        ret['want_hostcollections'] = want_hostcollections
+        ret['want_facts'] = want_facts
+
+        if want_ansible_ssh_host:
+            ret['compose'] = {'ansible_ssh_host': "foreman['ip6'] | default(foreman['ip'], true)"}
+        ret['keyed_groups'] = [group_by_hostvar[grouping_name] for grouping_name in group_by_hostvar]
+
+        def form_keyed_group(group_pattern):
+            """
+            Converts foreman group_pattern to
+            inventory plugin keyed_group
+
+            e.g. {app_param}-{tier_param}-{dc_param}
+                 becomes
+                 "%s-%s-%s" | format(app_param, tier_param, dc_param)
+            """
+            if type(group_pattern) is not str:
+                return None
+            params = re.findall('{[^}]*}', group_pattern)
+            if len(params) == 0:
+                return None
+
+            param_names = []
+            for p in params:
+                param_names.append(p[1:-1].strip())  # strip braces and space
+
+            # form keyed_group key by
+            # replacing curly braces with '%s'
+            # (for use with jinja's format filter)
+            key = group_pattern
+            for p in params:
+                key = key.replace(p, '%s', 1)
+
+            # apply jinja filter to key
+            key = '"{}" | format({})'.format(key, ', '.join(param_names))
+
+            keyed_group = {'key': key,
+                           'separator': ''}
+            return keyed_group
+
+        try:
+            group_patterns = json.loads(group_patterns)
+
+            if type(group_patterns) is list:
+                for group_pattern in group_patterns:
+                    keyed_group = form_keyed_group(group_pattern)
+                    if keyed_group:
+                        ret['keyed_groups'].append(keyed_group)
+        except json.JSONDecodeError:
+            logger.warning('Could not parse group_patterns. Expected JSON-formatted string, found: {}'
+                           .format(group_patterns))
+
+        return ret
+
+
+class tower(PluginFileInjector):
+    plugin_name = 'tower'
+    namespace = 'awx'
+    collection = 'awx'
+
+    def inventory_as_dict(self, inventory_source, private_data_dir):
+        ret = super(tower, self).inventory_as_dict(inventory_source, private_data_dir)
+        # Credentials injected as env vars, same as script
+        try:
+            # plugin can take an actual int type
+            identifier = int(inventory_source.instance_filters)
+        except ValueError:
+            # inventory_id could be a named URL
+            identifier = iri_to_uri(inventory_source.instance_filters)
+        ret['inventory_id'] = identifier
+        ret['include_metadata'] = True  # used for license check
+        return ret
+
+
+for cls in PluginFileInjector.__subclasses__():
+    FrozenInjectors[cls.__name__] = cls

awx/main/models/credential/__init__.py

@@ -96,6 +96,10 @@ class Credential(PasswordFieldsModel, CommonModelNameNotUnique, ResourceMixin):
         help_text=_('Specify the type of credential you want to create. Refer '
                     'to the Ansible Tower documentation for details on each type.')
     )
+    managed_by_tower = models.BooleanField(
+        default=False,
+        editable=False
+    )
     organization = models.ForeignKey(
         'Organization',
         null=True,
@@ -331,6 +335,7 @@ class CredentialType(CommonModelNameNotUnique):
         ('insights', _('Insights')),
         ('external', _('External')),
         ('kubernetes', _('Kubernetes')),
+        ('galaxy', _('Galaxy/Automation Hub')),
     )
 
     kind = models.CharField(
@@ -876,33 +881,6 @@ ManagedCredentialType(
     }
 )
 
-ManagedCredentialType(
-    namespace='cloudforms',
-    kind='cloud',
-    name=ugettext_noop('Red Hat CloudForms'),
-    managed_by_tower=True,
-    inputs={
-        'fields': [{
-            'id': 'host',
-            'label': ugettext_noop('CloudForms URL'),
-            'type': 'string',
-            'help_text': ugettext_noop('Enter the URL for the virtual machine that '
-                                       'corresponds to your CloudForms instance. '
-                                       'For example, https://cloudforms.example.org')
-        }, {
-            'id': 'username',
-            'label': ugettext_noop('Username'),
-            'type': 'string'
-        }, {
-            'id': 'password',
-            'label': ugettext_noop('Password'),
-            'type': 'string',
-            'secret': True,
-        }],
-        'required': ['host', 'username', 'password'],
-    }
-)
-
 ManagedCredentialType(
     namespace='gce',
     kind='cloud',
@@ -1173,6 +1151,38 @@ ManagedCredentialType(
 )
 
 
+ManagedCredentialType(
+    namespace='galaxy_api_token',
+    kind='galaxy',
+    name=ugettext_noop('Ansible Galaxy/Automation Hub API Token'),
+    inputs={
+        'fields': [{
+            'id': 'url',
+            'label': ugettext_noop('Galaxy Server URL'),
+            'type': 'string',
+            'help_text': ugettext_noop('The URL of the Galaxy instance to connect to.')
+        },{
+            'id': 'auth_url',
+            'label': ugettext_noop('Auth Server URL'),
+            'type': 'string',
+            'help_text': ugettext_noop(
+                'The URL of a Keycloak server token_endpoint, if using '
+                'SSO auth.'
+            )
+        },{
+            'id': 'token',
+            'label': ugettext_noop('API Token'),
+            'type': 'string',
+            'secret': True,
+            'help_text': ugettext_noop(
+                'A token to use for authentication against the Galaxy instance.'
+            )
+        }],
+        'required': ['url'],
+    }
+)
+
+
 class CredentialInputSource(PrimordialModel):
 
     class Meta:

awx/main/models/events.py

@@ -4,6 +4,8 @@ import datetime
 import logging
 from collections import defaultdict
 
+from django.conf import settings
+from django.core.exceptions import ObjectDoesNotExist
 from django.db import models, DatabaseError, connection
 from django.utils.dateparse import parse_datetime
 from django.utils.text import Truncator
@@ -57,7 +59,18 @@ def create_host_status_counts(event_data):
     return dict(host_status_counts)
 
 
+MINIMAL_EVENTS = set([
+    'playbook_on_play_start', 'playbook_on_task_start',
+    'playbook_on_stats', 'EOF'
+])
+
+
 def emit_event_detail(event):
+    if (
+        settings.UI_LIVE_UPDATES_ENABLED is False and
+        event.event not in MINIMAL_EVENTS
+    ):
+        return
     cls = event.__class__
     relation = {
         JobEvent: 'job_id',
@@ -337,41 +350,47 @@ class BasePlaybookEvent(CreatedModifiedModel):
                 pass
 
             if isinstance(self, JobEvent):
-                hostnames = self._hostnames()
-                self._update_host_summary_from_stats(set(hostnames))
-                if self.job.inventory:
-                    try:
-                        self.job.inventory.update_computed_fields()
-                    except DatabaseError:
-                        logger.exception('Computed fields database error saving event {}'.format(self.pk))
+                try:
+                    job = self.job
+                except ObjectDoesNotExist:
+                    job = None
+                if job:
+                    hostnames = self._hostnames()
+                    self._update_host_summary_from_stats(set(hostnames))
+                    if job.inventory:
+                        try:
+                            job.inventory.update_computed_fields()
+                        except DatabaseError:
+                            logger.exception('Computed fields database error saving event {}'.format(self.pk))
 
-                # find parent links and progagate changed=T and failed=T
-                changed = self.job.job_events.filter(changed=True).exclude(parent_uuid=None).only('parent_uuid').values_list('parent_uuid', flat=True).distinct()  # noqa
-                failed = self.job.job_events.filter(failed=True).exclude(parent_uuid=None).only('parent_uuid').values_list('parent_uuid', flat=True).distinct()  # noqa
+                    # find parent links and progagate changed=T and failed=T
+                    changed = job.job_events.filter(changed=True).exclude(parent_uuid=None).only('parent_uuid').values_list('parent_uuid', flat=True).distinct()  # noqa
+                    failed = job.job_events.filter(failed=True).exclude(parent_uuid=None).only('parent_uuid').values_list('parent_uuid', flat=True).distinct()  # noqa
 
-                JobEvent.objects.filter(
-                    job_id=self.job_id, uuid__in=changed
-                ).update(changed=True)
-                JobEvent.objects.filter(
-                    job_id=self.job_id, uuid__in=failed
-                ).update(failed=True)
+                    JobEvent.objects.filter(
+                        job_id=self.job_id, uuid__in=changed
+                    ).update(changed=True)
+                    JobEvent.objects.filter(
+                        job_id=self.job_id, uuid__in=failed
+                    ).update(failed=True)
 
-                # send success/failure notifications when we've finished handling the playbook_on_stats event
-                from awx.main.tasks import handle_success_and_failure_notifications  # circular import
+                    # send success/failure notifications when we've finished handling the playbook_on_stats event
+                    from awx.main.tasks import handle_success_and_failure_notifications  # circular import
 
-                def _send_notifications():
-                    handle_success_and_failure_notifications.apply_async([self.job.id])
-                connection.on_commit(_send_notifications)
+                    def _send_notifications():
+                        handle_success_and_failure_notifications.apply_async([job.id])
+                    connection.on_commit(_send_notifications)
 
 
         for field in ('playbook', 'play', 'task', 'role'):
             value = force_text(event_data.get(field, '')).strip()
             if value != getattr(self, field):
                 setattr(self, field, value)
-        analytics_logger.info(
-            'Event data saved.',
-            extra=dict(python_objects=dict(job_event=self))
-        )
+        if settings.LOG_AGGREGATOR_ENABLED:
+            analytics_logger.info(
+                'Event data saved.',
+                extra=dict(python_objects=dict(job_event=self))
+            )
 
     @classmethod
     def create_from_data(cls, **kwargs):
@@ -484,7 +503,11 @@ class JobEvent(BasePlaybookEvent):
 
     def _update_host_summary_from_stats(self, hostnames):
         with ignore_inventory_computed_fields():
-            if not self.job or not self.job.inventory:
+            try:
+                if not self.job or not self.job.inventory:
+                    logger.info('Event {} missing job or inventory, host summaries not updated'.format(self.pk))
+                    return
+            except ObjectDoesNotExist:
                 logger.info('Event {} missing job or inventory, host summaries not updated'.format(self.pk))
                 return
             job = self.job
@@ -520,13 +543,21 @@ class JobEvent(BasePlaybookEvent):
                 (summary['host_id'], summary['id'])
                 for summary in JobHostSummary.objects.filter(job_id=job.id).values('id', 'host_id')
             )
+            updated_hosts = set()
             for h in all_hosts:
                 # if the hostname *shows up* in the playbook_on_stats event
                 if h.name in hostnames:
                     h.last_job_id = job.id
+                    updated_hosts.add(h)
                 if h.id in host_mapping:
                     h.last_job_host_summary_id = host_mapping[h.id]
-            Host.objects.bulk_update(all_hosts, ['last_job_id', 'last_job_host_summary_id'])
+                    updated_hosts.add(h)
+
+            Host.objects.bulk_update(
+                list(updated_hosts),
+                ['last_job_id', 'last_job_host_summary_id'],
+                batch_size=100
+            )
 
 
     @property

awx/main/models/ha.py

@@ -12,6 +12,7 @@ from django.utils.translation import ugettext_lazy as _
 from django.conf import settings
 from django.utils.timezone import now, timedelta
 
+import redis
 from solo.models import SingletonModel
 
 from awx import __version__ as awx_application_version
@@ -23,7 +24,7 @@ from awx.main.models.unified_jobs import UnifiedJob
 from awx.main.utils import get_cpu_capacity, get_mem_capacity, get_system_task_capacity
 from awx.main.models.mixins import RelatedJobsMixin
 
-__all__ = ('Instance', 'InstanceGroup', 'TowerScheduleState', 'TowerAnalyticsState')
+__all__ = ('Instance', 'InstanceGroup', 'TowerScheduleState')
 
 
 class HasPolicyEditsMixin(HasEditsMixin):
@@ -152,6 +153,14 @@ class Instance(HasPolicyEditsMixin, BaseModel):
             self.capacity = get_system_task_capacity(self.capacity_adjustment)
         else:
             self.capacity = 0
+
+        try:
+            # if redis is down for some reason, that means we can't persist
+            # playbook event data; we should consider this a zero capacity event
+            redis.Redis.from_url(settings.BROKER_URL).ping()
+        except redis.ConnectionError:
+            self.capacity = 0
+
         self.cpu = cpu[0]
         self.memory = mem[0]
         self.cpu_capacity = cpu[1]
@@ -252,18 +261,20 @@ class InstanceGroup(HasPolicyEditsMixin, BaseModel, RelatedJobsMixin):
         app_label = 'main'
 
 
-    def fit_task_to_most_remaining_capacity_instance(self, task):
+    @staticmethod
+    def fit_task_to_most_remaining_capacity_instance(task, instances):
         instance_most_capacity = None
-        for i in self.instances.filter(capacity__gt=0, enabled=True).order_by('hostname'):
+        for i in instances:
             if i.remaining_capacity >= task.task_impact and \
                     (instance_most_capacity is None or
                      i.remaining_capacity > instance_most_capacity.remaining_capacity):
                 instance_most_capacity = i
         return instance_most_capacity
 
-    def find_largest_idle_instance(self):
+    @staticmethod
+    def find_largest_idle_instance(instances):
         largest_instance = None
-        for i in self.instances.filter(capacity__gt=0, enabled=True).order_by('hostname'):
+        for i in instances:
             if i.jobs_running == 0:
                 if largest_instance is None:
                     largest_instance = i
@@ -287,10 +298,6 @@ class TowerScheduleState(SingletonModel):
     schedule_last_run = models.DateTimeField(auto_now_add=True)
 
 
-class TowerAnalyticsState(SingletonModel):
-    last_run = models.DateTimeField(auto_now_add=True)
-
-
 def schedule_policy_task():
     from awx.main.tasks import apply_cluster_membership_policies
     connection.on_commit(lambda: apply_cluster_membership_policies.apply_async())

awx/main/models/jobs.py

@@ -798,6 +798,10 @@ class Job(UnifiedJob, JobOptions, SurveyJobMixin, JobNotificationMixin, TaskMana
         if self.project:
             for name in ('awx', 'tower'):
                 r['{}_project_revision'.format(name)] = self.project.scm_revision
+                r['{}_project_scm_branch'.format(name)] = self.project.scm_branch
+        if self.scm_branch:
+            for name in ('awx', 'tower'):
+                r['{}_job_scm_branch'.format(name)] = self.scm_branch
         if self.job_template:
             for name in ('awx', 'tower'):
                 r['{}_job_template_id'.format(name)] = self.job_template.pk

awx/main/models/notifications.py

@@ -393,7 +393,11 @@ class JobNotificationMixin(object):
             'job': job_context,
             'job_friendly_name': self.get_notification_friendly_name(),
             'url': self.get_ui_url(),
-            'job_metadata': json.dumps(self.notification_data(), indent=4)
+            'job_metadata': json.dumps(
+                self.notification_data(),
+                ensure_ascii=False,
+                indent=4
+            )
         }
 
         def build_context(node, fields, allowed_fields):

awx/main/models/organization.py

@@ -45,6 +45,12 @@ class Organization(CommonModel, NotificationFieldsModel, ResourceMixin, CustomVi
         blank=True,
         through='OrganizationInstanceGroupMembership'
     )
+    galaxy_credentials = OrderedManyToManyField(
+        'Credential',
+        blank=True,
+        through='OrganizationGalaxyCredentialMembership',
+        related_name='%(class)s_galaxy_credentials'
+    )
     max_hosts = models.PositiveIntegerField(
         blank=True,
         default=0,
@@ -108,6 +114,23 @@ class Organization(CommonModel, NotificationFieldsModel, ResourceMixin, CustomVi
         return UnifiedJob.objects.non_polymorphic().filter(organization=self)
 
 
+class OrganizationGalaxyCredentialMembership(models.Model):
+
+    organization = models.ForeignKey(
+        'Organization',
+        on_delete=models.CASCADE
+    )
+    credential = models.ForeignKey(
+        'Credential',
+        on_delete=models.CASCADE
+    )
+    position = models.PositiveIntegerField(
+        null=True,
+        default=None,
+        db_index=True,
+    )
+
+
 class Team(CommonModelNameNotUnique, ResourceMixin):
     '''
     A team is a group of users that work on common projects.

awx/main/models/projects.py

@@ -52,7 +52,6 @@ class ProjectOptions(models.Model):
     SCM_TYPE_CHOICES = [
         ('', _('Manual')),
         ('git', _('Git')),
-        ('hg', _('Mercurial')),
         ('svn', _('Subversion')),
         ('insights', _('Red Hat Insights')),
         ('archive', _('Remote Archive')),

awx/main/models/schedules.py

@@ -205,10 +205,15 @@ class Schedule(PrimordialModel, LaunchTimeConfig):
                     'A valid TZID must be provided (e.g., America/New_York)'
                 )
 
-        if fast_forward and ('MINUTELY' in rrule or 'HOURLY' in rrule):
+        if (
+            fast_forward and
+            ('MINUTELY' in rrule or 'HOURLY' in rrule) and
+            'COUNT=' not in rrule
+        ):
             try:
                 first_event = x[0]
-                if first_event < now():
+                # If the first event was over a week ago...
+                if (now() - first_event).days > 7:
                     # hourly/minutely rrules with far-past DTSTART values
                     # are *really* slow to precompute
                     # start *from* one week ago to speed things up drastically

awx/main/models/unified_jobs.py

@@ -873,7 +873,13 @@ class UnifiedJob(PolymorphicModel, PasswordFieldsModel, CommonModelNameNotUnique
 
         # If status changed, update the parent instance.
         if self.status != status_before:
-            self._update_parent_instance()
+            # Update parent outside of the transaction for Job w/ allow_simultaneous=True
+            # This dodges lock contention at the expense of the foreign key not being
+            # completely correct.
+            if getattr(self, 'allow_simultaneous', False):
+                connection.on_commit(self._update_parent_instance)
+            else:
+                self._update_parent_instance()
 
         # Done.
         return result

awx/main/models/workflow.py

@@ -674,7 +674,7 @@ class WorkflowJob(UnifiedJob, WorkflowJobOptions, SurveyJobMixin, JobNotificatio
         return self.status == 'running'
 
 
-class WorkflowApprovalTemplate(UnifiedJobTemplate):
+class WorkflowApprovalTemplate(UnifiedJobTemplate, RelatedJobsMixin):
 
     FIELDS_TO_PRESERVE_AT_COPY = ['description', 'timeout',]
 
@@ -702,6 +702,12 @@ class WorkflowApprovalTemplate(UnifiedJobTemplate):
     def workflow_job_template(self):
         return self.workflowjobtemplatenodes.first().workflow_job_template
 
+    '''
+    RelatedJobsMixin
+    '''
+    def _get_related_jobs(self):
+        return UnifiedJob.objects.filter(unified_job_template=self)
+
 
 class WorkflowApproval(UnifiedJob, JobNotificationMixin):
     class Meta:
@@ -776,6 +782,10 @@ class WorkflowApproval(UnifiedJob, JobNotificationMixin):
         self.send_approval_notification('running')
         return can_start
 
+    @property
+    def event_processing_finished(self):
+        return True
+
     def send_approval_notification(self, approval_status):
         from awx.main.tasks import send_notifications  # avoid circular import
         if self.workflow_job_template is None:

awx/main/notifications/webhook_backend.py

@@ -57,6 +57,7 @@ class WebhookBackend(AWXBaseEmailBackend, CustomNotificationBase):
 
     def send_messages(self, messages):
         sent_messages = 0
+        self.headers['Content-Type'] = 'application/json'
         if 'User-Agent' not in self.headers:
             self.headers['User-Agent'] = "Tower {}".format(get_awx_version())
         if self.http_method.lower() not in ['put','post']:
@@ -68,7 +69,7 @@ class WebhookBackend(AWXBaseEmailBackend, CustomNotificationBase):
                 auth = (self.username, self.password)
             r = chosen_method("{}".format(m.recipients()[0]),
                               auth=auth,
-                              json=m.body,
+                              data=json.dumps(m.body, ensure_ascii=False).encode('utf-8'),
                               headers=self.headers,
                               verify=(not self.disable_ssl_verification))
             if r.status_code >= 400:

awx/main/redact.py

@@ -1,8 +1,6 @@
 import re
 import urllib.parse as urlparse
 
-from django.conf import settings
-
 REPLACE_STR = '$encrypted$'
 
 
@@ -12,12 +10,6 @@ class UriCleaner(object):
 
     @staticmethod
     def remove_sensitive(cleartext):
-        # exclude_list contains the items that will _not_ be redacted
-        exclude_list = [settings.PUBLIC_GALAXY_SERVER['url']]
-        if settings.PRIMARY_GALAXY_URL:
-            exclude_list += [settings.PRIMARY_GALAXY_URL]
-        if settings.FALLBACK_GALAXY_SERVERS:
-            exclude_list += [server['url'] for server in settings.FALLBACK_GALAXY_SERVERS]
         redactedtext = cleartext
         text_index = 0
         while True:
@@ -25,10 +17,6 @@ class UriCleaner(object):
             if not match:
                 break
             uri_str = match.group(1)
-            # Do not redact items from the exclude list
-            if any(uri_str.startswith(exclude_uri) for exclude_uri in exclude_list):
-                text_index = match.start() + len(uri_str)
-                continue
             try:
                 # May raise a ValueError if invalid URI for one reason or another
                 o = urlparse.urlsplit(uri_str)

awx/main/scheduler/kubernetes.py

@@ -12,6 +12,24 @@ from awx.main.utils.common import parse_yaml_or_json
 logger = logging.getLogger('awx.main.scheduler')
 
 
+def deepmerge(a, b):
+    """
+    Merge dict structures and return the result.
+
+    >>> a = {'first': {'all_rows': {'pass': 'dog', 'number': '1'}}}
+    >>> b = {'first': {'all_rows': {'fail': 'cat', 'number': '5'}}}
+    >>> import pprint; pprint.pprint(deepmerge(a, b))
+    {'first': {'all_rows': {'fail': 'cat', 'number': '5', 'pass': 'dog'}}}
+    """
+    if isinstance(a, dict) and isinstance(b, dict):
+        return dict([(k, deepmerge(a.get(k), b.get(k)))
+                     for k in set(a.keys()).union(b.keys())])
+    elif b is None:
+        return a
+    else:
+        return b
+
+
 class PodManager(object):
 
     def __init__(self, task=None):
@@ -128,11 +146,13 @@ class PodManager(object):
         pod_spec = {**default_pod_spec, **pod_spec_override}
 
         if self.task:
-            pod_spec['metadata']['name'] = self.pod_name
-            pod_spec['metadata']['labels'] = {
-                'ansible-awx': settings.INSTALL_UUID,
-                'ansible-awx-job-id': str(self.task.id)
-            }
+            pod_spec['metadata'] = deepmerge(
+                pod_spec.get('metadata', {}),
+                dict(name=self.pod_name,
+                     labels={
+                         'ansible-awx': settings.INSTALL_UUID,
+                         'ansible-awx-job-id': str(self.task.id)
+                     }))
             pod_spec['spec']['containers'][0]['name'] = self.pod_name
 
         return pod_spec

awx/main/scheduler/task_manager.py

@@ -7,11 +7,14 @@ import logging
 import uuid
 import json
 import random
+from types import SimpleNamespace
 
 # Django
 from django.db import transaction, connection
 from django.utils.translation import ugettext_lazy as _, gettext_noop
 from django.utils.timezone import now as tz_now
+from django.conf import settings
+from django.db.models import Q
 
 # AWX
 from awx.main.dispatch.reaper import reap_job
@@ -44,11 +47,46 @@ logger = logging.getLogger('awx.main.scheduler')
 class TaskManager():
 
     def __init__(self):
+        '''
+        Do NOT put database queries or other potentially expensive operations
+        in the task manager init. The task manager object is created every time a
+        job is created, transitions state, and every 30 seconds on each tower node.
+        More often then not, the object is destroyed quickly because the NOOP case is hit.
+
+        The NOOP case is short-circuit logic. If the task manager realizes that another instance
+        of the task manager is already running, then it short-circuits and decides not to run.
+        '''
         self.graph = dict()
+        # start task limit indicates how many pending jobs can be started on this
+        # .schedule() run. Starting jobs is expensive, and there is code in place to reap
+        # the task manager after 5 minutes. At scale, the task manager can easily take more than
+        # 5 minutes to start pending jobs. If this limit is reached, pending jobs
+        # will no longer be started and will be started on the next task manager cycle.
+        self.start_task_limit = settings.START_TASK_LIMIT
+
+    def after_lock_init(self):
+        '''
+        Init AFTER we know this instance of the task manager will run because the lock is acquired.
+        '''
+        instances = Instance.objects.filter(~Q(hostname=None), capacity__gt=0, enabled=True)
+        self.real_instances = {i.hostname: i for i in instances}
+
+        instances_partial = [SimpleNamespace(obj=instance,
+                                             remaining_capacity=instance.remaining_capacity,
+                                             capacity=instance.capacity,
+                                             jobs_running=instance.jobs_running,
+                                             hostname=instance.hostname) for instance in instances]
+
+        instances_by_hostname = {i.hostname: i for i in instances_partial}
+
         for rampart_group in InstanceGroup.objects.prefetch_related('instances'):
             self.graph[rampart_group.name] = dict(graph=DependencyGraph(rampart_group.name),
                                                   capacity_total=rampart_group.capacity,
-                                                  consumed_capacity=0)
+                                                  consumed_capacity=0,
+                                                  instances=[])
+            for instance in rampart_group.instances.filter(capacity__gt=0, enabled=True).order_by('hostname'):
+                if instance.hostname in instances_by_hostname:
+                    self.graph[rampart_group.name]['instances'].append(instances_by_hostname[instance.hostname])
 
     def is_job_blocked(self, task):
         # TODO: I'm not happy with this, I think blocking behavior should be decided outside of the dependency graph
@@ -189,6 +227,10 @@ class TaskManager():
         return result
 
     def start_task(self, task, rampart_group, dependent_tasks=None, instance=None):
+        self.start_task_limit -= 1
+        if self.start_task_limit == 0:
+            # schedule another run immediately after this task manager
+            schedule_task_manager()
         from awx.main.tasks import handle_work_error, handle_work_success
 
         dependent_tasks = dependent_tasks or []
@@ -243,7 +285,7 @@ class TaskManager():
                 for group in InstanceGroup.objects.all():
                     if group.is_containerized or group.controller_id:
                         continue
-                    match = group.fit_task_to_most_remaining_capacity_instance(task)
+                    match = group.fit_task_to_most_remaining_capacity_instance(task, group.instances.all())
                     if match:
                         break
                 task.instance_group = rampart_group
@@ -448,12 +490,13 @@ class TaskManager():
     def process_pending_tasks(self, pending_tasks):
         running_workflow_templates = set([wf.unified_job_template_id for wf in self.get_running_workflow_jobs()])
         for task in pending_tasks:
+            if self.start_task_limit <= 0:
+                break
             if self.is_job_blocked(task):
                 logger.debug("{} is blocked from running".format(task.log_format))
                 continue
             preferred_instance_groups = task.preferred_instance_groups
             found_acceptable_queue = False
-            idle_instance_that_fits = None
             if isinstance(task, WorkflowJob):
                 if task.unified_job_template_id in running_workflow_templates:
                     if not task.allow_simultaneous:
@@ -470,24 +513,24 @@ class TaskManager():
                     found_acceptable_queue = True
                     break
 
-                if idle_instance_that_fits is None:
-                    idle_instance_that_fits = rampart_group.find_largest_idle_instance()
                 remaining_capacity = self.get_remaining_capacity(rampart_group.name)
                 if not rampart_group.is_containerized and self.get_remaining_capacity(rampart_group.name) <= 0:
                     logger.debug("Skipping group {}, remaining_capacity {} <= 0".format(
                                  rampart_group.name, remaining_capacity))
                     continue
 
-                execution_instance = rampart_group.fit_task_to_most_remaining_capacity_instance(task)
-                if execution_instance:
-                    logger.debug("Starting {} in group {} instance {} (remaining_capacity={})".format(
-                                 task.log_format, rampart_group.name, execution_instance.hostname, remaining_capacity))
-                elif not execution_instance and idle_instance_that_fits:
+                execution_instance = InstanceGroup.fit_task_to_most_remaining_capacity_instance(task, self.graph[rampart_group.name]['instances']) or \
+                    InstanceGroup.find_largest_idle_instance(self.graph[rampart_group.name]['instances'])
+
+                if execution_instance or rampart_group.is_containerized:
                     if not rampart_group.is_containerized:
-                        execution_instance = idle_instance_that_fits
+                        execution_instance.remaining_capacity = max(0, execution_instance.remaining_capacity - task.task_impact)
+                        execution_instance.jobs_running += 1
                         logger.debug("Starting {} in group {} instance {} (remaining_capacity={})".format(
                                      task.log_format, rampart_group.name, execution_instance.hostname, remaining_capacity))
-                if execution_instance or rampart_group.is_containerized:
+
+                    if execution_instance:
+                        execution_instance = self.real_instances[execution_instance.hostname]
                     self.graph[rampart_group.name]['graph'].add_job(task)
                     self.start_task(task, rampart_group, task.get_jobs_fail_chain(), execution_instance)
                     found_acceptable_queue = True
@@ -559,6 +602,9 @@ class TaskManager():
     def _schedule(self):
         finished_wfjs = []
         all_sorted_tasks = self.get_tasks()
+
+        self.after_lock_init()
+
         if len(all_sorted_tasks) > 0:
             # TODO: Deal with
             # latest_project_updates = self.get_latest_project_update_tasks(all_sorted_tasks)

awx/main/tasks.py

@@ -50,8 +50,9 @@ import ansible_runner
 
 # AWX
 from awx import __version__ as awx_application_version
-from awx.main.constants import PRIVILEGE_ESCALATION_METHODS, STANDARD_INVENTORY_UPDATE_ENV, GALAXY_SERVER_FIELDS
+from awx.main.constants import PRIVILEGE_ESCALATION_METHODS, STANDARD_INVENTORY_UPDATE_ENV
 from awx.main.access import access_registry
+from awx.main.analytics import all_collectors, expensive_collectors
 from awx.main.redact import UriCleaner
 from awx.main.models import (
     Schedule, TowerScheduleState, Instance, InstanceGroup,
@@ -62,7 +63,7 @@ from awx.main.models import (
     build_safe_env, enforce_bigint_pk_migration
 )
 from awx.main.constants import ACTIVE_STATES
-from awx.main.exceptions import AwxTaskError
+from awx.main.exceptions import AwxTaskError, PostRunError
 from awx.main.queue import CallbackQueueDispatcher
 from awx.main.isolated import manager as isolated_manager
 from awx.main.dispatch.publish import task
@@ -72,11 +73,12 @@ from awx.main.utils import (update_scm_url,
                             ignore_inventory_group_removal, extract_ansible_vars, schedule_task_manager,
                             get_awx_version)
 from awx.main.utils.ansible import read_ansible_config
-from awx.main.utils.common import _get_ansible_version, get_custom_venv_choices
+from awx.main.utils.common import get_custom_venv_choices
 from awx.main.utils.external_logging import reconfigure_rsyslog
 from awx.main.utils.safe_yaml import safe_dump, sanitize_jinja
 from awx.main.utils.reload import stop_local_services
 from awx.main.utils.pglock import advisory_lock
+from awx.main.utils.handlers import SpecialInventoryHandler
 from awx.main.consumers import emit_channel_notification
 from awx.main import analytics
 from awx.conf import settings_registry
@@ -311,7 +313,7 @@ def delete_project_files(project_path):
 
 @task(queue='tower_broadcast_all')
 def profile_sql(threshold=1, minutes=1):
-    if threshold == 0:
+    if threshold <= 0:
         cache.delete('awx-profile-sql-threshold')
         logger.error('SQL PROFILING DISABLED')
     else:
@@ -354,6 +356,26 @@ def send_notifications(notification_list, job_id=None):
 
 @task(queue=get_local_queuename)
 def gather_analytics():
+    def _gather_and_ship(subset, since, until):
+        tgzfiles = []
+        try:
+            tgzfiles = analytics.gather(subset=subset, since=since, until=until)
+            # empty analytics without raising an exception is not an error
+            if not tgzfiles:
+                return True
+            logger.info('Gathered analytics from {} to {}: {}'.format(since, until, tgzfiles))
+            for tgz in tgzfiles:
+                analytics.ship(tgz)
+        except Exception:
+            logger.exception('Error gathering and sending analytics for {} to {}.'.format(since,until))
+            return False
+        finally:
+            if tgzfiles:
+                for tgz in tgzfiles:
+                    if os.path.exists(tgz):
+                        os.remove(tgz)
+        return True
+
     from awx.conf.models import Setting
     from rest_framework.fields import DateTimeField
     if not settings.INSIGHTS_TRACKING_STATE:
@@ -372,16 +394,29 @@ def gather_analytics():
             if acquired is False:
                 logger.debug('Not gathering analytics, another task holds lock')
                 return
-            try:
-                tgz = analytics.gather()
-                if not tgz:
-                    return
-                logger.info('gathered analytics: {}'.format(tgz))
-                analytics.ship(tgz)
-                settings.AUTOMATION_ANALYTICS_LAST_GATHER = gather_time
-            finally:
-                if os.path.exists(tgz):
-                    os.remove(tgz)
+            subset = list(all_collectors().keys())
+            incremental_collectors = []
+            for collector in expensive_collectors():
+                if collector in subset:
+                    subset.remove(collector)
+                    incremental_collectors.append(collector)
+
+            # Cap gathering at 4 weeks of data if there has been no data gathering
+            since = last_time or (gather_time - timedelta(weeks=4))
+
+            if incremental_collectors:
+                start = since
+                until = None
+                while start < gather_time:
+                    until = start + timedelta(hours = 4)
+                    if (until > gather_time):
+                        until = gather_time
+                    if not _gather_and_ship(incremental_collectors, since=start, until=until):
+                        break
+                    start = until
+                    settings.AUTOMATION_ANALYTICS_LAST_GATHER = until
+            if subset:
+                _gather_and_ship(subset, since=since, until=gather_time)
 
 
 @task(queue=get_local_queuename)
@@ -840,25 +875,12 @@ class BaseTask(object):
                 logger.error('Failed to update %s after %d retries.',
                              self.model._meta.object_name, _attempt)
 
-    def get_ansible_version(self, instance):
-        if not hasattr(self, '_ansible_version'):
-            self._ansible_version = _get_ansible_version(
-                ansible_path=self.get_path_to_ansible(instance, executable='ansible'))
-        return self._ansible_version
-
     def get_path_to(self, *args):
         '''
         Return absolute path relative to this file.
         '''
         return os.path.abspath(os.path.join(os.path.dirname(__file__), *args))
 
-    def get_path_to_ansible(self, instance, executable='ansible-playbook', **kwargs):
-        venv_path = getattr(instance, 'ansible_virtualenv_path', settings.ANSIBLE_VENV_PATH)
-        venv_exe = os.path.join(venv_path, 'bin', executable)
-        if os.path.exists(venv_exe):
-            return venv_exe
-        return shutil.which(executable)
-
     def build_private_data(self, instance, private_data_dir):
         '''
         Return SSH private key data (only if stored in DB as ssh_key_data).
@@ -1203,6 +1225,13 @@ class BaseTask(object):
         Ansible runner puts a parent_uuid on each event, no matter what the type.
         AWX only saves the parent_uuid if the event is for a Job.
         '''
+        # cache end_line locally for RunInventoryUpdate tasks
+        # which generate job events from two 'streams':
+        # ansible-inventory and the awx.main.commands.inventory_import
+        # logger
+        if isinstance(self, RunInventoryUpdate):
+            self.end_line = event_data['end_line']
+
         if event_data.get(self.event_data_key, None):
             if self.event_data_key != 'job_id':
                 event_data.pop('parent_uuid', None)
@@ -1231,7 +1260,7 @@ class BaseTask(object):
             # so it *should* have a negligible performance impact
             task = event_data.get('event_data', {}).get('task_action')
             try:
-                if task in ('git', 'hg', 'svn'):
+                if task in ('git', 'svn'):
                     event_data_json = json.dumps(event_data)
                     event_data_json = UriCleaner.remove_sensitive(event_data_json)
                     event_data = json.loads(event_data_json)
@@ -1484,6 +1513,8 @@ class BaseTask(object):
                 self.instance.job_explanation = "Job terminated due to timeout"
                 status = 'failed'
                 extra_update_fields['job_explanation'] = self.instance.job_explanation
+                # ensure failure notification sends even if playbook_on_stats event is not triggered
+                handle_success_and_failure_notifications.apply_async([self.instance.job.id])
 
         except InvalidVirtualenvError as e:
             extra_update_fields['job_explanation'] = e.message
@@ -1497,6 +1528,12 @@ class BaseTask(object):
 
         try:
             self.post_run_hook(self.instance, status)
+        except PostRunError as exc:
+            if status == 'successful':
+                status = exc.status
+                extra_update_fields['job_explanation'] = exc.args[0]
+                if exc.tb:
+                    extra_update_fields['result_traceback'] = exc.tb
         except Exception:
             logger.exception('{} Post run hook errored.'.format(self.instance.log_format))
 
@@ -1630,21 +1667,10 @@ class RunJob(BaseTask):
 
         return passwords
 
-    def add_ansible_venv(self, venv_path, env, isolated=False):
-        super(RunJob, self).add_ansible_venv(venv_path, env, isolated=isolated)
-        # Add awx/lib to PYTHONPATH.
-        env['PYTHONPATH'] = env.get('PYTHONPATH', '') + self.get_path_to('..', 'lib') + ':'
-
     def build_env(self, job, private_data_dir, isolated=False, private_data_files=None):
         '''
         Build environment dictionary for ansible-playbook.
         '''
-        plugin_dir = self.get_path_to('..', 'plugins', 'callback')
-        plugin_dirs = [plugin_dir]
-        if hasattr(settings, 'AWX_ANSIBLE_CALLBACK_PLUGINS') and \
-                settings.AWX_ANSIBLE_CALLBACK_PLUGINS:
-            plugin_dirs.extend(settings.AWX_ANSIBLE_CALLBACK_PLUGINS)
-        plugin_path = ':'.join(plugin_dirs)
         env = super(RunJob, self).build_env(job, private_data_dir,
                                             isolated=isolated,
                                             private_data_files=private_data_files)
@@ -1655,20 +1681,13 @@ class RunJob(BaseTask):
         # callbacks to work.
         env['JOB_ID'] = str(job.pk)
         env['INVENTORY_ID'] = str(job.inventory.pk)
-        if job.use_fact_cache:
-            library_path = env.get('ANSIBLE_LIBRARY')
-            env['ANSIBLE_LIBRARY'] = ':'.join(
-                filter(None, [
-                    library_path,
-                    self.get_path_to('..', 'plugins', 'library')
-                ])
-            )
         if job.project:
             env['PROJECT_REVISION'] = job.project.scm_revision
         env['ANSIBLE_RETRY_FILES_ENABLED'] = "False"
         env['MAX_EVENT_RES'] = str(settings.MAX_EVENT_RES_DATA)
         if not isolated:
-            env['ANSIBLE_CALLBACK_PLUGINS'] = plugin_path
+            if hasattr(settings, 'AWX_ANSIBLE_CALLBACK_PLUGINS') and settings.AWX_ANSIBLE_CALLBACK_PLUGINS:
+                env['ANSIBLE_CALLBACK_PLUGINS'] = ':'.join(settings.AWX_ANSIBLE_CALLBACK_PLUGINS)
             env['AWX_HOST'] = settings.TOWER_URL_BASE
 
         # Create a directory for ControlPath sockets that is unique to each
@@ -2043,38 +2062,27 @@ class RunProjectUpdate(BaseTask):
         # like https://github.com/ansible/ansible/issues/30064
         env['TMP'] = settings.AWX_PROOT_BASE_PATH
         env['PROJECT_UPDATE_ID'] = str(project_update.pk)
-        env['ANSIBLE_CALLBACK_PLUGINS'] = self.get_path_to('..', 'plugins', 'callback')
         if settings.GALAXY_IGNORE_CERTS:
             env['ANSIBLE_GALAXY_IGNORE'] = True
-        # Set up the public Galaxy server, if enabled
-        galaxy_configured = False
-        if settings.PUBLIC_GALAXY_ENABLED:
-            galaxy_servers = [settings.PUBLIC_GALAXY_SERVER]  # static setting
-        else:
-            galaxy_configured = True
-            galaxy_servers = []
-        # Set up fallback Galaxy servers, if configured
-        if settings.FALLBACK_GALAXY_SERVERS:
-            galaxy_configured = True
-            galaxy_servers = settings.FALLBACK_GALAXY_SERVERS + galaxy_servers
-        # Set up the primary Galaxy server, if configured
-        if settings.PRIMARY_GALAXY_URL:
-            galaxy_configured = True
-            galaxy_servers = [{'id': 'primary_galaxy'}] + galaxy_servers
-            for key in GALAXY_SERVER_FIELDS:
-                value = getattr(settings, 'PRIMARY_GALAXY_{}'.format(key.upper()))
-                if value:
-                    galaxy_servers[0][key] = value
-        if galaxy_configured:
-            for server in galaxy_servers:
-                for key in GALAXY_SERVER_FIELDS:
-                    if not server.get(key):
-                        continue
-                    env_key = ('ANSIBLE_GALAXY_SERVER_{}_{}'.format(server.get('id', 'unnamed'), key)).upper()
-                    env[env_key] = server[key]
-            if galaxy_servers:
-                # now set the precedence of galaxy servers
-                env['ANSIBLE_GALAXY_SERVER_LIST'] = ','.join([server.get('id', 'unnamed') for server in galaxy_servers])
+
+        # build out env vars for Galaxy credentials (in order)
+        galaxy_server_list = []
+        if project_update.project.organization:
+            for i, cred in enumerate(
+                project_update.project.organization.galaxy_credentials.all()
+            ):
+                env[f'ANSIBLE_GALAXY_SERVER_SERVER{i}_URL'] = cred.get_input('url')
+                auth_url = cred.get_input('auth_url', default=None)
+                token = cred.get_input('token', default=None)
+                if token:
+                    env[f'ANSIBLE_GALAXY_SERVER_SERVER{i}_TOKEN'] = token
+                if auth_url:
+                    env[f'ANSIBLE_GALAXY_SERVER_SERVER{i}_AUTH_URL'] = auth_url
+                galaxy_server_list.append(f'server{i}')
+
+        if galaxy_server_list:
+            env['ANSIBLE_GALAXY_SERVER_LIST'] = ','.join(galaxy_server_list)
+
         return env
 
     def _build_scm_url_extra_vars(self, project_update):
@@ -2146,19 +2154,32 @@ class RunProjectUpdate(BaseTask):
             elif not scm_branch:
                 raise RuntimeError('Could not determine a revision to run from project.')
         elif not scm_branch:
-            scm_branch = {'hg': 'tip'}.get(project_update.scm_type, 'HEAD')
+            scm_branch = 'HEAD'
+
+        galaxy_creds_are_defined = (
+            project_update.project.organization and
+            project_update.project.organization.galaxy_credentials.exists()
+        )
+        if not galaxy_creds_are_defined and (
+            settings.AWX_ROLES_ENABLED or settings.AWX_COLLECTIONS_ENABLED
+        ):
+            logger.warning(
+                'Galaxy role/collection syncing is enabled, but no '
+                f'credentials are configured for {project_update.project.organization}.'
+            )
+
         extra_vars.update({
             'projects_root': settings.PROJECTS_ROOT.rstrip('/'),
             'local_path': os.path.basename(project_update.project.local_path),
             'project_path': project_update.get_project_path(check_if_exists=False),  # deprecated
             'insights_url': settings.INSIGHTS_URL_BASE,
-            'awx_license_type': get_license(show_key=False).get('license_type', 'UNLICENSED'),
+            'awx_license_type': get_license().get('license_type', 'UNLICENSED'),
             'awx_version': get_awx_version(),
             'scm_url': scm_url,
             'scm_branch': scm_branch,
             'scm_clean': project_update.scm_clean,
-            'roles_enabled': settings.AWX_ROLES_ENABLED,
-            'collections_enabled': settings.AWX_COLLECTIONS_ENABLED,
+            'roles_enabled': galaxy_creds_are_defined and settings.AWX_ROLES_ENABLED,
+            'collections_enabled': galaxy_creds_are_defined and settings.AWX_COLLECTIONS_ENABLED,
         })
         # apply custom refspec from user for PR refs and the like
         if project_update.scm_refspec:
@@ -2169,7 +2190,7 @@ class RunProjectUpdate(BaseTask):
         self._write_extra_vars_file(private_data_dir, extra_vars)
 
     def build_cwd(self, project_update, private_data_dir):
-        return self.get_path_to('..', 'playbooks')
+        return os.path.join(private_data_dir, 'project')
 
     def build_playbook_path_relative_to_cwd(self, project_update, private_data_dir):
         return os.path.join('project_update.yml')
@@ -2310,6 +2331,12 @@ class RunProjectUpdate(BaseTask):
             shutil.rmtree(stage_path)
         os.makedirs(stage_path)  # presence of empty cache indicates lack of roles or collections
 
+        # the project update playbook is not in a git repo, but uses a vendoring directory
+        # to be consistent with the ansible-runner model,
+        # that is moved into the runner projecct folder here
+        awx_playbooks = self.get_path_to('..', 'playbooks')
+        copy_tree(awx_playbooks, os.path.join(private_data_dir, 'project'))
+
     @staticmethod
     def clear_project_cache(cache_dir, keep_value):
         if os.path.isdir(cache_dir):
@@ -2403,9 +2430,10 @@ class RunProjectUpdate(BaseTask):
                 shutil.rmtree(stage_path)  # cannot trust content update produced
 
             if self.job_private_data_dir:
-                # copy project folder before resetting to default branch
-                # because some git-tree-specific resources (like submodules) might matter
-                self.make_local_copy(instance, self.job_private_data_dir)
+                if status == 'successful':
+                    # copy project folder before resetting to default branch
+                    # because some git-tree-specific resources (like submodules) might matter
+                    self.make_local_copy(instance, self.job_private_data_dir)
                 if self.original_branch:
                     # for git project syncs, non-default branches can be problems
                     # restore to branch the repo was on before this run
@@ -2447,9 +2475,17 @@ class RunInventoryUpdate(BaseTask):
     event_model = InventoryUpdateEvent
     event_data_key = 'inventory_update_id'
 
+    # TODO: remove once inv updates run in containers
+    def should_use_proot(self, inventory_update):
+        '''
+        Return whether this task should use proot.
+        '''
+        return getattr(settings, 'AWX_PROOT_ENABLED', False)
+
+    # TODO: remove once inv updates run in containers
     @property
     def proot_show_paths(self):
-        return [self.get_path_to('..', 'plugins', 'inventory'), settings.AWX_ANSIBLE_COLLECTIONS_PATHS]
+        return [settings.AWX_ANSIBLE_COLLECTIONS_PATHS]
 
     def build_private_data(self, inventory_update, private_data_dir):
         """
@@ -2467,19 +2503,15 @@ class RunInventoryUpdate(BaseTask):
         If no private data is needed, return None.
         """
         if inventory_update.source in InventorySource.injectors:
-            injector = InventorySource.injectors[inventory_update.source](self.get_ansible_version(inventory_update))
+            injector = InventorySource.injectors[inventory_update.source]()
             return injector.build_private_data(inventory_update, private_data_dir)
 
     def build_env(self, inventory_update, private_data_dir, isolated, private_data_files=None):
-        """Build environment dictionary for inventory import.
+        """Build environment dictionary for ansible-inventory.
 
-        This used to be the mechanism by which any data that needs to be passed
-        to the inventory update script is set up. In particular, this is how
-        inventory update is aware of its proper credentials.
-
-        Most environment injection is now accomplished by the credential
-        injectors. The primary purpose this still serves is to
-        still point to the inventory update INI or config file.
+        Most environment variables related to credentials or configuration
+        are accomplished by the inventory source injectors (in this method)
+        or custom credential type injectors (in main run method).
         """
         env = super(RunInventoryUpdate, self).build_env(inventory_update,
                                                         private_data_dir,
@@ -2487,15 +2519,18 @@ class RunInventoryUpdate(BaseTask):
                                                         private_data_files=private_data_files)
         if private_data_files is None:
             private_data_files = {}
-        self.add_awx_venv(env)
-        # Pass inventory source ID to inventory script.
+        # TODO: remove once containers replace custom venvs
+        self.add_ansible_venv(inventory_update.ansible_virtualenv_path, env, isolated=isolated)
+
+        # Legacy environment variables, were used as signal to awx-manage command
+        # now they are provided in case some scripts may be relying on them
         env['INVENTORY_SOURCE_ID'] = str(inventory_update.inventory_source_id)
         env['INVENTORY_UPDATE_ID'] = str(inventory_update.pk)
         env.update(STANDARD_INVENTORY_UPDATE_ENV)
 
         injector = None
         if inventory_update.source in InventorySource.injectors:
-            injector = InventorySource.injectors[inventory_update.source](self.get_ansible_version(inventory_update))
+            injector = InventorySource.injectors[inventory_update.source]()
 
         if injector is not None:
             env = injector.build_env(inventory_update, env, private_data_dir, private_data_files)
@@ -2551,52 +2586,25 @@ class RunInventoryUpdate(BaseTask):
         if inventory is None:
             raise RuntimeError('Inventory Source is not associated with an Inventory.')
 
-        # Piece together the initial command to run via. the shell.
-        args = ['awx-manage', 'inventory_import']
-        args.extend(['--inventory-id', str(inventory.pk)])
+        args = ['ansible-inventory', '--list', '--export']
 
-        # Add appropriate arguments for overwrite if the inventory_update
-        # object calls for it.
-        if inventory_update.overwrite:
-            args.append('--overwrite')
-        if inventory_update.overwrite_vars:
-            args.append('--overwrite-vars')
+        # Add arguments for the source inventory file/script/thing
+        source_location = self.pseudo_build_inventory(inventory_update, private_data_dir)
+        args.append('-i')
+        args.append(source_location)
 
-        # Declare the virtualenv the management command should activate
-        # as it calls ansible-inventory
-        args.extend(['--venv', inventory_update.ansible_virtualenv_path])
+        args.append('--output')
+        args.append(os.path.join(private_data_dir, 'artifacts', 'output.json'))
+
+        if os.path.isdir(source_location):
+            playbook_dir = source_location
+        else:
+            playbook_dir = os.path.dirname(source_location)
+        args.extend(['--playbook-dir', playbook_dir])
+
+        if inventory_update.verbosity:
+            args.append('-' + 'v' * min(5, inventory_update.verbosity * 2 + 1))
 
-        src = inventory_update.source
-        # Add several options to the shell arguments based on the
-        # inventory-source-specific setting in the AWX configuration.
-        # These settings are "per-source"; it's entirely possible that
-        # they will be different between cloud providers if an AWX user
-        # actively uses more than one.
-        if getattr(settings, '%s_ENABLED_VAR' % src.upper(), False):
-            args.extend(['--enabled-var',
-                        getattr(settings, '%s_ENABLED_VAR' % src.upper())])
-        if getattr(settings, '%s_ENABLED_VALUE' % src.upper(), False):
-            args.extend(['--enabled-value',
-                        getattr(settings, '%s_ENABLED_VALUE' % src.upper())])
-        if getattr(settings, '%s_GROUP_FILTER' % src.upper(), False):
-            args.extend(['--group-filter',
-                         getattr(settings, '%s_GROUP_FILTER' % src.upper())])
-        if getattr(settings, '%s_HOST_FILTER' % src.upper(), False):
-            args.extend(['--host-filter',
-                         getattr(settings, '%s_HOST_FILTER' % src.upper())])
-        if getattr(settings, '%s_EXCLUDE_EMPTY_GROUPS' % src.upper()):
-            args.append('--exclude-empty-groups')
-        if getattr(settings, '%s_INSTANCE_ID_VAR' % src.upper(), False):
-            args.extend(['--instance-id-var',
-                        "'{}'".format(getattr(settings, '%s_INSTANCE_ID_VAR' % src.upper())),])
-        # Add arguments for the source inventory script
-        args.append('--source')
-        args.append(self.pseudo_build_inventory(inventory_update, private_data_dir))
-        if src == 'custom':
-            args.append("--custom")
-        args.append('-v%d' % inventory_update.verbosity)
-        if settings.DEBUG:
-            args.append('--traceback')
         return args
 
     def build_inventory(self, inventory_update, private_data_dir):
@@ -2613,7 +2621,7 @@ class RunInventoryUpdate(BaseTask):
 
         injector = None
         if inventory_update.source in InventorySource.injectors:
-            injector = InventorySource.injectors[src](self.get_ansible_version(inventory_update))
+            injector = InventorySource.injectors[src]()
 
         if injector is not None:
             content = injector.inventory_contents(inventory_update, private_data_dir)
@@ -2636,11 +2644,9 @@ class RunInventoryUpdate(BaseTask):
 
     def build_cwd(self, inventory_update, private_data_dir):
         '''
-        There are two cases where the inventory "source" is in a different
+        There is one case where the inventory "source" is in a different
         location from the private data:
-         - deprecated vendored inventory scripts in awx/plugins/inventory
          - SCM, where source needs to live in the project folder
-        in these cases, the inventory does not exist in the standard tempdir
         '''
         src = inventory_update.source
         if src == 'scm' and inventory_update.source_project_update:
@@ -2698,6 +2704,75 @@ class RunInventoryUpdate(BaseTask):
             # This follows update, not sync, so make copy here
             RunProjectUpdate.make_local_copy(source_project, private_data_dir)
 
+    def post_run_hook(self, inventory_update, status):
+        if status != 'successful':
+            return # nothing to save, step out of the way to allow error reporting
+
+        private_data_dir = inventory_update.job_env['AWX_PRIVATE_DATA_DIR']
+        expected_output = os.path.join(private_data_dir, 'artifacts', 'output.json')
+        with open(expected_output) as f:
+            data = json.load(f)
+
+        # build inventory save options
+        options = dict(
+            overwrite=inventory_update.overwrite,
+            overwrite_vars=inventory_update.overwrite_vars,
+        )
+        src = inventory_update.source
+
+        if inventory_update.enabled_var:
+            options['enabled_var'] = inventory_update.enabled_var
+            options['enabled_value'] = inventory_update.enabled_value
+        else:
+            if getattr(settings, '%s_ENABLED_VAR' % src.upper(), False):
+                options['enabled_var'] = getattr(settings, '%s_ENABLED_VAR' % src.upper())
+            if getattr(settings, '%s_ENABLED_VALUE' % src.upper(), False):
+                options['enabled_value'] = getattr(settings, '%s_ENABLED_VALUE' % src.upper())
+
+        if inventory_update.host_filter:
+            options['host_filter'] = inventory_update.host_filter
+
+        if getattr(settings, '%s_EXCLUDE_EMPTY_GROUPS' % src.upper()):
+            options['exclude_empty_groups'] = True
+        if getattr(settings, '%s_INSTANCE_ID_VAR' % src.upper(), False):
+            options['instance_id_var'] = getattr(settings, '%s_INSTANCE_ID_VAR' % src.upper())
+
+        # Verbosity is applied to saving process, as well as ansible-inventory CLI option
+        if inventory_update.verbosity:
+            options['verbosity'] = inventory_update.verbosity
+
+        handler = SpecialInventoryHandler(
+            self.event_handler, self.cancel_callback,
+            verbosity=inventory_update.verbosity,
+            job_timeout=self.get_instance_timeout(self.instance),
+            start_time=inventory_update.started,
+            counter=self.event_ct, initial_line=self.end_line
+        )
+        inv_logger = logging.getLogger('awx.main.commands.inventory_import')
+        formatter = inv_logger.handlers[0].formatter
+        formatter.job_start = inventory_update.started
+        handler.formatter = formatter
+        inv_logger.handlers[0] = handler
+
+        from awx.main.management.commands.inventory_import import Command as InventoryImportCommand
+        cmd = InventoryImportCommand()
+        try:
+            # save the inventory data to database.
+            # canceling exceptions will be handled in the global post_run_hook
+            cmd.perform_update(options, data, inventory_update)
+        except PermissionDenied as exc:
+            logger.exception('License error saving {} content'.format(inventory_update.log_format))
+            raise PostRunError(str(exc), status='error')
+        except PostRunError:
+            logger.exception('Error saving {} content, rolling back changes'.format(inventory_update.log_format))
+            raise
+        except Exception:
+            logger.exception('Exception saving {} content, rolling back changes.'.format(
+                inventory_update.log_format))
+            raise PostRunError(
+                'Error occured while saving inventory data, see traceback or server logs',
+                status='error', tb=traceback.format_exc())
+
 
 @task(queue=get_local_queuename)
 class RunAdHocCommand(BaseTask):
@@ -2756,7 +2831,6 @@ class RunAdHocCommand(BaseTask):
         '''
         Build environment dictionary for ansible.
         '''
-        plugin_dir = self.get_path_to('..', 'plugins', 'callback')
         env = super(RunAdHocCommand, self).build_env(ad_hoc_command, private_data_dir,
                                                      isolated=isolated,
                                                      private_data_files=private_data_files)
@@ -2766,7 +2840,6 @@ class RunAdHocCommand(BaseTask):
         env['AD_HOC_COMMAND_ID'] = str(ad_hoc_command.pk)
         env['INVENTORY_ID'] = str(ad_hoc_command.inventory.pk)
         env['INVENTORY_HOSTVARS'] = str(True)
-        env['ANSIBLE_CALLBACK_PLUGINS'] = plugin_dir
         env['ANSIBLE_LOAD_CALLBACK_PLUGINS'] = '1'
         env['ANSIBLE_SFTP_BATCH_MODE'] = 'False'
 

awx/main/tests/data/inventory/plugins/azure_rm/files/azure_rm.yml

@@ -1,43 +0,0 @@
-conditional_groups:
-  azure: true
-default_host_filters: []
-exclude_host_filters:
-- resource_group not in ['foo_resources', 'bar_resources']
-- '"Creator" not in tags.keys()'
-- tags["Creator"] != "jmarshall"
-- '"peanutbutter" not in tags.keys()'
-- tags["peanutbutter"] != "jelly"
-- location not in ['southcentralus', 'westus']
-fail_on_template_errors: false
-hostvar_expressions:
-  ansible_host: private_ipv4_addresses[0]
-  computer_name: name
-  private_ip: private_ipv4_addresses[0] if private_ipv4_addresses else None
-  provisioning_state: provisioning_state | title
-  public_ip: public_ipv4_addresses[0] if public_ipv4_addresses else None
-  public_ip_id: public_ip_id if public_ip_id is defined else None
-  public_ip_name: public_ip_name if public_ip_name is defined else None
-  tags: tags if tags else None
-  type: resource_type
-keyed_groups:
-- key: location
-  prefix: ''
-  separator: ''
-- key: tags.keys() | list if tags else []
-  prefix: ''
-  separator: ''
-- key: security_group
-  prefix: ''
-  separator: ''
-- key: resource_group
-  prefix: ''
-  separator: ''
-- key: os_disk.operating_system_type
-  prefix: ''
-  separator: ''
-- key: dict(tags.keys() | map("regex_replace", "^(.*)$", "\1_") | list | zip(tags.values() | list)) if tags else []
-  prefix: ''
-  separator: ''
-plain_host_names: true
-plugin: azure.azcollection.azure_rm
-use_contrib_script_compatible_sanitization: true

awx/main/tests/data/inventory/plugins/ec2/files/aws_ec2.yml

@@ -1,81 +0,0 @@
-boto_profile: /tmp/my_boto_stuff
-compose:
-  ansible_host: public_dns_name
-  ec2_account_id: owner_id
-  ec2_ami_launch_index: ami_launch_index | string
-  ec2_architecture: architecture
-  ec2_block_devices: dict(block_device_mappings | map(attribute='device_name') | list | zip(block_device_mappings | map(attribute='ebs.volume_id') | list))
-  ec2_client_token: client_token
-  ec2_dns_name: public_dns_name
-  ec2_ebs_optimized: ebs_optimized
-  ec2_eventsSet: events | default("")
-  ec2_group_name: placement.group_name
-  ec2_hypervisor: hypervisor
-  ec2_id: instance_id
-  ec2_image_id: image_id
-  ec2_instance_profile: iam_instance_profile | default("")
-  ec2_instance_type: instance_type
-  ec2_ip_address: public_ip_address
-  ec2_kernel: kernel_id | default("")
-  ec2_key_name: key_name
-  ec2_launch_time: launch_time | regex_replace(" ", "T") | regex_replace("(\+)(\d\d):(\d)(\d)$", ".\g<2>\g<3>Z")
-  ec2_monitored: monitoring.state in ['enabled', 'pending']
-  ec2_monitoring_state: monitoring.state
-  ec2_persistent: persistent | default(false)
-  ec2_placement: placement.availability_zone
-  ec2_platform: platform | default("")
-  ec2_private_dns_name: private_dns_name
-  ec2_private_ip_address: private_ip_address
-  ec2_public_dns_name: public_dns_name
-  ec2_ramdisk: ramdisk_id | default("")
-  ec2_reason: state_transition_reason
-  ec2_region: placement.region
-  ec2_requester_id: requester_id | default("")
-  ec2_root_device_name: root_device_name
-  ec2_root_device_type: root_device_type
-  ec2_security_group_ids: security_groups | map(attribute='group_id') | list |  join(',')
-  ec2_security_group_names: security_groups | map(attribute='group_name') | list |  join(',')
-  ec2_sourceDestCheck: source_dest_check | default(false) | lower | string
-  ec2_spot_instance_request_id: spot_instance_request_id | default("")
-  ec2_state: state.name
-  ec2_state_code: state.code
-  ec2_state_reason: state_reason.message if state_reason is defined else ""
-  ec2_subnet_id: subnet_id | default("")
-  ec2_tag_Name: tags.Name
-  ec2_virtualization_type: virtualization_type
-  ec2_vpc_id: vpc_id | default("")
-filters:
-  instance-state-name:
-  - running
-groups:
-  ec2: true
-hostnames:
-- dns-name
-iam_role_arn: arn:aws:iam::123456789012:role/test-role
-keyed_groups:
-- key: placement.availability_zone
-  parent_group: zones
-  prefix: ''
-  separator: ''
-- key: instance_type | regex_replace("[^A-Za-z0-9\_]", "_")
-  parent_group: types
-  prefix: type
-- key: placement.region
-  parent_group: regions
-  prefix: ''
-  separator: ''
-- key: dict(tags.keys() | map("regex_replace", "[^A-Za-z0-9\_]", "_") | list | zip(tags.values() | map("regex_replace", "[^A-Za-z0-9\_]", "_") | list))
-  parent_group: tags
-  prefix: tag
-- key: tags.keys() | map("regex_replace", "[^A-Za-z0-9\_]", "_") | list
-  parent_group: tags
-  prefix: tag
-- key: placement.availability_zone
-  parent_group: '{{ placement.region }}'
-  prefix: ''
-  separator: ''
-plugin: amazon.aws.aws_ec2
-regions:
-- us-east-2
-- ap-south-1
-use_contrib_script_compatible_sanitization: true

awx/main/tests/data/inventory/plugins/gce/files/gcp_compute.yml

@@ -1,50 +0,0 @@
-auth_kind: serviceaccount
-compose:
-  ansible_ssh_host: networkInterfaces[0].accessConfigs[0].natIP | default(networkInterfaces[0].networkIP)
-  gce_description: description if description else None
-  gce_id: id
-  gce_image: image
-  gce_machine_type: machineType
-  gce_metadata: metadata.get("items", []) | items2dict(key_name="key", value_name="value")
-  gce_name: name
-  gce_network: networkInterfaces[0].network.name
-  gce_private_ip: networkInterfaces[0].networkIP
-  gce_public_ip: networkInterfaces[0].accessConfigs[0].natIP | default(None)
-  gce_status: status
-  gce_subnetwork: networkInterfaces[0].subnetwork.name
-  gce_tags: tags.get("items", [])
-  gce_zone: zone
-hostnames:
-- name
-- public_ip
-- private_ip
-keyed_groups:
-- key: gce_subnetwork
-  prefix: network
-- key: gce_private_ip
-  prefix: ''
-  separator: ''
-- key: gce_public_ip
-  prefix: ''
-  separator: ''
-- key: machineType
-  prefix: ''
-  separator: ''
-- key: zone
-  prefix: ''
-  separator: ''
-- key: gce_tags
-  prefix: tag
-- key: status | lower
-  prefix: status
-- key: image
-  prefix: ''
-  separator: ''
-plugin: google.cloud.gcp_compute
-projects:
-- fooo
-retrieve_image_info: true
-use_contrib_script_compatible_sanitization: true
-zones:
-- us-east4-a
-- us-west1-b

awx/main/tests/data/inventory/plugins/openstack/files/file_reference

@@ -1,7 +1,3 @@
-ansible:
-  expand_hostvars: true
-  fail_on_errors: true
-  use_hostnames: false
 clouds:
   devstack:
     auth:
@@ -11,5 +7,5 @@ clouds:
       project_domain_name: fooo
       project_name: fooo
       username: fooo
-    private: false
+    private: true
     verify: false

awx/main/tests/data/inventory/plugins/openstack/files/openstack.yml

@@ -1,4 +0,0 @@
-expand_hostvars: true
-fail_on_errors: true
-inventory_hostname: uuid
-plugin: openstack.cloud.openstack

awx/main/tests/data/inventory/plugins/rhv/files/ovirt.yml

@@ -1,20 +0,0 @@
-base_source_var: value_of_var
-compose:
-  ansible_host: (devices.values() | list)[0][0] if devices else None
-groups:
-  dev: '"dev" in tags'
-keyed_groups:
-- key: cluster
-  prefix: cluster
-  separator: _
-- key: status
-  prefix: status
-  separator: _
-- key: tags
-  prefix: tag
-  separator: _
-ovirt_hostname_preference:
-- name
-- fqdn
-ovirt_insecure: false
-plugin: ovirt.ovirt.ovirt

awx/main/tests/data/inventory/plugins/satellite6/files/foreman.yml

@@ -1,30 +0,0 @@
-base_source_var: value_of_var
-compose:
-  ansible_ssh_host: foreman['ip6'] | default(foreman['ip'], true)
-group_prefix: foo_group_prefix
-keyed_groups:
-- key: foreman['environment_name'] | lower | regex_replace(' ', '') | regex_replace('[^A-Za-z0-9_]', '_') | regex_replace('none', '')
-  prefix: foo_group_prefixenvironment_
-  separator: ''
-- key: foreman['location_name'] | lower | regex_replace(' ', '') | regex_replace('[^A-Za-z0-9_]', '_')
-  prefix: foo_group_prefixlocation_
-  separator: ''
-- key: foreman['organization_name'] | lower | regex_replace(' ', '') | regex_replace('[^A-Za-z0-9_]', '_')
-  prefix: foo_group_prefixorganization_
-  separator: ''
-- key: foreman['content_facet_attributes']['lifecycle_environment_name'] | lower | regex_replace(' ', '') | regex_replace('[^A-Za-z0-9_]', '_')
-  prefix: foo_group_prefixlifecycle_environment_
-  separator: ''
-- key: foreman['content_facet_attributes']['content_view_name'] | lower | regex_replace(' ', '') | regex_replace('[^A-Za-z0-9_]', '_')
-  prefix: foo_group_prefixcontent_view_
-  separator: ''
-- key: '"%s-%s-%s" | format(app, tier, color)'
-  separator: ''
-- key: '"%s-%s" | format(app, color)'
-  separator: ''
-legacy_hostvars: true
-plugin: theforeman.foreman.foreman
-validate_certs: false
-want_facts: true
-want_hostcollections: true
-want_params: true

awx/main/tests/data/inventory/plugins/tower/files/tower.yml

@@ -1,3 +0,0 @@
-include_metadata: true
-inventory_id: 42
-plugin: awx.awx.tower

awx/main/tests/data/inventory/plugins/vmware/files/vmware_vm_inventory.yml

@@ -1,55 +0,0 @@
-compose:
-  ansible_host: guest.ipAddress
-  ansible_ssh_host: guest.ipAddress
-  ansible_uuid: 99999999 | random | to_uuid
-  availablefield: availableField
-  configissue: configIssue
-  configstatus: configStatus
-  customvalue: customValue
-  effectiverole: effectiveRole
-  guestheartbeatstatus: guestHeartbeatStatus
-  layoutex: layoutEx
-  overallstatus: overallStatus
-  parentvapp: parentVApp
-  recenttask: recentTask
-  resourcepool: resourcePool
-  rootsnapshot: rootSnapshot
-  triggeredalarmstate: triggeredAlarmState
-filters:
-- config.zoo == "DC0_H0_VM0"
-hostnames:
-- config.foo
-keyed_groups:
-- key: config.asdf
-  prefix: ''
-  separator: ''
-plugin: community.vmware.vmware_vm_inventory
-properties:
-- availableField
-- configIssue
-- configStatus
-- customValue
-- datastore
-- effectiveRole
-- guestHeartbeatStatus
-- layout
-- layoutEx
-- name
-- network
-- overallStatus
-- parentVApp
-- permission
-- recentTask
-- resourcePool
-- rootSnapshot
-- snapshot
-- triggeredAlarmState
-- value
-- capability
-- config
-- guest
-- runtime
-- storage
-- summary
-strict: false
-with_nested_properties: true

awx/main/tests/factories/README.md

@@ -52,11 +52,11 @@ patterns
 --------
 
 `mk` functions are single object fixtures. They should create only a single object with the minimum deps.
-They should also accept a `persited` flag, if they must be persisted to work, they raise an error if persisted=False
+They should also accept a `persisted` flag, if they must be persisted to work, they raise an error if persisted=False
 
 `generate` and `apply` functions are helpers that build up the various parts of a `create` functions objects. These
 should be useful for more than one create function to use and should explicitly accept all of the values needed
-to execute. These functions should also be robust and have very speciifc error reporting about constraints and/or
+to execute. These functions should also be robust and have very specific error reporting about constraints and/or
 bad values.
 
 `create` functions compose many of the `mk` and `generate` functions to make different object

awx/main/tests/functional/analytics/test_collectors.py

@@ -1,6 +1,7 @@
 import pytest
 import tempfile
 import os
+import re
 import shutil
 import csv
 
@@ -27,7 +28,8 @@ def sqlite_copy_expert(request):
 
     def write_stdout(self, sql, fd):
         # Would be cool if we instead properly disected the SQL query and verified
-        # it that way. But instead, we just take the nieve approach here.
+        # it that way. But instead, we just take the naive approach here.
+        sql = sql.strip()
         assert sql.startswith("COPY (")
         assert sql.endswith(") TO STDOUT WITH CSV HEADER")
 
@@ -35,6 +37,10 @@ def sqlite_copy_expert(request):
         sql = sql.replace(") TO STDOUT WITH CSV HEADER", "")
         # sqlite equivalent
         sql = sql.replace("ARRAY_AGG", "GROUP_CONCAT")
+        # SQLite doesn't support isoformatted dates, because that would be useful
+        sql = sql.replace("+00:00", "")
+        i = re.compile(r'(?P<date>\d\d\d\d-\d\d-\d\d)T')
+        sql = i.sub(r'\g<date> ', sql)
 
         # Remove JSON style queries
         # TODO: could replace JSON style queries with sqlite kind of equivalents
@@ -86,7 +92,7 @@ def test_copy_tables_unified_job_query(
     job_name = job_template.create_unified_job().name
 
     with tempfile.TemporaryDirectory() as tmpdir:
-        collectors.copy_tables(time_start, tmpdir, subset="unified_jobs")
+        collectors.unified_jobs_table(time_start, tmpdir, until = now() + timedelta(seconds=1))
         with open(os.path.join(tmpdir, "unified_jobs_table.csv")) as f:
             lines = "".join([line for line in f])
 
@@ -134,7 +140,7 @@ def test_copy_tables_workflow_job_node_query(sqlite_copy_expert, workflow_job):
     time_start = now() - timedelta(hours=9)
 
     with tempfile.TemporaryDirectory() as tmpdir:
-        collectors.copy_tables(time_start, tmpdir, subset="workflow_job_node_query")
+        collectors.workflow_job_node_table(time_start, tmpdir, until = now() + timedelta(seconds=1))
         with open(os.path.join(tmpdir, "workflow_job_node_table.csv")) as f:
             reader = csv.reader(f)
             # Pop the headers

awx/main/tests/functional/analytics/test_core.py

@@ -10,17 +10,17 @@ from awx.main.analytics import gather, register
 
 
 @register('example', '1.0')
-def example(since):
+def example(since, **kwargs):
     return {'awx': 123}
 
 
 @register('bad_json', '1.0')
-def bad_json(since):
+def bad_json(since, **kwargs):
     return set()
 
 
 @register('throws_error', '1.0')
-def throws_error(since):
+def throws_error(since, **kwargs):
     raise ValueError()
     
 
@@ -39,9 +39,9 @@ def mock_valid_license():
 def test_gather(mock_valid_license):
     settings.INSIGHTS_TRACKING_STATE = True
     
-    tgz = gather(module=importlib.import_module(__name__))
+    tgzfiles = gather(module=importlib.import_module(__name__))
     files = {}
-    with tarfile.open(tgz, "r:gz") as archive:
+    with tarfile.open(tgzfiles[0], "r:gz") as archive:
         for member in archive.getmembers():
             files[member.name] = archive.extractfile(member)
 
@@ -53,7 +53,8 @@ def test_gather(mock_valid_license):
         assert './bad_json.json' not in files.keys()
         assert './throws_error.json' not in files.keys()
     try:
-        os.remove(tgz)
+        for tgz in tgzfiles:
+            os.remove(tgz)
     except Exception:
         pass
         

awx/main/tests/functional/analytics/test_projects_by_scm_type.py

@@ -1,34 +0,0 @@
-import pytest
-import random
-
-from awx.main.models import Project
-from awx.main.analytics import collectors
-
-
-@pytest.mark.django_db
-def test_empty():
-    assert collectors.projects_by_scm_type(None) == {
-        'manual': 0,
-        'git': 0,
-        'svn': 0,
-        'hg': 0,
-        'insights': 0,
-        'archive': 0,
-    }
-
-
-@pytest.mark.django_db
-@pytest.mark.parametrize('scm_type', [t[0] for t in Project.SCM_TYPE_CHOICES])
-def test_multiple(scm_type):
-    expected = {
-        'manual': 0,
-        'git': 0,
-        'svn': 0,
-        'hg': 0,
-        'insights': 0,
-        'archive': 0,
-    }
-    for i in range(random.randint(0, 10)):
-        Project(scm_type=scm_type).save()
-        expected[scm_type or 'manual'] += 1
-    assert collectors.projects_by_scm_type(None) == expected

awx/main/tests/functional/api/test_credential.py

@@ -675,33 +675,6 @@ def test_net_create_ok(post, organization, admin):
     assert cred.inputs['authorize'] is True
 
 
-#
-# Cloudforms Credentials
-#
-@pytest.mark.django_db
-def test_cloudforms_create_ok(post, organization, admin):
-    params = {
-        'credential_type': 1,
-        'name': 'Best credential ever',
-        'inputs': {
-            'host': 'some_host',
-            'username': 'some_username',
-            'password': 'some_password',
-        }
-    }
-    cloudforms = CredentialType.defaults['cloudforms']()
-    cloudforms.save()
-    params['organization'] = organization.id
-    response = post(reverse('api:credential_list'), params, admin)
-    assert response.status_code == 201
-
-    assert Credential.objects.count() == 1
-    cred = Credential.objects.all()[:1].get()
-    assert cred.inputs['host'] == 'some_host'
-    assert cred.inputs['username'] == 'some_username'
-    assert decrypt_field(cred, 'password') == 'some_password'
-
-
 #
 # GCE Credentials
 #

awx/main/tests/functional/api/test_credential_type.py

@@ -220,7 +220,7 @@ def test_create_valid_kind(kind, get, post, admin):
 
 
 @pytest.mark.django_db
-@pytest.mark.parametrize('kind', ['ssh', 'vault', 'scm', 'insights', 'kubernetes'])
+@pytest.mark.parametrize('kind', ['ssh', 'vault', 'scm', 'insights', 'kubernetes', 'galaxy'])
 def test_create_invalid_kind(kind, get, post, admin):
     response = post(reverse('api:credential_type_list'), {
         'kind': kind,

awx/main/tests/functional/api/test_inventory.py

@@ -1,5 +1,6 @@
 # -*- coding: utf-8 -*-
 import pytest
+import json
 from unittest import mock
 
 from django.core.exceptions import ValidationError
@@ -8,8 +9,6 @@ from awx.api.versioning import reverse
 
 from awx.main.models import InventorySource, Inventory, ActivityStream
 
-import json
-
 
 @pytest.fixture
 def scm_inventory(inventory, project):
@@ -522,7 +521,8 @@ class TestInventorySourceCredential:
             data={
                 'inventory': inventory.pk, 'name': 'fobar', 'source': 'scm',
                 'source_project': project.pk, 'source_path': '',
-                'credential': vault_credential.pk
+                'credential': vault_credential.pk,
+                'source_vars': 'plugin: a.b.c',
             },
             expect=400,
             user=admin_user
@@ -561,7 +561,7 @@ class TestInventorySourceCredential:
             data={
                 'inventory': inventory.pk, 'name': 'fobar', 'source': 'scm',
                 'source_project': project.pk, 'source_path': '',
-                'credential': os_cred.pk
+                'credential': os_cred.pk, 'source_vars': 'plugin: a.b.c',
             },
             expect=201,
             user=admin_user
@@ -636,8 +636,14 @@ class TestControlledBySCM:
         assert scm_inventory.inventory_sources.count() == 0
 
     def test_adding_inv_src_ok(self, post, scm_inventory, project, admin_user):
-        post(reverse('api:inventory_inventory_sources_list', kwargs={'pk': scm_inventory.id}),
-             {'name': 'new inv src', 'source_project': project.pk, 'update_on_project_update': False, 'source': 'scm', 'overwrite_vars': True},
+        post(reverse('api:inventory_inventory_sources_list',
+             kwargs={'pk': scm_inventory.id}),
+             {'name': 'new inv src',
+              'source_project': project.pk,
+              'update_on_project_update': False,
+              'source': 'scm',
+              'overwrite_vars': True,
+              'source_vars': 'plugin: a.b.c'},
              admin_user, expect=201)
 
     def test_adding_inv_src_prohibited(self, post, scm_inventory, project, admin_user):
@@ -657,7 +663,7 @@ class TestControlledBySCM:
     def test_adding_inv_src_without_proj_access_prohibited(self, post, project, inventory, rando):
         inventory.admin_role.members.add(rando)
         post(reverse('api:inventory_inventory_sources_list', kwargs={'pk': inventory.id}),
-             {'name': 'new inv src', 'source_project': project.pk, 'source': 'scm', 'overwrite_vars': True},
+             {'name': 'new inv src', 'source_project': project.pk, 'source': 'scm', 'overwrite_vars': True, 'source_vars': 'plugin: a.b.c'},
              rando, expect=403)
 
 

awx/main/tests/functional/api/test_job_runtime_params.py

@@ -359,6 +359,71 @@ def test_job_launch_fails_with_missing_vault_password(machine_credential, vault_
     assert response.data['passwords_needed_to_start'] == ['vault_password']
 
 
+@pytest.mark.django_db
+def test_job_launch_with_added_cred_and_vault_password(credential, machine_credential, vault_credential,
+                                                       deploy_jobtemplate, post, admin):
+    # see: https://github.com/ansible/awx/issues/8202
+    vault_credential.inputs['vault_password'] = 'ASK'
+    vault_credential.save()
+    payload = {
+        'credentials': [vault_credential.id, machine_credential.id],
+        'credential_passwords': {'vault_password': 'vault-me'},
+    }
+
+    deploy_jobtemplate.ask_credential_on_launch = True
+    deploy_jobtemplate.credentials.remove(credential)
+    deploy_jobtemplate.credentials.add(vault_credential)
+    deploy_jobtemplate.save()
+
+    with mock.patch.object(Job, 'signal_start') as signal_start:
+        post(
+            reverse('api:job_template_launch', kwargs={'pk': deploy_jobtemplate.pk}),
+            payload,
+            admin,
+            expect=201,
+        )
+        signal_start.assert_called_with(**{
+            'vault_password': 'vault-me'
+        })
+
+
+@pytest.mark.django_db
+def test_job_launch_with_multiple_launch_time_passwords(credential, machine_credential, vault_credential,
+                                                        deploy_jobtemplate, post, admin):
+    # see: https://github.com/ansible/awx/issues/8202
+    deploy_jobtemplate.ask_credential_on_launch = True
+    deploy_jobtemplate.credentials.remove(credential)
+    deploy_jobtemplate.credentials.add(machine_credential)
+    deploy_jobtemplate.credentials.add(vault_credential)
+    deploy_jobtemplate.save()
+
+    second_machine_credential = Credential(
+        name='SSH #2',
+        credential_type=machine_credential.credential_type,
+        inputs={'password': 'ASK'}
+    )
+    second_machine_credential.save()
+
+    vault_credential.inputs['vault_password'] = 'ASK'
+    vault_credential.save()
+    payload = {
+        'credentials': [vault_credential.id, second_machine_credential.id],
+        'credential_passwords': {'ssh_password': 'ssh-me', 'vault_password': 'vault-me'},
+    }
+
+    with mock.patch.object(Job, 'signal_start') as signal_start:
+        post(
+            reverse('api:job_template_launch', kwargs={'pk': deploy_jobtemplate.pk}),
+            payload,
+            admin,
+            expect=201,
+        )
+        signal_start.assert_called_with(**{
+            'ssh_password': 'ssh-me',
+            'vault_password': 'vault-me',
+        })
+
+
 @pytest.mark.django_db
 @pytest.mark.parametrize('launch_kwargs', [
     {'vault_password.abc': 'vault-me-1', 'vault_password.xyz': 'vault-me-2'},

awx/main/tests/functional/api/test_organizations.py

@@ -9,7 +9,7 @@ from django.conf import settings
 import pytest
 
 # AWX
-from awx.main.models import ProjectUpdate
+from awx.main.models import ProjectUpdate, CredentialType, Credential
 from awx.api.versioning import reverse
 
 
@@ -288,3 +288,90 @@ def test_organization_delete_with_active_jobs(delete, admin, organization, organ
 
     assert resp.data['error'] == u"Resource is being used by running jobs."
     assert resp_sorted == expect_sorted
+
+
+@pytest.mark.django_db
+def test_galaxy_credential_association_forbidden(alice, organization, post):
+    galaxy = CredentialType.defaults['galaxy_api_token']()
+    galaxy.save()
+
+    cred = Credential.objects.create(
+        credential_type=galaxy,
+        name='Public Galaxy',
+        organization=organization,
+        inputs={
+            'url': 'https://galaxy.ansible.com/'
+        }
+    )
+    url = reverse('api:organization_galaxy_credentials_list', kwargs={'pk': organization.id})
+    post(
+        url,
+        {'associate': True, 'id': cred.pk},
+        user=alice,
+        expect=403
+    )
+
+
+@pytest.mark.django_db
+def test_galaxy_credential_type_enforcement(admin, organization, post):
+    ssh = CredentialType.defaults['ssh']()
+    ssh.save()
+
+    cred = Credential.objects.create(
+        credential_type=ssh,
+        name='SSH Credential',
+        organization=organization,
+    )
+    url = reverse('api:organization_galaxy_credentials_list', kwargs={'pk': organization.id})
+    resp = post(
+        url,
+        {'associate': True, 'id': cred.pk},
+        user=admin,
+        expect=400
+    )
+    assert resp.data['msg'] == 'Credential must be a Galaxy credential, not Machine.'
+
+
+@pytest.mark.django_db
+def test_galaxy_credential_association(alice, admin, organization, post, get):
+    galaxy = CredentialType.defaults['galaxy_api_token']()
+    galaxy.save()
+
+    for i in range(5):
+        cred = Credential.objects.create(
+            credential_type=galaxy,
+            name=f'Public Galaxy {i + 1}',
+            organization=organization,
+            inputs={
+                'url': 'https://galaxy.ansible.com/'
+            }
+        )
+        url = reverse('api:organization_galaxy_credentials_list', kwargs={'pk': organization.id})
+        post(
+            url,
+            {'associate': True, 'id': cred.pk},
+            user=admin,
+            expect=204
+        )
+    resp = get(url, user=admin)
+    assert [cred['name'] for cred in resp.data['results']] == [
+        'Public Galaxy 1',
+        'Public Galaxy 2',
+        'Public Galaxy 3',
+        'Public Galaxy 4',
+        'Public Galaxy 5',
+    ]
+
+    post(
+        url,
+        {'disassociate': True, 'id': Credential.objects.get(name='Public Galaxy 3').pk},
+        user=admin,
+        expect=204
+    )
+    resp = get(url, user=admin)
+    assert [cred['name'] for cred in resp.data['results']] == [
+        'Public Galaxy 1',
+        'Public Galaxy 2',
+        'Public Galaxy 4',
+        'Public Galaxy 5',
+    ]