* Defer ansible_facts in HostManager to avoid fetching large JSON column in host list queries (AAP-68023)
The host list endpoint (GET /api/v2/hosts/) fetches the ansible_facts
JSON column unnecessarily, contributing to the 7.8s median query time
at scale. This column can be very large and is not used by the list
serializer.
Changes:
- HostManager.get_queryset() now defers ansible_facts
- finish_fact_cache call site uses .only(*HOST_FACTS_FIELDS) to eagerly
load ansible_facts when actually needed, avoiding N+1 queries
- Unit test mocks updated to support .only() queryset chaining
- Points DAB dependency at the RBAC query optimization branch for
combined testing
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Django 5.2 restricts LogoutView to POST only (deprecated in 4.1,
removed in 5.0+). Without this fix, GET requests to /api/logout/
return 405 Method Not Allowed.
Add http_method_names override and a get() method that delegates
to post() where auth_logout() actually runs
* AAP-57817 Add Redis connection retry using redis-py 7.0+ built-in mechanism
* Refactor Redis client helpers to use settings and eliminate code duplication
* Create awx/main/utils/redis.py and move Redis client functions to avoid circular imports
* Fix subsystem_metrics to share Redis connection pool between
client and pipeline
* Cache Redis clients in RelayConsumer and RelayWebsocketStatsManager to avoid creating new connection pools on every call
* Add cap and base config
* Add Redis retry logic with exponential backoff to handle connection failures during long-running operations
* Add REDIS_BACKOFF_CAP and REDIS_BACKOFF_BASE settings to allow
adjustment of retry timing in worst-case scenarios without code changes
* Simplify Redis retry tests by removing unnecessary reload logic
* Fix CI: Use Python 3.13 for ansible-test compatibility
ansible-test only supports Python 3.11, 3.12, and 3.13.
Changed collection-integration jobs from '3.x' to '3.13'
to avoid using Python 3.14 which is not supported.
* Fix ansible-test Python version for CI integration tests
ansible-test only supports Python 3.11, 3.12, and 3.13.
Added ANSIBLE_TEST_PYTHON_VERSION variable to explicitly pass
--python 3.13 flag to ansible-test integration command.
This prevents ansible-test from auto-detecting and using
Python 3.14.0, which is not supported.
* Fix CI: Execute ansible-test with Python 3.13 to avoid
unsupported Python 3.14
* Fix CI: Use Python 3.13 across all jobs to avoid Python
3.14 compatibility issues
* Fix CI: Use 'python' and 'ansible.test' module for Python
3.13 compatibility
* Fix CI: Use 'python' instead of 'python3' for Python 3.13
compatibility
* Fix CI: Ensure ansible-test uses Python 3.13 environment
explicitly
* Fix: Remove silent failure check for ansible-core in test suite
* Fix CI: Export PYTHONPATH to make awxkit available to ansible-test
* Fix CI: Use 'python' in run_awx_devel to maintain Python
3.13 environment
* Fix CI: Remove setup-python from awx_devel_image that was resetting Python 3.13 to 3.14
* Move logic to unified job model instead of view
* Refine logic to only apply to double escaped characters to prevent touching unicord chars
* Refine logic to only apply to stdout so that it does not impact webhook notifications
* Revise naming to reflect correction to escapes, not just escape quotes
* Update code comments to reflect fixing double escapes vs double escaped quotes specifically
* Add regex for 5 most common python escape chars to make fix more robust
* Revise start_fact_cache and finish_fact_cache to use JSON file (#15970)
* Revise start_fact_cache and finish_fact_cache to use JSON file with host list inside it
* Revise artifacts path to be relative to the job private_data_dir
* Update calls to start_fact_cache and finish_fact_cache to agree with new reference to artifacts_dir
* Prevents unnecessary updates to ansible_facts_modified, fixing timestamp-related test failures.
* Import bulk_update_sorted_by_id
* Removed assert that calls ansible_facts_new which was removed in the backported pr
* Add import of Host back
* Fix some patterns in collection test playbooks
* Revert change to ansible.builtin.user
* Revert change to WFJT for dup label error
* Add error handling and fix references
* Add back lookup organization
* Fix all remainingfailing syntax in workflow_job_template
* Allow creating galaxy credential types without an organization (#16077)
* remove requirement for galaxy credentials to belong to an organization
* remove organization check for galaxy credential type
---------
Co-authored-by: AlanCoding <arominge@redhat.com>
Co-authored-by: Peter Braun <pbraun@redhat.com>
* Bug fix for AAP-47771 this data migration updates existing CredentialType entries
in the database and changes the kind from github_app to github_app_lookup
* Combine migration 0203 into 0202
* Add test to ensure reconciliation issue has been resolved
* Fix collection task breaking collection ci checks
* Patch ansible.module_utils.basic._ANSIBLE_PROFILE directly
* Conditionalize other santity assertions
* Remove added blank lines and identifier from Fail if absent and no identifier set
Sort both bulk updates and add batch size to facts bulk update to resolve deadlock issue
Update tests to expect batch_size to agree with changes
Add utility method to bulk update and sort hosts and applied that to the appropriate locations
Update functional tests to use bulk_update_sorted_by_id since update_hosts has been deleted
Add comment NOSONAR to get rid of Sonarqube warning since this is just a test and it's not actually a security issue
Fix failing test test_finish_job_fact_cache_clear & test_finish_job_fact_cache_with_existing_data
---------
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
Co-authored-by: Alan Rominger <arominge@redhat.com>
Co-authored-by: Seth Foster <fosterbseth@gmail.com>
* Revise start_fact_cache and finish_fact_cache to use JSON file with host list inside it
* Revise artifacts path to be relative to the job private_data_dir
* Update calls to start_fact_cache and finish_fact_cache to agree with new reference to artifacts_dir
* Prevents unnecessary updates to ansible_facts_modified, fixing timestamp-related test failures.
* Demo of sorting hosts live test
* Sort both bulk updates and add batch size to facts bulk update to resolve deadlock issue
* Update tests to expect batch_size to agree with changes
* Add utility method to bulk update and sort hosts and applied that to the appropriate locations
Remove unused imports
Add utility method for sorting bulk updates
Remove try except OperationalError for loop
Remove unused import of django.db.OperationalError
Remove batch size as it is now on the bulk update utility method as 100
Remove batch size here since it is specified in sortedbulkupdate
Add transaction.atomic to have entire transaction is run as a signle transaction before committing to the db
Revert change to bulk update as it's not needed here and just sort instead
Move bulk_sorted utility method into db.py and updated name to not be specific to Hosts
Revise to import bulk_update_sorted.. rather than calling it as an argument
Fix way I'm importing bulk_update_sorted.. Remove unneeded Host import and remove calls to bul_update as args
Rebise calls to bulk_update_sorted.. to include Host in the args
REmove raw_update_hosts method and replace with bulk_update_sorted_by_id in update_hosts
Remove update_hosts function and replace with bulk_update_sorted_by_id
Update live tests to use bulk_update_sorted_by_id
Fix the fields in bulk_update to agree with test
* Update functional tests to use bulk_update_sorted_by_id since update_hosts has been deleted
Replace update_hosts with bulk_update_sorted_by_id
Remove referenes to update_hosts
Update corresponding fact cachin tests to use bulk_update_sorted_by_id
Remove import of bulk_sorted_update
Add code comment to live test to silence Sonarqube hotspot
* Add comment NOSONAR to get rid of Sonarqube warning since this is just a test and it's not actually a security issue
Get test_finish_job_fact_cache_with_existing_data passing
Get test_finish_job_fact_cache_clear passing
Remove reference to raw_update and replace with new bulk update utility method
Add pytest.mark.django_db to appropriate tests
Corrent which model is called in bulk_update_sorted_by_id
Remove now unused Host import
Point to where bulk_update_sorted_by_id to where that is actually being used
Correct import of bulk_update_sorted_by_id
Revert changes in this file to avoid db calls issue
Remove @pytest.mark.django_db from unit tests
Remove commented out host sorting suggested fix
Fix failing tests test_pre_post_run_hook_facts_deleted_sliced & test_pre_post_run_hook_facts
Remove atomic transaction line, add return, and add docstring
* Fix failing test test_finish_job_fact_cache_clear & test_finish_job_fact_cache_with_existing_data
---------
Co-authored-by: Alan Rominger <arominge@redhat.com>
* Added test_jobs.py to the model unit test folder in orther to show the undesired behaviour with fact cache
* Added test_jobs.py to the model unit test folder in orther to show the undesired behaviour with fact cache
* Solved undesired behaviour with fact_cache
* Solved bug with slices
* Remove unused imports
Remove now unused line of code which was commented out by the contributor
Revert "Remove now unused line of code which was commented out by the contributor"
This reverts commit f1a056a2356d56bc7256957a18503bd14dcfd8aa.
* Add back line that had been commented out as this line makes hosts specific to the particular slice when applicable
Revise private_data_dir fixture to see if it improves code coverage
Checked out awx/main/tests/unit/models/test_jobs.py in devel to see if it resolves git diff issue
* Fix formatting in awx/main/tests/unit/models/test_jobs.py
Rename for loop from host in hosts to hosts in hosts_cahced and remove unneeded continue
Revise finish_fact_cache to utilize inventory rather than hosts
Remove local var hosts that was assigned but unused
Revert change in start_fact_cache hosts_cached back to hosts
Revise the way we are handling hosts_cached and joining the file
Revert "Revise the way we are handling hosts_cached and joining the file"
This reverts commit e6e3d2f09c1b79a9bce3647a72e3dad97fe0aed8.
Reapply "Revise the way we are handling hosts_cached and joining the file"
This reverts commit a42b7ae69133fee24d3a5f1b456d9c343d111df9.
Revert some of my changes to get back to a better working state
Rename for loop to host in hosts_cached and remove unneeded continue
Remove jobs job.get_hosts_for_fact_cache() from post run hook, fix if statement after continue block, and revise how we are calling hosts in finish for loop
Add test_invalid_host_facts to test_jobs to increase code coverage
Update method signature to use hosts_cached and updated other references to hosts in finish_facts_cached
Rename hosts iterator to hosts_cached to agree with naming elsewhere
Revise test_invalid_host_facts to get more code coverage
Revise test_invalid_host_facts to increase codecov
Revise test_pre_post_run_hook_facts_deleted_sliced to ensure we are hitting the assertionerror for code cov
Revise mock_inventory.hosts. to hit assert failure
Add revision of hosts and facts to force failure to satisfy code cov
Fix failure in test_pre_post_run_hook_facts_deleted_sliced
Add back for loop to create failures and add assert to hit them
Remove hosts.iterator() from both start_fact_cache and finish_fact_cache
Remove unused import of Queryset to satisfy api-lint
Fix typo in docstring hasnot to has not
Move hosts_cached.append(host) to outer loop in start_fact_cache
Add class to help support cached hosts resolving host.name issue with hosts_cached
* Add live tests for ansible facts
Remove fixture needed for local work only maybe
Revert "Add class to help support cached hosts resolving host.name issue with hosts_cached"
This reverts commit 99d998cfb9960baafe887de80bd9b01af50513ec.
* Move hosts_cached.append(host) outside of try except
* Move hosts_cached.append(host) to the beginning of start_fact_cache
---------
Signed-off-by: onetti7 <davonebat@gmail.com>
Co-authored-by: onetti7 <davonebat@gmail.com>
Co-authored-by: Alan Rominger <arominge@redhat.com>
* feat: 38589 GitHub App Authentication (#15807)
* feat: 38589 GitHub App Authentication
Allows both git@<personal-token> and x-access-token@<github-access-token> when authenticating using git.
This allows GitHub App tokens to work without interfering with existing authentication types.
---------
Co-authored-by: Jake Jackson <thedoubl3j@Jakes-MacBook-Pro.local>
* revert change made to allow UI to accept x-access-token, just use htt… (#15851)
revert change made to allow UI to accept x-access-token, just use https:// instead
* Add Github dep for new cred support if used (#15850)
* Add pygithub for new app token support
* fixed git requirements file with new
* added new github dep and relevant deps it needs
* add required licenses
* Add artifacts to satisfy license check
* Remove duplicated license
---------
Co-authored-by: Andrea Restle-Lay <arestlel@redhat.com>
Co-authored-by: Alan Rominger <arominge@redhat.com>
* Remove deps update it came with the cherry-pick and is not needed in this version
Remove unneeded deps updates from requirements.in
Remove point to awx-plugins as it is not needed in tower
* Add a credential plugin that uses GitHub Apps to get tokens
* Add github app tests
* Ran requirements updater script
Ran black on github_app_test to fix formatting issue
Add scm_github_app to managed credentials
Ran updater script to reflect new deps
Added github app info to def build_passwords in jobs.py, cred now appears in credential types
Update ManagedCredentialType for GitHub App to match what we have in awx-plugins
Update inputs in maManagedCredentialType to github_app_inputs to communicate with awx/main/credential_plugins/github_app.py
Revert incorrect change in ManagedCredentialType, change github_app_lookup to call inputs instead of github_app_inputs
Updated namespace to github_app_lookup to agree with nomenclature used in the rest of the implementation and to resolve failing API test
Remove import pointing to awx plugins and update to point to credential_plugins
Remove references to gh_app_plugin_mod and change to github_app
Remove from awx_plugins.interfaces._temporary_private_django_api import ( # noqa: WPS436 to resolve failing test
Remove flake8 typing & typing references that do not exist in this version of Tower
Remove references in jobs.py and __init__.py since this is an external cred type and registered it in setup.cfg instead
Remove blank line
REvise name in cfg from github_app_lookup to github_app to see if that ifxes module not found error
Revise first declaration of github_app to agree with file name to see if that resolves issue
Rename line 174 to agree with what's in config
Fix reference to github_app_lookup to github_app
Linters compliaining about the github_app in __all__ not being defined, renamed to see if that aligns them
Fix naming in test to correspond to naming of cred type
Update naming to be more specific and add blank line to setup.cfg
Remove __all__ from githubapp.py to satisfy linters
Revert formatting change since it is not needed in this repository
* Add blank line at the end of requirements.in
---------
Co-authored-by: Andrea Restle-Lay <andrearestlelay@gmail.com>
Co-authored-by: Jake Jackson <thedoubl3j@Jakes-MacBook-Pro.local>
Co-authored-by: Jake Jackson <jljacks93@gmail.com>
Co-authored-by: Andrea Restle-Lay <arestlel@redhat.com>
Co-authored-by: Alan Rominger <arominge@redhat.com>
* Added test_jobs.py to the model unit test folder in orther to show the undesired behaviour with fact cache
Signed-off-by: onetti7 <davonebat@gmail.com>
* Added test_jobs.py to the model unit test folder in orther to show the undesired behaviour with fact cache
Signed-off-by: onetti7 <davonebat@gmail.com>
* Solved undesired behaviour with fact_cache
Signed-off-by: onetti7 <davonebat@gmail.com>
* Solved bug with slices
Signed-off-by: onetti7 <davonebat@gmail.com>
* Remove unused imports
Remove now unused line of code which was commented out by the contributor
Revert "Remove now unused line of code which was commented out by the contributor"
This reverts commit f1a056a2356d56bc7256957a18503bd14dcfd8aa.
* Add back line that had been commented out as this line makes hosts specific to the particular slice when applicable
Revise private_data_dir fixture to see if it improves code coverage
Checked out awx/main/tests/unit/models/test_jobs.py in devel to see if it resolves git diff issue
* Fix formatting in awx/main/tests/unit/models/test_jobs.py
Rename for loop from host in hosts to hosts in hosts_cahced and remove unneeded continue
Revise finish_fact_cache to utilize inventory rather than hosts
Remove local var hosts that was assigned but unused
Revert change in start_fact_cache hosts_cached back to hosts
Revise the way we are handling hosts_cached and joining the file
Revert "Revise the way we are handling hosts_cached and joining the file"
This reverts commit e6e3d2f09c1b79a9bce3647a72e3dad97fe0aed8.
Reapply "Revise the way we are handling hosts_cached and joining the file"
This reverts commit a42b7ae69133fee24d3a5f1b456d9c343d111df9.
Revert some of my changes to get back to a better working state
Rename for loop to host in hosts_cached and remove unneeded continue
Remove jobs job.get_hosts_for_fact_cache() from post run hook, fix if statement after continue block, and revise how we are calling hosts in finish for loop
Add test_invalid_host_facts to test_jobs to increase code coverage
Update method signature to use hosts_cached and updated other references to hosts in finish_facts_cached
Rename hosts iterator to hosts_cached to agree with naming elsewhere
Revise test_invalid_host_facts to get more code coverage
Revise test_invalid_host_facts to increase codecov
Revise test_pre_post_run_hook_facts_deleted_sliced to ensure we are hitting the assertionerror for code cov
Revise mock_inventory.hosts. to hit assert failure
Add revision of hosts and facts to force failure to satisfy code cov
Fix failure in test_pre_post_run_hook_facts_deleted_sliced
Add back for loop to create failures and add assert to hit them
Remove hosts.iterator() from both start_fact_cache and finish_fact_cache
Remove unused import of Queryset to satisfy api-lint
Fix typo in docstring hasnot to has not
Move hosts_cached.append(host) to outer loop in start_fact_cache
Add class to help support cached hosts resolving host.name issue with hosts_cached
* Add live tests for ansible facts
Remove fixture needed for local work only maybe
Revert "Add class to help support cached hosts resolving host.name issue with hosts_cached"
This reverts commit 99d998cfb9960baafe887de80bd9b01af50513ec.
* Move hosts_cached.append(host) outside of try except
* Move hosts_cached.append(host) to the beginning of start_fact_cache
---------
Signed-off-by: onetti7 <davonebat@gmail.com>
Co-authored-by: onetti7 <davonebat@gmail.com>
Co-authored-by: Alan Rominger <arominge@redhat.com>
* Add descriptions for plugin names
* Update serializers to display plugin and plugin description
* Add function to extract plugin name descriptions
* Add description for scm
* Conditionalize scm and file descriptions
* Add username and password to handle_auth and update exception message
Revise naming of ldap username and password
* Add url for LDAP and userpass to method_auth
* Add information regarding LDAP and username and password to credential plugins documentation
Revise ldap_auth to userpass_auth and revised exception to better reflect functionality
* Revise method_auth to ensure certs can be used with username and ensure namespace functionality is not hindered
rename migration function set_peers_from_control_nodes_true to automatically_peer_from_control_plane
import settings and only run function if settings.IS_K8S is true
set listener_port for control nodes to None
Add hop node support to awx collections
- add peers and peers_from_control_nodes fields
- show new node_type "hop"
- add tests for adding hop nodes via collections
Co-authored-by: Seth Foster <fosterseth@users.noreply.github.com>
* Created new rsyslog launch file.
* Rsyslog conf work.
* Refining how we're calling rsyslog conf.
* Removed rsyslog so it no longer launches in the web container.
* Added the new launch_awx_rsyslog.sh to the /usr/bin
Lila Yasin
