* Fix cartesian product in organization user/admin count queries
The organizations list and detail endpoints annotated each org with user and admin counts using two Count() calls that traverse the Role.members M2M. Django generated two LEFT JOINs on the same through table, crossing every member row with every admin row before COUNT(DISTINCT) reduced the product.
At scale (2,617 members × 46,233 admins) this produced 120M intermediate rows and 96-second query times, causing 504 timeouts.
Replace with independent Subquery expressions that each query main_rbac_roles_members separately - no cross product.
Fixes: AAP-72817
Fixes: AAP-72480
* Fix variable names which do not meet coding standards
* Fix formatting inconsistency in organization detail subquery annotation
Break the long .annotate() line across multiple lines to match the style used in mixin.py.
* Rewrite org count subqueries to use DAB RBAC models
Replace old RBAC Role.members.through subqueries with
RoleUserAssignment-based correlated subqueries, querying
managed RoleDefinitions ('Organization Member' / 'Organization Admin')
directly. This aligns with the DAB RBAC migration direction and
eliminates dependency on the deprecated ImplicitRoleField M2M tables
for these counts.
Update test fixtures to use RoleDefinition.give_permission() and
add setup_managed_roles where needed.
* Fix collection tests: set up managed role definitions
The DAB RBAC migration to use RoleUserAssignment subqueries in
organization views requires managed role definitions (Organization
Member, Organization Admin) to exist in the test database.
Add an autouse fixture to the collection test conftest that calls
setup_managed_role_definitions() before each test.
* Add setup_managed_roles fixture to functional tests hitting org views
Tests that hit organization list/detail views now require the
setup_managed_roles fixture to pre-create the Organization Member
and Organization Admin RoleDefinition objects used by the DAB RBAC
subqueries.
* Revert setup_managed_roles from ext_auditor tests
The setup_managed_roles fixture conflicts with the ext_auditor_rd
fixture by deleting the Alien Auditor role definition. These tests
don't need it — the defensive view code handles missing role
definitions gracefully.
* Handle missing Organization Member/Admin role definitions gracefully
Use filter().first() instead of get() for RoleDefinition lookups in
organization list and detail views. Returns 0 for user/admin counts
when role definitions are not yet created, preventing 500 errors in
environments where post_migrate signals haven't run.
* Cast OuterRef('pk') to TextField for RoleUserAssignment.object_id comparison
RoleUserAssignment.object_id is a TextField, but OuterRef('pk') on
Organization produces an integer. PostgreSQL strictly rejects text = integer
comparisons. Use Cast() to explicitly convert the PK to text.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Restore oauth_token backward compatibility for collection token auth
The aap_token rename (c8981e321e) restored module-level token auth but
left two interfaces from earlier collection releases broken:
- The lookup (controller_api) and inventory (controller) plugins
previously declared an oauth_token option. Add oauth_token as an
alias of aap_token in the auth_plugin doc fragment and in
AUTH_ARGSPEC so query(..., oauth_token=...) and inventory YAML keys
keep working.
- tower_cli.cfg-style config files used an oauth_token key under
[general]; it was silently ignored after the rename, quietly
degrading auth. load_config() now also reads the legacy oauth_token
key and maps it to aap_token, with the new aap_token key winning when
both are present. aap_token remains the canonical attribute used by
_parse_aap_token() and the Bearer header logic.
Also make the test helper compatible with ansible-core 2.21+, which
requires a serialization profile alongside _ANSIBLE_ARGS, and extend
the tests to cover the oauth_token alias and legacy config file key.
No changelog fragment added: awx_collection has no changelogs/
directory on devel.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Document oauth_token alias in module auth doc fragment
The oauth_token alias was added to aap_token in AUTH_ARGSPEC but not to
the module doc fragment, failing the validate-modules sanity check
(undocumented argument alias).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Generalize version references in compat comments
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The aap_token parameter was added to the collection argspec and docs
in #16025, but nothing consumed it after token auth was removed in
#15623: modules silently ignored the token and fell back to basic
auth, breaking token authentication through the AAP gateway.
Wire it up so requests authenticate with the provided token (e.g. one
issued by the AAP gateway, which validates it and proxies to the
controller):
- Send "Authorization: Bearer <token>" in make_request when aap_token
is set, skipping the basic-auth login probe; basic auth is unchanged
when no token is given
- Accept the token as a string or as the dict set as a fact by the
ansible.platform.token module ({token: ..., id: ...}), which is the
documented cross-collection mint/use/delete workflow
- Restore controller_oauthtoken and tower_oauthtoken as aliases for
back-compat with pre-#15623 playbooks, matching downstream
- Forward aap_token through the controller_api lookup and controller
inventory plugins via short_params, and add the missing
CONTROLLER_OAUTH_TOKEN/TOWER_OAUTH_TOKEN env sources to the plugin
doc fragment (plugins resolve env vars from doc fragments, not
env_fallback); AAP_TOKEN is no longer marked deprecated there
- Support tokens in the awxkit-based export/import modules
- Add unit tests covering the Bearer header for both token forms, the
aliases, the bad-dict failure, and the basic-auth fallback
Verified end-to-end against a live gateway-fronted AAP 2.7 deployment:
modules, the lookup plugin, both aliases, all env sources, dict-form
tokens, job launch/wait, and a clean HTTP 401 on an invalid token.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fix awx-collection tests for ansible-core 2.21.0 compat
ansible-core 2.21.0 introduced _PARSED_MODULE_ARGS in
module_utils/basic.py, used by _return_formatted() to conditionally
include invocation data. The awx_collection test harness bypasses
normal arg parsing via _load_params mock, leaving this variable as
None and causing AttributeError on every exit_json/fail_json call.
Mock _PARSED_MODULE_ARGS with _ansible_inject_invocation=True to
match the pre-2.21 behavior of always including invocation data.
* resolve bug and add simple unit tests
* Update awx_collection/plugins/modules/license.py
Co-authored-by: Andrew Potozniak <tyraziel@gmail.com>
---------
Co-authored-by: Andrew Potozniak <tyraziel@gmail.com>
Bump migrations and delete some files
Resolve remaining conflicts
Fix requirements
Flake8 fixes
Prefer devel changes for schema
Use correct versions
Remove sso connected stuff
Update to modern actions and collection fixes
Remove unwated alias
Version problems in actions
Fix more versioning problems
Update warning string
Messed it up again
Shorten exception
More removals
Remove pbr license
Remove tests deleted in devel
Remove unexpected files
Remove some content missed in the rebase
Use sleep_task from devel
Restore devel live conftest file
Add in settings that got missed
Prefer devel version of collection test
Finish repairing .github path
Remove unintended test file duplication
Undo more unintended file additions
* Working branch for testing DAB RBAC changes
* AAP-48392 Handle DAB RBAC either before or after new type model (for merge) (#16045)
* Handle DAB RBAC either before or after new type model
* Translate CT to DAB CT
* Fix for rearrangement of post_migration methods
* Directly include RBAC service URLs
* Add a run before remote permission additions
* Sync old rbac to remote rbac (#7025)
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
* Set DAB requirement back to devel
---------
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
Co-authored-by: Seth Foster <fosterseth@users.noreply.github.com>
* Fix issue where export module does not honor CONTROLLER_OPTIONAL_API_URLPATTERN_PREFIX
* Add unit test and handle leading/trailing slashes
* Reformat
* Refactor for clarity
* Remove unused import
* Fix issue where export module does not honor CONTROLLER_OPTIONAL_API_URLPATTERN_PREFIX
* Add unit test and handle leading/trailing slashes
* Reformat
* Refactor for clarity
* Remove unused import
* Fix collection task breaking collection ci checks
* Patch ansible.module_utils.basic._ANSIBLE_PROFILE directly
* Conditionalize other santity assertions
* Remove added blank lines and identifier from Fail if absent and no identifier set
* Update collection args (#16025)
* update collection arguments
* Add integration testing for new param
* fix: sanity check failures
---------
Co-authored-by: Sean Sullivan <ssulliva@redhat.com>
Co-authored-by: Alan Rominger <arominge@redhat.com>
* update formatting for sanity testing
* fixing indentation for sanity suite
* adjust tests to use new token name
* update tests to use aap_token instead of controller_oauthtoken
* add back aliases for backward compat
* we have integration tests that still leverage the old token name
* while we can rename these, this tells me that customers might still
have them in the wild and breaking them in a z stream is no bueno
* revert alias changes
---------
Co-authored-by: Peter Braun <pbraun@redhat.com>
Co-authored-by: Sean Sullivan <ssulliva@redhat.com>
Co-authored-by: Alan Rominger <arominge@redhat.com>
* Handle DAB RBAC either before or after new type model
* Translate CT to DAB CT
* Fixes for content type switch
* Use more compatible coding pattern
* Deeper purge of content_type_id
* revert, turns out that did not work
* More content type replacements
* Revert changes to serializer
* Revert another content_type change
* Fix for rearrangement of post_migration methods
* Remove thing I am not going to do
* Revert branch pin that was temporary
* Fix bug where collectstatic could error due to dispatcherd config
* Revert test because it will not work in test suite
* New publish mocking system
* Remove import of unused
* Fix default publish broker
Use dynamic AWX max_workers value
Make basic --status and --running commands work
Make feature flag enabled true by default for development
* [dispatcherd] Dispatcher socket-based `--status` demo working (#15908)
* Fix Task Decorator to Work With and Without Feature Flag (AAP-41775) (#15911)
* refactor(system): extract common heartbeat helpers and split cluster_node_heartbeat
Extract common heartbeat logic into helper functions: _heartbeat_instance_management: consolidates instance management, health checks, and lost-instance detection. _heartbeat_check_versions: compares instance versions and initiates shutdown when necessary. _heartbeat_handle_lost_instances: reaps jobs and marks lost instances offline.
Refactor the original cluster_node_heartbeat to use these helpers and retain legacy behavior (using bind_kwargs).
Introduce adispatch_cluster_node_heartbeat for dispatcherd: uses the control API to retrieve running tasks and reaps them.
Link the two implementations by attaching adispatch_cluster_node_heartbeat as the _new_method on cluster_node_heartbeat.
* feat(publish): delegate heartbeat task submission to new dispatcherd implementation
Update apply_async to check at runtime if FEATURE_NEW_DISPATCHER is enabled.
When the task is cluster_node_heartbeat and a _new_method is attached, delegate the task submission to the new dispatcherd implementation.
Preserve the original behavior for all other tasks and fallback on error.
* refactor(system): extract task ID retrieval from dispatcherd into helper function
Improves readability of adispatch_cluster_node_heartbeat by extracting
the complex UUID parsing logic into a dedicated helper function.
Adds clearer error handling and follows established code patterns.
* fix(dispatcher): Enable task decorator to work with and without feature flag
Implemented a new approach for handling task execution with feature flags
by attaching alternative implementations to apply_async._new_method. This
allows cluster_node_heartbeat to work correctly with both the legacy and
new dispatcher systems without modifying core decorator logic.
AAP-41775
* fix(dispatcher): Improve error handling and logging in feature flag implementation
- Add error handling when attaching alternative dispatcher implementation
- Fix method self-reference in apply_async to properly use cls.apply_async
- Document limitations of this targeted approach for specific tasks
- Add logging for better debugging of dispatcher selection
- Ensure decorator timing by keeping method attachment after function definitions
This completes the robust implementation for switching between dispatcher
implementations based on feature flags.
AAP-41775
* fix(dispatcher): Implement registry pattern for dispatcher feature flag compatibility
Replaces direct method attribute assignment with a global registry for
alternative implementations. The original approach tried to attach new
methods directly to apply_async bound methods, which fails because bound
methods don't support attribute assignment in Python.
The registry pattern:
- Creates a global ALTERNATIVE_TASK_IMPLEMENTATIONS dict in publish.py
- Registers alternative implementations by task name
- Modifies apply_async to check the registry when feature flag is enabled
- Adds extensive logging throughout the process for debugging
This enables cluster_node_heartbeat to work correctly with both the legacy
and new dispatcher implementations based on the FEATURE_NEW_DISPATCHER flag.
AAP-41775
* refactor(dispatcher): Remove excessive logging from dispatcher implementation
Reduces verbose debugging logs while maintaining essential logging for critical
operations. Preserves:
- Task implementation selection based on feature flag
- Registration success/failure messages
- Critical error reporting
Removed:
- Registry content debugging messages
- Repetitive task diagnostics
- Non-essential information logging
AAP-41775
* fix(dispatcher): Fix shallow copy in dispatcher schedule conversion
This resolves "AttributeError: 'float' object has no attribute 'total_seconds'"
errors when the dispatcher is restarted.
Refs: AAP-41775
* Use IPC mechanism to get running tasks (#15926)
* Allow tasks from tasks
* Fix failure to limit to waiting jobs
* Get job record with lock
* Fix failures in dispatcherd feature branch (#15930)
* Fully handle DispatcherCancel
* Complete rest of preload import work
* Complete dispatcherd integration & job cancellation (AAP-43033) (#15941)
* feat(dispatcher): Implement job cancellation for new dispatcher
Adds feature-flag-aware job cancellation that routes cancel requests to either
the legacy dispatcher or the new dispatcherd library based on the
FEATURE_NEW_DISPATCHER flag.
- Updates cancel_dispatcher_process() to use dispatcherd's control API when enabled
- Handles both direct cancellation and task manager workflow cancellation cases
- Works with DispatcherCancel exception handling to properly handle SIGUSR1 signals
AAP-43033
* fix(dispatcher): Update run_dispatcher.py to properly handle task cancellation
Modifies the cancel command in run_dispatcher.py to properly cancel tasks
when the FEATURE_NEW_DISPATCHER flag is enabled, rather than just listing
running tasks.
The implementation translates each task UUID to the appropriate
filter format expected by the dispatcherd control API, maintaining the same
behavior as the original implementation.
Part of: AAP-43033
* refactor(system): Refactor dispatch_startup() to extract common startup logic and branch based on feature flag
This commit refactors the dispatch_startup() function to improve clarity and consistency across the legacy
and new dispatcherd flows.
No dispatcher-specific functionality is needed beyond the changes made, so this refactoring improves robustness without
altering core behavior.
* refactor(system): Refactor inform_cluster_of_shutdown() for clarity
* refactor(tasks): Replace @task with @task_awx across 22 tasks for dispatcher compatibility
- Migrated all task decorators to use @task_awx, ensuring dispatcher-aware behavior.
- Tested each task with the new dispatcherd, verifying that tasks using the registry pattern execute correctly without needing binder‐based alternative implementations.
- Removed redundant logging and outdated comments.
- Legacy tasks that do not require special parameter extraction continue to use their original logic.
- This commit reflects our complete journey of testing and verifying dispatcherd compatibility across all 22 tasks.
* refactor(publish): fix linter
* Fix bug from the branch rebase
* AAP-43763 Add tests for connection management in dispatcherd workers (#15949)
* Add test for job cancel in live tests
* Fix bug from the branch rebase
* Add test for connection recovery after connection broke
* Add test for breaking connection
* Fix dispatcherd bugs: schedule aliases, job kwargs handling, cancel handling (#15960)
* Put in job kwargs handling, not done before
* AAP-44382 [dispatcherd] Fixes for running with feature flag off (#15973)
* Use correct decorator for test of tasks
* Finalize dispatcherd feature branch (#15975)
* Work dispatcherd into dependency management system
* Use util methods from DAB
* Rename the dispatcherd feature flag, and flip default to not-enabled
* Move to new submit_task method
* Update the location of the sock file
* AAP-44381 Make dispatcherd config loading more lazy (#15979)
* Make dispatcherd config loading more lazy
* Make submission error more obvious
* Fix signal handling gap, hijack SIGUSR1 from dispatcherd (#15983)
* Fix signal handling gap, hijack SIGUSR1 from dispatcherd
* Minor adjustments to dispatcherd status command
* [dispatcherd] Get rid of alternative task registry (#15984)
Get rid of alternative task registry
* Fix deadlock error and other cleanup errors (#15987)
* Move to proper error handling location
---------
Co-authored-by: artem_tiupin <70763601+art-tapin@users.noreply.github.com>
* - add new entry points
- add logic to check what version of the project is running
* remove former discovery method
* update custom_injectors and remove unused import
* fix how we load external creds
* remove stale code to match devel
* fix cloudforms test and move credential loading
* add load credentials method to get tests passing
* Conditionalize integration tests if the cred is present
* remove inventory source test
* inventory source is covered in the workflow job template target
* Remove oauth provider
This removes the oauth provider functionality from awx. The
oauth2_provider app and all references to it have been removed.
Migrations to delete the two tables that locally overwrote
oauth2_provider tables are included. This change does not include
migrations to delete the tables provided by the oauth2_provider app.
Also not included here are changes to awxkit, awx_collection or the ui.
* Fix linters
* Update migrations after rebase
* Update collection tests for auth changes
The changes in https://github.com/ansible/awx/pull/15554 will cause a
few collection tests to fail, depending on what the test configuration
is. This changes the tests to look for a specific warning rather than
counting the number of warnings emitted.
* Update migration
* Removed unused oauth_scopes references
---------
Co-authored-by: Mike Graves <mgraves@redhat.com>
Co-authored-by: Alan Rominger <arominge@redhat.com>
In essence, this configures Python to turn any warnings emitted in
runtime into errors[[1]]. This is the best practice that allows
reacting to future deprecation announcements that are coming from the
dependencies (direct, or transitive, or even CPython itself)[[2]].
The typical workflow looks like this:
1. If a dependency is updated an a warning is hit in tests, the
deprecated thing should be replaced with newer APIs.
2. If a dependency is transitive or we have no control over it
otherwise, the specific warning and a regex matching its message,
plus the module reference (where possible) can be added to the
list of temporary ignores in `pytest.ini`.
3. The list of temporary ignores should be reevaluated periodically,
including when dependency re-pinning in lockfile is happening.
[1]: https://docs.python.org/3/using/cmdline.html#cmdoption-W
[2]: https://pytest-with-eric.com/configuration/pytest-ignore-warnings/
Adds new modules for CRUD operations on the
following endpoints:
- api/v2/role_definitions
- api/v2/role_user_assignments
- api/v2/role_team_assignments
Note: assignment is Create or Delete only
Additional changes:
- Currently DAB endpoints do not have "type"
field on the resource list items. So this modifies
the create_or_update_if_needed to allow manually
specifying item type.
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
Develop ability to list permissions for existing roles
Create a model registry for RBAC-tracked models
Write the data migration logic for creating
the preloaded role definitions
Write migration to migrate old Role into ObjectRole model
This loops over the old Role model, knowing it is unique
on object and role_field
Most of the logic is concerned with identifying the
needed permissions, and then corresponding role definition
As needed, object roles are created and users then teams
are assigned
Write re-computation of cache logic for teams
and then for object role permissions
Migrate new RBAC internals to ansible_base
Migrate tests to ansible_base
Implement solution for visible_roles
Expose URLs for DAB RBAC
After removing CRUD from receptor addresses, we need
to remove the module.
- remove receptor_address module
- Add listener_port to instance module
- Add peers_from_control_nodes to instance module
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
- Add receptor_address module which allows
users to create addresses for instances
- Update awx_collection functional and integration
tests to support new peering design
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
* Move awxkit import code into a pytest fixture to better control when
the import happens
* Ensure /awx_devel/awxkit is added to sys path before awxkit import
runs
* Basic export tests
* Added test that highlights a problem with running Schedule exports as
non-root user. We rely on the POST key in the OPTIONS response to
determine the fields to export for a resource. The POST key is not
present if a user does NOT have create privileges.
* Fixed up forwarding all headers from the API server back to the test
code. This was causing a problem in awxkit code that checks for
allowed HTTP Verbs in the headers.
* Adding hosts bulk deletion feature
Signed-off-by: Avi Layani <alayani@redhat.com>
* fix the type of the argument
Signed-off-by: Avi Layani <alayani@redhat.com>
* fixing activity_entry tracking
Signed-off-by: Avi Layani <alayani@redhat.com>
* Revert "fixing activity_entry tracking"
This reverts commit c8eab52c2ccc5abe215d56d1704ba1157e5fbbd0.
Since the bulk_delete is not related to an inventory, only hosts which
can be from different inventories.
* get only needed vars to reduce memory consumption
Signed-off-by: Avi Layani <alayani@redhat.com>
* filtering the data to reduce memory increase the number of queries
Signed-off-by: Avi Layani <alayani@redhat.com>
* update the activity stream for inventories
Signed-off-by: Avi Layani <alayani@redhat.com>
* fix the changes dict initialiazation
Signed-off-by: Avi Layani <alayani@redhat.com>
---------
Signed-off-by: Avi Layani <alayani@redhat.com>
Add hop node support to awx collections
- add peers and peers_from_control_nodes fields
- show new node_type "hop"
- add tests for adding hop nodes via collections
Co-authored-by: Seth Foster <fosterseth@users.noreply.github.com>
* Use separate module for test settings
* Further refine some pre-existing comments in settings
* Add CACHES to setting snapshot exceptions to accommodate changed load order
