* AAP-83773 — Use unfiltered count for activity stream pagination
Forward-port of tower#7606 (stable-2.6).
The RBAC-filtered COUNT(*) on activity_stream takes ~36 min per call
on large tables (713K rows) due to the pk__in subquery shape introduced
by the AAP-81860 LEFT JOIN fix. Replace with an unfiltered table count
for pagination -- an approximate over-count is harmless for UI page
navigation while the actual page results remain RBAC-filtered.
* AAP-83773 — Add unit tests for ActivityStreamPagination
Cover ActivityStreamPaginator and ActivityStreamPagination to satisfy
SonarCloud coverage gate on new lines in awx/api/pagination.py.
* AAP-83773 — Save/restore django_paginator_class in base Pagination
Address review feedback from lallen92: use a save/restore pattern in
Pagination.paginate_queryset() so subclasses only need to set the class
attribute. Remove the now-redundant paginate_queryset override from
ActivityStreamPagination.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Liam Allen <lallen@redhat.com>
* AAP-84057: Set PostgreSQL statement_timeout on web worker DB connections
When uwsgi's harakiri kills a worker, the PostgreSQL backend continues
running the query indefinitely. These abandoned queries accumulate and
create resource contention for all other queries.
Add a connection_created signal handler that sets statement_timeout on
new DB connections. Under uwsgi, the timeout is auto-derived from the
harakiri value (minus 5s margin so PostgreSQL cancels the query before
uwsgi kills the worker). Outside uwsgi (task workers, migrations),
no timeout is applied. A manual DATABASE_STATEMENT_TIMEOUT setting
is available as a fallback for non-uwsgi deployments.
* Remove timeout value caching because it brings no significant gains
* Use proportional margin between statement_timeout and harakiri timeout
* Fix zero-harakiri test to patch fake uwsgi module instead of None
* Refactor statement_timeout from signal handler to connection string
Move statement_timeout configuration from a connection_created signal
handler (extra SQL round-trip per connection) to a dynaconf merge
function that sets it via the libpq OPTIONS connection string parameter.
This mirrors the existing merge_application_name() pattern and
eliminates the SET statement on every new connection.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The TeamAccess.filtered_queryset() method combined two access paths
(org membership and direct read permission) using Q(…) | Q(…), which
PostgreSQL could not optimize — it scanned all teams for each OR
branch. Replacing the OR with UNION lets each branch use the
RoleEvaluation 3-column index independently. The UNION result is
wrapped in pk__in= so the outer queryset remains compatible with
BaseAccess.get_queryset()'s select_related() call.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* AAP-82745 — Replace O(N) correlated subqueries in org list with bulk aggregation
The org list endpoint computed per-org user and admin counts using two
correlated subqueries against roleuserassignment — one per org per role
type. At scale this was the #1 worst query in the customer DB (677ms
mean, 336K calls, 3,801 min total DB time).
Replace with a single flat conditional-aggregation query that scans
roleuserassignment once regardless of org count. Also simplify the
detail view to use direct .count() calls and add a query count
regression test.
* Remove dead code: org_counts is always populated after refactor
* Address review feedback on detail view and test
- Detail view: combine two .count() calls into single .aggregate()
with conditional counts, matching the list-view pattern
- Use .update() in else branch for consistency with if branch
- Remove redundant setup_managed_roles fixture (transitive via
organization_resource_creator)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Replace legacy Role M2M traversal in UserAccess.filtered_queryset()
with DAB's visible_users(), which queries organizational membership
through RoleUserAssignment subqueries instead of LEFT OUTER JOINs
through the objectrole/roleevaluation tables.
Also replace role__in=actor.has_roles.all() in can_admin() with
RoleEvaluation._actor_role_filter() to avoid the objectrole JOIN
in per-object permission checks.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* AAP-81860 — Eliminate LEFT OUTER JOINs in ActivityStream RBAC query
ActivityStreamAccess.filtered_queryset() used field-traversal Q objects
(e.g. Q(host__inventory__in=...)) across 18 M2M relationships, which
Django translates into LEFT OUTER JOINs. This forces PostgreSQL to join
all intermediary tables before filtering, making it the #2 DB consumer
at 483s in a 10-minute Scale Lab window.
Replace all M2M field traversals with pk__in subqueries using .through
intermediary tables, following the same pattern proven in AAP-81082
(28% improvement for unified job RBAC). This changes the SQL from
LEFT OUTER JOIN to IN (SELECT ...) semi-joins, allowing PostgreSQL to
skip the unconditional joins. Also:
- Remove .distinct() (no longer needed without M2M JOINs)
- Remove unnecessary if-truthy checks that evaluated subqueries as
boolean before including them (5 extra DB roundtrips per request)
- Migrate accessible_pk_qs(user, 'read_role') to access_ids_qs(user,
'view') for direct DAB RBAC API usage
* Use .exists() instead of queryset truth-test for auditing_orgs guard
Avoids evaluating the full queryset when checking whether the user has
auditing orgs — .exists() issues a cheaper SELECT 1 … LIMIT 1 instead.
* Restore .exists() guards for resource-type Q branches
Re-add conditional guards around inventory, credential, project,
job template, workflow job template, and team Q branches. Uses
.exists() (cheap SELECT 1 LIMIT 1) instead of the old queryset
truth-test to avoid unnecessary query evaluation while still
preventing generation of an overly complex query when the user
has no access to a given resource type.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
AAP-81517 — Optimize HostList API: conditional DISTINCT + composite index
Make .distinct() conditional on host_filter being set — without it the
RBAC IN subquery on a direct FK cannot produce duplicates, so DISTINCT
is pure overhead. Add composite index (host_id, id DESC) on
main_jobhostsummary so the with_latest_summary_id() correlated subquery
can use an index-only top-1 scan instead of scanning and sorting.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* AAP-81082 Eliminate LEFT OUTER JOINs in unified job RBAC query
UnifiedJobAccess.filtered_queryset() used field-traversal Q objects
(inventoryupdate__inventory_source__inventory__id__in=...) which Django
translates into LEFT OUTER JOINs. This forces PostgreSQL to scan all
rows in main_unifiedjob before filtering — at scale, 99.84% are
discarded. Replace with pk__in subqueries that generate IN (SELECT ...)
instead, allowing PostgreSQL to skip the unconditional join. EXPLAIN
ANALYZE shows a 28% reduction in execution time (496ms -> 355ms), with
larger gains expected under concurrent load.
* AAP-81082 Eliminate LEFT OUTER JOINs in RBAC filtered_queryset methods
Replace field-traversal Q objects with pk__in subqueries across all Access
classes that query polymorphic or M2M tables, preventing Django from
generating unconditional LEFT OUTER JOINs. Also migrate legacy
_accessible_pk_qs / accessible_pk_qs calls to DAB RBAC access_ids_qs.
Affected: UnifiedJobAccess, UnifiedJobTemplateAccess, JobAccess,
JobEventAccess, LabelAccess.
* Fix docstring
* Use _actor_role_filter() in UnifiedJobTemplate.accessible_pk_qs()
Replace `role__in=accessor.has_roles.all()` with the optimized `_actor_role_filter()` subquery pattern from django-ansible-base.
The old pattern causes a 3-table JOIN through RoleUserAssignment -> ObjectRole -> RoleEvaluation on every non-superuser request. _actor_role_filter() skips the ObjectRole table entirely by using a direct subquery on RoleUserAssignment.object_role_id, eliminating the intermediate JOIN and reducing query time for /api/v2/unified_jobs/ requests by non-superusers.
* Fix cartesian product in organization user/admin count queries
The organizations list and detail endpoints annotated each org with user and admin counts using two Count() calls that traverse the Role.members M2M. Django generated two LEFT JOINs on the same through table, crossing every member row with every admin row before COUNT(DISTINCT) reduced the product.
At scale (2,617 members × 46,233 admins) this produced 120M intermediate rows and 96-second query times, causing 504 timeouts.
Replace with independent Subquery expressions that each query main_rbac_roles_members separately - no cross product.
Fixes: AAP-72817
Fixes: AAP-72480
* Fix variable names which do not meet coding standards
* Fix formatting inconsistency in organization detail subquery annotation
Break the long .annotate() line across multiple lines to match the style used in mixin.py.
* Rewrite org count subqueries to use DAB RBAC models
Replace old RBAC Role.members.through subqueries with
RoleUserAssignment-based correlated subqueries, querying
managed RoleDefinitions ('Organization Member' / 'Organization Admin')
directly. This aligns with the DAB RBAC migration direction and
eliminates dependency on the deprecated ImplicitRoleField M2M tables
for these counts.
Update test fixtures to use RoleDefinition.give_permission() and
add setup_managed_roles where needed.
* Fix collection tests: set up managed role definitions
The DAB RBAC migration to use RoleUserAssignment subqueries in
organization views requires managed role definitions (Organization
Member, Organization Admin) to exist in the test database.
Add an autouse fixture to the collection test conftest that calls
setup_managed_role_definitions() before each test.
* Add setup_managed_roles fixture to functional tests hitting org views
Tests that hit organization list/detail views now require the
setup_managed_roles fixture to pre-create the Organization Member
and Organization Admin RoleDefinition objects used by the DAB RBAC
subqueries.
* Revert setup_managed_roles from ext_auditor tests
The setup_managed_roles fixture conflicts with the ext_auditor_rd
fixture by deleting the Alien Auditor role definition. These tests
don't need it — the defensive view code handles missing role
definitions gracefully.
* Handle missing Organization Member/Admin role definitions gracefully
Use filter().first() instead of get() for RoleDefinition lookups in
organization list and detail views. Returns 0 for user/admin counts
when role definitions are not yet created, preventing 500 errors in
environments where post_migrate signals haven't run.
* Cast OuterRef('pk') to TextField for RoleUserAssignment.object_id comparison
RoleUserAssignment.object_id is a TextField, but OuterRef('pk') on
Organization produces an integer. PostgreSQL strictly rejects text = integer
comparisons. Use Cast() to explicitly convert the PK to text.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* [AAP-78392] Optimize HostManager.active_count() with cache and functional index
active_count() runs a full sequential scan with LOWER()+DISTINCT on main_host for every license check. At customer scale this consumed 74.5 minutes of DB time over 4 hours (47K calls at 93ms avg).
Add a 60-second Redis-backed cache via the existing memoize decorator to reduce call volume by ~99.5%. Add a functional btree index on LOWER(name) to eliminate the sequential scan for the remaining calls.
* Use AddIndexConcurrently instead of AddIndex in the migration for host name lower index
* Revert AddIndexConcurrently to AddIndex for CI compatibility
The api-migrations CI job runs against SQLite which does not support PostgreSQL-specific AddIndexConcurrently. Standard AddIndex works across all backends and the brief write lock during production upgrades is acceptable for this table size.
* Remove functional index, keep cache-only fix per reviewer feedback
Drop the LOWER(name) functional index and migration to minimize
the change footprint.
----
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* [AAP-74343] Decouple installed_collections and ansible_version from indirect node counting flag
The indirect_instance_count callback plugin and its artifact processing
were entirely gated behind FEATURE_INDIRECT_NODE_COUNTING_ENABLED. This
caused installed_collections and ansible_version to remain unpopulated
when the flag was off, even though these are baseline analytics fields
unrelated to indirect host counting.
Always run the callback plugin and persist installed_collections and
ansible_version to the database. Only the indirect-counting-specific
parts (EventQuery creation, event_queries_processed flag, and vendor
collections) remain gated behind the feature flag.
* [AAP-74343] Read callbacks_enabled from ansible.cfg so user-configured callbacks are preserved
The check for 'callbacks_enabled' in config_values was dead code because
read_ansible_config was never asked to read that setting. Now that the
callback registration runs unconditionally, fix this by including
'callbacks_enabled' in the variables of interest.
* [AAP-74343] Use comma delimiter for ANSIBLE_CALLBACKS_ENABLED
Ansible's CALLBACKS_ENABLED config is type list and splits on commas.
The colon delimiter would cause combined callback names to be treated
as a single invalid name.
* [AAP-74343] Add tests for ANSIBLE_CALLBACKS_ENABLED configuration
Verify that indirect_instance_count is always set, user-configured
callbacks from ansible.cfg are preserved, and the comma delimiter
is used as ansible-core expects.
* [AAP-74343] Use public API for namespace package path access
Replace library.__path__._path[0] with library.__path__[0] to avoid
relying on a private CPython implementation detail of _NamespacePath.
* [AAP-74343] Skip host query scanning when indirect counting flag is off
The indirect_instance_count callback plugin now checks AWX_COLLECT_HOST_QUERIES
to decide whether to scan for host query files. When the feature flag is off,
the plugin only collects collection metadata (name + version) and ansible_version,
skipping the expensive embedded/external query file discovery.
* [AAP-74343] Set AWX_COLLECT_HOST_QUERIES in query discovery tests
The TestExternalQueryDiscovery tests exercise the host query scanning
path, which now requires AWX_COLLECT_HOST_QUERIES=1 in the environment.
* [AAP-74343] Use Ansible plugin config system for collect_host_queries
Declare collect_host_queries as a formal plugin option in DOCUMENTATION
with env var AWX_COLLECT_HOST_QUERIES, replacing the raw os.getenv() call
with self.get_option(). This follows the standard Ansible plugin
configuration pattern.
* [AAP-74343] Add test for disabled collect_host_queries path
Verify that when collect_host_queries is false, the plugin still
enumerates collections for metadata but skips host query file scanning.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* AAP-41742: Fix workflow node update failing when JT has unprompted labels
PATCH extra_data on a workflow node fails with
{"labels":["Field is not configured to prompt on launch."]}
when the node has labels associated but the JT has
ask_labels_on_launch=False.
The serializer was passing all persisted M2M state from prompts_dict()
to _accept_or_ignore_job_kwargs() on every PATCH, re-validating
unchanged fields. Fix scopes validation to only the fields in the
request; full re-validation still occurs when unified_job_template
is being changed.
* Capture attrs keys before _build_mock_obj mutates them
_build_mock_obj() pops pseudo-fields (limit, scm_branch, job_tags,
etc.) from attrs. Computing requested_prompt_fields after the pop
would miss those fields and skip their ask_on_launch validation.
* Include survey_passwords when validating extra_vars prompts
prompts_dict() emits survey_passwords alongside extra_vars.
_accept_or_ignore_job_kwargs uses it to decrypt encrypted survey
values before validation. Without it, encrypted password blobs
are validated as-is against the survey spec.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Only remove the collection directory the fixture created
(redhat/indirect_accounting) instead of the entire
/var/lib/awx/vendor_collections/ root, so we don't accidentally
delete vendor collections that may have been installed by the
build process.
Forward-port of ansible/tower#7350.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Delete existing all-group vars on inventory sync (with overwrite-vars=True) instead of merging them.
* Implementation of inv var handling with file as db.
* Improve serialization to file of inv vars for src update
* Include inventory-level variable editing into inventory source update handling
* Add group vars to inventory source update handling
* Add support for overwrite_vars to new inventory source handling
* Persist inventory var history in the database instead of a file.
* Remove logging which was needed during development.
* Remove further debugging code and improve comments
* Move special handling for user edits of variables into serializers
* Relate the inventory variable history model to its inventory
* Allow for inventory variables to have the value 'None'
* Fix KeyError in new inventory variable handling
* Add unique-together constraint for new model InventoryGroupVariablesWithHistory
* Use only one special invsrc_id for initial update and manual updates
* Fix internal server error when creating a new inventory
* Print the empty string for a variable with value 'None'
* Fix comment which incorrectly states old behaviour
* Fix inventory_group_variables_update tests which did not take the new handling of None into account
* Allow any type for Ansible-core variable values
* Refactor misleading method names
* Fix internal server error when savig vars from group form
* Remove superfluous json conversion in front of JSONField
* Call variable update from create/update instead from validate
* Use group_id instead of group_name in model InventoryGroupVariablesWithHistory
* Disable new variable update handling for all regular (non-'all') groups
* Add live test to verify AAP-17690 (inv var deleted from source)
* Add functional tests to verify inventory variables update logic
* Fix migration which was corrupted by a rebase
* Add a more complex live test and resolve linter complaints
* Force overwrite_vars=False for updates from source on all-group
* Change behavior with respect to overwrite_vars
* Delete existing all-group vars on inventory sync (with overwrite-vars=True) instead of merging them.
* Implementation of inv var handling with file as db.
* Improve serialization to file of inv vars for src update
* Include inventory-level variable editing into inventory source update handling
* Add group vars to inventory source update handling
* Add support for overwrite_vars to new inventory source handling
* Persist inventory var history in the database instead of a file.
* Remove logging which was needed during development.
* Remove further debugging code and improve comments
* Move special handling for user edits of variables into serializers
* Relate the inventory variable history model to its inventory
* Allow for inventory variables to have the value 'None'
* Fix KeyError in new inventory variable handling
* Add unique-together constraint for new model InventoryGroupVariablesWithHistory
* Use only one special invsrc_id for initial update and manual updates
* Fix internal server error when creating a new inventory
* Print the empty string for a variable with value 'None'
* Fix comment which incorrectly states old behaviour
* Fix inventory_group_variables_update tests which did not take the new handling of None into account
* Allow any type for Ansible-core variable values
* Refactor misleading method names
* Fix internal server error when savig vars from group form
* Remove superfluous json conversion in front of JSONField
* Call variable update from create/update instead from validate
* Use group_id instead of group_name in model InventoryGroupVariablesWithHistory
* Disable new variable update handling for all regular (non-'all') groups
* Add live test to verify AAP-17690 (inv var deleted from source)
* Add functional tests to verify inventory variables update logic
* Fix migration which was corrupted by a rebase
* Add a more complex live test and resolve linter complaints
* Force overwrite_vars=False for updates from source on all-group
* Change behavior with respect to overwrite_vars
* Move call to django_validate_password to the correct method were the user object is available.
* Added tests for the Django password validation functionality.
* Move call to django_validate_password to the correct method were the user object is available.
* Added tests for the Django password validation functionality.
