* AAP-81173 — Replace 4-way OR with UNION in unified job list RBAC query
The UnifiedJobAccess.filtered_queryset() method used a 4-way OR to
determine which unified jobs a user can see. Under load, the OR forces
PostgreSQL to evaluate all four branches in a single plan, preventing
branch-specific index optimization and causing 35 hours of DB time per
30-minute Scale Lab window.
Split each RBAC branch (template read_role, inventory update, ad-hoc
command, org auditor) into separate querysets combined with UNION, giving
the planner an independent optimal plan per branch. The UNION result is
wrapped in pk__in= for compatibility with BaseAccess.get_queryset()
prefetch_related and the workflowapproval filter.
This follows the same pattern proven in AAP-83319 (team list UNION fix).
* AAP-81173 — Add UnifiedJobPagination to prevent COUNT regression
The pk__in UNION pattern used for RBAC filtering forces the large
outer table as the driving table for COUNT(*), requiring PostgreSQL
to materialize all subquery result sets. On large deployments this
produces catastrophic query times (see AAP-83773 for the identical
issue on activity_stream).
Override the paginator count to use an unfiltered
UnifiedJob.objects.count() — the over-count is acceptable for
pagination UI. Also fix test_unified_job_list_rando_sees_nothing
to assert on results length instead of count, since count is now
unfiltered.
* AAP-81173 — Add unit tests for UnifiedJobPagination coverage
Cover the UnifiedJobPaginator.count cached property and the
count_disabled branch in UnifiedJobPagination.paginate_queryset
to satisfy SonarCloud's 90% new-code coverage gate.
* AAP-81173 — Address review feedback: save/restore paginator class, format consistency
- Fix Pagination.paginate_queryset() to save/restore django_paginator_class
instead of hardcoding DjangoPaginator in the finally block. This lets
subclasses set the class attribute without needing to override the method.
- Remove UnifiedJobPagination.paginate_queryset() override — now only needs
to set django_paginator_class = UnifiedJobPaginator as a class attribute.
- Format by_org_auditor consistently with the other three UNION branches.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
awxkit has its own tox.ini and when `tox -e linters` runs, awxkit/.tox/
can get populated with a py3 virtualenv. Since flake8 is configured to
scan the `awxkit` directory, it recurses into
awxkit/.tox/py3/lib/python3.12/site-packages/ and reports hundreds of
false positives (F405, E265, E266) from third-party packages like PyYAML.
Adding .tox to the flake8 exclude list prevents this — matching the
existing exclusion of `env` for virtualenvs.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
dict() with keyword arguments is idiomatic for Ansible module
argument_spec definitions. Suppress the "use literal instead of
constructor" rule (python:S7498) for awx_collection/plugins/modules/.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Jake Jackson <jljacks93@gmail.com>
* fix: prefer scm_revision as the cache_id if available
* add tests
* fix: make sure galaxy requirements are always updated when a new revision is pulled
* make sure existing project cache id is used when available
---------
Co-authored-by: Liam Allen <lallen@redhat.com>
* AAP-83773 — Use unfiltered count for activity stream pagination
Forward-port of tower#7606 (stable-2.6).
The RBAC-filtered COUNT(*) on activity_stream takes ~36 min per call
on large tables (713K rows) due to the pk__in subquery shape introduced
by the AAP-81860 LEFT JOIN fix. Replace with an unfiltered table count
for pagination -- an approximate over-count is harmless for UI page
navigation while the actual page results remain RBAC-filtered.
* AAP-83773 — Add unit tests for ActivityStreamPagination
Cover ActivityStreamPaginator and ActivityStreamPagination to satisfy
SonarCloud coverage gate on new lines in awx/api/pagination.py.
* AAP-83773 — Save/restore django_paginator_class in base Pagination
Address review feedback from lallen92: use a save/restore pattern in
Pagination.paginate_queryset() so subclasses only need to set the class
attribute. Remove the now-redundant paginate_queryset override from
ActivityStreamPagination.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Liam Allen <lallen@redhat.com>
* AAP-84057: Set PostgreSQL statement_timeout on web worker DB connections
When uwsgi's harakiri kills a worker, the PostgreSQL backend continues
running the query indefinitely. These abandoned queries accumulate and
create resource contention for all other queries.
Add a connection_created signal handler that sets statement_timeout on
new DB connections. Under uwsgi, the timeout is auto-derived from the
harakiri value (minus 5s margin so PostgreSQL cancels the query before
uwsgi kills the worker). Outside uwsgi (task workers, migrations),
no timeout is applied. A manual DATABASE_STATEMENT_TIMEOUT setting
is available as a fallback for non-uwsgi deployments.
* Remove timeout value caching because it brings no significant gains
* Use proportional margin between statement_timeout and harakiri timeout
* Fix zero-harakiri test to patch fake uwsgi module instead of None
* Refactor statement_timeout from signal handler to connection string
Move statement_timeout configuration from a connection_created signal
handler (extra SQL round-trip per connection) to a dynaconf merge
function that sets it via the libpq OPTIONS connection string parameter.
This mirrors the existing merge_application_name() pattern and
eliminates the SET statement on every new connection.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Django's cascade collector materializes all JobHostSummary IDs into a
single UPDATE ... IN (...) to SET_NULL on Host.last_job_host_summary.
With many jobs x hosts this exceeds PostgreSQL's max memory allocation
size for a single query (1GB).
Pre-delete JHS rows and clear Host FK references in chunks of 1000 job
IDs using raw SQL before Django's .delete() runs, so the cascade
collector finds nothing to collect.
Assisted-by: Claude Code via Google Vertex AI
The TeamAccess.filtered_queryset() method combined two access paths
(org membership and direct read permission) using Q(…) | Q(…), which
PostgreSQL could not optimize — it scanned all teams for each OR
branch. Replacing the OR with UNION lets each branch use the
RoleEvaluation 3-column index independently. The UNION result is
wrapped in pk__in= so the outer queryset remains compatible with
BaseAccess.get_queryset()'s select_related() call.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
AAP-82221 — Skip reverse sync during post-migrate role definition setup
The dab_post_migrate signal handler added in 64dc097914 calls
setup_managed_role_definitions on every awx-manage migrate run.
During fresh installs this deletes stale managed role definitions
and triggers a reverse sync to the gateway, which returns 423
Locked because migrate_service_data has not finished yet. The
installer retries 5 times but each attempt fires the same signal
and hits the same 423, failing the controller init.
* AAP-76460 — Prioritize REDHAT_CANDLEPIN_HOST over rhsm.conf for subscription validation
When REDHAT_CANDLEPIN_HOST is explicitly configured (Satellite/disconnected
environments), use it as the subscription host instead of falling back to
rhsm.conf. This fixes subscription loading failures in containerized
deployments where rhsm.conf defaults to subscription.rhsm.redhat.com but
the system cannot reach it.
Also guard against double port-appending in get_satellite_subs when the
host URL already includes a port.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Liam Allen <lallen@redhat.com>
* Optimize HostList API: conditional DISTINCT + composite index on JobHostSummary (#16530)
AAP-81517 — Optimize HostList API: conditional DISTINCT + composite index
Make .distinct() conditional on host_filter being set — without it the
RBAC IN subquery on a direct FK cannot produce duplicates, so DISTINCT
is pure overhead. Add composite index (host_id, id DESC) on
main_jobhostsummary so the with_latest_summary_id() correlated subquery
can use an index-only top-1 scan instead of scanning and sorting.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Fix: Increase awx-operator molecule timeout to 20m (#16534)
The awx-operator molecule kind test intermittently times out after
15 minutes on GitHub Actions runners, causing flaky CI failures.
Bump the bash-level timeout from 15m to 20m (step-level
timeout-minutes: 60 is unchanged).
Closes: AAP-81583
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* AAP-81860 — Eliminate LEFT OUTER JOINs in ActivityStream RBAC query (#16533)
* AAP-81860 — Eliminate LEFT OUTER JOINs in ActivityStream RBAC query
ActivityStreamAccess.filtered_queryset() used field-traversal Q objects
(e.g. Q(host__inventory__in=...)) across 18 M2M relationships, which
Django translates into LEFT OUTER JOINs. This forces PostgreSQL to join
all intermediary tables before filtering, making it the #2 DB consumer
at 483s in a 10-minute Scale Lab window.
Replace all M2M field traversals with pk__in subqueries using .through
intermediary tables, following the same pattern proven in AAP-81082
(28% improvement for unified job RBAC). This changes the SQL from
LEFT OUTER JOIN to IN (SELECT ...) semi-joins, allowing PostgreSQL to
skip the unconditional joins. Also:
- Remove .distinct() (no longer needed without M2M JOINs)
- Remove unnecessary if-truthy checks that evaluated subqueries as
boolean before including them (5 extra DB roundtrips per request)
- Migrate accessible_pk_qs(user, 'read_role') to access_ids_qs(user,
'view') for direct DAB RBAC API usage
* Use .exists() instead of queryset truth-test for auditing_orgs guard
Avoids evaluating the full queryset when checking whether the user has
auditing orgs — .exists() issues a cheaper SELECT 1 … LIMIT 1 instead.
* Restore .exists() guards for resource-type Q branches
Re-add conditional guards around inventory, credential, project,
job template, workflow job template, and team Q branches. Uses
.exists() (cheap SELECT 1 LIMIT 1) instead of the old queryset
truth-test to avoid unnecessary query evaluation while still
preventing generation of an overly complex query when the user
has no access to a given resource type.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* chore: Upgrade kubernetes package to 36.0.2
* feat: remove extra_vars from jobs and unified_jobs list endpoint. Add… (#16461)
feat: add exclude query parameter to unified_jobs and jobs list endpoints
Allow clients to exclude heavy fields (artifacts, extra_vars) from list
responses via ?exclude=artifacts,extra_vars. These fields are now included
by default instead of being stripped.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* AAP-80457 - Fix credential_input_source to handle state: absent when target crede… (#16523)
Fix credential_input_source to handle state: absent when target credential is missing
The credential_input_source module would error when trying to delete a
credential input source (state: absent) if the target credential didn't
exist. This breaks idempotent playbook runs where cleanup tasks assume
resources may already be gone.
The fix only applies to state: absent; state: present still correctly
fails when the target credential doesn't exist.
Co-authored-by: Liam Allen <lallen@redhat.com>
* Add additional test coverage
* get_satellite_subs() reads verify from rhsm.conf, never from REDHAT_CANDLEPIN_VERIFY.
* Address coderabbit feedback
---------
Signed-off-by: Liam Allen <lallen@redhat.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Dirk Julich <djulich@redhat.com>
Co-authored-by: Rodrigo Toshiaki Horie <rodrigo.horie@hotmail.com>
Co-authored-by: Adrià Sala <22398818+adrisala@users.noreply.github.com>
Co-authored-by: Peter Braun <pbraun@redhat.com>
Co-authored-by: Nick Meyer <nick.a.meyer@icloud.com>
* Balance controller node selection during capacity ties
When multiple control-plane instances have equal remaining capacity,
the instance selection algorithm always picks the first instance in
iteration order. This causes burst workloads to concentrate all job
management on a single controller pod while others remain idle.
Add a tie-breaking criterion: when would_be_remaining capacity is
equal, prefer the instance with fewer jobs_running. This distributes
the control overhead (event processing, callbacks, output streaming)
more evenly across available controller pods during burst scenarios.
The change is backwards-compatible: when capacity clearly differs,
the existing "most remaining capacity" logic dominates unchanged.
Signed-off-by: Alexey Masolov <amasolov@redhat.com>
Signed-off-by: Alexey Masolov <alexey.masolov@gmail.com>
Made-with: Cursor
* Track jobs_running on controller node for container-group tasks
Address review feedback: container-group tasks only set controller_node
(execution_node is empty), so the previous consume_capacity call on the
control path never incremented jobs_running. This meant the tie-breaker
could not differentiate between nodes.
Now pass job_impact=True on the control path when the controller is not
also the execution node (avoiding double-counting for hybrid nodes).
Also improve test coverage:
- Fix test case to prove capacity dominates over jobs_running
- Add dedicated tests for container-group controller distribution
- Add test verifying equal-capacity tie-breaking behaviour
Signed-off-by: Alexey Masolov <amasolov@redhat.com>
Signed-off-by: Alexey Masolov <alexey.masolov@gmail.com>
Made-with: Cursor
* Add docstrings to consume_capacity and fit_task_to_most_remaining_capacity_instance
Address CodeRabbit docstring coverage warning by documenting the two
methods modified in this PR. No functional changes.
Signed-off-by: Alexey Masolov <amasolov@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
---------
Signed-off-by: Alexey Masolov <amasolov@redhat.com>
Signed-off-by: Alexey Masolov <alexey.masolov@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Liam Allen <lallen@redhat.com>
* Optimize HostList API: conditional DISTINCT + composite index on JobHostSummary (#16530)
AAP-81517 — Optimize HostList API: conditional DISTINCT + composite index
Make .distinct() conditional on host_filter being set — without it the
RBAC IN subquery on a direct FK cannot produce duplicates, so DISTINCT
is pure overhead. Add composite index (host_id, id DESC) on
main_jobhostsummary so the with_latest_summary_id() correlated subquery
can use an index-only top-1 scan instead of scanning and sorting.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Fix: Increase awx-operator molecule timeout to 20m (#16534)
The awx-operator molecule kind test intermittently times out after
15 minutes on GitHub Actions runners, causing flaky CI failures.
Bump the bash-level timeout from 15m to 20m (step-level
timeout-minutes: 60 is unchanged).
Closes: AAP-81583
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* AAP-81860 — Eliminate LEFT OUTER JOINs in ActivityStream RBAC query (#16533)
* AAP-81860 — Eliminate LEFT OUTER JOINs in ActivityStream RBAC query
ActivityStreamAccess.filtered_queryset() used field-traversal Q objects
(e.g. Q(host__inventory__in=...)) across 18 M2M relationships, which
Django translates into LEFT OUTER JOINs. This forces PostgreSQL to join
all intermediary tables before filtering, making it the #2 DB consumer
at 483s in a 10-minute Scale Lab window.
Replace all M2M field traversals with pk__in subqueries using .through
intermediary tables, following the same pattern proven in AAP-81082
(28% improvement for unified job RBAC). This changes the SQL from
LEFT OUTER JOIN to IN (SELECT ...) semi-joins, allowing PostgreSQL to
skip the unconditional joins. Also:
- Remove .distinct() (no longer needed without M2M JOINs)
- Remove unnecessary if-truthy checks that evaluated subqueries as
boolean before including them (5 extra DB roundtrips per request)
- Migrate accessible_pk_qs(user, 'read_role') to access_ids_qs(user,
'view') for direct DAB RBAC API usage
* Use .exists() instead of queryset truth-test for auditing_orgs guard
Avoids evaluating the full queryset when checking whether the user has
auditing orgs — .exists() issues a cheaper SELECT 1 … LIMIT 1 instead.
* Restore .exists() guards for resource-type Q branches
Re-add conditional guards around inventory, credential, project,
job template, workflow job template, and team Q branches. Uses
.exists() (cheap SELECT 1 LIMIT 1) instead of the old queryset
truth-test to avoid unnecessary query evaluation while still
preventing generation of an overly complex query when the user
has no access to a given resource type.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* chore: Upgrade kubernetes package to 36.0.2
* feat: remove extra_vars from jobs and unified_jobs list endpoint. Add… (#16461)
feat: add exclude query parameter to unified_jobs and jobs list endpoints
Allow clients to exclude heavy fields (artifacts, extra_vars) from list
responses via ?exclude=artifacts,extra_vars. These fields are now included
by default instead of being stripped.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* AAP-80457 - Fix credential_input_source to handle state: absent when target crede… (#16523)
Fix credential_input_source to handle state: absent when target credential is missing
The credential_input_source module would error when trying to delete a
credential input source (state: absent) if the target credential didn't
exist. This breaks idempotent playbook runs where cleanup tasks assume
resources may already be gone.
The fix only applies to state: absent; state: present still correctly
fails when the target credential doesn't exist.
Co-authored-by: Liam Allen <lallen@redhat.com>
* AAP-83163 — Eliminate LEFT JOIN fan-out in user list RBAC query (#16546)
Replace legacy Role M2M traversal in UserAccess.filtered_queryset()
with DAB's visible_users(), which queries organizational membership
through RoleUserAssignment subqueries instead of LEFT OUTER JOINs
through the objectrole/roleevaluation tables.
Also replace role__in=actor.has_roles.all() in can_admin() with
RoleEvaluation._actor_role_filter() to avoid the objectrole JOIN
in per-object permission checks.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* alter_skip_tags_to_textfield
* Bump migration number
---------
Co-authored-by: Dirk Julich <djulich@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Rodrigo Toshiaki Horie <rodrigo.horie@hotmail.com>
Co-authored-by: Adrià Sala <22398818+adrisala@users.noreply.github.com>
Co-authored-by: Peter Braun <pbraun@redhat.com>
Co-authored-by: Nick Meyer <nick.a.meyer@icloud.com>
* AAP-82745 — Replace O(N) correlated subqueries in org list with bulk aggregation
The org list endpoint computed per-org user and admin counts using two
correlated subqueries against roleuserassignment — one per org per role
type. At scale this was the #1 worst query in the customer DB (677ms
mean, 336K calls, 3,801 min total DB time).
Replace with a single flat conditional-aggregation query that scans
roleuserassignment once regardless of org count. Also simplify the
detail view to use direct .count() calls and add a query count
regression test.
* Remove dead code: org_counts is always populated after refactor
* Address review feedback on detail view and test
- Detail view: combine two .count() calls into single .aggregate()
with conditional counts, matching the list-view pattern
- Use .update() in else branch for consistency with if branch
- Remove redundant setup_managed_roles fixture (transitive via
organization_resource_creator)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Replace legacy Role M2M traversal in UserAccess.filtered_queryset()
with DAB's visible_users(), which queries organizational membership
through RoleUserAssignment subqueries instead of LEFT OUTER JOINs
through the objectrole/roleevaluation tables.
Also replace role__in=actor.has_roles.all() in can_admin() with
RoleEvaluation._actor_role_filter() to avoid the objectrole JOIN
in per-object permission checks.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Fix credential_input_source to handle state: absent when target credential is missing
The credential_input_source module would error when trying to delete a
credential input source (state: absent) if the target credential didn't
exist. This breaks idempotent playbook runs where cleanup tasks assume
resources may already be gone.
The fix only applies to state: absent; state: present still correctly
fails when the target credential doesn't exist.
Co-authored-by: Liam Allen <lallen@redhat.com>
feat: add exclude query parameter to unified_jobs and jobs list endpoints
Allow clients to exclude heavy fields (artifacts, extra_vars) from list
responses via ?exclude=artifacts,extra_vars. These fields are now included
by default instead of being stripped.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* AAP-81860 — Eliminate LEFT OUTER JOINs in ActivityStream RBAC query
ActivityStreamAccess.filtered_queryset() used field-traversal Q objects
(e.g. Q(host__inventory__in=...)) across 18 M2M relationships, which
Django translates into LEFT OUTER JOINs. This forces PostgreSQL to join
all intermediary tables before filtering, making it the #2 DB consumer
at 483s in a 10-minute Scale Lab window.
Replace all M2M field traversals with pk__in subqueries using .through
intermediary tables, following the same pattern proven in AAP-81082
(28% improvement for unified job RBAC). This changes the SQL from
LEFT OUTER JOIN to IN (SELECT ...) semi-joins, allowing PostgreSQL to
skip the unconditional joins. Also:
- Remove .distinct() (no longer needed without M2M JOINs)
- Remove unnecessary if-truthy checks that evaluated subqueries as
boolean before including them (5 extra DB roundtrips per request)
- Migrate accessible_pk_qs(user, 'read_role') to access_ids_qs(user,
'view') for direct DAB RBAC API usage
* Use .exists() instead of queryset truth-test for auditing_orgs guard
Avoids evaluating the full queryset when checking whether the user has
auditing orgs — .exists() issues a cheaper SELECT 1 … LIMIT 1 instead.
* Restore .exists() guards for resource-type Q branches
Re-add conditional guards around inventory, credential, project,
job template, workflow job template, and team Q branches. Uses
.exists() (cheap SELECT 1 LIMIT 1) instead of the old queryset
truth-test to avoid unnecessary query evaluation while still
preventing generation of an overly complex query when the user
has no access to a given resource type.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The awx-operator molecule kind test intermittently times out after
15 minutes on GitHub Actions runners, causing flaky CI failures.
Bump the bash-level timeout from 15m to 20m (step-level
timeout-minutes: 60 is unchanged).
Closes: AAP-81583
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
AAP-81517 — Optimize HostList API: conditional DISTINCT + composite index
Make .distinct() conditional on host_filter being set — without it the
RBAC IN subquery on a direct FK cannot produce duplicates, so DISTINCT
is pure overhead. Add composite index (host_id, id DESC) on
main_jobhostsummary so the with_latest_summary_id() correlated subquery
can use an index-only top-1 scan instead of scanning and sorting.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Replace hardcoded credentials in subscriptions module examples
The EXAMPLES block contained values that looked like real client
credentials. Replace them with obvious placeholders and add a
username/password example.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* AAP-81082 Eliminate LEFT OUTER JOINs in unified job RBAC query
UnifiedJobAccess.filtered_queryset() used field-traversal Q objects
(inventoryupdate__inventory_source__inventory__id__in=...) which Django
translates into LEFT OUTER JOINs. This forces PostgreSQL to scan all
rows in main_unifiedjob before filtering — at scale, 99.84% are
discarded. Replace with pk__in subqueries that generate IN (SELECT ...)
instead, allowing PostgreSQL to skip the unconditional join. EXPLAIN
ANALYZE shows a 28% reduction in execution time (496ms -> 355ms), with
larger gains expected under concurrent load.
* AAP-81082 Eliminate LEFT OUTER JOINs in RBAC filtered_queryset methods
Replace field-traversal Q objects with pk__in subqueries across all Access
classes that query polymorphic or M2M tables, preventing Django from
generating unconditional LEFT OUTER JOINs. Also migrate legacy
_accessible_pk_qs / accessible_pk_qs calls to DAB RBAC access_ids_qs.
Affected: UnifiedJobAccess, UnifiedJobTemplateAccess, JobAccess,
JobEventAccess, LabelAccess.
* Fix docstring
* Refactor result_stdout_raw_handle to use parameterized COPY SQL.
Replace f-string SQL construction with psycopg.sql composables and bound
parameters so security scans no longer flag formatted raw SQL in the
unified jobs stdout path.
Fix sqlite_copy mock rendering for psycopg3 SQL composables.
* Fix sqlite_copy mock without psycopg SQL internals.
Load stdout from the first populated event table instead of rendering
psycopg composables, which use version-specific private attributes.
* Use sql.Literal in COPY query for Django cursor.copy compatibility.
Django's cursor.copy() does not forward bind parameters to psycopg,
which caused stdout API 500s against real PostgreSQL.
* Fix SonarCloud security rating by removing user-controlled data from sqlite filepath
Replace os.path.basename(sys.argv[0]) with a hardcoded 'unknown' fallback
in RecordedQueryLog.write() to eliminate path injection via CLI arguments.
This resolves SonarCloud rule pythonsecurity:S8706 and helps restore the
AWX security rating from C to A.
Closes: AAP-80006
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Remove unused sys import from test_db.py
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Forward-port of ansible/tower#7537 for the devel branch.
When running awxkit's tox tests, pytest picks up the root pytest.ini
which pulls in pytest-django options (--reuse-db, --nomigrations,
DJANGO_SETTINGS_MODULE) and Django-specific filterwarnings. Since the
awxkit tox environment does not install Django or pytest-django, these
cause test collection to fail.
The root cause is that pytest.ini has absolute priority in pytest's
config discovery — it searches all ancestor directories for pytest.ini
before falling back to tox.ini's [pytest] section. A [pytest] section
in awxkit/tox.ini alone cannot prevent the root config from being used.
Fix by:
- Adding awxkit/pytest.ini to act as the primary config boundary
(pytest.ini has the highest priority in config discovery, so its
presence in awxkit/ stops the upward search before reaching root)
- Adding explicit `test` path argument to the pytest command in
awxkit/tox.ini so pytest discovers tests correctly
- Adding `testpaths` and `python_files` to the [pytest] section in
awxkit/tox.ini as a secondary config boundary
- Adding awxkit/conftest.py that registers the Django-specific CLI
options and INI keys as harmless no-ops, as a further safety net
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Fix: handle cursor-paginated API responses in get_one()
DAB PR #1025 switched role_team_assignments and role_user_assignments
endpoints from PageNumberPagination to CursorPagination. Cursor
pagination returns {results, next, previous} without a count field,
causing get_one() to fail with "The endpoint did not provide count
and results".
When the response includes results but no count, infer count from
len(results). Also guard get_all_endpoint() against missing count.
Instances reporting cpu=0 or memory=0 with no errors would be
transitioned to READY state. Treat zero cpu/memory as an error
so the node stays offline until a valid health check is received.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat(analytics): ANSTRAT-2268 log request_id, account_number, org_id from ingress API response
Log the ingress API success response fields (request_id, account_number,
org_id) in the controller task log when gather_analytics tarballs are
uploaded. This enables support engineers to trace uploads through to
Kibana without source code modifications.
* style(analytics): use f-strings in _log_shipping_response to match codebase conventions
* style(analytics): fix black formatting in test assertions
* Use _actor_role_filter() in UnifiedJobTemplate.accessible_pk_qs()
Replace `role__in=accessor.has_roles.all()` with the optimized `_actor_role_filter()` subquery pattern from django-ansible-base.
The old pattern causes a 3-table JOIN through RoleUserAssignment -> ObjectRole -> RoleEvaluation on every non-superuser request. _actor_role_filter() skips the ObjectRole table entirely by using a direct subquery on RoleUserAssignment.object_role_id, eliminating the intermediate JOIN and reducing query time for /api/v2/unified_jobs/ requests by non-superusers.
fix: use GPG-signed commits in spec sync workflow
Switch from unsigned GitHub API commits to GPG-signed git commits
using the aap-api-bot GPG key (OPENAPI_SPEC_SYNC_GPG_PRIVATE_KEY).
The aap-openapi-specs repo requires signed commits via org ruleset.
The previous API-based approach didn't sign commits because GitHub
only auto-signs API commits for GitHub App tokens, not user PATs.
This matches the pattern used by EDA and Gateway teams for their
spec sync workflows.
Also fixes template injection risk by using env vars instead of
direct ${{ }} expansion in shell context.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Fix cartesian product in organization user/admin count queries
The organizations list and detail endpoints annotated each org with user and admin counts using two Count() calls that traverse the Role.members M2M. Django generated two LEFT JOINs on the same through table, crossing every member row with every admin row before COUNT(DISTINCT) reduced the product.
At scale (2,617 members × 46,233 admins) this produced 120M intermediate rows and 96-second query times, causing 504 timeouts.
Replace with independent Subquery expressions that each query main_rbac_roles_members separately - no cross product.
Fixes: AAP-72817
Fixes: AAP-72480
* Fix variable names which do not meet coding standards
* Fix formatting inconsistency in organization detail subquery annotation
Break the long .annotate() line across multiple lines to match the style used in mixin.py.
* Rewrite org count subqueries to use DAB RBAC models
Replace old RBAC Role.members.through subqueries with
RoleUserAssignment-based correlated subqueries, querying
managed RoleDefinitions ('Organization Member' / 'Organization Admin')
directly. This aligns with the DAB RBAC migration direction and
eliminates dependency on the deprecated ImplicitRoleField M2M tables
for these counts.
Update test fixtures to use RoleDefinition.give_permission() and
add setup_managed_roles where needed.
* Fix collection tests: set up managed role definitions
The DAB RBAC migration to use RoleUserAssignment subqueries in
organization views requires managed role definitions (Organization
Member, Organization Admin) to exist in the test database.
Add an autouse fixture to the collection test conftest that calls
setup_managed_role_definitions() before each test.
* Add setup_managed_roles fixture to functional tests hitting org views
Tests that hit organization list/detail views now require the
setup_managed_roles fixture to pre-create the Organization Member
and Organization Admin RoleDefinition objects used by the DAB RBAC
subqueries.
* Revert setup_managed_roles from ext_auditor tests
The setup_managed_roles fixture conflicts with the ext_auditor_rd
fixture by deleting the Alien Auditor role definition. These tests
don't need it — the defensive view code handles missing role
definitions gracefully.
* Handle missing Organization Member/Admin role definitions gracefully
Use filter().first() instead of get() for RoleDefinition lookups in
organization list and detail views. Returns 0 for user/admin counts
when role definitions are not yet created, preventing 500 errors in
environments where post_migrate signals haven't run.
* Cast OuterRef('pk') to TextField for RoleUserAssignment.object_id comparison
RoleUserAssignment.object_id is a TextField, but OuterRef('pk') on
Organization produces an integer. PostgreSQL strictly rejects text = integer
comparisons. Use Cast() to explicitly convert the PK to text.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* [AAP-78392] Optimize HostManager.active_count() with cache and functional index
active_count() runs a full sequential scan with LOWER()+DISTINCT on main_host for every license check. At customer scale this consumed 74.5 minutes of DB time over 4 hours (47K calls at 93ms avg).
Add a 60-second Redis-backed cache via the existing memoize decorator to reduce call volume by ~99.5%. Add a functional btree index on LOWER(name) to eliminate the sequential scan for the remaining calls.
* Use AddIndexConcurrently instead of AddIndex in the migration for host name lower index
* Revert AddIndexConcurrently to AddIndex for CI compatibility
The api-migrations CI job runs against SQLite which does not support PostgreSQL-specific AddIndexConcurrently. Standard AddIndex works across all backends and the brief write lock during production upgrades is acceptable for this table size.
* Remove functional index, keep cache-only fix per reviewer feedback
Drop the LOWER(name) functional index and migration to minimize
the change footprint.
----
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The aap-openapi-specs repo requires commit signatures via org ruleset.
Switch from git commit+push to the GitHub Git Data API which
automatically signs commits, satisfying the required_signatures rule.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
feat: inject x-ai-description from overlay file during schema generation
Many endpoints have human-readable AI descriptions that were added
downstream in aap-mcp-server (PRs #73 and #119) but never backported
as @extend_schema_if_available decorators. This causes 470 out of 631
x-ai-description entries to be lost every time the spec is regenerated.
Add a JSON overlay file (openapi_ai_descriptions.json) containing the
missing descriptions keyed by operationId, and a drf-spectacular
postprocessing hook that merges them into the generated schema for any
operation that doesn't already have x-ai-description from a decorator.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Restore oauth_token backward compatibility for collection token auth
The aap_token rename (c8981e321e) restored module-level token auth but
left two interfaces from earlier collection releases broken:
- The lookup (controller_api) and inventory (controller) plugins
previously declared an oauth_token option. Add oauth_token as an
alias of aap_token in the auth_plugin doc fragment and in
AUTH_ARGSPEC so query(..., oauth_token=...) and inventory YAML keys
keep working.
- tower_cli.cfg-style config files used an oauth_token key under
[general]; it was silently ignored after the rename, quietly
degrading auth. load_config() now also reads the legacy oauth_token
key and maps it to aap_token, with the new aap_token key winning when
both are present. aap_token remains the canonical attribute used by
_parse_aap_token() and the Bearer header logic.
Also make the test helper compatible with ansible-core 2.21+, which
requires a serialization profile alongside _ANSIBLE_ARGS, and extend
the tests to cover the oauth_token alias and legacy config file key.
No changelog fragment added: awx_collection has no changelogs/
directory on devel.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Document oauth_token alias in module auth doc fragment
The oauth_token alias was added to aap_token in AUTH_ARGSPEC but not to
the module doc fragment, failing the validate-modules sanity check
(undocumented argument alias).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Generalize version references in compat comments
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
The aap_token parameter was added to the collection argspec and docs
in #16025, but nothing consumed it after token auth was removed in
#15623: modules silently ignored the token and fell back to basic
auth, breaking token authentication through the AAP gateway.
Wire it up so requests authenticate with the provided token (e.g. one
issued by the AAP gateway, which validates it and proxies to the
controller):
- Send "Authorization: Bearer <token>" in make_request when aap_token
is set, skipping the basic-auth login probe; basic auth is unchanged
when no token is given
- Accept the token as a string or as the dict set as a fact by the
ansible.platform.token module ({token: ..., id: ...}), which is the
documented cross-collection mint/use/delete workflow
- Restore controller_oauthtoken and tower_oauthtoken as aliases for
back-compat with pre-#15623 playbooks, matching downstream
- Forward aap_token through the controller_api lookup and controller
inventory plugins via short_params, and add the missing
CONTROLLER_OAUTH_TOKEN/TOWER_OAUTH_TOKEN env sources to the plugin
doc fragment (plugins resolve env vars from doc fragments, not
env_fallback); AAP_TOKEN is no longer marked deprecated there
- Support tokens in the awxkit-based export/import modules
- Add unit tests covering the Bearer header for both token forms, the
aliases, the bad-dict failure, and the basic-auth fallback
Verified end-to-end against a live gateway-fronted AAP 2.7 deployment:
modules, the lookup plugin, both aliases, all env sources, dict-form
tokens, job launch/wait, and a clean HTTP 401 on an invalid token.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Move plugin loading to lazy-on-first-access, DB sync to dispatcher
Remove credential type and inventory plugin loading from Django's
app.ready() path. In-memory registries (ManagedCredentialType.registry
and InventorySourceOptions.injectors) are now populated lazily on first
access via LazyLoadDict, a dict subclass that calls a loader function
on the first read operation. This ensures web workers, dispatcher
workers, and management commands all get the registries populated
exactly when needed, without eager loading at startup.
The DB sync (CredentialType.setup_tower_managed_defaults) is moved to
the dispatcher's startup task, where it only needs to run once per
deployment rather than in every Django process.
Co-Authored-By: Alan Rominger <arominge@redhat.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* AAP-65883: Refactor clear_setting_cache to use DAB shared utility
Delegate cache invalidation logic to ansible_base.lib.cache.tasks.clear_cache,
passing AWX-specific dependent key resolution (settings_registry) and
post-invalidation hook (LOG_AGGREGATOR_LEVEL reconfiguration) as callbacks.
Requires: ansible/django-ansible-base AAP-65883/dab-cache-invalidation-job
Assisted-by: Claude Code / Opus 4.6 (Anthropic)
* AAP-65883: Extract helper functions to module level
Move _resolve_setting_dependents and _post_setting_invalidation out of
clear_setting_cache for better stack traces and independent testability
per review feedback (John Westcott).
Assisted-by: Claude Code / Opus 4.6 (Anthropic)
* Move PG version check to check_db command
Move to utils, check in pre_migrate signal
* Add back in environment var skip
* Add tests for compliance
tests Assisted-By: claude
