Compare commits

...

154 Commits

Author SHA1 Message Date
red-hat-konflux-kflux-prd-rh02[bot]
1bc78e1117 Update quay.io/aap-ci/tekton-catalog/pipeline/test/aap-api-tests:0.1 Docker digest to d88df86
Signed-off-by: red-hat-konflux-kflux-prd-rh02 <190377777+red-hat-konflux-kflux-prd-rh02[bot]@users.noreply.github.com>
2026-08-06 00:01:43 +00:00
Alan Rominger
3f04ed4707 [AAP-82668] Skip old RBAC sync on cascade-deleted assignments (#16559)
* [AAP-82668] Skip old RBAC sync on cascade-deleted assignments

When a non-RBAC parent (e.g. Organization) is deleted, Django cascades
to RoleUserAssignment/RoleTeamAssignment and fires post_delete for each.
The sync_assignments_to_old_rbac_delete handler would then do 3-4
FK/GFK queries per assignment to sync removals to the old Role model —
entirely redundant since the old Role M2M tables cascade-delete from the
same content object.

Use Django's post_delete `origin` kwarg (4.1+) to detect this: when
origin is a Model instance whose app_label is not dab_rbac, the delete
is a cascade from a non-RBAC parent and the sync is skipped.

Measured on 150 teams / 321 users / 48K assignments:
  Baseline:      78.8s
  With fix:      20.4s (via service-index endpoint)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-08-03 14:43:03 -04:00
Alan Rominger
4996c91d12 AAP-85067 Old RBAC sync regression tests for bulk claims (#16565)
* test: add regression tests for old RBAC sync after bulk claims

Tests that save_user_claims (which uses bulk_create, skipping signals)
correctly syncs old Role.members when run through AwxJWTAuthentication.
Covers add, remove, and multi-org/team scenarios.

* test: call process_permissions instead of internal _sync_old_rbac

Tests now exercise the public AwxJWTAuthentication.process_permissions()
API with the JWT layer mocked, rather than calling the private
_sync_old_rbac method directly.

The test_bulk_claims_skips_old_rbac_signals test asserts that
save_user_claims does NOT populate old Role.members via signals.
On current DAB, save_user_claims uses the serial give_permission
path which fires signals normally, so old Role.members IS populated.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-08-03 14:39:55 -04:00
Dirk Julich
c84575c215 AAP-81173 — Replace 4-way OR with UNION in unified job list RBAC query (#16555)
* AAP-81173 — Replace 4-way OR with UNION in unified job list RBAC query

The UnifiedJobAccess.filtered_queryset() method used a 4-way OR to
determine which unified jobs a user can see. Under load, the OR forces
PostgreSQL to evaluate all four branches in a single plan, preventing
branch-specific index optimization and causing 35 hours of DB time per
30-minute Scale Lab window.

Split each RBAC branch (template read_role, inventory update, ad-hoc
command, org auditor) into separate querysets combined with UNION, giving
the planner an independent optimal plan per branch. The UNION result is
wrapped in pk__in= for compatibility with BaseAccess.get_queryset()
prefetch_related and the workflowapproval filter.

This follows the same pattern proven in AAP-83319 (team list UNION fix).

* AAP-81173 — Add UnifiedJobPagination to prevent COUNT regression

The pk__in UNION pattern used for RBAC filtering forces the large
outer table as the driving table for COUNT(*), requiring PostgreSQL
to materialize all subquery result sets.  On large deployments this
produces catastrophic query times (see AAP-83773 for the identical
issue on activity_stream).

Override the paginator count to use an unfiltered
UnifiedJob.objects.count() — the over-count is acceptable for
pagination UI.  Also fix test_unified_job_list_rando_sees_nothing
to assert on results length instead of count, since count is now
unfiltered.

* AAP-81173 — Add unit tests for UnifiedJobPagination coverage

Cover the UnifiedJobPaginator.count cached property and the
count_disabled branch in UnifiedJobPagination.paginate_queryset
to satisfy SonarCloud's 90% new-code coverage gate.

* AAP-81173 — Address review feedback: save/restore paginator class, format consistency

- Fix Pagination.paginate_queryset() to save/restore django_paginator_class
  instead of hardcoding DjangoPaginator in the finally block. This lets
  subclasses set the class attribute without needing to override the method.
- Remove UnifiedJobPagination.paginate_queryset() override — now only needs
  to set django_paginator_class = UnifiedJobPaginator as a class attribute.
- Format by_org_auditor consistently with the other three UNION branches.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-08-03 16:27:46 +02:00
Hao Liu
9d67c302fd Exclude .tox from flake8 linter scope (#16566)
awxkit has its own tox.ini and when `tox -e linters` runs, awxkit/.tox/
can get populated with a py3 virtualenv. Since flake8 is configured to
scan the `awxkit` directory, it recurses into
awxkit/.tox/py3/lib/python3.12/site-packages/ and reports hundreds of
false positives (F405, E265, E266) from third-party packages like PyYAML.

Adding .tox to the flake8 exclude list prevents this — matching the
existing exclusion of `env` for virtualenvs.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-08-03 09:25:46 +02:00
Seth Foster
1e8944676b Suppress SonarQube S7498 for Ansible collection modules (#16532)
dict() with keyword arguments is idiomatic for Ansible module
argument_spec definitions. Suppress the "use literal instead of
constructor" rule (python:S7498) for awx_collection/plugins/modules/.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Jake Jackson <jljacks93@gmail.com>
2026-07-31 13:17:13 -04:00
Peter Braun
bee4470fb9 fix: prefer scm_revision as the cache_id if available (#16564)
* fix: prefer scm_revision as the cache_id if available

* add tests

* fix: make sure galaxy requirements are always updated when a new revision is pulled

* make sure existing project cache id is used when available

---------

Co-authored-by: Liam Allen <lallen@redhat.com>
2026-07-31 09:20:18 +00:00
Dirk Julich
10f2f11fe2 [devel][forwardport] AAP-83773 — Use unfiltered count for activity stream pagination (#16557)
* AAP-83773 — Use unfiltered count for activity stream pagination

Forward-port of tower#7606 (stable-2.6).

The RBAC-filtered COUNT(*) on activity_stream takes ~36 min per call
on large tables (713K rows) due to the pk__in subquery shape introduced
by the AAP-81860 LEFT JOIN fix.  Replace with an unfiltered table count
for pagination -- an approximate over-count is harmless for UI page
navigation while the actual page results remain RBAC-filtered.

* AAP-83773 — Add unit tests for ActivityStreamPagination

Cover ActivityStreamPaginator and ActivityStreamPagination to satisfy
SonarCloud coverage gate on new lines in awx/api/pagination.py.

* AAP-83773 — Save/restore django_paginator_class in base Pagination

Address review feedback from lallen92: use a save/restore pattern in
Pagination.paginate_queryset() so subclasses only need to set the class
attribute. Remove the now-redundant paginate_queryset override from
ActivityStreamPagination.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Liam Allen <lallen@redhat.com>
2026-07-31 11:16:06 +02:00
Dirk Julich
2a28b80ec6 AAP-84057: Set PostgreSQL statement_timeout on web worker DB connections (#16558)
* AAP-84057: Set PostgreSQL statement_timeout on web worker DB connections

When uwsgi's harakiri kills a worker, the PostgreSQL backend continues
running the query indefinitely.  These abandoned queries accumulate and
create resource contention for all other queries.

Add a connection_created signal handler that sets statement_timeout on
new DB connections.  Under uwsgi, the timeout is auto-derived from the
harakiri value (minus 5s margin so PostgreSQL cancels the query before
uwsgi kills the worker).  Outside uwsgi (task workers, migrations),
no timeout is applied.  A manual DATABASE_STATEMENT_TIMEOUT setting
is available as a fallback for non-uwsgi deployments.

* Remove timeout value caching because it brings no significant gains

* Use proportional margin between statement_timeout and harakiri timeout

* Fix zero-harakiri test to patch fake uwsgi module instead of None

* Refactor statement_timeout from signal handler to connection string

Move statement_timeout configuration from a connection_created signal
handler (extra SQL round-trip per connection) to a dynaconf merge
function that sets it via the libpq OPTIONS connection string parameter.
This mirrors the existing merge_application_name() pattern and
eliminates the SET statement on every new connection.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-31 10:27:33 +02:00
Elijah DeLee
bf6a5f6b21 AAP-36932 Batch-delete JobHostSummary before cleanup to avoid invalid memory alloc (#16563)
Django's cascade collector materializes all JobHostSummary IDs into a
single UPDATE ... IN (...) to SET_NULL on Host.last_job_host_summary.
With many jobs x hosts this exceeds PostgreSQL's max memory allocation
size for a single query (1GB).

Pre-delete JHS rows and clear Host FK references in chunks of 1000 job
IDs using raw SQL before Django's .delete() runs, so the cascade
collector finds nothing to collect.

Assisted-by: Claude Code via Google Vertex AI
2026-07-30 10:13:35 -04:00
Dirk Julich
f1a3e13df7 AAP-83319 — Replace OR with UNION in team list RBAC query (#16554)
The TeamAccess.filtered_queryset() method combined two access paths
(org membership and direct read permission) using Q(…) | Q(…), which
PostgreSQL could not optimize — it scanned all teams for each OR
branch. Replacing the OR with UNION lets each branch use the
RoleEvaluation 3-column index independently. The UNION result is
wrapped in pk__in= so the outer queryset remains compatible with
BaseAccess.get_queryset()'s select_related() call.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-29 20:44:17 +02:00
Rodrigo Toshiaki Horie
78a55b25ec [AAP-82221] Skip reverse sync during post-migrate role definition setup (#16561)
AAP-82221 — Skip reverse sync during post-migrate role definition setup

The dab_post_migrate signal handler added in 64dc097914 calls
setup_managed_role_definitions on every awx-manage migrate run.
During fresh installs this deletes stale managed role definitions
and triggers a reverse sync to the gateway, which returns 423
Locked because migrate_service_data has not finished yet. The
installer retries 5 times but each attempt fires the same signal
and hits the same 423, failing the controller init.
2026-07-27 15:25:03 -03:00
Liam Allen
54e5c948fe AAP-76460 Prioritize REDHAT_CANDLEPIN_HOST over rhsm.conf for subscription validation (#16543)
* AAP-76460 — Prioritize REDHAT_CANDLEPIN_HOST over rhsm.conf for subscription validation

When REDHAT_CANDLEPIN_HOST is explicitly configured (Satellite/disconnected
environments), use it as the subscription host instead of falling back to
rhsm.conf. This fixes subscription loading failures in containerized
deployments where rhsm.conf defaults to subscription.rhsm.redhat.com but
the system cannot reach it.

Also guard against double port-appending in get_satellite_subs when the
host URL already includes a port.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Liam Allen <lallen@redhat.com>

* Optimize HostList API: conditional DISTINCT + composite index on JobHostSummary (#16530)

AAP-81517 — Optimize HostList API: conditional DISTINCT + composite index

Make .distinct() conditional on host_filter being set — without it the
RBAC IN subquery on a direct FK cannot produce duplicates, so DISTINCT
is pure overhead. Add composite index (host_id, id DESC) on
main_jobhostsummary so the with_latest_summary_id() correlated subquery
can use an index-only top-1 scan instead of scanning and sorting.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* Fix: Increase awx-operator molecule timeout to 20m (#16534)

The awx-operator molecule kind test intermittently times out after
15 minutes on GitHub Actions runners, causing flaky CI failures.
Bump the bash-level timeout from 15m to 20m (step-level
timeout-minutes: 60 is unchanged).

Closes: AAP-81583

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* AAP-81860 — Eliminate LEFT OUTER JOINs in ActivityStream RBAC query (#16533)

* AAP-81860 — Eliminate LEFT OUTER JOINs in ActivityStream RBAC query

ActivityStreamAccess.filtered_queryset() used field-traversal Q objects
(e.g. Q(host__inventory__in=...)) across 18 M2M relationships, which
Django translates into LEFT OUTER JOINs. This forces PostgreSQL to join
all intermediary tables before filtering, making it the #2 DB consumer
at 483s in a 10-minute Scale Lab window.

Replace all M2M field traversals with pk__in subqueries using .through
intermediary tables, following the same pattern proven in AAP-81082
(28% improvement for unified job RBAC). This changes the SQL from
LEFT OUTER JOIN to IN (SELECT ...) semi-joins, allowing PostgreSQL to
skip the unconditional joins. Also:

- Remove .distinct() (no longer needed without M2M JOINs)
- Remove unnecessary if-truthy checks that evaluated subqueries as
  boolean before including them (5 extra DB roundtrips per request)
- Migrate accessible_pk_qs(user, 'read_role') to access_ids_qs(user,
  'view') for direct DAB RBAC API usage

* Use .exists() instead of queryset truth-test for auditing_orgs guard

Avoids evaluating the full queryset when checking whether the user has
auditing orgs — .exists() issues a cheaper SELECT 1 … LIMIT 1 instead.

* Restore .exists() guards for resource-type Q branches

Re-add conditional guards around inventory, credential, project,
job template, workflow job template, and team Q branches. Uses
.exists() (cheap SELECT 1 LIMIT 1) instead of the old queryset
truth-test to avoid unnecessary query evaluation while still
preventing generation of an overly complex query when the user
has no access to a given resource type.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* chore: Upgrade kubernetes package to 36.0.2

* feat: remove extra_vars from jobs and unified_jobs list endpoint. Add… (#16461)

feat: add exclude query parameter to unified_jobs and jobs list endpoints

Allow clients to exclude heavy fields (artifacts, extra_vars) from list
responses via ?exclude=artifacts,extra_vars. These fields are now included
by default instead of being stripped.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* AAP-80457 - Fix credential_input_source to handle state: absent when target crede… (#16523)

Fix credential_input_source to handle state: absent when target credential is missing

The credential_input_source module would error when trying to delete a
credential input source (state: absent) if the target credential didn't
exist. This breaks idempotent playbook runs where cleanup tasks assume
resources may already be gone.

The fix only applies to state: absent; state: present still correctly
fails when the target credential doesn't exist.

Co-authored-by: Liam Allen <lallen@redhat.com>

* Add additional test coverage

* get_satellite_subs() reads verify from rhsm.conf, never from REDHAT_CANDLEPIN_VERIFY.

* Address coderabbit feedback

---------

Signed-off-by: Liam Allen <lallen@redhat.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Dirk Julich <djulich@redhat.com>
Co-authored-by: Rodrigo Toshiaki Horie <rodrigo.horie@hotmail.com>
Co-authored-by: Adrià Sala <22398818+adrisala@users.noreply.github.com>
Co-authored-by: Peter Braun <pbraun@redhat.com>
Co-authored-by: Nick Meyer <nick.a.meyer@icloud.com>
2026-07-27 14:24:44 +01:00
tvo
8812569f92 Add stable branch-cut checklist reference to release process docs (#16535) 2026-07-22 16:10:01 -06:00
Lila Yasin
64dc097914 [AAP-82221] Add member_organization to specialized Organization *Admin roles (#16545) 2026-07-22 15:25:43 -04:00
Alexey Masolov
fcc1aa56d1 Balance controller node selection during capacity ties (#16417)
* Balance controller node selection during capacity ties

When multiple control-plane instances have equal remaining capacity,
the instance selection algorithm always picks the first instance in
iteration order. This causes burst workloads to concentrate all job
management on a single controller pod while others remain idle.

Add a tie-breaking criterion: when would_be_remaining capacity is
equal, prefer the instance with fewer jobs_running. This distributes
the control overhead (event processing, callbacks, output streaming)
more evenly across available controller pods during burst scenarios.

The change is backwards-compatible: when capacity clearly differs,
the existing "most remaining capacity" logic dominates unchanged.

Signed-off-by: Alexey Masolov <amasolov@redhat.com>
Signed-off-by: Alexey Masolov <alexey.masolov@gmail.com>
Made-with: Cursor

* Track jobs_running on controller node for container-group tasks

Address review feedback: container-group tasks only set controller_node
(execution_node is empty), so the previous consume_capacity call on the
control path never incremented jobs_running. This meant the tie-breaker
could not differentiate between nodes.

Now pass job_impact=True on the control path when the controller is not
also the execution node (avoiding double-counting for hybrid nodes).

Also improve test coverage:
- Fix test case to prove capacity dominates over jobs_running
- Add dedicated tests for container-group controller distribution
- Add test verifying equal-capacity tie-breaking behaviour

Signed-off-by: Alexey Masolov <amasolov@redhat.com>
Signed-off-by: Alexey Masolov <alexey.masolov@gmail.com>
Made-with: Cursor

* Add docstrings to consume_capacity and fit_task_to_most_remaining_capacity_instance

Address CodeRabbit docstring coverage warning by documenting the two
methods modified in this PR. No functional changes.

Signed-off-by: Alexey Masolov <amasolov@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

---------

Signed-off-by: Alexey Masolov <amasolov@redhat.com>
Signed-off-by: Alexey Masolov <alexey.masolov@gmail.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Liam Allen <lallen@redhat.com>
2026-07-21 15:46:47 +01:00
Peter Braun
d8e7a711d0 fix: delete inventory hosts in batches to avoid memory exhaustion (#16547)
* fix: delete inventory hosts in batches to avoid memory exhaustion

* update tests

* fix failing test
2026-07-21 11:38:44 +00:00
Liam Allen
9acf3d1887 AAP-73393 - Remove skip_tags character limit by converting to TextField (#16552)
* Optimize HostList API: conditional DISTINCT + composite index on JobHostSummary (#16530)

AAP-81517 — Optimize HostList API: conditional DISTINCT + composite index

Make .distinct() conditional on host_filter being set — without it the
RBAC IN subquery on a direct FK cannot produce duplicates, so DISTINCT
is pure overhead. Add composite index (host_id, id DESC) on
main_jobhostsummary so the with_latest_summary_id() correlated subquery
can use an index-only top-1 scan instead of scanning and sorting.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* Fix: Increase awx-operator molecule timeout to 20m (#16534)

The awx-operator molecule kind test intermittently times out after
15 minutes on GitHub Actions runners, causing flaky CI failures.
Bump the bash-level timeout from 15m to 20m (step-level
timeout-minutes: 60 is unchanged).

Closes: AAP-81583

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* AAP-81860 — Eliminate LEFT OUTER JOINs in ActivityStream RBAC query (#16533)

* AAP-81860 — Eliminate LEFT OUTER JOINs in ActivityStream RBAC query

ActivityStreamAccess.filtered_queryset() used field-traversal Q objects
(e.g. Q(host__inventory__in=...)) across 18 M2M relationships, which
Django translates into LEFT OUTER JOINs. This forces PostgreSQL to join
all intermediary tables before filtering, making it the #2 DB consumer
at 483s in a 10-minute Scale Lab window.

Replace all M2M field traversals with pk__in subqueries using .through
intermediary tables, following the same pattern proven in AAP-81082
(28% improvement for unified job RBAC). This changes the SQL from
LEFT OUTER JOIN to IN (SELECT ...) semi-joins, allowing PostgreSQL to
skip the unconditional joins. Also:

- Remove .distinct() (no longer needed without M2M JOINs)
- Remove unnecessary if-truthy checks that evaluated subqueries as
  boolean before including them (5 extra DB roundtrips per request)
- Migrate accessible_pk_qs(user, 'read_role') to access_ids_qs(user,
  'view') for direct DAB RBAC API usage

* Use .exists() instead of queryset truth-test for auditing_orgs guard

Avoids evaluating the full queryset when checking whether the user has
auditing orgs — .exists() issues a cheaper SELECT 1 … LIMIT 1 instead.

* Restore .exists() guards for resource-type Q branches

Re-add conditional guards around inventory, credential, project,
job template, workflow job template, and team Q branches. Uses
.exists() (cheap SELECT 1 LIMIT 1) instead of the old queryset
truth-test to avoid unnecessary query evaluation while still
preventing generation of an overly complex query when the user
has no access to a given resource type.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* chore: Upgrade kubernetes package to 36.0.2

* feat: remove extra_vars from jobs and unified_jobs list endpoint. Add… (#16461)

feat: add exclude query parameter to unified_jobs and jobs list endpoints

Allow clients to exclude heavy fields (artifacts, extra_vars) from list
responses via ?exclude=artifacts,extra_vars. These fields are now included
by default instead of being stripped.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* AAP-80457 - Fix credential_input_source to handle state: absent when target crede… (#16523)

Fix credential_input_source to handle state: absent when target credential is missing

The credential_input_source module would error when trying to delete a
credential input source (state: absent) if the target credential didn't
exist. This breaks idempotent playbook runs where cleanup tasks assume
resources may already be gone.

The fix only applies to state: absent; state: present still correctly
fails when the target credential doesn't exist.

Co-authored-by: Liam Allen <lallen@redhat.com>

* AAP-83163 — Eliminate LEFT JOIN fan-out in user list RBAC query (#16546)

Replace legacy Role M2M traversal in UserAccess.filtered_queryset()
with DAB's visible_users(), which queries organizational membership
through RoleUserAssignment subqueries instead of LEFT OUTER JOINs
through the objectrole/roleevaluation tables.

Also replace role__in=actor.has_roles.all() in can_admin() with
RoleEvaluation._actor_role_filter() to avoid the objectrole JOIN
in per-object permission checks.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* alter_skip_tags_to_textfield

* Bump migration number

---------

Co-authored-by: Dirk Julich <djulich@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Rodrigo Toshiaki Horie <rodrigo.horie@hotmail.com>
Co-authored-by: Adrià Sala <22398818+adrisala@users.noreply.github.com>
Co-authored-by: Peter Braun <pbraun@redhat.com>
Co-authored-by: Nick Meyer <nick.a.meyer@icloud.com>
2026-07-21 10:00:21 +01:00
Dirk Julich
7a7a6224c0 AAP-82745 — Replace O(N) correlated subqueries in org list with bulk aggregation (#16548)
* AAP-82745 — Replace O(N) correlated subqueries in org list with bulk aggregation

The org list endpoint computed per-org user and admin counts using two
correlated subqueries against roleuserassignment — one per org per role
type. At scale this was the #1 worst query in the customer DB (677ms
mean, 336K calls, 3,801 min total DB time).

Replace with a single flat conditional-aggregation query that scans
roleuserassignment once regardless of org count. Also simplify the
detail view to use direct .count() calls and add a query count
regression test.

* Remove dead code: org_counts is always populated after refactor

* Address review feedback on detail view and test

- Detail view: combine two .count() calls into single .aggregate()
  with conditional counts, matching the list-view pattern
- Use .update() in else branch for consistency with if branch
- Remove redundant setup_managed_roles fixture (transitive via
  organization_resource_creator)

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-20 15:41:20 +02:00
Dirk Julich
ea14ee1563 AAP-83163 — Eliminate LEFT JOIN fan-out in user list RBAC query (#16546)
Replace legacy Role M2M traversal in UserAccess.filtered_queryset()
with DAB's visible_users(), which queries organizational membership
through RoleUserAssignment subqueries instead of LEFT OUTER JOINs
through the objectrole/roleevaluation tables.

Also replace role__in=actor.has_roles.all() in can_admin() with
RoleEvaluation._actor_role_filter() to avoid the objectrole JOIN
in per-object permission checks.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-17 19:24:13 +02:00
Nick Meyer
354fa35860 AAP-80457 - Fix credential_input_source to handle state: absent when target crede… (#16523)
Fix credential_input_source to handle state: absent when target credential is missing

The credential_input_source module would error when trying to delete a
credential input source (state: absent) if the target credential didn't
exist. This breaks idempotent playbook runs where cleanup tasks assume
resources may already be gone.

The fix only applies to state: absent; state: present still correctly
fails when the target credential doesn't exist.

Co-authored-by: Liam Allen <lallen@redhat.com>
2026-07-09 13:42:06 +01:00
Peter Braun
d0576d7823 feat: remove extra_vars from jobs and unified_jobs list endpoint. Add… (#16461)
feat: add exclude query parameter to unified_jobs and jobs list endpoints

Allow clients to exclude heavy fields (artifacts, extra_vars) from list
responses via ?exclude=artifacts,extra_vars. These fields are now included
by default instead of being stripped.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-08 17:05:41 +02:00
Adrià Sala
9d5bf22f90 chore: Upgrade kubernetes package to 36.0.2 2026-07-08 16:02:07 +02:00
Dirk Julich
f3b04125a6 AAP-81860 — Eliminate LEFT OUTER JOINs in ActivityStream RBAC query (#16533)
* AAP-81860 — Eliminate LEFT OUTER JOINs in ActivityStream RBAC query

ActivityStreamAccess.filtered_queryset() used field-traversal Q objects
(e.g. Q(host__inventory__in=...)) across 18 M2M relationships, which
Django translates into LEFT OUTER JOINs. This forces PostgreSQL to join
all intermediary tables before filtering, making it the #2 DB consumer
at 483s in a 10-minute Scale Lab window.

Replace all M2M field traversals with pk__in subqueries using .through
intermediary tables, following the same pattern proven in AAP-81082
(28% improvement for unified job RBAC). This changes the SQL from
LEFT OUTER JOIN to IN (SELECT ...) semi-joins, allowing PostgreSQL to
skip the unconditional joins. Also:

- Remove .distinct() (no longer needed without M2M JOINs)
- Remove unnecessary if-truthy checks that evaluated subqueries as
  boolean before including them (5 extra DB roundtrips per request)
- Migrate accessible_pk_qs(user, 'read_role') to access_ids_qs(user,
  'view') for direct DAB RBAC API usage

* Use .exists() instead of queryset truth-test for auditing_orgs guard

Avoids evaluating the full queryset when checking whether the user has
auditing orgs — .exists() issues a cheaper SELECT 1 … LIMIT 1 instead.

* Restore .exists() guards for resource-type Q branches

Re-add conditional guards around inventory, credential, project,
job template, workflow job template, and team Q branches. Uses
.exists() (cheap SELECT 1 LIMIT 1) instead of the old queryset
truth-test to avoid unnecessary query evaluation while still
preventing generation of an overly complex query when the user
has no access to a given resource type.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-07 16:52:11 +02:00
Rodrigo Toshiaki Horie
1bd07b981a Fix: Increase awx-operator molecule timeout to 20m (#16534)
The awx-operator molecule kind test intermittently times out after
15 minutes on GitHub Actions runners, causing flaky CI failures.
Bump the bash-level timeout from 15m to 20m (step-level
timeout-minutes: 60 is unchanged).

Closes: AAP-81583

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-07-06 19:51:21 +00:00
Dirk Julich
41545cfcf0 Optimize HostList API: conditional DISTINCT + composite index on JobHostSummary (#16530)
AAP-81517 — Optimize HostList API: conditional DISTINCT + composite index

Make .distinct() conditional on host_filter being set — without it the
RBAC IN subquery on a direct FK cannot produce duplicates, so DISTINCT
is pure overhead. Add composite index (host_id, id DESC) on
main_jobhostsummary so the with_latest_summary_id() correlated subquery
can use an index-only top-1 scan instead of scanning and sorting.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-06 17:01:20 +02:00
Seth Foster
72a1922a9c Replace realistic looking credentials in subscriptions module examples (#16531)
Replace hardcoded credentials in subscriptions module examples

The EXAMPLES block contained values that looked like real client
credentials. Replace them with obvious placeholders and add a
username/password example.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-07-02 15:52:26 -04:00
Dirk Julich
f8fa690de3 AAP-81082 — Eliminate LEFT OUTER JOINs in unified job RBAC query (#16527)
* AAP-81082 Eliminate LEFT OUTER JOINs in unified job RBAC query

UnifiedJobAccess.filtered_queryset() used field-traversal Q objects
(inventoryupdate__inventory_source__inventory__id__in=...) which Django
translates into LEFT OUTER JOINs. This forces PostgreSQL to scan all
rows in main_unifiedjob before filtering — at scale, 99.84% are
discarded. Replace with pk__in subqueries that generate IN (SELECT ...)
instead, allowing PostgreSQL to skip the unconditional join. EXPLAIN
ANALYZE shows a 28% reduction in execution time (496ms -> 355ms), with
larger gains expected under concurrent load.

* AAP-81082 Eliminate LEFT OUTER JOINs in RBAC filtered_queryset methods

Replace field-traversal Q objects with pk__in subqueries across all Access
classes that query polymorphic or M2M tables, preventing Django from
generating unconditional LEFT OUTER JOINs. Also migrate legacy
_accessible_pk_qs / accessible_pk_qs calls to DAB RBAC access_ids_qs.

Affected: UnifiedJobAccess, UnifiedJobTemplateAccess, JobAccess,
JobEventAccess, LabelAccess.

* Fix docstring
2026-07-01 13:18:09 +02:00
Hamzah Yousuf
8ab5deb54a FIX: Refactor formatted raw SQL in unified_jobs result_stdout_raw_handle (#16522)
* Refactor result_stdout_raw_handle to use parameterized COPY SQL.
Replace f-string SQL construction with psycopg.sql composables and bound
parameters so security scans no longer flag formatted raw SQL in the
unified jobs stdout path.
Fix sqlite_copy mock rendering for psycopg3 SQL composables.

* Fix sqlite_copy mock without psycopg SQL internals.
Load stdout from the first populated event table instead of rendering
psycopg composables, which use version-specific private attributes.

* Use sql.Literal in COPY query for Django cursor.copy compatibility.
Django's cursor.copy() does not forward bind parameters to psycopg,
which caused stdout API 500s against real PostgreSQL.
2026-06-25 13:49:22 +00:00
Rodrigo Toshiaki Horie
843f23f4cb Fix SonarCloud security rating: remove user-controlled data from sqlite filepath (#16516)
* Fix SonarCloud security rating by removing user-controlled data from sqlite filepath

Replace os.path.basename(sys.argv[0]) with a hardcoded 'unknown' fallback
in RecordedQueryLog.write() to eliminate path injection via CLI arguments.
This resolves SonarCloud rule pythonsecurity:S8706 and helps restore the
AWX security rating from C to A.

Closes: AAP-80006

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Remove unused sys import from test_db.py

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-06-25 10:08:54 -03:00
Lila Yasin
6d665dda33 Fix awxkit tox test failure caused by root pytest.ini config bleed (#16520)
Forward-port of ansible/tower#7537 for the devel branch.

When running awxkit's tox tests, pytest picks up the root pytest.ini
which pulls in pytest-django options (--reuse-db, --nomigrations,
DJANGO_SETTINGS_MODULE) and Django-specific filterwarnings.  Since the
awxkit tox environment does not install Django or pytest-django, these
cause test collection to fail.

The root cause is that pytest.ini has absolute priority in pytest's
config discovery — it searches all ancestor directories for pytest.ini
before falling back to tox.ini's [pytest] section.  A [pytest] section
in awxkit/tox.ini alone cannot prevent the root config from being used.

Fix by:
- Adding awxkit/pytest.ini to act as the primary config boundary
  (pytest.ini has the highest priority in config discovery, so its
  presence in awxkit/ stops the upward search before reaching root)
- Adding explicit `test` path argument to the pytest command in
  awxkit/tox.ini so pytest discovers tests correctly
- Adding `testpaths` and `python_files` to the [pytest] section in
  awxkit/tox.ini as a secondary config boundary
- Adding awxkit/conftest.py that registers the Django-specific CLI
  options and INI keys as harmless no-ops, as a further safety net

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-24 10:42:23 -04:00
Stevenson Michel
17dc7f898a Fix: handle cursor-paginated responses in get_one() (#16521)
* Fix: handle cursor-paginated API responses in get_one()

DAB PR #1025 switched role_team_assignments and role_user_assignments
endpoints from PageNumberPagination to CursorPagination. Cursor
pagination returns {results, next, previous} without a count field,
causing get_one() to fail with "The endpoint did not provide count
and results".

When the response includes results but no count, infer count from
len(results). Also guard get_all_endpoint() against missing count.
2026-06-24 09:56:26 -04:00
Seth Foster
f25e436bef Do not mark node READY when health check reports zero cpu or memory (#16511)
Instances reporting cpu=0 or memory=0 with no errors would be
transitioned to READY state. Treat zero cpu/memory as an error
so the node stays offline until a valid health check is received.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-06-22 10:23:52 -04:00
Dave
67048a609a feat(analytics): log request_id, account_number, org_id from ingress API response (#16512)
* feat(analytics): ANSTRAT-2268 log request_id, account_number, org_id from ingress API response

Log the ingress API success response fields (request_id, account_number,
org_id) in the controller task log when gather_analytics tarballs are
uploaded. This enables support engineers to trace uploads through to
Kibana without source code modifications.

* style(analytics): use f-strings in _log_shipping_response to match codebase conventions

* style(analytics): fix black formatting in test assertions
2026-06-22 10:03:57 -04:00
Dirk Julich
23b7ad5067 [devel backport][AAP-70902] Use _actor_role_filter() in UnifiedJobTemplate.accessible_pk_qs() (#16513)
* Use _actor_role_filter() in UnifiedJobTemplate.accessible_pk_qs()

Replace `role__in=accessor.has_roles.all()` with the optimized `_actor_role_filter()` subquery pattern from django-ansible-base.

The old pattern causes a 3-table JOIN through RoleUserAssignment -> ObjectRole -> RoleEvaluation on every non-superuser request. _actor_role_filter() skips the ObjectRole table entirely by using a direct subquery on RoleUserAssignment.object_role_id, eliminating the intermediate JOIN and reducing query time for /api/v2/unified_jobs/ requests by non-superusers.
2026-06-19 16:45:14 +02:00
Rodrigo Toshiaki Horie
0dfc168a5f fix: pipe blob content via stdin to avoid ARG_MAX limit in spec sync (#16510)
fix: use GPG-signed commits in spec sync workflow

Switch from unsigned GitHub API commits to GPG-signed git commits
using the aap-api-bot GPG key (OPENAPI_SPEC_SYNC_GPG_PRIVATE_KEY).

The aap-openapi-specs repo requires signed commits via org ruleset.
The previous API-based approach didn't sign commits because GitHub
only auto-signs API commits for GitHub App tokens, not user PATs.

This matches the pattern used by EDA and Gateway teams for their
spec sync workflows.

Also fixes template injection risk by using env vars instead of
direct ${{ }} expansion in shell context.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-06-18 15:56:03 -03:00
Ryan Williams
25115ed7a8 chore: update tekton pipeline bundle digest and renovate config (#16494)
Signed-off-by: Ryan Williams <3375653+ryankwilliams@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-06-18 14:46:40 -04:00
thanujdesu11
f8fc3d107f AAP-78420: Updating the remediation condition in system.py to check for cpu and memory values set to 0. (#16499)
Updating the remediation condition on line 636 to check for cpu and memory values set to 0.
2026-06-18 13:05:04 -04:00
Dirk Julich
c1bd2eb338 [AAP-72817] Fix cartesian product in organization user/admin count queries (#16501)
* Fix cartesian product in organization user/admin count queries

The organizations list and detail endpoints annotated each org with user and admin counts using two Count() calls that traverse the Role.members M2M. Django generated two LEFT JOINs on the same through table, crossing every member row with every admin row before COUNT(DISTINCT) reduced the product.

At scale (2,617 members × 46,233 admins) this produced 120M intermediate rows and 96-second query times, causing 504 timeouts.

Replace with independent Subquery expressions that each query main_rbac_roles_members separately - no cross product.

Fixes: AAP-72817
Fixes: AAP-72480

* Fix variable names which do not meet coding standards

* Fix formatting inconsistency in organization detail subquery annotation

Break the long .annotate() line across multiple lines to match the style used in mixin.py.

* Rewrite org count subqueries to use DAB RBAC models

Replace old RBAC Role.members.through subqueries with
RoleUserAssignment-based correlated subqueries, querying
managed RoleDefinitions ('Organization Member' / 'Organization Admin')
directly. This aligns with the DAB RBAC migration direction and
eliminates dependency on the deprecated ImplicitRoleField M2M tables
for these counts.

Update test fixtures to use RoleDefinition.give_permission() and
add setup_managed_roles where needed.

* Fix collection tests: set up managed role definitions

The DAB RBAC migration to use RoleUserAssignment subqueries in
organization views requires managed role definitions (Organization
Member, Organization Admin) to exist in the test database.

Add an autouse fixture to the collection test conftest that calls
setup_managed_role_definitions() before each test.

* Add setup_managed_roles fixture to functional tests hitting org views

Tests that hit organization list/detail views now require the
setup_managed_roles fixture to pre-create the Organization Member
and Organization Admin RoleDefinition objects used by the DAB RBAC
subqueries.

* Revert setup_managed_roles from ext_auditor tests

The setup_managed_roles fixture conflicts with the ext_auditor_rd
fixture by deleting the Alien Auditor role definition. These tests
don't need it — the defensive view code handles missing role
definitions gracefully.

* Handle missing Organization Member/Admin role definitions gracefully

Use filter().first() instead of get() for RoleDefinition lookups in
organization list and detail views. Returns 0 for user/admin counts
when role definitions are not yet created, preventing 500 errors in
environments where post_migrate signals haven't run.

* Cast OuterRef('pk') to TextField for RoleUserAssignment.object_id comparison

RoleUserAssignment.object_id is a TextField, but OuterRef('pk') on
Organization produces an integer. PostgreSQL strictly rejects text = integer
comparisons. Use Cast() to explicitly convert the PK to text.

---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-18 18:35:22 +02:00
Dirk Julich
61d17673d9 Optimize HostManager.active_count() with cache (#16505)
* [AAP-78392] Optimize HostManager.active_count() with cache and functional index

active_count() runs a full sequential scan with LOWER()+DISTINCT on main_host for every license check. At customer scale this consumed 74.5 minutes of DB time over 4 hours (47K calls at 93ms avg).

Add a 60-second Redis-backed cache via the existing memoize decorator to reduce call volume by ~99.5%. Add a functional btree index on LOWER(name) to eliminate the sequential scan for the remaining calls.

* Use AddIndexConcurrently instead of AddIndex in the migration for host name lower index

* Revert AddIndexConcurrently to AddIndex for CI compatibility

The api-migrations CI job runs against SQLite which does not support PostgreSQL-specific AddIndexConcurrently. Standard AddIndex works across all backends and the brief write lock during production upgrades is acceptable for this table size.

* Remove functional index, keep cache-only fix per reviewer feedback

Drop the LOWER(name) functional index and migration to minimize
the change footprint.

----
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-18 17:48:29 +02:00
Stevenson Michel
2a197cc627 Update CI schedule to 11/17 UTC and add stable-2.7 dispatch (#16507)
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-06-18 09:02:00 -04:00
Rodrigo Toshiaki Horie
4b4fafc79f fix: use GitHub API for signed commits in spec sync workflow (#16509)
The aap-openapi-specs repo requires commit signatures via org ruleset.
Switch from git commit+push to the GitHub Git Data API which
automatically signs commits, satisfying the required_signatures rule.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-06-17 18:44:05 +00:00
Rodrigo Toshiaki Horie
242f008f44 feat: restore x-ai-description entries in OpenAPI schema (#16502)
feat: inject x-ai-description from overlay file during schema generation

Many endpoints have human-readable AI descriptions that were added
downstream in aap-mcp-server (PRs #73 and #119) but never backported
as @extend_schema_if_available decorators. This causes 470 out of 631
x-ai-description entries to be lost every time the spec is regenerated.

Add a JSON overlay file (openapi_ai_descriptions.json) containing the
missing descriptions keyed by operationId, and a drf-spectacular
postprocessing hook that merges them into the generated schema for any
operation that doesn't already have x-ai-description from a decorator.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-06-17 15:26:24 -03:00
Rodrigo Toshiaki Horie
34f34e058b AAP-79142 fix: resolve SonarCloud security rating C (#16503)
fix: resolve SonarCloud security rating C (AAP-79142)

Replace regex-based substring checks with plain `in` operator in awxkit
page.py to eliminate ReDoS vulnerability (python:S5852). Remove stray
empty Pipfile that triggered missing lockfile warning (text:S8565).

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-06-15 17:41:53 -03:00
Hao Liu
849f5f796c Restore oauth_token backward compatibility for collection token auth (#16500)
* Restore oauth_token backward compatibility for collection token auth

The aap_token rename (c8981e321e) restored module-level token auth but
left two interfaces from earlier collection releases broken:

- The lookup (controller_api) and inventory (controller) plugins
  previously declared an oauth_token option. Add oauth_token as an
  alias of aap_token in the auth_plugin doc fragment and in
  AUTH_ARGSPEC so query(..., oauth_token=...) and inventory YAML keys
  keep working.

- tower_cli.cfg-style config files used an oauth_token key under
  [general]; it was silently ignored after the rename, quietly
  degrading auth. load_config() now also reads the legacy oauth_token
  key and maps it to aap_token, with the new aap_token key winning when
  both are present. aap_token remains the canonical attribute used by
  _parse_aap_token() and the Bearer header logic.

Also make the test helper compatible with ansible-core 2.21+, which
requires a serialization profile alongside _ANSIBLE_ARGS, and extend
the tests to cover the oauth_token alias and legacy config file key.

No changelog fragment added: awx_collection has no changelogs/
directory on devel.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Document oauth_token alias in module auth doc fragment

The oauth_token alias was added to aap_token in AUTH_ARGSPEC but not to
the module doc fragment, failing the validate-modules sanity check
(undocumented argument alias).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* Generalize version references in compat comments

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-12 18:32:07 +00:00
Hao Liu
c8981e321e Make aap_token functional for collection token auth (#16498)
The aap_token parameter was added to the collection argspec and docs
in #16025, but nothing consumed it after token auth was removed in
#15623: modules silently ignored the token and fell back to basic
auth, breaking token authentication through the AAP gateway.

Wire it up so requests authenticate with the provided token (e.g. one
issued by the AAP gateway, which validates it and proxies to the
controller):

- Send "Authorization: Bearer <token>" in make_request when aap_token
  is set, skipping the basic-auth login probe; basic auth is unchanged
  when no token is given
- Accept the token as a string or as the dict set as a fact by the
  ansible.platform.token module ({token: ..., id: ...}), which is the
  documented cross-collection mint/use/delete workflow
- Restore controller_oauthtoken and tower_oauthtoken as aliases for
  back-compat with pre-#15623 playbooks, matching downstream
- Forward aap_token through the controller_api lookup and controller
  inventory plugins via short_params, and add the missing
  CONTROLLER_OAUTH_TOKEN/TOWER_OAUTH_TOKEN env sources to the plugin
  doc fragment (plugins resolve env vars from doc fragments, not
  env_fallback); AAP_TOKEN is no longer marked deprecated there
- Support tokens in the awxkit-based export/import modules
- Add unit tests covering the Bearer header for both token forms, the
  aliases, the bad-dict failure, and the basic-auth fallback

Verified end-to-end against a live gateway-fronted AAP 2.7 deployment:
modules, the lookup plugin, both aliases, all env sources, dict-form
tokens, job launch/wait, and a clean HTTP 401 on an invalid token.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-06-10 18:18:25 -04:00
Seth Foster
d5e5ea3670 Lazy-load plugin registries, move DB sync to dispatcher (#16483)
Move plugin loading to lazy-on-first-access, DB sync to dispatcher

Remove credential type and inventory plugin loading from Django's
app.ready() path. In-memory registries (ManagedCredentialType.registry
and InventorySourceOptions.injectors) are now populated lazily on first
access via LazyLoadDict, a dict subclass that calls a loader function
on the first read operation. This ensures web workers, dispatcher
workers, and management commands all get the registries populated
exactly when needed, without eager loading at startup.

The DB sync (CredentialType.setup_tower_managed_defaults) is moved to
the dispatcher's startup task, where it only needs to run once per
deployment rather than in every Django process.

Co-Authored-By: Alan Rominger <arominge@redhat.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-06-09 12:45:50 -04:00
Bryan Havenstein
d566f71ae0 AAP-65883: Refactor clear_setting_cache to use DAB utility (#16469)
* AAP-65883: Refactor clear_setting_cache to use DAB shared utility

Delegate cache invalidation logic to ansible_base.lib.cache.tasks.clear_cache,
passing AWX-specific dependent key resolution (settings_registry) and
post-invalidation hook (LOG_AGGREGATOR_LEVEL reconfiguration) as callbacks.

Requires: ansible/django-ansible-base AAP-65883/dab-cache-invalidation-job

Assisted-by: Claude Code / Opus 4.6 (Anthropic)

* AAP-65883: Extract helper functions to module level

Move _resolve_setting_dependents and _post_setting_invalidation out of
clear_setting_cache for better stack traces and independent testability
per review feedback (John Westcott).

Assisted-by: Claude Code / Opus 4.6 (Anthropic)
2026-06-09 11:56:58 -04:00
Peter Braun
c8cb465fde fix: do not allow exec instances to be added to the control plane (#16477)
Co-authored-by: Stevenson Michel <iamstevensonmichel@outlook.com>
2026-06-09 09:04:10 -04:00
Alan Rominger
49e21d7c1c Move PG version check to awx-manage check_db & migrate commands (#15463)
* Move PG version check to check_db command

Move to utils, check in pre_migrate signal

* Add back in environment var skip

* Add tests for compliance

tests Assisted-By: claude
2026-06-08 10:57:55 -04:00
Ryan Williams
b531151931 chore: bump ATF pipeline bundle sha (#16480)
Signed-off-by: Ryan Williams <3375653+ryankwilliams@users.noreply.github.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-06-04 14:24:39 -04:00
Stevenson Michel
54857c7a82 [Devel] Bump kubernetes client to >=36.0.0 to fix NO_PROXY handling (#16478)
Bump kubernetes client to >=36.0.0 to fix NO_PROXY handling
2026-06-04 13:21:55 -04:00
Hamzah Yousuf
e03899b581 FIX: Add Organization Parameter to Demo Credential (#16474)
Fix demo credential organization in create_preload_data
2026-06-03 18:32:16 +00:00
Lila Yasin
b4f27de4a2 [AAP-53283] Use getproxies() for analytics API proxy configuration (#16475)
Revert "[AAP-53283] Fix analytics API requests to respect proxy environment variables (#16451)"

This reverts commit 45480941f8.
2026-06-02 15:38:18 -04:00
Seth Foster
5cc467d4cf [AAP-74497] Reset orphaned waiting jobs when controller node is deprovisioned (#16467)
Reset orphaned waiting jobs when controller node is deprovisioned

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-06-02 10:46:52 -04:00
Stevenson Michel
b14b9e1771 [Devel] Reject RRULE with INTERVAL=0 to Prevent Scheduler Hang (#16464)
* added interval null rrule check and updated tests

* Added secondly to the expected errors
2026-06-02 10:22:36 -04:00
Ryan Williams
c4c2779976 fix: invalid renovate config to bump tekton pipeline bundles (#16472)
Signed-off-by: Ryan Williams <3375653+ryankwilliams@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-06-02 15:34:55 +02:00
Tomas Z
4bdb11c2a6 [AAP-65882] feat: switch to DAB Redis cache backend (#16471)
[AAP-65882] Switch to DAB Redis cache backend

Replace awx.main.cache.AWXRedisCache with the equivalent
ansible_base.lib.cache.redis_cache.DABRedisCache and remove the
now-redundant AWX driver. The DAB driver was migrated from this
exact code and provides the same fault-tolerant cache behavior.

Signed-off-by: Tomas Z <tznamena@redhat.com>
2026-06-01 17:15:20 -04:00
Alan Rominger
80f8ee1dec Fix skip of push operations which are still failing in mirrors (#16468)
* Fix bash operator precedence in repo ownership check

The condition had || operators outside proper test block grouping,
which could cause the check to fail with a shell error. Wrap the
OR conditions in parentheses with explicit [[ ]] tests.

Assisted-by: Claude Haiku 4.5 <noreply@anthropic.com>

* Replace reusable workflow with direct if conditions for repo ownership check

The reusable workflow with job dependencies had a timing/evaluation issue
where jobs would still execute even when should_run=false. Using direct
if conditions with github context variables (repository, ref_name) is more
reliable and ensures jobs are properly skipped on fork pushes.

Assisted-by: Claude Haiku 4.5 <noreply@anthropic.com>

* Remove unused repo-owns-branch reusable workflow

No longer needed after replacing with direct if conditions.

Assisted-by: Claude Haiku 4.5 <noreply@anthropic.com>
2026-06-01 13:25:27 -04:00
Chris Meyers
f22df56e44 Fix TypeError when update_fields=None in model save methods (#16466)
Django allows passing update_fields=None to model.save() to mean 'update all fields'.
However, kwargs.get('update_fields', []) returns None when the key exists with a None value,
rather than the default empty list.

This caused crashes in mark_field_for_save() and other code that assumes update_fields is
a list when doing membership tests ('field' not in update_fields) or append operations.

Changed pattern from:
  update_fields = kwargs.get('update_fields', [])

To:
  update_fields = kwargs.get('update_fields') or []

This correctly handles:
- Key missing: get() returns None → or [] gives []
- Key present with None: get() returns None → or [] gives []
- Key present with list: get() returns list → or [] keeps the list

Fixed in 12 locations across 8 model files:
- awx/main/models/base.py (3 instances)
- awx/main/models/unified_jobs.py (2)
- awx/main/models/jobs.py (2)
- awx/main/models/ad_hoc_commands.py
- awx/main/models/inventory.py
- awx/main/models/notifications.py
- awx/main/models/projects.py
- awx/main/models/workflow.py

Fixes task manager crashes with:
TypeError: argument of type 'NoneType' is not iterable
2026-05-29 14:14:16 -04:00
Alan Rominger
fccb6744f9 Address even more pytest warnings, and migrate smart inventory tests (#16330)
* Address even more pytest warnings, co-authored with Opus 4.6

* Upgrade pyparsing

* Attempt to update smart inventory logic

* Move smart inventory tests here

* Fix some failing dev env tests

Assisted-by: claude

* Use shared fixture for teardown

* Fix test goof

assisted-by: claude Opus 4.6
2026-05-28 16:17:13 -04:00
Alan Rominger
200a68aefa [AAP-57274] Fix creator permissions for models without old-style roles (#16457)
* [AAP-57274] Fix creator permissions for models without old-style roles

NotificationTemplate has no old-style ImplicitRoleField (like admin_role)
because notification permissions were historically org-level only.
When a non-admin user creates a notification template,
give_creator_permissions tries to sync the DAB RBAC assignment back
to the old role system and hits an AttributeError.

Catch the AttributeError so the DAB RBAC assignment still succeeds.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-28 16:06:03 -04:00
Alan Rominger
9b922f70ed Make dispatcherd min_workers literally 4 (#16421)
assisted-by: claude
2026-05-28 09:58:13 -04:00
Lila Yasin
e4fa4810eb Expose JOB_VARIABLE_PREFIXES as a configurable setting (#16452)
*  INCLUDE_AWX_VAR_PREFIX to USE_TOWER_VAR_PREFIX boolean toggle replace the include legacy prefix boolean with a tower-or-awx toggle USE_TOWER_VAR_PREFIX=True (default) emits only tower_ prefixed variables, false emits only awx_ (deprecated)

* Clean up dead constant and cache get_job_variable_prefixes() calls

* Revise tests to reflect new behavior

* Fix fragile fallback test to actually exercise getattr default

* Fix mock target for settings fallback test
2026-05-26 16:43:50 -04:00
Dirk Julich
b37f3892b6 [AAP-74343] Decouple installed_collections/ansible_version from indirect node counting flag (#16453)
* [AAP-74343] Decouple installed_collections and ansible_version from indirect node counting flag

The indirect_instance_count callback plugin and its artifact processing
were entirely gated behind FEATURE_INDIRECT_NODE_COUNTING_ENABLED. This
caused installed_collections and ansible_version to remain unpopulated
when the flag was off, even though these are baseline analytics fields
unrelated to indirect host counting.

Always run the callback plugin and persist installed_collections and
ansible_version to the database. Only the indirect-counting-specific
parts (EventQuery creation, event_queries_processed flag, and vendor
collections) remain gated behind the feature flag.

* [AAP-74343] Read callbacks_enabled from ansible.cfg so user-configured callbacks are preserved

The check for 'callbacks_enabled' in config_values was dead code because
read_ansible_config was never asked to read that setting. Now that the
callback registration runs unconditionally, fix this by including
'callbacks_enabled' in the variables of interest.

* [AAP-74343] Use comma delimiter for ANSIBLE_CALLBACKS_ENABLED

Ansible's CALLBACKS_ENABLED config is type list and splits on commas.
The colon delimiter would cause combined callback names to be treated
as a single invalid name.

* [AAP-74343] Add tests for ANSIBLE_CALLBACKS_ENABLED configuration

Verify that indirect_instance_count is always set, user-configured
callbacks from ansible.cfg are preserved, and the comma delimiter
is used as ansible-core expects.

* [AAP-74343] Use public API for namespace package path access

Replace library.__path__._path[0] with library.__path__[0] to avoid
relying on a private CPython implementation detail of _NamespacePath.

* [AAP-74343] Skip host query scanning when indirect counting flag is off

The indirect_instance_count callback plugin now checks AWX_COLLECT_HOST_QUERIES
to decide whether to scan for host query files. When the feature flag is off,
the plugin only collects collection metadata (name + version) and ansible_version,
skipping the expensive embedded/external query file discovery.

* [AAP-74343] Set AWX_COLLECT_HOST_QUERIES in query discovery tests

The TestExternalQueryDiscovery tests exercise the host query scanning
path, which now requires AWX_COLLECT_HOST_QUERIES=1 in the environment.

* [AAP-74343] Use Ansible plugin config system for collect_host_queries

Declare collect_host_queries as a formal plugin option in DOCUMENTATION
with env var AWX_COLLECT_HOST_QUERIES, replacing the raw os.getenv() call
with self.get_option(). This follows the standard Ansible plugin
configuration pattern.

* [AAP-74343] Add test for disabled collect_host_queries path

Verify that when collect_host_queries is false, the plugin still
enumerates collections for metadata but skips host query file scanning.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-22 14:45:08 +02:00
Stevenson Michel
ec85902b37 Fix awx-collection tests for ansible-core 2.21.0 (#16455)
Fix awx-collection tests for ansible-core 2.21.0 compat

ansible-core 2.21.0 introduced _PARSED_MODULE_ARGS in
module_utils/basic.py, used by _return_formatted() to conditionally
include invocation data. The awx_collection test harness bypasses
normal arg parsing via _load_params mock, leaving this variable as
None and causing AttributeError on every exit_json/fail_json call.

Mock _PARSED_MODULE_ARGS with _ansible_inject_invocation=True to
match the pre-2.21 behavior of always including invocation data.
2026-05-21 13:38:52 -04:00
Adrià Sala
5eeb854620 feat: increase parallel count for atf runs (#16450)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-18 15:32:15 +02:00
Lila Yasin
45480941f8 [AAP-53283] Fix analytics API requests to respect proxy environment variables (#16451)
Fix analytics API requests to respect proxy environment variables 
Assisted-by: Claude
2026-05-15 13:13:15 -04:00
jessicamack
90b7d35554 Implement Candlepin certificate integration (#16388)
* AAP-12516 [option 2] Handle nested workflow artifacts via root node `ancestor_artifacts` (#16381)

* Add new test for artfact precedence upstream node vs outer workflow

* Fix bugs, upstream artifacts come first for precedence

* Track nested artifacts path through ancestor_artifacts on root nodes

* Fix case where first root node did not get the vars

* touchup comment

* Prevent conflict with sliced jobs hack

* Reorder URLs so that Django debug toolbar can work (#16352)

* Reorder URLs so that Django debug toolbar can work

* Move comment with URL move

* feat: support for oidc credential /test endpoint (#16370)

Adds support for testing external credentials that use OIDC workload identity tokens.
When FEATURE_OIDC_WORKLOAD_IDENTITY_ENABLED is enabled, the /test endpoints return
JWT payload details alongside test results.

- Add OIDC credential test endpoints with job template selection
- Return JWT payload and secret value in test response
- Maintain backward compatibility (detail field for errors)
- Add comprehensive unit and functional tests
- Refactor shared error handling logic

Co-authored-by: Daniel Finca <dfinca@redhat.com>
Co-authored-by: melissalkelly <melissalkelly1@gmail.com>

* Bind the install bundle to the ansible.receptor collection 2.0.8 version (#16396)

* [Devel] Config Endpoint Optimization (#16389)

* Improved performance of the config endpoint by reducing database queries in GET /api/controller/v2/config/

* Fix OIDC workload identity for inventory sync (#16390)

The cloud credential used by inventory updates was not going through
the OIDC workload identity token flow because it lives outside the
normal _credentials list. This overrides populate_workload_identity_tokens
in RunInventoryUpdate to include the cloud credential as an
additional_credentials argument to the base implementation, and
patches get_cloud_credential on the instance so the injector picks up
the credential with OIDC context intact.

Co-authored-by: Alan Rominger <arominge@redhat.com>
Co-authored-by: Dave Mulford <dmulford@redhat.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: integrate awx-tui to the awx_devel image (#16399)

* Aap 45980 (#16395)

* support bitbucket_dc webhooks

* add test

* update docs

* fix import for refactored method (#16394)

retrieve_workload_identity_jwt_with_claims is now
in a separate utility file, not in jobs.py

Signed-off-by: Seth Foster <fosterbseth@gmail.com>

* AAP-70257 controller collection should retry transient HTTP errors with exponential backoff. (#16415)

controller collection should retry transient HTTP errors with exponential backoff

* AAP-71844 Fix rrule fast-forward across DST boundaries (#16407)

Fix rrule fast-forward producing wrong occurrences across DST boundaries

The UTC round-trip in _fast_forward_rrule shifts the dtstart's local
hour when the original and fast-forwarded times are in different DST
periods. Since dateutil generates HOURLY occurrences by stepping in
local time, the shifted hour changes the set of reachable hours. With
BYHOUR constraints this causes a ValueError crash; without BYHOUR,
occurrences are silently shifted by 1 hour.

Fix by performing all arithmetic in the dtstart's original timezone.
Python aware-datetime subtraction already computes absolute elapsed
time regardless of timezone, so the UTC conversion was unnecessary
for correctness and actively harmful during fall-back ambiguity.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Correctly restrict push actions to ownership repos (#16398)

* Correctly restrict push actions to ownership repos

* Use standard action to see if push actions should run

* Run spec job for 2.6 and higher

* Be even more restrictve, do not push if on a fork

* [Devel] Performance Optimization for Select Hosts Query (#16413)

* Fixed black reformating

* Make test simulate 500k hosts in real world scenario

* feat: improve unauthorized response on aap deployments (#16422)

* fix: do not include secret values in the credentials test endpoint an… (#16425)

fix: do not include secret values in the credentials test endpoint and add a guard to make sure credentials are testable

* [devel backport] AAP-41742: Fix workflow node update failing when JT has unprompted labels (#16426)

* AAP-41742: Fix workflow node update failing when JT has unprompted labels

PATCH extra_data on a workflow node fails with
{"labels":["Field is not configured to prompt on launch."]}
when the node has labels associated but the JT has
ask_labels_on_launch=False.

The serializer was passing all persisted M2M state from prompts_dict()
to _accept_or_ignore_job_kwargs() on every PATCH, re-validating
unchanged fields. Fix scopes validation to only the fields in the
request; full re-validation still occurs when unified_job_template
is being changed.

* Capture attrs keys before _build_mock_obj mutates them

_build_mock_obj() pops pseudo-fields (limit, scm_branch, job_tags,
etc.) from attrs. Computing requested_prompt_fields after the pop
would miss those fields and skip their ask_on_launch validation.

* Include survey_passwords when validating extra_vars prompts

prompts_dict() emits survey_passwords alongside extra_vars.
_accept_or_ignore_job_kwargs uses it to decrypt encrypted survey
values before validation. Without it, encrypted password blobs
are validated as-is against the survey spec.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add test to ensure credential secret values are not returned (#16434)

* AAP-68024 perf: derive last_job_host_summary from query instead of denormalized FK (#16332)

* perf: stop eagerly updating Host.last_job_host_summary on every job completion

The playbook_on_stats wrapup path bulk-updates last_job_host_summary_id
on every host touched by a job. In the Q4CY25 scale lab this query had
a median execution time of 75 seconds due to index churn on main_host.

Replace all reads of the denormalized FK with a new classmethod
JobHostSummary.latest_for_host(host_id) that queries for the most
recent summary on demand. This eliminates the write-side bulk_update
of last_job_host_summary_id entirely.

Changes:
- Add JobHostSummary.latest_for_host() classmethod
- Serializer: use latest_for_host() instead of obj.last_job_host_summary
- Dashboard view: use subquery instead of FK traversal for failed hosts
- Inventory.update_computed_fields: use subquery for failed host count
- events.py: remove last_job_host_summary_id from bulk_update
- signals.py: simplify _update_host_last_jhs to only update last_job
- access.py/managers.py: remove select_related/defer through the FK

The FK field on Host is left in place for now (removal requires a
migration) but is no longer written to.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix .pk AttributeError, add job_template annotations, annotate host sublists

- Add 'pk' to AnnotatedSummary dynamic type (fixes AttributeError in get_related)
- Add job_template_id and job_template_name to subquery annotations so list
  views include these fields in summary_fields.last_job (matching detail views)
- Traverse job__ FK from JobHostSummary instead of using separate UnifiedJob
  subquery with OuterRef on another annotation (cleaner SQL, avoids alias issue)
- Annotate all host sublist views (InventoryHostsList, GroupHostsList,
  GroupAllHostsList, InventorySourceHostsList) to prevent N+1 queries

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Update test_events to use JobHostSummary.latest_for_host instead of stale FKs

Tests were asserting host.last_job_id and host.last_job_host_summary_id
which are no longer updated. Use JobHostSummary.latest_for_host() to
derive the same data, matching the new read-time derivation approach.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Remove stale failures_url from deprecated DashboardView

The failures_url linked to ?last_job_host_summary__failed=True which
filters on the now-stale FK. The dashboard count itself was already
fixed to use a subquery annotation. Since DashboardView is deprecated
and has_active_failures is a SerializerMethodField (not filterable),
remove the failures_url entirely rather than creating a custom filter.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Apply black formatting to changed files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Refactor: replace 10 subquery annotations with bulk prefetch

Instead of annotating every host queryset with 10 correlated subqueries
(summary + job + job_template fields), annotate only _latest_summary_id
and bulk-fetch the full JobHostSummary objects after pagination via
select_related('job', 'job__job_template').

This reduces the SQL from 10 correlated subqueries to 1 subquery + 1 IN
query, addressing review feedback about annotation overhead on host list
views.

- _annotate_host_latest_summary: only annotates _latest_summary_id
- _prefetch_latest_summaries: bulk-fetches and attaches to host objects
- HostSummaryPrefetchMixin: hooks into list() after pagination
- Serializer uses real JobHostSummary objects (no more AnnotatedSummary)
- to_representation always overwrites stale FK values

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Refactor: move latest summary to QuerySet._fetch_all + Host.latest_summary

Per review feedback, replace the view-level HostSummaryPrefetchMixin
with a custom QuerySet that bulk-attaches summaries at evaluation time
(like prefetch_related), and a Host.latest_summary property as the
single access point.

- HostLatestSummaryQuerySet: overrides _fetch_all() to bulk-fetch
  JobHostSummary objects with select_related after queryset evaluation
- HostManager now inherits from the custom queryset via from_queryset()
- Host.latest_summary property: uses cache if available, falls back to
  individual query
- Remove _annotate_host_latest_summary, _prefetch_latest_summaries,
  HostSummaryPrefetchMixin from views — no more list() override needed
- Remove last_job/last_job_host_summary from SUMMARIZABLE_FK_FIELDS
- Serializer uses obj.latest_summary and DEFAULT_SUMMARY_FIELDS loop

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix: scope annotation to views, restore license_error/canceled_on

- Remove with_latest_summary_id() from HostManager.get_queryset() to
  avoid applying the correlated subquery to every Host query globally
  (count, exists, internal relations)
- Apply with_latest_summary_id() in get_queryset() of the 6
  host-serving views only
- Restore license_error and canceled_on to last_job summary fields
  to avoid breaking API change

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Guard _fetch_all() to skip bulk-attach on non-annotated querysets

Without this guard, _fetch_all() would set _latest_summary_cache=None
on every host in non-annotated querysets (e.g. Host.objects.filter()),
masking the per-object fallback query in Host.latest_summary.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Remove name from last_job_host_summary and canceled_on from last_job summary

Per reviewer feedback: these fields were not in the original API contract
via SUMMARIZABLE_FK_FIELDS and their addition would be an API change.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add functional tests for HostLatestSummaryQuerySet and Host.latest_summary

Tests cover:
- with_latest_summary_id() annotation and most-recent selection
- _fetch_all() bulk-attach behavior on annotated querysets
- _fetch_all() skips non-annotated querysets (preserves fallback)
- .count() and .exists() do NOT trigger _fetch_all
- Host.latest_summary cache hits (zero queries) and fallback
- Host.latest_job property
- select_related on bulk-attached summaries (no N+1)
- Chaining preserves annotation
- Multiple jobs / partial host coverage

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Apply black formatting to test_host_queryset.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ben Thomasson <bthomass@redhat.com>

* Fix flake8 F841: remove unused job1/job2 variables in tests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ben Thomasson <bthomass@redhat.com>

* Add comment explaining why Prefetch was not used for host latest summary

Django Prefetch cannot handle latest per group -- [:1] slicing fetches
1 record globally, not per host (Django ticket #26780). The custom
_fetch_all override uses the same 2-query pattern as prefetch_related
internally, customized for this use case.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix null handling to keep old behavior

---------

Signed-off-by: Ben Thomasson <bthomass@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: AlanCoding <arominge@redhat.com>

* [AAP-72722] Use url instead of jwt_aud for workload identity audience (#16432)

* [AAP-72722] Use url instead of jwt_aud for workload identity audience

The OIDC credential plugin's jwt_aud field is being removed. Use the
plugin's url field as the audience when requesting workload identity
tokens, since the target service URL is the appropriate audience value.

Assisted-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [Devel] Optimize host_list_rbac query  (#16408)

* Defer ansible_facts in HostManager to avoid fetching large JSON column in host list queries (AAP-68023)

The host list endpoint (GET /api/v2/hosts/) fetches the ansible_facts
JSON column unnecessarily, contributing to the 7.8s median query time
at scale. This column can be very large and is not used by the list
serializer.

Changes:
- HostManager.get_queryset() now defers ansible_facts
- finish_fact_cache call site uses .only(*HOST_FACTS_FIELDS) to eagerly
  load ansible_facts when actually needed, avoiding N+1 queries
- Unit test mocks updated to support .only() queryset chaining
- Points DAB dependency at the RBAC query optimization branch for
  combined testing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------

* fix: constructed inventories no longer increase the host count (#16433)

* Fix version worktree (#16431)

* git worktree friendly precomit install

* worktrees don't have a .git directory. Before, docker-compose would
  trigger pre-commit install and fail.

* make docker-compose work in git worktree

* AWX tries to discover the version via info stored in .git/ dir.
  setuptools-scm is capable of finding the .git/ dir, starting from a
  worktree, but is unable because only the worktree is mapped into the
  container, not the .git/ dir itself. Thus, we have to detect and pass
  the version into the container from outside. That is why this change
  landed in the Makefile.

* fix: as_user() gateway session cookie fallback (#16437)

Add a fallback that checks for `gateway_sessionid` when no cookie
matches `session_cookie_name`, mirroring the existing fallback in
`Connection.login()`. The finally block now cleans up whichever
cookie name was actually used.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* Pass setting to dispatcherd so it can be configured (#16438)

* fix: allow blank password field to fix OpenAPI schema validation (#16440)

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* first pass porting over metrics

* move settings to defaults

* add more unit tests

* update unit tests

* lint fixes

* more lint fixes

* refactor and address feedback

* remove the api views

* remove model and move helper functions out of licensing

* add settings to API, fix tests, refactoring

* fix circular import

* update tests

* remove duplicate code, handle edge cases, use clearer naming, add test coverage

* update test for changes in ship()

* remove unneeded setting

* _discover_org should account for verify-tls=False

* directly assign settings, detect url, update tests

* log errors close to occurance

* rename function for clarity, focus on critical tests

* rename for clarity, lint fixes

* fix test params, priority for org discovery

* fix test failures and linting

---------

Signed-off-by: Seth Foster <fosterbseth@gmail.com>
Signed-off-by: Ben Thomasson <bthomass@redhat.com>
Co-authored-by: Alan Rominger <arominge@redhat.com>
Co-authored-by: Daniel Finca <dfinca@redhat.com>
Co-authored-by: melissalkelly <melissalkelly1@gmail.com>
Co-authored-by: Tong He <68936428+unnecessary-username@users.noreply.github.com>
Co-authored-by: Stevenson Michel <iamstevensonmichel@outlook.com>
Co-authored-by: Seth Foster <fosterseth@users.noreply.github.com>
Co-authored-by: Dave Mulford <dmulford@redhat.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Adrià Sala <22398818+adrisala@users.noreply.github.com>
Co-authored-by: Peter Braun <pbraun@redhat.com>
Co-authored-by: Sean Sullivan <ssulliva@redhat.com>
Co-authored-by: Dirk Julich <djulich@redhat.com>
Co-authored-by: Ben Thomasson <bthomass@redhat.com>
Co-authored-by: Dan Leehr <dleehr@users.noreply.github.com>
Co-authored-by: Lila Yasin <lyasin@redhat.com>
Co-authored-by: Chris Meyers <chrismeyersfsu@users.noreply.github.com>
2026-05-14 09:33:48 -04:00
Alan Rominger
9606366625 Consolidate validation rules for same-org restrictions (#16427)
* Consolidate implementation of same-org validation rule

* Update tests for the simplified validation

* Still do validation with deferance to the new callback

* Correctly falsy handling in view logic
2026-05-12 08:59:45 -04:00
Alan Rominger
188c10c7d6 Only request read permission for PR scan (#16295) 2026-05-11 18:09:55 +00:00
Ryan Williams
2d02a72218 feat: add tekton pipeline run atf tests on pull request (#16443)
Signed-off-by: Ryan Williams <3375653+ryankwilliams@users.noreply.github.com>
2026-05-11 12:50:38 +02:00
Lila Yasin
d3b40cb57e [devel]AAP-74276: Replace setuptools with packaging in awxkit install_requires (#16444)
AAP-74276: Replace setuptools with packaging in awxkit install_requires

The Python 3.12 upgrade replaced distutils.version.LooseVersion with
packaging.version.Version but did not update awxkit's install_requires.
setuptools is no longer needed at runtime since pkg_resources was also
replaced with importlib.metadata. This causes ModuleNotFoundError on
standalone CLI installs where packaging is not present.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-06 11:25:52 -04:00
Alan Rominger
6179b16987 AAP-72269 Change fact processing loop to use file listing (#16403)
* Change fact processing loop to use file listing

* Fix some test

* Address coderabbit comments

* Handle saving facts in batches to keep memory low

* Improve log about mismatch in response to review comment
2026-05-05 15:35:46 +02:00
jessicamack
cbbd683720 AAP-70294: Migrate Unit Test Candidates from ATF to Upstream (#16385)
* add converted atf tests

* fix bulk settings test
2026-05-04 15:07:46 +00:00
Adrià Sala
2451156fc6 fix: allow blank password field to fix OpenAPI schema validation (#16440)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-05-04 16:03:36 +02:00
Alan Rominger
83f60cddc2 Pass setting to dispatcherd so it can be configured (#16438) 2026-04-30 15:58:52 -04:00
Adrià Sala
c67d93218f fix: as_user() gateway session cookie fallback (#16437)
Add a fallback that checks for `gateway_sessionid` when no cookie
matches `session_cookie_name`, mirroring the existing fallback in
`Connection.login()`. The finally block now cleans up whichever
cookie name was actually used.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-30 10:43:08 +02:00
Chris Meyers
eac8968217 Fix version worktree (#16431)
* git worktree friendly precomit install

* worktrees don't have a .git directory. Before, docker-compose would
  trigger pre-commit install and fail.

* make docker-compose work in git worktree

* AWX tries to discover the version via info stored in .git/ dir.
  setuptools-scm is capable of finding the .git/ dir, starting from a
  worktree, but is unable because only the worktree is mapped into the
  container, not the .git/ dir itself. Thus, we have to detect and pass
  the version into the container from outside. That is why this change
  landed in the Makefile.
2026-04-28 16:07:14 -04:00
Peter Braun
df771d0e9d fix: constructed inventories no longer increase the host count (#16433) 2026-04-28 20:01:21 +00:00
Lila Yasin
1213ea6f62 [Devel] Optimize host_list_rbac query (#16408)
* Defer ansible_facts in HostManager to avoid fetching large JSON column in host list queries (AAP-68023)

The host list endpoint (GET /api/v2/hosts/) fetches the ansible_facts
JSON column unnecessarily, contributing to the 7.8s median query time
at scale. This column can be very large and is not used by the list
serializer.

Changes:
- HostManager.get_queryset() now defers ansible_facts
- finish_fact_cache call site uses .only(*HOST_FACTS_FIELDS) to eagerly
  load ansible_facts when actually needed, avoiding N+1 queries
- Unit test mocks updated to support .only() queryset chaining
- Points DAB dependency at the RBAC query optimization branch for
  combined testing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
2026-04-28 14:00:13 -04:00
Dan Leehr
b66c0105ae [AAP-72722] Use url instead of jwt_aud for workload identity audience (#16432)
* [AAP-72722] Use url instead of jwt_aud for workload identity audience

The OIDC credential plugin's jwt_aud field is being removed. Use the
plugin's url field as the audience when requesting workload identity
tokens, since the target service URL is the appropriate audience value.

Assisted-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-28 10:53:09 -04:00
Ben Thomasson
d1b3ae53ae AAP-68024 perf: derive last_job_host_summary from query instead of denormalized FK (#16332)
* perf: stop eagerly updating Host.last_job_host_summary on every job completion

The playbook_on_stats wrapup path bulk-updates last_job_host_summary_id
on every host touched by a job. In the Q4CY25 scale lab this query had
a median execution time of 75 seconds due to index churn on main_host.

Replace all reads of the denormalized FK with a new classmethod
JobHostSummary.latest_for_host(host_id) that queries for the most
recent summary on demand. This eliminates the write-side bulk_update
of last_job_host_summary_id entirely.

Changes:
- Add JobHostSummary.latest_for_host() classmethod
- Serializer: use latest_for_host() instead of obj.last_job_host_summary
- Dashboard view: use subquery instead of FK traversal for failed hosts
- Inventory.update_computed_fields: use subquery for failed host count
- events.py: remove last_job_host_summary_id from bulk_update
- signals.py: simplify _update_host_last_jhs to only update last_job
- access.py/managers.py: remove select_related/defer through the FK

The FK field on Host is left in place for now (removal requires a
migration) but is no longer written to.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix .pk AttributeError, add job_template annotations, annotate host sublists

- Add 'pk' to AnnotatedSummary dynamic type (fixes AttributeError in get_related)
- Add job_template_id and job_template_name to subquery annotations so list
  views include these fields in summary_fields.last_job (matching detail views)
- Traverse job__ FK from JobHostSummary instead of using separate UnifiedJob
  subquery with OuterRef on another annotation (cleaner SQL, avoids alias issue)
- Annotate all host sublist views (InventoryHostsList, GroupHostsList,
  GroupAllHostsList, InventorySourceHostsList) to prevent N+1 queries

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Update test_events to use JobHostSummary.latest_for_host instead of stale FKs

Tests were asserting host.last_job_id and host.last_job_host_summary_id
which are no longer updated. Use JobHostSummary.latest_for_host() to
derive the same data, matching the new read-time derivation approach.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Remove stale failures_url from deprecated DashboardView

The failures_url linked to ?last_job_host_summary__failed=True which
filters on the now-stale FK. The dashboard count itself was already
fixed to use a subquery annotation. Since DashboardView is deprecated
and has_active_failures is a SerializerMethodField (not filterable),
remove the failures_url entirely rather than creating a custom filter.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Apply black formatting to changed files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Refactor: replace 10 subquery annotations with bulk prefetch

Instead of annotating every host queryset with 10 correlated subqueries
(summary + job + job_template fields), annotate only _latest_summary_id
and bulk-fetch the full JobHostSummary objects after pagination via
select_related('job', 'job__job_template').

This reduces the SQL from 10 correlated subqueries to 1 subquery + 1 IN
query, addressing review feedback about annotation overhead on host list
views.

- _annotate_host_latest_summary: only annotates _latest_summary_id
- _prefetch_latest_summaries: bulk-fetches and attaches to host objects
- HostSummaryPrefetchMixin: hooks into list() after pagination
- Serializer uses real JobHostSummary objects (no more AnnotatedSummary)
- to_representation always overwrites stale FK values

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Refactor: move latest summary to QuerySet._fetch_all + Host.latest_summary

Per review feedback, replace the view-level HostSummaryPrefetchMixin
with a custom QuerySet that bulk-attaches summaries at evaluation time
(like prefetch_related), and a Host.latest_summary property as the
single access point.

- HostLatestSummaryQuerySet: overrides _fetch_all() to bulk-fetch
  JobHostSummary objects with select_related after queryset evaluation
- HostManager now inherits from the custom queryset via from_queryset()
- Host.latest_summary property: uses cache if available, falls back to
  individual query
- Remove _annotate_host_latest_summary, _prefetch_latest_summaries,
  HostSummaryPrefetchMixin from views — no more list() override needed
- Remove last_job/last_job_host_summary from SUMMARIZABLE_FK_FIELDS
- Serializer uses obj.latest_summary and DEFAULT_SUMMARY_FIELDS loop

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix: scope annotation to views, restore license_error/canceled_on

- Remove with_latest_summary_id() from HostManager.get_queryset() to
  avoid applying the correlated subquery to every Host query globally
  (count, exists, internal relations)
- Apply with_latest_summary_id() in get_queryset() of the 6
  host-serving views only
- Restore license_error and canceled_on to last_job summary fields
  to avoid breaking API change

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Guard _fetch_all() to skip bulk-attach on non-annotated querysets

Without this guard, _fetch_all() would set _latest_summary_cache=None
on every host in non-annotated querysets (e.g. Host.objects.filter()),
masking the per-object fallback query in Host.latest_summary.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Remove name from last_job_host_summary and canceled_on from last_job summary

Per reviewer feedback: these fields were not in the original API contract
via SUMMARIZABLE_FK_FIELDS and their addition would be an API change.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Add functional tests for HostLatestSummaryQuerySet and Host.latest_summary

Tests cover:
- with_latest_summary_id() annotation and most-recent selection
- _fetch_all() bulk-attach behavior on annotated querysets
- _fetch_all() skips non-annotated querysets (preserves fallback)
- .count() and .exists() do NOT trigger _fetch_all
- Host.latest_summary cache hits (zero queries) and fallback
- Host.latest_job property
- select_related on bulk-attached summaries (no N+1)
- Chaining preserves annotation
- Multiple jobs / partial host coverage

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Apply black formatting to test_host_queryset.py

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ben Thomasson <bthomass@redhat.com>

* Fix flake8 F841: remove unused job1/job2 variables in tests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ben Thomasson <bthomass@redhat.com>

* Add comment explaining why Prefetch was not used for host latest summary

Django Prefetch cannot handle latest per group -- [:1] slicing fetches
1 record globally, not per host (Django ticket #26780). The custom
_fetch_all override uses the same 2-query pattern as prefetch_related
internally, customized for this use case.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix null handling to keep old behavior

---------

Signed-off-by: Ben Thomasson <bthomass@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: AlanCoding <arominge@redhat.com>
2026-04-28 10:47:22 -04:00
Peter Braun
f3b7d442c3 feat: add test to ensure credential secret values are not returned (#16434) 2026-04-27 12:50:51 +00:00
Dirk Julich
376f964a40 [devel backport] AAP-41742: Fix workflow node update failing when JT has unprompted labels (#16426)
* AAP-41742: Fix workflow node update failing when JT has unprompted labels

PATCH extra_data on a workflow node fails with
{"labels":["Field is not configured to prompt on launch."]}
when the node has labels associated but the JT has
ask_labels_on_launch=False.

The serializer was passing all persisted M2M state from prompts_dict()
to _accept_or_ignore_job_kwargs() on every PATCH, re-validating
unchanged fields. Fix scopes validation to only the fields in the
request; full re-validation still occurs when unified_job_template
is being changed.

* Capture attrs keys before _build_mock_obj mutates them

_build_mock_obj() pops pseudo-fields (limit, scm_branch, job_tags,
etc.) from attrs. Computing requested_prompt_fields after the pop
would miss those fields and skip their ask_on_launch validation.

* Include survey_passwords when validating extra_vars prompts

prompts_dict() emits survey_passwords alongside extra_vars.
_accept_or_ignore_job_kwargs uses it to decrypt encrypted survey
values before validation. Without it, encrypted password blobs
are validated as-is against the survey spec.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-24 16:17:04 +02:00
Peter Braun
c71a49e044 fix: do not include secret values in the credentials test endpoint an… (#16425)
fix: do not include secret values in the credentials test endpoint and add a guard to make sure credentials are testable
2026-04-24 12:35:12 +00:00
Adrià Sala
99ac0d39dc feat: improve unauthorized response on aap deployments (#16422) 2026-04-23 14:35:45 +00:00
Stevenson Michel
55ad29ac68 [Devel] Performance Optimization for Select Hosts Query (#16413)
* Fixed black reformating

* Make test simulate 500k hosts in real world scenario
2026-04-22 12:05:36 -04:00
Alan Rominger
3fd3b741b6 Correctly restrict push actions to ownership repos (#16398)
* Correctly restrict push actions to ownership repos

* Use standard action to see if push actions should run

* Run spec job for 2.6 and higher

* Be even more restrictve, do not push if on a fork
2026-04-21 11:26:04 -04:00
Seth Foster
1636abd669 AAP-71844 Fix rrule fast-forward across DST boundaries (#16407)
Fix rrule fast-forward producing wrong occurrences across DST boundaries

The UTC round-trip in _fast_forward_rrule shifts the dtstart's local
hour when the original and fast-forwarded times are in different DST
periods. Since dateutil generates HOURLY occurrences by stepping in
local time, the shifted hour changes the set of reachable hours. With
BYHOUR constraints this causes a ValueError crash; without BYHOUR,
occurrences are silently shifted by 1 hour.

Fix by performing all arithmetic in the dtstart's original timezone.
Python aware-datetime subtraction already computes absolute elapsed
time regardless of timezone, so the UTC conversion was unnecessary
for correctness and actively harmful during fall-back ambiguity.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-21 10:54:42 -04:00
Sean Sullivan
d21e0141ce AAP-70257 controller collection should retry transient HTTP errors with exponential backoff. (#16415)
controller collection should retry transient HTTP errors with exponential backoff
2026-04-21 08:12:08 -06:00
Seth Foster
e5bae59f5a fix import for refactored method (#16394)
retrieve_workload_identity_jwt_with_claims is now
in a separate utility file, not in jobs.py

Signed-off-by: Seth Foster <fosterbseth@gmail.com>
2026-04-15 10:47:17 -04:00
Peter Braun
a8afbd1ca3 Aap 45980 (#16395)
* support bitbucket_dc webhooks

* add test

* update docs
2026-04-14 14:00:59 +00:00
Adrià Sala
da996c01a0 feat: integrate awx-tui to the awx_devel image (#16399) 2026-04-14 10:08:19 +02:00
Seth Foster
b8c9ae73cd Fix OIDC workload identity for inventory sync (#16390)
The cloud credential used by inventory updates was not going through
the OIDC workload identity token flow because it lives outside the
normal _credentials list. This overrides populate_workload_identity_tokens
in RunInventoryUpdate to include the cloud credential as an
additional_credentials argument to the base implementation, and
patches get_cloud_credential on the instance so the injector picks up
the credential with OIDC context intact.

Co-authored-by: Alan Rominger <arominge@redhat.com>
Co-authored-by: Dave Mulford <dmulford@redhat.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 16:26:18 -04:00
Stevenson Michel
d71f18fa44 [Devel] Config Endpoint Optimization (#16389)
* Improved performance of the config endpoint by reducing database queries in GET /api/controller/v2/config/
2026-04-09 16:24:03 -04:00
Tong He
e82a4246f3 Bind the install bundle to the ansible.receptor collection 2.0.8 version (#16396) 2026-04-09 17:09:26 +02:00
Daniel Finca
b83019bde6 feat: support for oidc credential /test endpoint (#16370)
Adds support for testing external credentials that use OIDC workload identity tokens.
When FEATURE_OIDC_WORKLOAD_IDENTITY_ENABLED is enabled, the /test endpoints return
JWT payload details alongside test results.

- Add OIDC credential test endpoints with job template selection
- Return JWT payload and secret value in test response
- Maintain backward compatibility (detail field for errors)
- Add comprehensive unit and functional tests
- Refactor shared error handling logic

Co-authored-by: Daniel Finca <dfinca@redhat.com>
Co-authored-by: melissalkelly <melissalkelly1@gmail.com>
2026-04-06 15:56:11 -04:00
Alan Rominger
6d94aa84e7 Reorder URLs so that Django debug toolbar can work (#16352)
* Reorder URLs so that Django debug toolbar can work

* Move comment with URL move
2026-04-03 10:22:21 -04:00
Alan Rominger
7155400efc AAP-12516 [option 2] Handle nested workflow artifacts via root node ancestor_artifacts (#16381)
* Add new test for artfact precedence upstream node vs outer workflow

* Fix bugs, upstream artifacts come first for precedence

* Track nested artifacts path through ancestor_artifacts on root nodes

* Fix case where first root node did not get the vars

* touchup comment

* Prevent conflict with sliced jobs hack
2026-04-02 15:18:11 -04:00
melissalkelly
e80ce43f87 Fix workload identity project updates (#16373)
* fix: enable workload identity credentials for project updates

* Add explanatory comment for credential context handling

* Revert build_passwords
2026-03-31 14:48:27 +02:00
Stevenson Michel
595e093bbf [CI-Fix] Pin setuptools_scm<10 to fix api-lint failure (#16376)
Fix CI: Pin setuptools_scm<10 to fix api-lint build failure

setuptools-scm 10.0.5 (with its new vcs-versioning dependency) requires
a [tool.setuptools_scm] or [tool.vcs-versioning] section in pyproject.toml.
AWX intentionally omits this section because it uses a custom version
resolution via setup.cfg (version = attr: awx.get_version). The new major
version of setuptools-scm treats the missing section as a fatal error when
building the sdist in tox's isolated build, causing the linters environment
to fail.

Pinning to <10 restores compatibility with the existing version resolution
strategy.

Failing run: https://github.com/ansible/awx/actions/runs/23744310714
Branch: devel

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-30 14:37:16 -04:00
TVo
cd7f6f602f Fix OpenAPI schema validation message mismatch (#16372) 2026-03-25 12:36:10 -06:00
Chris Meyers
310dd3e18f Update dispatcherd to version 2026.3.25 (#16369)
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 10:57:01 -04:00
Matthew Sandoval
7c75788b0a AAP-67740 Pass plugin_description through to CredentialType.description (#16364)
* Pass plugin_description through to CredentialType.description

Propagate the plugin_description field from credential plugins into the
CredentialType description when loading and creating managed credential
types, including updates to existing records.

Assisted-by: Claude

* Add unit tests for plugin_description passthrough to CredentialType

Tests cover load_plugin, get_creation_params, and
_setup_tower_managed_defaults handling of the description field.

Assisted-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: PabloHiro <palonso@redhat.com>
2026-03-25 11:03:11 +01:00
Peter Braun
ab294385ad fix: avoid delete in loop in inventory import (#16366) 2026-03-24 15:37:59 +00:00
Alan Rominger
377dfce197 Record whether a file was written for fact cache (#16361) 2026-03-20 12:53:34 -04:00
Matthew Sandoval
ff68d6196d Add feature flag for OIDC workload identity credential types (#16348)
Add install-time feature flag for OIDC workload identity credential types

Implements FEATURE_OIDC_WORKLOAD_IDENTITY_ENABLED feature flag to gate
HashiCorp Vault OIDC credential types as a Technology Preview feature.

When the feature flag is disabled (default), OIDC credential types are
not loaded into the plugin registry at application startup and do not
exist in the database.

When enabled, OIDC credential types are loaded normally and function
as expected.

Changes:
- Add FEATURE_OIDC_WORKLOAD_IDENTITY_ENABLED setting (defaults to False)
- Add OIDC_CREDENTIAL_TYPE_NAMESPACES constant for maintainability
- Modify load_credentials() to skip OIDC types when flag is disabled
- Add test coverage (2 test cases)

This is an install-time flag that requires application restart to take
effect. The flag is checked during application startup when credential
types are loaded from plugins.

Fixes: AAP-64510

Assisted-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2026-03-18 22:40:33 -04:00
Ben Thomasson
bfefee5aef fix: NameError in wsrelay when JSON decode fails with DEBUG logging (#16340)
* fix: NameError in wsrelay when JSON decode fails with DEBUG logging

run_connection() referenced payload in the JSONDecodeError handler,
but payload was never assigned because json.loads() is what failed.
Use msg.data instead to log the raw message content.

Fixes: AAP-68045

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Fix other instance of undefined payload

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: AlanCoding <arominge@redhat.com>
2026-03-18 17:17:25 +00:00
Alan Rominger
0aaca1bffd Fix job cancel chain bugs (#16325)
* Fix job cancel chain bugs

* Early relief valve for canceled jobs, ATF related changes

* Add test and fix for approval nodes as well

* Revert unwanted change

* Refactor workflow approval nodes to make it more clean

* Revert data structure changes

* Delete local utility file

* Review comment addressing

* Use canceled status in websocket

* Delete slop

* Add agent marker

* Bugbot comment about status websocket mismatch
2026-03-18 12:08:27 -04:00
Rodrigo Toshiaki Horie
679e48cbe8 [AAP-68258] Fix SonarCloud Reliability issues (#16354)
* Fix SonarCloud Reliability issues: time-dependent class attrs and dict comprehensions

- Move last_stats/last_flush from class body to __init__ in CallbackBrokerWorker
  (S8434: time-dependent expressions evaluated at class definition)
- Replace dict comprehensions with dict.fromkeys() in has_create.py
  (S7519: constant-value dict should use fromkeys)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Fix callback receiver tests to use flush(force=True)

Tests were implicitly relying on last_flush being a stale class-level
timestamp. Now that last_flush is set in __init__, the time-based flush
condition isn't met when flush() is called immediately after construction.
Use force=True to explicitly trigger an immediate flush in tests.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-18 14:21:39 +00:00
Alan Rominger
c591eb4a7a Do not ignore errors on activity stream connection (#16344) 2026-03-17 17:04:13 -04:00
Alan Rominger
cc2fbf332c Stop writing tmp test files that are not cleaned up (#16358)
* Stop writing tmp test files that are not cleaned up
2026-03-17 17:02:47 -04:00
Alan Rominger
1646694258 Fix jinja2 error from newer ansible versions (#16356)
* Fix error from newer ansible versions

* Include fix for setting cachable

* Revert "Include fix for setting cachable"

This reverts commit 477293c258.
2026-03-17 13:36:33 -04:00
Jake Jackson
643a9849df update editable_dependencies readme (#15471)
fixed typo/project naming to match example.
2026-03-12 19:36:18 +00:00
Rodrigo Toshiaki Horie
8bd8bcda94 [AAP-68258] Fix SonarCloud Reliability Rating issue in Common exception constructor (#16351)
Fix SonarCloud Reliability Rating issue in Common exception constructor

The constructor had code paths where attributes were not consistently
initialized and super().__init__() was not called, which was flagged
as a Reliability Rating issue by SonarCloud. Ensures all branches
properly set self.status_string and self.msg, and call super().__init__().

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-12 15:21:01 -03:00
Alan Rominger
63f3c735ea Delete unused contains method (#16346) 2026-03-12 12:00:29 -04:00
Tong He
7e29f9e3f2 Enrich tests against is_ha_environment() 2026-03-11 11:43:12 +01:00
Pablo H.
c115e0168a Align log message of workload identity tokens with other task messages (#16345) 2026-03-10 14:05:31 -04:00
Andrea Restle-Lay
619d8c67a9 [AAP-63314] P4.4: Controller - Pass Workload TTL to Gateway (#16303)
* Pass workload TTL to Gateway (minimal changes) assisted-by: Claude

* lint
Assisted-by: Claude

* fix unit tests assisted-by claude

* use existing functions assisted-by: Claude

* fix test assisted-by: Claude

* fixes for sonarcloud assisted-by: Claude

* nit

* nit

* address feedback

* feedback from pr review assisted-by: Claude

* feedback from pr review assisted-by: Claude

* Apply suggestion from @dleehr

Co-authored-by: Dan Leehr <dleehr@users.noreply.github.com>

* lint assisted-by: Claude

* fix: narrow vendor_collections_dir fixture teardown scope (#16326)

Only remove the collection directory the fixture created
(redhat/indirect_accounting) instead of the entire
/var/lib/awx/vendor_collections/ root, so we don't accidentally
delete vendor collections that may have been installed by the
build process.

Forward-port of ansible/tower#7350.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* AAP-67436 Remove pbr from requirements (#16337)

* Remove pbr from requirements

pbr was temporarily added to support ansible-runner installed from a git
branch. It is no longer needed as a direct dependency.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Retrigger CI

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* [AAP-64062] Enforce JWT-only authentication for Controller when deployed as part of AAP (#16283)

After all settings are loaded, override DEFAULT_AUTHENTICATION_CLASSES
to only allow Gateway JWT authentication when RESOURCE_SERVER__URL is
set. This makes the lockdown immutable — no configuration file or
environment variable can re-enable legacy auth methods (Basic, Session,
OAuth2, Token).

This is the same pattern used by Hub (galaxy_ng) and EDA (eda-server)
for ANSTRAT-1840.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Re-trigger CI

Made-with: Cursor

* Re-trigger CI

Made-with: Cursor

* [AAP-63314] Pass job timeout as workload_ttl_seconds to Gateway    Assisted-by: Claude

* Additional unit test requested at review  Assisted-by: Claude

* Revert profiled_pg/base.py rebase error, unrelated to AAP-63314

* revert requirements changes introduced by testing

* revert

* revert

* docstring nit from coderabbit

---------

Co-authored-by: Dan Leehr <dleehr@users.noreply.github.com>
Co-authored-by: Dirk Julich <djulich@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Hao Liu <44379968+TheRealHaoLiu@users.noreply.github.com>
2026-03-10 08:54:28 -04:00
Hao Liu
0d08a4da60 [AAP-64062] Enforce JWT-only authentication for Controller when deployed as part of AAP (#16283)
After all settings are loaded, override DEFAULT_AUTHENTICATION_CLASSES
to only allow Gateway JWT authentication when RESOURCE_SERVER__URL is
set. This makes the lockdown immutable — no configuration file or
environment variable can re-enable legacy auth methods (Basic, Session,
OAuth2, Token).

This is the same pattern used by Hub (galaxy_ng) and EDA (eda-server)
for ANSTRAT-1840.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-09 17:46:31 +01:00
Hao Liu
36a1121cd8 AAP-67436 Remove pbr from requirements (#16337)
* Remove pbr from requirements

pbr was temporarily added to support ansible-runner installed from a git
branch. It is no longer needed as a direct dependency.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* Retrigger CI

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-09 15:53:38 +00:00
Dirk Julich
212546f92b fix: narrow vendor_collections_dir fixture teardown scope (#16326)
Only remove the collection directory the fixture created
(redhat/indirect_accounting) instead of the entire
/var/lib/awx/vendor_collections/ root, so we don't accidentally
delete vendor collections that may have been installed by the
build process.

Forward-port of ansible/tower#7350.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 14:19:24 +01:00
Adrià Sala
fad4881280 fix: include awxkit coverage in sonarqube 2026-03-06 09:41:38 +01:00
Adrià Sala
65b1867114 fix: help-context detection to ignore option values 2026-03-06 09:41:38 +01:00
Adrià Sala
1a3085ff40 fix: nitpick and upload awxkit coverage to ci 2026-03-06 09:41:38 +01:00
Adrià Sala
51ed59c506 fix: awxkit help flags for detailed & general help
Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>
2026-03-06 09:41:38 +01:00
Alan Rominger
670dfeed25 Address more ignored pytest warnings, co-authored with Opus 4.6 (#16298)
* Address more ignored pytest warnings

* Fix what we can with CI results

* Add new migration file
2026-03-05 13:51:19 -05:00
Alan Rominger
7384c73c9a Update hash for related action update (#16328) 2026-03-05 18:34:34 +00:00
Alan Rominger
25b43deec0 Address unused variables issue (#16327) 2026-03-05 12:39:18 -05:00
Seth Foster
f74f82e30c Forward port external query files from stable-2.6 (#16312)
* Revert "AAP-58452 Add version fallback for external query files (#16309)"

This reverts commit 0f2692b504.

* AAP-58441: Add runtime integration for external query collection (#7208)

Extend build_private_data_files() to copy vendor collections from
/var/lib/awx/vendor_collections/ to the job's private_data_dir,
making external query files available to the indirect node counting
callback plugin in execution environments.

Changes:
- Copy vendor_collections to private_data_dir during job preparation
- Add vendor_collections path to ANSIBLE_COLLECTIONS_PATH in build_env()
- Gracefully handle missing source directory with warning log
- Feature gated by FEATURE_INDIRECT_NODE_COUNTING_ENABLED flag

This enables external query file discovery for indirect node counting
across all deployment types (RPM, Podman, OpenShift, Kubernetes) using
the existing private_data_dir mechanism.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

* [stable-2.6] AAP-58451: Add callback plugin discovery for external query files (#7223)

* AAP-58451: Add callback plugin discovery for external query files

Extend the indirect_instance_count callback plugin to discover and load
external query files from the bundled redhat.indirect_accounting collection
when embedded queries are not present in the target collection.

Changes:
- Add external query discovery with precedence (embedded queries first)
- External query path: redhat.indirect_accounting/extensions/audit/
  external_queries/{namespace}.{name}.{version}.yml
- Use self._display.v() for external query messages (visible with -v)
- Use self._display.vv() for embedded query messages (visible with -vv)
- Fix: Change .exists() to .is_file() per Traversable ABC
- Handle missing external query collection gracefully (ModuleNotFoundError)

Note: This implements exact version match only. Version fallback logic
is covered in AAP-58452.

* fix CI error when using Traversable.is_file

* Add minimal implementation for AAP-58451

* Fix formatting

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>

* AAP-58452 Add version fallback for external query files (#7254)

* AAP-58456 unit test suite for external query handling (#7283)

* Add unit tests for external query handling

* Refactor unit tests for external query handling

* Refactor indirect node counting callback code to improve testing code

* Refactor unit tests for external query handling for improved callback code

* Fix test for majore version boundary check

* Fix weaknesses in some unit tests

* Make callback plugin module self contained, independent from awx

* AAP-58470 integration tests (core) for external queries (#7278)

* Add collection for testing external queries

* Add query files for testing external query file runtime integration

* Add live tests for external query file runtime integration

* Remove redundant wait for events and refactor test data folders

* Fix unit tests: mock flag_enabled to avoid DB access

The AAP-58441 cherry-pick added a flag_enabled() call in
BaseTask.build_private_data_files(), which is called by all task types.
Tests for RunInventoryUpdate and RunJob credentials now hit this code
path and need the flag mocked to avoid database access in unit tests.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: attempt exact query file match before Version parsing (#7345)

The exact-version filename check does not require PEP440 parsing, but
Version() was called first, causing early return on non-PEP440 version
strings even when an exact file exists on disk. Move the exact file
check before Version parsing so fallback logic only parses when needed.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* Do no longer mutate global sys.modules (#7337)

* [stable-2.6] AAP-58452 fix: Add queries_dir guard (#7338)

* Add queries_dir guard

* fix: update unit tests to mock _get_query_file_dir instead of files

The TestVersionFallback tests mocked `files()` with chainable path
mocks, but `find_external_query_with_fallback` now uses
`_get_query_file_dir()` which returns the queries directory directly.
Mock the helper instead for simpler, correct tests.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* fix: remove unused EXTERNAL_QUERY_PATH constant (#7336)

The constant was defined but never referenced — the path is constructed
inline via Traversable's `/` operator which requires individual segments,
not a slash-separated string.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

* fix: restore original feature flag state in test fixture (#7347)

The enable_indirect_host_counting fixture unconditionally disabled the
FEATURE_INDIRECT_NODE_COUNTING_ENABLED flag on teardown, even when it
was already enabled before the test (as is the case in development via
development_defaults.py). This caused test_indirect_host_counting to
fail when run after the external query tests, because the callback
plugin was no longer enabled.

Save and restore the original flag state instead.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Dirk Julich <djulich@redhat.com>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-03-05 14:05:58 +01:00
Alan Rominger
be5fbf365e AAP-65054 Fix bugs where concurrent jobs would clear facts of unrelated hosts (#16318)
* Add new tests for bug saving concurrent facts

* Fix first bug and improve tests

* Fix new bug where concurrent job clears facts from other job in unwanted way

* minor test fixes

* Add in missing playbook

* Fix host reference for constructed inventory

* Increase speed for concurrent fact tests

* Make test a bit faster

* Fix linters

* Add some functional tests

* Remove the sanity test

* Agent markers added

* Address SonarCloud

* Do backdating method, resolving stricter assertions

* Address coderabbit comments

* Address review comment with qs only method

* Delete missed sleep statement

* Add more coverage
2026-03-05 07:58:32 -05:00
Peter Braun
0995f7c5fe update bindep.txt to versionles dependencies (#16316) 2026-03-04 16:25:09 -07:00
Alan Rominger
3fbc71e6c8 Null value handling, discovered in production logs (#16323) 2026-03-04 14:48:21 -05:00
Alan Rominger
143d4cee34 Pin container versions (#16322)
* Pin container versions

* downgrades from coderabbit

* Add notes for future upgrades
2026-03-04 14:47:50 -05:00
Lila Yasin
af7fbea854 Fix LoggedLogoutView for Django 5.2 GET removal (#16317)
Django 5.2 restricts LogoutView to POST only (deprecated in 4.1,
  removed in 5.0+). Without this fix, GET requests to /api/logout/
  return 405 Method Not Allowed.

  Add http_method_names override and a get() method that delegates
  to post() where auth_logout() actually runs
2026-03-04 14:05:32 -05:00
Pablo H.
57f9eb093a feat: workload identity credentials integration (#16286)
* feat: workload identity credentials integration

* feat: cache credentials and add context property to Credential

Assisted-by: Claude

* feat: include safeguard in case feature flag is disabled

* feat: tests to validate workload identity credentials integration

* fix: affected tests by the credential cache mechanism

* feat: remove word cache from variables and comments, use standard library decorators

* fix: reorder tests in correct files

* Use better error catching mechanisms

* Adjust logic to support multiple credential input sources and use internal field

* Remove hardcoded credential type names

* Add tests for the internal field

Assited-by: Claude
2026-03-04 10:22:27 -05:00
Dirk Julich
8d191046b5 AAP-63318: Internal/developer documentation for indirect query files (#16319)
Add documentation for indirect query files
2026-03-04 15:45:44 +01:00
Stevenson Michel
7a5f0998d2 [Devel] Added HTTP_X_FORWARDED_FOR in Devel for production (#16314)
Added HTTP_X_FORWARDED_FOR in Devel for production
2026-03-03 17:08:38 -05:00
Stevenson Michel
d1f4fc3e97 [Devel] Addition of logic to trigger worflow dispatch on release_4.6 and stable-2.6 (#16315)
* Added worflow dispatch to trigger ci on release_4.6 and stable-2.6

* fix error cascading to subsequent jobs for cron

* made compose_tag resolve to the correct branch name
2026-03-02 18:48:42 -05:00
Lila Yasin
0f2692b504 AAP-58452 Add version fallback for external query files (#16309)
Forward port of 5c4653fbbca4b542fea05f070efea0396b034839 from stable-2.6.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 15:08:27 -05:00
Alan Rominger
e1e2c60f2e AAP-66379 Include scaledown fix from dispatcherd (#16305)
Include scaledown fix from dispatcherd
2026-02-27 14:45:57 -05:00
Alan Rominger
d8a2aa1dc3 Do not write to dispatcher.log from AWX application code (#16302) 2026-02-27 14:45:27 -05:00
Alan Rominger
9d61e42ede Bump action version (#16306) 2026-02-27 09:52:38 -05:00
Seth Foster
2c71bcda32 Improve transactional integrity for starting controller jobs in dispatcherd (#16300)
Remove SELECT FOR UPDATE from job dispatch to reduce transaction rollbacks
                                                                                                                                                                                                                                                                                           
  Move status transition from BaseTask.transition_status (which used
  SELECT FOR UPDATE inside transaction.atomic()) into                                                                                                                                                                                                                                      
  dispatch_waiting_jobs. The new approach uses filter().update() which                                                                                                                                                                                                                     
  is atomic at the database level without requiring explicit row locks,
  reducing transaction contention and rollbacks observed in perfscale
  testing.

  The transition_status method was an artifact of the feature flag era
  where we needed to support both old and new code paths. Since
  dispatch_waiting_jobs is already a singleton
  (on_duplicate='queue_one') scoped to the local node, the
  de-duplication logic is unnecessary.

  Status is updated after task submission to dispatcherd, so the job's
  UUID is in the dispatch pipeline before being marked running —
  preventing the reaper from incorrectly reaping jobs during the
  handoff window. RunJob.run() handles the race where a worker picks
  up the task before the status update lands by accepting waiting and
  transitioning it to running itself.

  Signed-off-by: Seth Foster <fosterbseth@gmail.com>
  Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-02-26 14:16:36 -05:00
Stevenson Michel
a21f9fbdb8 Addition of Cron Schedule (#16301)
Added cron schedule for ci jobs
2026-02-23 16:31:34 -05:00
Daniel Finca Martínez
2a35ce5524 AAP-62693 Integrate workload identity client to request JWTs (#16296)
* Add retrieve_workload_identity_jwt to jobs.py and tests

* Apply linting

* Add precondition to client retrieval

* Add test case for client not configured

* Remove trailing period in match string
2026-02-19 09:13:32 -05:00
Alan Rominger
567a980a03 Give error details of sliced jobs if they error in live tests (#16273) 2026-02-18 15:12:12 -05:00
Alan Rominger
9059cfbda6 Fix some pytest warnings using Opus 4.6 (#16269)
* Fix some pytest warnings using Opus 4.6

* Fix review comments

* Use raw-strings and regex markers for matching exception pattern

Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>

* Make regex work

* Undo always true assertion edit

---------

Co-authored-by: 🇺🇦 Sviatoslav Sydorenko (Святослав Сидоренко) <wk.cvs.github@sydorenko.org.ua>
2026-02-18 15:11:41 -05:00
Jake Jackson
d8fd953732 Update PR template to add steps to repro (#16294)
* Update PR template to add steps to repro

* updated bottom section to include steps to reproduce to help with
  debugging and triage
2026-02-18 15:01:36 -05:00
Stevenson Michel
39851c392a [Devel] Removal of Token Auth From Devel (#16293)
* Revert "[Devel][AAP-65384]Restoration of Token Authentication for AWX CLI (#16281)"

This reverts commit 994a2b3c04.
2026-02-18 14:32:44 -05:00
Chris Meyers
aeba4a1a3f Revert "Change remote host finding logic"
This reverts commit 08f1507f70.
2026-02-17 14:46:45 -05:00
Adrià Sala
915deca78c fix: awxkit user creation through gw 2026-02-17 15:38:20 +01:00
Peter Braun
1a79e853fe do not add optional survey fields with empty strings that are not bac… (#16289)
* do not add optional survey fields with empty strings that are not backed by extra_vars

* exclude password fields from skipping if not defined
2026-02-17 12:59:38 +01:00
239 changed files with 15123 additions and 1893 deletions

View File

@@ -24,7 +24,7 @@ in as the first entry for your PR title.
##### ADDITIONAL INFORMATION
##### STEPS TO REPRODUCE AND EXTRA INFO
<!---
Include additional information to help people understand the change here.
For bugs that don't have a linked bug report, a step-by-step reproduction

View File

@@ -4,14 +4,54 @@ env:
LC_ALL: "C.UTF-8" # prevent ERROR: Ansible could not initialize the preferred locale: unsupported locale setting
CI_GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
DEV_DOCKER_OWNER: ${{ github.repository_owner }}
COMPOSE_TAG: ${{ github.base_ref || 'devel' }}
COMPOSE_TAG: ${{ github.base_ref || github.ref_name || 'devel' }}
UPSTREAM_REPOSITORY_ID: 91594105
on:
pull_request:
push:
branches:
- devel # needed to publish code coverage post-merge
schedule:
- cron: '0 11,17 * * 1-5'
workflow_dispatch: {}
jobs:
trigger-release-branches:
name: "Dispatch CI to release branches"
if: github.event_name == 'schedule'
runs-on: ubuntu-latest
permissions:
actions: write
steps:
- name: Trigger CI on release_4.6
id: dispatch_release_46
continue-on-error: true
run: gh workflow run ci.yml --ref release_4.6
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GH_REPO: ${{ github.repository }}
- name: Trigger CI on stable-2.6
id: dispatch_stable_26
continue-on-error: true
run: gh workflow run ci.yml --ref stable-2.6
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GH_REPO: ${{ github.repository }}
- name: Trigger CI on stable-2.7
id: dispatch_stable_27
continue-on-error: true
run: gh workflow run ci.yml --ref stable-2.7
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GH_REPO: ${{ github.repository }}
- name: Check dispatch results
if: steps.dispatch_release_46.outcome == 'failure' || steps.dispatch_stable_26.outcome == 'failure' || steps.dispatch_stable_27.outcome == 'failure'
run: |
echo "One or more dispatches failed:"
echo " release_4.6: ${{ steps.dispatch_release_46.outcome }}"
echo " stable-2.6: ${{ steps.dispatch_stable_26.outcome }}"
echo " stable-2.7: ${{ steps.dispatch_stable_27.outcome }}"
exit 1
common-tests:
name: ${{ matrix.tests.name }}
runs-on: ubuntu-latest
@@ -62,7 +102,11 @@ jobs:
run: |
if [ -f "reports/coverage.xml" ]; then
sed -i '2i<!-- PR ${{ github.event.pull_request.number }} -->' reports/coverage.xml
echo "Injected PR number ${{ github.event.pull_request.number }} into coverage.xml"
echo "Injected PR number ${{ github.event.pull_request.number }} into reports/coverage.xml"
fi
if [ -f "awxkit/coverage.xml" ]; then
sed -i '2i<!-- PR ${{ github.event.pull_request.number }} -->' awxkit/coverage.xml
echo "Injected PR number ${{ github.event.pull_request.number }} into awxkit/coverage.xml"
fi
- name: Upload test coverage to Codecov
@@ -109,7 +153,9 @@ jobs:
uses: actions/upload-artifact@v4
with:
name: ${{ matrix.tests.name }}-artifacts
path: reports/coverage.xml
path: |
reports/coverage.xml
awxkit/coverage.xml
retention-days: 5
- name: >-
@@ -122,7 +168,7 @@ jobs:
&& github.event_name == 'push'
&& env.UPSTREAM_REPOSITORY_ID == github.repository_id
&& github.ref_name == github.event.repository.default_branch
uses: ansible/gh-action-record-test-results@cd5956ead39ec66351d0779470c8cff9638dd2b8
uses: ansible/gh-action-record-test-results@3784db66a1b7fb3809999a7251c8a7203a7ffbe8
with:
aggregation-server-url: ${{ vars.PDE_ORG_RESULTS_AGGREGATOR_UPLOAD_URL }}
http-auth-password: >-
@@ -214,7 +260,7 @@ jobs:
continue-on-error: true
run: |
set +e
timeout 15m bash -elc '
timeout 20m bash -elc '
python -m pip install -r molecule/requirements.txt
python -m pip install PyYAML # for awx/tools/scripts/rewrite-awx-operator-requirements.py
$(realpath ../awx/tools/scripts/rewrite-awx-operator-requirements.py) molecule/requirements.yml $(realpath ../awx)
@@ -296,7 +342,7 @@ jobs:
&& github.event_name == 'push'
&& env.UPSTREAM_REPOSITORY_ID == github.repository_id
&& github.ref_name == github.event.repository.default_branch
uses: ansible/gh-action-record-test-results@cd5956ead39ec66351d0779470c8cff9638dd2b8
uses: ansible/gh-action-record-test-results@3784db66a1b7fb3809999a7251c8a7203a7ffbe8
with:
aggregation-server-url: ${{ vars.PDE_ORG_RESULTS_AGGREGATOR_UPLOAD_URL }}
http-auth-password: >-

View File

@@ -13,6 +13,10 @@ on:
- stable-*
jobs:
push-development-images:
if: |
github.event_name == 'workflow_dispatch' ||
(github.repository == 'ansible/awx' && (github.ref_name == 'devel' || startsWith(github.ref_name, 'feature_'))) ||
(github.repository == 'ansible/tower' && (startsWith(github.ref_name, 'stable-') || startsWith(github.ref_name, 'release_')))
runs-on: ubuntu-latest
timeout-minutes: 120
permissions:
@@ -30,12 +34,6 @@ jobs:
make-target: awx-kube-buildx
steps:
- name: Skipping build of awx image for non-awx repository
run: |
echo "Skipping build of awx image for non-awx repository"
exit 0
if: matrix.build-targets.image-name == 'awx' && !endsWith(github.repository, '/awx')
- uses: actions/checkout@v4
with:
show-progress: false

View File

@@ -12,7 +12,7 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 20
permissions:
packages: write
packages: read
contents: read
steps:
- name: Check for each of the lines

View File

@@ -16,9 +16,15 @@ on:
push:
branches:
- devel
- 'stable-2.[6-9]'
- 'stable-2.[1-9][0-9]'
workflow_dispatch: # Allow manual triggering for testing
jobs:
sync-openapi-spec:
if: |
github.event_name == 'workflow_dispatch' ||
(github.repository == 'ansible/awx' && (github.ref_name == 'devel' || startsWith(github.ref_name, 'feature_'))) ||
(github.repository == 'ansible/tower' && (startsWith(github.ref_name, 'stable-') || startsWith(github.ref_name, 'release_')))
name: Sync OpenAPI spec to central repo
runs-on: ubuntu-latest
permissions:
@@ -37,9 +43,12 @@ jobs:
private-github-key: ${{ secrets.PRIVATE_GITHUB_KEY }}
- name: Generate API Schema
env:
REF_NAME: ${{ github.ref_name }}
BASE_REF: ${{ github.base_ref }}
run: |
DEV_DOCKER_TAG_BASE=ghcr.io/${OWNER_LC} \
COMPOSE_TAG=${{ github.base_ref || github.ref_name }} \
COMPOSE_TAG=${BASE_REF:-${REF_NAME}} \
docker run -u $(id -u) --rm -v ${{ github.workspace }}:/awx_devel/:Z \
--workdir=/awx_devel `make print-DEVEL_IMAGE_NAME` /start_tests.sh genschema
@@ -66,9 +75,11 @@ jobs:
- name: Fail if branch doesn't exist
if: steps.checkout_spec_repo.outcome == 'failure'
env:
REF_NAME: ${{ github.ref_name }}
run: |
echo "##[error]❌ Branch '${{ github.ref_name }}' does not exist in the central spec repository."
echo "##[error]Expected branch: ${{ github.ref_name }}"
echo "##[error]❌ Branch '${REF_NAME}' does not exist in the central spec repository."
echo "##[error]Expected branch: ${REF_NAME}"
echo "##[error]This branch must be created in the spec repo before specs can be synced."
exit 1
@@ -106,31 +117,47 @@ jobs:
working-directory: spec-repo
env:
GH_TOKEN: ${{ secrets.OPENAPI_SPEC_SYNC_TOKEN }}
GPG_PRIVATE_KEY: ${{ secrets.OPENAPI_SPEC_SYNC_GPG_PRIVATE_KEY }}
COMMIT_MESSAGE: ${{ github.event.head_commit.message }}
SPEC_REPO: ansible-automation-platform/aap-openapi-specs
REF_NAME: ${{ github.ref_name }}
GITHUB_SHA_FULL: ${{ github.sha }}
GITHUB_REPO: ${{ github.repository }}
IS_NEW_FILE: ${{ steps.compare.outputs.is_new_file }}
run: |
# Configure git
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
# Import GPG key and configure git for signed commits
echo "$GPG_PRIVATE_KEY" | gpg --batch --import 2>/dev/null
GPG_KEY_ID=$(gpg --list-secret-keys --keyid-format long 2>/dev/null | grep sec | head -1 | awk '{print $2}' | cut -d'/' -f2)
if [ -z "$GPG_KEY_ID" ]; then
echo "❌ Failed to import GPG key or extract key ID"
exit 1
fi
git config user.name "aap-api-bot"
git config user.email "aap-api-bot@redhat.com"
git config commit.gpgsign true
git config user.signingkey "$GPG_KEY_ID"
# Configure git to use the token for push
git remote set-url origin "https://x-access-token:${GH_TOKEN}@github.com/${SPEC_REPO}.git"
SHORT_SHA="${GITHUB_SHA_FULL:0:7}"
BRANCH_NAME="update-Controller-${REF_NAME}-${SHORT_SHA}"
# Create branch for PR
SHORT_SHA="${{ github.sha }}"
SHORT_SHA="${SHORT_SHA:0:7}"
BRANCH_NAME="update-Controller-${{ github.ref_name }}-${SHORT_SHA}"
git checkout -b "$BRANCH_NAME"
# Add and commit changes
git add "controller.json"
if [ "${{ steps.compare.outputs.is_new_file }}" == "true" ]; then
COMMIT_MSG="Add Controller OpenAPI spec for ${{ github.ref_name }}"
if [ "${IS_NEW_FILE}" == "true" ]; then
COMMIT_MSG="Add Controller OpenAPI spec for ${REF_NAME}"
else
COMMIT_MSG="Update Controller OpenAPI spec for ${{ github.ref_name }}"
COMMIT_MSG="Update Controller OpenAPI spec for ${REF_NAME}"
fi
git commit -m "$COMMIT_MSG
git commit -m "${COMMIT_MSG}
Synced from ${{ github.repository }}@${{ github.sha }}
Source branch: ${{ github.ref_name }}
Synced from ${GITHUB_REPO}@${GITHUB_SHA_FULL}
Source branch: ${REF_NAME}
Co-Authored-By: github-actions[bot] <github-actions[bot]@users.noreply.github.com>"
@@ -138,17 +165,17 @@ jobs:
git push origin "$BRANCH_NAME"
# Create PR
PR_TITLE="[${{ github.ref_name }}] Update Controller spec from merged commit"
PR_TITLE="[${REF_NAME}] Update Controller spec from merged commit"
PR_BODY="## Summary
Automated OpenAPI spec sync from component repository merge.
**Source:** ${{ github.repository }}@${{ github.sha }}
**Branch:** \`${{ github.ref_name }}\`
**Source:** ${GITHUB_REPO}@${GITHUB_SHA_FULL}
**Branch:** \`${REF_NAME}\`
**Component:** \`Controller\`
**Spec File:** \`controller.json\`
## Changes
$(if [ "${{ steps.compare.outputs.is_new_file }}" == "true" ]; then echo "- 🆕 New spec file created"; else echo "- 📝 Spec file updated with latest changes"; fi)
$(if [ "${IS_NEW_FILE}" == "true" ]; then echo "- 🆕 New spec file created"; else echo "- 📝 Spec file updated with latest changes"; fi)
## Source Commit
\`\`\`
@@ -159,17 +186,20 @@ jobs:
🤖 This PR was automatically generated by the OpenAPI spec sync workflow."
gh pr create \
--repo "${SPEC_REPO}" \
--title "$PR_TITLE" \
--body "$PR_BODY" \
--base "${{ github.ref_name }}" \
--base "${REF_NAME}" \
--head "$BRANCH_NAME"
echo "✅ Created PR in spec repo"
- name: Report results
if: always()
env:
HAS_DIFF: ${{ steps.compare.outputs.has_diff }}
run: |
if [ "${{ steps.compare.outputs.has_diff }}" == "true" ]; then
if [ "${HAS_DIFF}" == "true" ]; then
echo "📝 Spec sync completed - PR created in spec repo"
else
echo "✅ Spec sync completed - no changes needed"

View File

@@ -14,6 +14,10 @@ on:
- stable-**
jobs:
push:
if: |
github.event_name == 'workflow_dispatch' ||
(github.repository == 'ansible/awx' && (github.ref_name == 'devel' || startsWith(github.ref_name, 'feature_'))) ||
(github.repository == 'ansible/tower' && (startsWith(github.ref_name, 'stable-') || startsWith(github.ref_name, 'release_')))
runs-on: ubuntu-latest
timeout-minutes: 60
permissions:

View File

@@ -0,0 +1,65 @@
---
apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
name: awx-atf-tests-pull-request
annotations:
build.appstudio.openshift.io/repo: https://github.com/{{repo_owner}}/{{repo_name}}?rev={{revision}}
build.appstudio.redhat.com/commit_sha: '{{revision}}'
build.appstudio.redhat.com/pull_request_number: '{{pull_request_number}}'
build.appstudio.redhat.com/target_branch: '{{target_branch}}'
pipelinesascode.tekton.dev/cancel-in-progress: 'true'
pipelinesascode.tekton.dev/max-keep-runs: "3"
pipelinesascode.tekton.dev/on-comment: "^/run-atf-tests$"
pipelinesascode.tekton.dev/target-namespace: ansible-ci-tenant
labels:
appstudio.openshift.io/application: '{{repo_owner}}'
appstudio.openshift.io/component: '{{repo_owner}}-{{repo_name}}'
pipelines.appstudio.openshift.io/type: build
spec:
timeouts:
pipeline: "8h"
tasks: "7h"
finally: "1h"
pipelineRef:
resolver: bundles
params:
- name: name
value: aap-api-tests
- name: bundle
value: quay.io/aap-ci/tekton-catalog/pipeline/test/aap-api-tests:0.1@sha256:d88df867d552bb7139897dd2167efa64181c595ad87c3e52eea70813b3c66868
- name: kind
value: pipeline
- name: secret
value: quay-aap-ci-viewer
taskRunTemplate:
serviceAccountName: konflux-integration-runner
params:
- name: git-url
value: "{{source_url}}"
- name: pipeline-github-org
value: "{{repo_owner}}"
- name: pipeline-github-repo
value: "{{repo_name}}"
- name: pipeline-github-target-branch
value: '{{target_branch}}'
- name: pipeline-github-pr-revision
value: "{{revision}}"
- name: pipeline-github-pr-number
value: "{{pull_request_number}}"
- name: aap-dev-component-source-name
value: "controller"
- name: pytest-number-of-parallel-processes
value: "6"
workspaces:
- name: workspace
volumeClaimTemplate:
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi

View File

@@ -103,6 +103,12 @@ When necessary, remove any AWX containers and images by running the following:
### Pre commit hooks
Install the pre-commit hook before contributing:
```
make pre-commit
```
When you attempt to perform a `git commit` there will be a pre-commit hook that gets run before the commit is allowed to your local repository. For example, python's [black](https://pypi.org/project/black/) will be run to test the formatting of any python files.
While you can use environment variables to skip the pre-commit hooks GitHub will run similar tests and prevent merging of PRs if the tests do not pass.

View File

@@ -10,6 +10,7 @@ KIND_BIN ?= $(shell which kind)
CHROMIUM_BIN=/tmp/chrome-linux/chrome
GIT_REPO_NAME ?= $(shell basename `git rev-parse --show-toplevel`)
GIT_BRANCH ?= $(shell git rev-parse --abbrev-ref HEAD)
GIT_IS_WORKTREE := $(shell test -f .git && echo yes)
MANAGEMENT_COMMAND ?= awx-manage
VERSION ?= $(shell $(PYTHON) tools/scripts/scm_version.py 2> /dev/null)
@@ -106,6 +107,15 @@ else
DOCKER_KUBE_CACHE_FLAG=$(DOCKER_CACHE)
endif
# AWX TUI variables
AWX_HOST ?= https://localhost:8043
AWX_USER ?= admin
AWX_PASSWORD ?= $$(awk -F"'" '/^admin_password:/{print $$2}' tools/docker-compose/_sources/secrets/admin_password.yml 2>/dev/null || echo "admin")
AWX_VERIFY_SSL ?= false
# For git worktree to find the referenced git dir
GIT_COMMON_DIR := $(shell git rev-parse --git-common-dir 2>/dev/null || echo .git)
.PHONY: awx-link clean clean-tmp clean-venv requirements requirements_dev \
update_requirements upgrade_requirements update_requirements_dev \
docker_update_requirements docker_upgrade_requirements docker_update_requirements_dev \
@@ -113,7 +123,7 @@ endif
receiver test test_unit test_coverage coverage_html \
sdist \
VERSION PYTHON_VERSION docker-compose-sources \
.git/hooks/pre-commit
pre-commit
clean-tmp:
rm -rf tmp/
@@ -342,11 +352,10 @@ black: reports
@command -v black >/dev/null 2>&1 || { echo "could not find black on your PATH, you may need to \`pip install black\`, or set AWX_IGNORE_BLACK=1" && exit 1; }
@(set -o pipefail && $@ $(BLACK_ARGS) awx awxkit awx_collection | tee reports/$@.report)
.git/hooks/pre-commit:
@echo "if [ -x pre-commit.sh ]; then" > .git/hooks/pre-commit
@echo " ./pre-commit.sh;" >> .git/hooks/pre-commit
@echo "fi" >> .git/hooks/pre-commit
@chmod +x .git/hooks/pre-commit
$(GIT_COMMON_DIR)/hooks/pre-commit:
ln -sf ../../pre-commit.sh $(GIT_COMMON_DIR)/hooks/pre-commit
pre-commit: $(GIT_COMMON_DIR)/hooks/pre-commit
genschema: awx-link reports
@if [ "$(VENV_BASE)" ]; then \
@@ -521,7 +530,7 @@ ifneq ($(ADMIN_PASSWORD),)
EXTRA_SOURCES_ANSIBLE_OPTS := -e admin_password=$(ADMIN_PASSWORD) $(EXTRA_SOURCES_ANSIBLE_OPTS)
endif
docker-compose-sources: .git/hooks/pre-commit
docker-compose-sources:
@if [ $(MINIKUBE_CONTAINER_GROUP) = true ]; then\
$(ANSIBLE_PLAYBOOK) -i tools/docker-compose/inventory -e minikube_setup=$(MINIKUBE_SETUP) tools/docker-compose-minikube/deploy.yml; \
fi;
@@ -553,7 +562,7 @@ docker-compose: awx/projects docker-compose-sources
$(MAKE) docker-compose-up
docker-compose-up:
$(DOCKER_COMPOSE) -f tools/docker-compose/_sources/docker-compose.yml $(COMPOSE_OPTS) up $(COMPOSE_UP_OPTS) --remove-orphans
$(if $(GIT_IS_WORKTREE),SETUPTOOLS_SCM_PRETEND_VERSION="$(VERSION)") $(DOCKER_COMPOSE) -f tools/docker-compose/_sources/docker-compose.yml $(COMPOSE_OPTS) up $(COMPOSE_UP_OPTS) --remove-orphans
docker-compose-down:
$(DOCKER_COMPOSE) -f tools/docker-compose/_sources/docker-compose.yml $(COMPOSE_OPTS) down --remove-orphans
@@ -571,6 +580,20 @@ docker-compose-runtest: awx/projects docker-compose-sources
docker-compose-build-schema: awx/projects docker-compose-sources
$(DOCKER_COMPOSE) -f tools/docker-compose/_sources/docker-compose.yml run --rm --service-ports --no-deps awx_1 make genschema
awx-tui:
@if ! command -v awx-tui > /dev/null 2>&1; then \
$(PYTHON) -m pip install awx-tui; \
fi
@if [ -f "$(HOME)/.config/awx-tui/config.yaml" ]; then \
$(PYTHON) -m awx_tui.main; \
else \
AWX_HOST=$(AWX_HOST) \
AWX_USER=$(AWX_USER) \
AWX_PASSWORD=$(AWX_PASSWORD) \
AWX_VERIFY_SSL=$(AWX_VERIFY_SSL) \
$(PYTHON) -m awx_tui.main --host $(AWX_HOST); \
fi
SCHEMA_DIFF_BASE_FOLDER ?= awx
SCHEMA_DIFF_BASE_BRANCH ?= devel
detect-schema-change: genschema
@@ -581,7 +604,7 @@ detect-schema-change: genschema
validate-openapi-schema: genschema
@echo "Validating OpenAPI schema from schema.json..."
@python3 -c "from openapi_spec_validator import validate; import json; spec = json.load(open('schema.json')); validate(spec); print('✓ OpenAPI Schema is valid!')"
@python3 -c "from openapi_spec_validator import validate; import json; spec = json.load(open('schema.json')); validate(spec); print('✓ Schema is valid')"
docker-compose-clean: awx/projects
$(DOCKER_COMPOSE) -f tools/docker-compose/_sources/docker-compose.yml rm -sf

View File

@@ -52,14 +52,6 @@ except ImportError: # pragma: no cover
MODE = 'production'
try:
import django # noqa: F401
except ImportError:
pass
else:
from django.db import connection
def prepare_env():
# Update the default settings environment variable based on current mode.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'awx.settings')
@@ -79,14 +71,6 @@ def manage():
from django.conf import settings
from django.core.management import execute_from_command_line
# enforce the postgres version is a minimum of 12 (we need this for partitioning); if not, then terminate program with exit code of 1
# In the future if we require a feature of a version of postgres > 12 this should be updated to reflect that.
# The return of connection.pg_version is something like 12013
if not os.getenv('SKIP_PG_VERSION_CHECK', False) and not MODE == 'development':
if (connection.pg_version // 10000) < 12:
sys.stderr.write("At a minimum, postgres version 12 is required\n")
sys.exit(1)
if len(sys.argv) >= 2 and sys.argv[1] in ('version', '--version'): # pragma: no cover
sys.stdout.write('%s\n' % __version__)
# If running as a user without permission to read settings, display an

View File

@@ -131,8 +131,14 @@ class LoggedLoginView(auth_views.LoginView):
class LoggedLogoutView(auth_views.LogoutView):
# Override http_method_names to allow GET requests (Django 5.2+ defaults to POST only)
http_method_names = ["get", "post", "options"]
success_url_allowed_hosts = set(settings.LOGOUT_ALLOWED_HOSTS.split(",")) if settings.LOGOUT_ALLOWED_HOSTS else set()
def get(self, request, *args, **kwargs):
"""Handle GET requests for logout (for backward compatibility)."""
return self.post(request, *args, **kwargs)
def dispatch(self, request, *args, **kwargs):
if is_proxied_request():
# 1) We intentionally don't obey ?next= here, just always redirect to platform login
@@ -266,7 +272,10 @@ class APIView(views.APIView):
response = self.handle_exception(self.__init_request_error__)
if response.status_code == 401:
if response.data and 'detail' in response.data:
response.data['detail'] += _(' To establish a login session, visit') + ' /api/login/.'
if getattr(settings, 'RESOURCE_SERVER__URL', None):
response.data['detail'] += _(' Direct access is not allowed, authenticate via the platform gateway.')
else:
response.data['detail'] += _(' To establish a login session, visit') + ' /api/login/.'
logger.info(status_msg)
else:
logger.warning(status_msg)

View File

@@ -0,0 +1,471 @@
{
"activity_stream_retrieve": "Retrieve an audit trail entry for tracking all changes within the system",
"ad_hoc_commands_activity_stream_list": "List activity stream of an ad hoc command",
"ad_hoc_commands_create": "Create an ad hoc command",
"ad_hoc_commands_destroy": "Delete an ad hoc command",
"ad_hoc_commands_events_list": "List events of an ad hoc command",
"ad_hoc_commands_list": "List ad hoc commands",
"ad_hoc_commands_notifications_list": "List notifications of an ad hoc command",
"ad_hoc_commands_retrieve": "Retrieve an ad hoc command",
"ad_hoc_commands_stdout_retrieve": "Retrieve a stdout output of an ad hoc command",
"analytics_adoption_rate_options_retrieve": "Retrieve single analytics adoption rate option",
"analytics_adoption_rate_retrieve": "Retrieve single analytics adoption rate",
"analytics_event_explorer_options_retrieve": "Retrieve single analytics event explorer option",
"analytics_event_explorer_retrieve": "Retrieve single analytics event explorer",
"analytics_host_explorer_options_retrieve": "Retrieve single analytics host explorer option",
"analytics_host_explorer_retrieve": "Retrieve single analytics host explorer",
"analytics_job_explorer_options_retrieve": "Retrieve single analytics job explorer option",
"analytics_job_explorer_retrieve": "Retrieve single analytics job explorer",
"analytics_probe_template_for_hosts_options_retrieve": "Retrieve single analytics probe template for hosts option",
"analytics_probe_template_for_hosts_retrieve": "Retrieve single analytics probe template for host",
"analytics_probe_templates_options_retrieve": "Retrieve single analytics probe templates option",
"analytics_probe_templates_retrieve": "Retrieve single analytics probe template",
"analytics_reports_retrieve": "Retrieve single analytics report",
"analytics_roi_templates_options_retrieve": "Retrieve single analytics roi templates option",
"analytics_roi_templates_retrieve": "Retrieve single analytics roi template",
"constructed_inventories_create": "Create a constructed inventory",
"constructed_inventories_destroy": "Delete a constructed inventory",
"constructed_inventories_partial_update": "Update a constructed inventory",
"constructed_inventories_retrieve": "Retrieve a constructed inventory",
"constructed_inventories_update": "Update a constructed inventory",
"credential_input_sources_create": "Create a credential input source",
"credential_input_sources_destroy": "Delete a credential input source",
"credential_input_sources_list": "List credential input sources",
"credential_input_sources_partial_update": "Update a credential input source",
"credential_input_sources_retrieve": "Retrieve a credential input source",
"credential_input_sources_update": "Update a credential input source",
"credential_types_credentials_create": "Create a credential of a credential type",
"credential_types_credentials_list": "List credentials of a credential type",
"credential_types_retrieve": "Retrieve a credential type",
"credential_types_test_retrieve": "Retrieve single test for a credential_type",
"credentials_destroy": "Delete a credential",
"credentials_input_sources_create": "Create new source for a credential",
"credentials_input_sources_list": "List all sources for a credential",
"credentials_object_roles_list": "List roles of a credential",
"credentials_owner_teams_list": "List all teams for a credential",
"credentials_owner_users_list": "List all users for a credential",
"credentials_partial_update": "Update a credential",
"credentials_retrieve": "Retrieve a credential",
"credentials_test_retrieve": "Retrieve a test external credential",
"credentials_update": "Update a credential",
"execution_environments_activity_stream_list": "List activity stream of an execution environment",
"execution_environments_copy_create": "Create new copy for an execution_environment",
"execution_environments_copy_retrieve": "Retrieve single copy for an execution_environment",
"execution_environments_retrieve": "Retrieve an execution environment",
"execution_environments_unified_job_templates_list": "List unified job templates using this execution environment",
"feature_flags_state_retrieve": "Retrieve single feature flags state",
"feature_flags_states_list": "List all feature flags states",
"feature_flags_states_retrieve": "Retrieve single feature flags state",
"groups_activity_stream_list": "List activity stream for a group",
"groups_ad_hoc_commands_create": "Create an ad hoc command for a group",
"groups_ad_hoc_commands_list": "List ad hoc commands for a group",
"groups_all_hosts_list": "List all hosts for a group",
"groups_children_create": "Create new child for a group",
"groups_children_list": "List all children for a group",
"groups_destroy": "Delete a group",
"groups_hosts_create": "Create a host of a group",
"groups_hosts_list": "List hosts of a group",
"groups_inventory_sources_list": "List inventory sources of a group",
"groups_job_events_list": "List job events for a group",
"groups_job_host_summaries_list": "List job host summaries for a group",
"groups_partial_update": "Update a group",
"groups_potential_children_list": "List all children for a group",
"groups_retrieve": "Retrieve a group",
"groups_update": "Update a group",
"groups_variable_data_partial_update": "Update a variable datum for a group",
"groups_variable_data_retrieve": "Retrieve a variable datum for a group",
"groups_variable_data_update": "Update a variable datum for a group",
"host_metric_summary_monthly_list": "List monthly summaries for host metrics",
"host_metrics_list": "List host metrics",
"host_metrics_retrieve": "Retrieve a host metric",
"hosts_activity_stream_list": "List activity stream for a host",
"hosts_ad_hoc_command_events_list": "List events of ad hoc command of a host",
"hosts_ad_hoc_commands_create": "Create an ad hoc command of a host",
"hosts_ad_hoc_commands_list": "List ad hoc commands of a host",
"hosts_all_groups_list": "List all groups for a host",
"hosts_create": "Create a host",
"hosts_groups_create": "Create the list of groups a host is directly a member of",
"hosts_groups_list": "List the list of groups a host is directly a member of",
"hosts_inventory_sources_list": "List inventory sources of a host",
"hosts_job_events_list": "List job events of a host",
"hosts_job_host_summaries_list": "List job summaries of a host",
"hosts_partial_update": "Update a host",
"hosts_retrieve": "Retrieve a host",
"hosts_smart_inventories_list": "List all inventories for a host",
"hosts_update": "Update a host",
"hosts_variable_data_partial_update": "Update a variable datum for a host",
"hosts_variable_data_update": "Update a variable datum for a host",
"instance_groups_destroy": "Delete an instance group",
"instance_groups_instances_create": "Create an instance of an instance group",
"instance_groups_instances_list": "List instance of an instance group",
"instance_groups_jobs_list": "List jobs of an instance group",
"instance_groups_object_roles_list": "List all roles for an instance_group",
"instance_groups_partial_update": "Update an instance group",
"instance_groups_retrieve": "Retrieve an instance group",
"instance_groups_update": "Update an instance group",
"instances_instance_groups_create": "Create an instance group of an instance",
"instances_instance_groups_list": "List instance groups of an instance",
"instances_jobs_list": "List jobs executed on an instance",
"instances_list": "List instances",
"instances_partial_update": "Update an instance",
"instances_peers_list": "List all peers for an instance",
"instances_retrieve": "Retrieve an instance",
"instances_update": "Update an instance",
"inventories_access_list_list": "List users who can access the inventory",
"inventories_ad_hoc_commands_create": "Create an ad hoc command for an inventory",
"inventories_ad_hoc_commands_list": "List ad hoc command for an inventory",
"inventories_copy_create": "Create a copy of an inventory",
"inventories_copy_retrieve": "Retrieve a copy of an inventory",
"inventories_create": "Create an inventory",
"inventories_destroy": "Delete an inventory",
"inventories_groups_create": "Create a group of an inventory",
"inventories_groups_list": "List groups of an inventory",
"inventories_hosts_create": "Create a host of an inventory",
"inventories_hosts_list": "List hosts of an inventory",
"inventories_instance_groups_create": "Create an instance group of an inventory",
"inventories_instance_groups_list": "List instance groups of an inventory",
"inventories_inventory_sources_create": "Create an inventory source",
"inventories_inventory_sources_list": "List inventory sources",
"inventories_job_templates_list": "List job templates using an inventory",
"inventories_labels_list": "List labels of an inventory",
"inventories_object_roles_list": "List roles of an inventory",
"inventories_partial_update": "Update an inventory",
"inventories_retrieve": "Retrieve an inventory",
"inventories_update": "Update an inventory",
"inventories_update_inventory_sources_retrieve": "Retrieve single source for an inventory",
"inventories_variable_data_partial_update": "Partially update existing datum for an inventory",
"inventories_variable_data_retrieve": "Retrieve single datum for an inventory",
"inventories_variable_data_update": "Update existing datum for an inventory",
"inventory_sources_activity_stream_list": "List activity stream of an inventory source",
"inventory_sources_create": "Create an inventory source",
"inventory_sources_credentials_create": "Create a credential of an inventory source",
"inventory_sources_credentials_list": "List credentials of an inventory source",
"inventory_sources_destroy": "Delete an inventory source",
"inventory_sources_groups_destroy": "Delete a group of an inventory source",
"inventory_sources_groups_list": "List groups of an inventory source",
"inventory_sources_hosts_destroy": "Delete a host of an inventory source",
"inventory_sources_hosts_list": "List hosts of an inventory source",
"inventory_sources_inventory_updates_list": "List inventory updates of an inventory source",
"inventory_sources_list": "List inventory sources",
"inventory_sources_notification_templates_error_list": "List notification templates triggered on inventory source update error",
"inventory_sources_notification_templates_started_list": "List notification templates triggered on inventory source update start",
"inventory_sources_notification_templates_success_list": "List notification templates triggered on inventory source update success",
"inventory_sources_partial_update": "Update an inventory source",
"inventory_sources_retrieve": "Retrieve an inventory source",
"inventory_sources_schedules_create": "Create a schedule of an inventory source",
"inventory_sources_schedules_list": "List schedules of an inventory source",
"inventory_sources_update": "Update an inventory source",
"inventory_sources_update_retrieve": "Retrieve an update for an inventory source",
"inventory_updates_cancel_create": "Create a cancel for an inventory update",
"inventory_updates_cancel_retrieve": "Retrieve a cancel for an inventory update",
"inventory_updates_credentials_list": "List credentials of an inventory update",
"inventory_updates_destroy": "Delete an inventory update",
"inventory_updates_events_list": "List events of an inventory update",
"inventory_updates_list": "List inventory updates",
"inventory_updates_notifications_list": "List notifications of an inventory update",
"inventory_updates_retrieve": "Retrieve an inventory update",
"inventory_updates_stdout_retrieve": "Retrieve a stdout output of an inventory update",
"job_events_children_list": "List child events of a job event",
"job_events_retrieve": "Retrieve a job event detail",
"job_host_summaries_retrieve": "Retrieve a job host summary detail",
"job_templates_access_list_list": "List users who can access a job template",
"job_templates_activity_stream_list": "List activity stream of a job template",
"job_templates_copy_create": "Create a copy a job template",
"job_templates_copy_retrieve": "Retrieve a copy a job template",
"job_templates_create": "Create a job template",
"job_templates_credentials_create": "Create a credential of a job template",
"job_templates_credentials_list": "List credentials of a job template",
"job_templates_destroy": "Delete a job template",
"job_templates_instance_groups_create": "Create an instance group of a job template",
"job_templates_instance_groups_list": "List instance groups of a job template",
"job_templates_jobs_list": "List jobs of a job template",
"job_templates_labels_list": "List labels of a job template",
"job_templates_launch_retrieve": "Retrieve single launch for a job_template",
"job_templates_notification_templates_error_create": "Create a notification templates triggered on job error",
"job_templates_notification_templates_error_list": "List notification templates triggered on job error",
"job_templates_notification_templates_started_create": "Create a notification templates triggered on job start",
"job_templates_notification_templates_started_list": "List notification templates triggered on job start",
"job_templates_notification_templates_success_create": "Create a notification templates triggered on job success",
"job_templates_notification_templates_success_list": "List notification templates triggered on job success",
"job_templates_object_roles_list": "List roles of a job template",
"job_templates_partial_update": "Update a job template",
"job_templates_retrieve": "Retrieve a job template",
"job_templates_schedules_create": "Create a schedule of a job template",
"job_templates_schedules_list": "List schedules of a job template",
"job_templates_slice_workflow_jobs_create": "Create new job for a job_template",
"job_templates_slice_workflow_jobs_list": "List all jobs for a job_template",
"job_templates_update": "Update a job template",
"jobs_activity_stream_list": "List activity stream of a job",
"jobs_cancel_retrieve": "Retrieve a cancel for a job",
"jobs_create_schedule_retrieve": "Retrieve single schedule for a job",
"jobs_credentials_list": "List credentials of a job",
"jobs_destroy": "Delete a job",
"jobs_job_events_list": "List job events of a job",
"jobs_job_host_summaries_list": "List job host summaries of a job",
"jobs_labels_list": "List labels of a job",
"jobs_notifications_list": "List notifications of a job",
"jobs_relaunch_retrieve": "Retrieve single relaunch for a job",
"jobs_retrieve": "Retrieve a job",
"labels_create": "Create a label",
"labels_list": "List labels",
"labels_partial_update": "Update a label",
"labels_retrieve": "Retrieve a label",
"labels_update": "Update a label",
"me_list": "List current authenticated user",
"notification_templates_copy_create": "Create a copy a notification template",
"notification_templates_copy_retrieve": "Retrieve a copy a notification template",
"notification_templates_notifications_list": "List notifications of a notification template",
"notification_templates_retrieve": "Retrieve a notification template",
"notifications_list": "List notifications",
"notifications_retrieve": "Retrieve a notification",
"organizations_access_list_list": "List users who can access the organization",
"organizations_activity_stream_list": "List activity stream for an organization",
"organizations_admins_create": "Create new admin for an organization",
"organizations_admins_list": "List all admins for an organization",
"organizations_create": "Create an organization",
"organizations_credentials_create": "Create a credential of an organization",
"organizations_credentials_list": "List credentials of an organization",
"organizations_destroy": "Delete an organization",
"organizations_execution_environments_create": "Create an execution environment of an organization",
"organizations_execution_environments_list": "List execution environments of an organization",
"organizations_galaxy_credentials_create": "Create new credential for an organization",
"organizations_galaxy_credentials_list": "List all credentials for an organization",
"organizations_instance_groups_create": "Create an instance group of an organization",
"organizations_instance_groups_list": "List instance groups of an organization",
"organizations_inventories_list": "List inventories of an organization",
"organizations_job_templates_create": "Create a job template of an organization",
"organizations_job_templates_list": "List job templates of an organization",
"organizations_notification_templates_approvals_create": "Create new approval for an organization",
"organizations_notification_templates_approvals_list": "List all approvals for an organization",
"organizations_notification_templates_create": "Create a notification template of an organization",
"organizations_notification_templates_error_create": "Create new error for an organization",
"organizations_notification_templates_error_list": "List all error for an organization",
"organizations_notification_templates_list": "List notification templates of an organization",
"organizations_notification_templates_started_create": "Create new started for an organization",
"organizations_notification_templates_started_list": "List all started for an organization",
"organizations_notification_templates_success_create": "Create new success for an organization",
"organizations_notification_templates_success_list": "List all success for an organization",
"organizations_object_roles_list": "List roles of an organization",
"organizations_partial_update": "Update an organization",
"organizations_projects_create": "Create a project of an organization",
"organizations_projects_list": "List projects of an organization",
"organizations_retrieve": "Retrieve an organization",
"organizations_retrieve_2": "Retrieve an organization",
"organizations_teams_create": "Create a team of an organization",
"organizations_teams_list": "List teams of an organization",
"organizations_update": "Update an organization",
"organizations_users_create": "Create a user of an organization",
"organizations_users_list": "List users of an organization",
"organizations_workflow_job_templates_create": "Create a workflow job template of an organization",
"organizations_workflow_job_templates_list": "List workflow job templates of an organization",
"project_updates_cancel_create": "Create new cancel for a project_update",
"project_updates_cancel_retrieve": "Retrieve single cancel for a project_update",
"project_updates_destroy": "Delete a project update",
"project_updates_events_list": "List all events for a project_update",
"project_updates_list": "List project updates",
"project_updates_notifications_list": "List notifications of a project update",
"project_updates_retrieve": "Retrieve a project update",
"project_updates_scm_inventory_updates_list": "List all updates for a project_update",
"project_updates_stdout_retrieve": "Retrieve single stdout for a project_update",
"projects_access_list_list": "List users who can access the project",
"projects_activity_stream_list": "List activity stream for a project",
"projects_copy_create": "Create a copy of a project",
"projects_copy_retrieve": "Retrieve a copy of a project",
"projects_create": "Create a project",
"projects_destroy": "Delete a project",
"projects_inventories_retrieve": "Retrieve an inventory from a project",
"projects_notification_templates_error_create": "Create a notification template for project error events",
"projects_notification_templates_error_list": "List notification templates for project error events",
"projects_notification_templates_started_create": "Create a notification template for project started events",
"projects_notification_templates_started_list": "List notification templates for project started events",
"projects_notification_templates_success_create": "Create a notification template for project success events",
"projects_notification_templates_success_list": "List notification templates for project success events",
"projects_object_roles_list": "List roles of a project",
"projects_partial_update": "Update a project",
"projects_playbooks_retrieve": "Retrieve single playbook for a project",
"projects_project_updates_list": "List project updates of a project",
"projects_retrieve": "Retrieve a project",
"projects_schedules_create": "Create a schedule of a project",
"projects_schedules_list": "List schedules of a project",
"projects_scm_inventory_sources_list": "List all sources for a project",
"projects_teams_list": "List teams with access to a project",
"projects_update": "Update a project",
"projects_update_retrieve": "Retrieve single update for a project",
"receptor_addresses_list": "List receptor addresses",
"receptor_addresses_retrieve": "Retrieve a receptor address",
"role_definitions_create": "Create a RBAC roles defining permissions that can be managed and assigned to users and teams",
"role_definitions_destroy": "Delete a RBAC roles defining permissions that can be managed and assigned to users and teams",
"role_definitions_list": "List RBAC roles defining permissions that can be managed and assigned to users and teams",
"role_definitions_partial_update": "Update a RBAC roles defining permissions that can be managed and assigned to users and teams",
"role_definitions_retrieve": "Retrieve a RBAC roles defining permissions that can be managed and assigned to users and teams",
"role_definitions_team_assignments_list": "List all assignments for a role_definition",
"role_definitions_update": "Update a RBAC roles defining permissions that can be managed and assigned to users and teams",
"role_definitions_user_assignments_list": "List all assignments for a role_definition",
"role_metadata_retrieve": "Retrieve single role metadatum",
"role_team_access_list": "List all role team access",
"role_team_access_list_2": "List all role team access",
"role_team_assignments_create": "Create a RBAC role grants assigning permissions to team for specific resources",
"role_team_assignments_destroy": "Delete a RBAC role grants assigning permissions to team for specific resources",
"role_team_assignments_list": "List RBAC role grants assigning permissions to teams for specific resources",
"role_team_assignments_retrieve": "Retrieve a RBAC role grants assigning permissions to team for specific resources",
"role_user_access_list": "List all role user access",
"role_user_access_list_2": "List all role user access",
"role_user_assignments_create": "Create a RBAC role grants assigning permissions to user for specific resources",
"role_user_assignments_destroy": "Delete a RBAC role grants assigning permissions to user for specific resources",
"role_user_assignments_list": "List RBAC role grants assigning permissions to users for specific resources",
"role_user_assignments_retrieve": "Retrieve a RBAC role grants assigning permissions to user for specific resources",
"roles_list": "List roles",
"roles_retrieve": "Retrieve a role",
"roles_teams_list": "List teams with a role",
"roles_users_list": "List users with a role",
"schedules_create": "Create a schedule",
"schedules_credentials_create": "Create a credential of a schedule",
"schedules_credentials_list": "List credentials of a schedule",
"schedules_destroy": "Delete a schedule",
"schedules_instance_groups_create": "Create an instance group of a schedule",
"schedules_instance_groups_list": "List instance groups of a schedule",
"schedules_jobs_list": "List jobs created by a schedule",
"schedules_labels_list": "List labels of a schedule",
"schedules_list": "List schedules",
"schedules_partial_update": "Update a schedule",
"schedules_retrieve": "Retrieve a schedule",
"schedules_update": "Update a schedule",
"service_index_metadata_retrieve": "Retrieve single service index metadatum",
"service_index_resource_types_list": "List all service index resource types",
"service_index_resource_types_manifest_retrieve": "Retrieve single manifest for a resource-type",
"service_index_resource_types_retrieve": "Retrieve single service index resource type",
"service_index_resources_create": "Create new service index resource",
"service_index_resources_destroy": "Delete existing service index resource",
"service_index_resources_list": "List all service index resources",
"service_index_resources_partial_update": "Partially update existing service index resource",
"service_index_resources_retrieve": "Retrieve single service index resource",
"service_index_resources_update": "Update existing service index resource",
"service_index_retrieve": "Retrieve single service index",
"service_index_role_permissions_list": "List all service index role permissions",
"service_index_role_team_assignments_assign_create": "Create new service index role team assignments assign",
"service_index_role_team_assignments_list": "List all service index role team assignments",
"service_index_role_team_assignments_unassign_create": "Create new service index role team assignments unassign",
"service_index_role_types_list": "List all service index role types",
"service_index_role_user_assignments_assign_create": "Create new service index role user assignments assign",
"service_index_role_user_assignments_list": "List all service index role user assignments",
"service_index_role_user_assignments_unassign_create": "Create new service index role user assignments unassign",
"settings_destroy": "Delete existing setting",
"settings_logging_test_create": "Create new settings logging test",
"settings_retrieve": "Retrieve single setting",
"settings_update": "Update existing setting",
"system_job_templates_jobs_list": "List system jobs of a system job template",
"system_job_templates_notification_templates_error_create": "Create a notification templates triggered on system job error",
"system_job_templates_notification_templates_error_list": "List notification templates triggered on system job error",
"system_job_templates_notification_templates_started_create": "Create a notification templates triggered on system job start",
"system_job_templates_notification_templates_started_list": "List notification templates triggered on system job start",
"system_job_templates_notification_templates_success_create": "Create a notification templates triggered on system job success",
"system_job_templates_notification_templates_success_list": "List notification templates triggered on system job success",
"system_job_templates_retrieve": "Retrieve a system job template",
"system_job_templates_schedules_create": "Create a schedule of a system job template",
"system_job_templates_schedules_list": "List schedules of a system job template",
"system_jobs_cancel_create": "Create a cancel for a system job",
"system_jobs_cancel_retrieve": "Retrieve a cancel for a system job",
"system_jobs_destroy": "Delete a system job",
"system_jobs_events_list": "List events of a system job",
"system_jobs_notifications_list": "List notifications of a system job",
"system_jobs_retrieve": "Retrieve a system job",
"teams_access_list_list": "List users who can access the team",
"teams_activity_stream_list": "List activity stream for a team",
"teams_create": "Create a team",
"teams_credentials_create": "Create a credentials owned by a team",
"teams_credentials_list": "List credentials owned by a team",
"teams_destroy": "Delete a team",
"teams_list": "List teams",
"teams_object_roles_list": "List object roles of a team",
"teams_partial_update": "Update a team",
"teams_projects_list": "List projects accessible to a team",
"teams_retrieve": "Retrieve a team",
"teams_roles_list": "List roles of a team",
"teams_update": "Update a team",
"teams_users_create": "Create a user of a team",
"teams_users_list": "List users of a team",
"unified_job_templates_list": "List unified job templates",
"unified_jobs_list": "List unified jobs",
"users_access_list_list": "List users who can access the user",
"users_activity_stream_list": "List activity stream for a user",
"users_admin_of_organizations_retrieve": "Retrieve single organization for an user",
"users_create": "Create a user",
"users_credentials_create": "Create a credentials owned by a user",
"users_credentials_list": "List credentials owned by a user",
"users_destroy": "Delete a user",
"users_list": "List users",
"users_organizations_retrieve": "Retrieve an organization of a user",
"users_partial_update": "Update a user",
"users_projects_list": "List projects accessible to a user",
"users_retrieve": "Retrieve a user",
"users_roles_list": "List roles of a user",
"users_teams_list": "List teams of a user",
"users_update": "Update a user",
"workflow_approval_templates_approvals_list": "List all approvals for a workflow_approval_template",
"workflow_approval_templates_destroy": "Delete a workflow approval template detail",
"workflow_approval_templates_partial_update": "Update a workflow approval template detail",
"workflow_approval_templates_retrieve": "Retrieve a workflow approval template detail",
"workflow_approval_templates_update": "Update a workflow approval template detail",
"workflow_approvals_approve_retrieve": "Retrieve single approve for a workflow_approval",
"workflow_approvals_deny_retrieve": "Retrieve single deny for a workflow_approval",
"workflow_approvals_destroy": "Delete a workflow approval",
"workflow_approvals_retrieve": "Retrieve a workflow approval",
"workflow_job_nodes_always_nodes_list": "List always nodes of a workflow job node",
"workflow_job_nodes_credentials_list": "List credentials of a workflow job node",
"workflow_job_nodes_failure_nodes_list": "List failure nodes of a workflow job node",
"workflow_job_nodes_instance_groups_create": "Create an instance group of a workflow job node",
"workflow_job_nodes_instance_groups_list": "List instance groups of a workflow job node",
"workflow_job_nodes_labels_list": "List labels of a workflow job node",
"workflow_job_nodes_list": "List workflow job nodes",
"workflow_job_nodes_retrieve": "Retrieve a workflow job node",
"workflow_job_nodes_success_nodes_list": "List success nodes of a workflow job node",
"workflow_job_template_nodes_always_nodes_create": "Create new node for a workflow_job_template_node",
"workflow_job_template_nodes_always_nodes_list": "List all nodes for a workflow_job_template_node",
"workflow_job_template_nodes_create": "Create a workflow job template node",
"workflow_job_template_nodes_create_approval_template_retrieve": "Retrieve single template for a workflow_job_template_node",
"workflow_job_template_nodes_credentials_create": "Create a credential of a workflow job template node",
"workflow_job_template_nodes_credentials_list": "List credentials of a workflow job template node",
"workflow_job_template_nodes_destroy": "Delete a workflow job template node",
"workflow_job_template_nodes_failure_nodes_create": "Create new node for a workflow_job_template_node",
"workflow_job_template_nodes_failure_nodes_list": "List all nodes for a workflow_job_template_node",
"workflow_job_template_nodes_instance_groups_create": "Create an instance group of a workflow job template node",
"workflow_job_template_nodes_instance_groups_list": "List instance groups of a workflow job template node",
"workflow_job_template_nodes_labels_list": "List labels of a workflow job template node",
"workflow_job_template_nodes_list": "List workflow job template nodes",
"workflow_job_template_nodes_partial_update": "Update a workflow job template node",
"workflow_job_template_nodes_retrieve": "Retrieve a workflow job template node",
"workflow_job_template_nodes_success_nodes_create": "Create new node for a workflow_job_template_node",
"workflow_job_template_nodes_success_nodes_list": "List all nodes for a workflow_job_template_node",
"workflow_job_template_nodes_update": "Update a workflow job template node",
"workflow_job_templates_access_list_list": "List users who can access a workflow job template",
"workflow_job_templates_activity_stream_list": "List activity stream of a workflow job template",
"workflow_job_templates_copy_create": "Create a copy a workflow job template",
"workflow_job_templates_create": "Create a workflow job template",
"workflow_job_templates_destroy": "Delete a workflow job template",
"workflow_job_templates_labels_list": "List labels of a workflow job template",
"workflow_job_templates_launch_retrieve": "Retrieve a launch a workflow job from a workflow job template",
"workflow_job_templates_notification_templates_approvals_create": "Create a notification templates triggered on workflow approval",
"workflow_job_templates_notification_templates_approvals_list": "List notification templates triggered on workflow approval",
"workflow_job_templates_notification_templates_error_create": "Create a notification templates triggered on workflow job error",
"workflow_job_templates_notification_templates_error_list": "List notification templates triggered on workflow job error",
"workflow_job_templates_notification_templates_started_create": "Create a notification templates triggered on workflow job start",
"workflow_job_templates_notification_templates_started_list": "List notification templates triggered on workflow job start",
"workflow_job_templates_notification_templates_success_create": "Create a notification templates triggered on workflow job success",
"workflow_job_templates_notification_templates_success_list": "List notification templates triggered on workflow job success",
"workflow_job_templates_object_roles_list": "List roles of a workflow job template",
"workflow_job_templates_partial_update": "Update a workflow job template",
"workflow_job_templates_retrieve": "Retrieve a workflow job template",
"workflow_job_templates_schedules_create": "Create a schedule of a workflow job template",
"workflow_job_templates_schedules_list": "List schedules of a workflow job template",
"workflow_job_templates_update": "Update a workflow job template",
"workflow_job_templates_workflow_jobs_list": "List workflow jobs of a workflow job template",
"workflow_job_templates_workflow_nodes_create": "Create new node for a workflow_job_template",
"workflow_job_templates_workflow_nodes_list": "List all nodes for a workflow_job_template",
"workflow_jobs_activity_stream_list": "List activity stream of a workflow job",
"workflow_jobs_cancel_retrieve": "Retrieve a cancel for a workflow job",
"workflow_jobs_destroy": "Delete a workflow job",
"workflow_jobs_labels_list": "List labels of a workflow job",
"workflow_jobs_notifications_list": "List notifications of a workflow job",
"workflow_jobs_retrieve": "Retrieve a workflow job",
"workflow_jobs_workflow_nodes_list": "List workflow nodes of a workflow job"
}

View File

@@ -6,12 +6,15 @@ from collections import OrderedDict
# Django REST Framework
from django.conf import settings
from django.core.paginator import Paginator as DjangoPaginator
from django.utils.functional import cached_property
from rest_framework import pagination
from rest_framework.response import Response
from rest_framework.utils.urls import replace_query_param
from rest_framework.settings import api_settings
from django.utils.translation import gettext_lazy as _
from awx.main.models import ActivityStream, UnifiedJob
class DisabledPaginator(DjangoPaginator):
@property
@@ -23,6 +26,22 @@ class DisabledPaginator(DjangoPaginator):
return 200
class ActivityStreamPaginator(DjangoPaginator):
"""Use unfiltered table count for activity stream pagination (AAP-83773)."""
@cached_property
def count(self):
return ActivityStream.objects.count()
class UnifiedJobPaginator(DjangoPaginator):
"""Use unfiltered table count for unified job pagination."""
@cached_property
def count(self):
return UnifiedJob.objects.count()
class Pagination(pagination.PageNumberPagination):
page_size_query_param = 'page_size'
max_page_size = settings.MAX_PAGE_SIZE
@@ -57,12 +76,13 @@ class Pagination(pagination.PageNumberPagination):
def paginate_queryset(self, queryset, request, **kwargs):
self.count_disabled = 'count_disabled' in request.query_params
original_paginator = self.django_paginator_class
try:
if self.count_disabled:
self.django_paginator_class = DisabledPaginator
return super(Pagination, self).paginate_queryset(queryset, request, **kwargs)
finally:
self.django_paginator_class = DjangoPaginator
self.django_paginator_class = original_paginator
def get_paginated_response(self, data):
if self.count_disabled:
@@ -70,6 +90,14 @@ class Pagination(pagination.PageNumberPagination):
return super(Pagination, self).get_paginated_response(data)
class ActivityStreamPagination(Pagination):
django_paginator_class = ActivityStreamPaginator
class UnifiedJobPagination(Pagination):
django_paginator_class = UnifiedJobPaginator
class LimitPagination(pagination.BasePagination):
default_limit = api_settings.PAGE_SIZE
limit_query_param = 'limit'

View File

@@ -1,3 +1,5 @@
import json
import os
import warnings
from rest_framework.permissions import IsAuthenticated
@@ -53,6 +55,37 @@ def filter_credential_type_schema(
return result
def inject_ai_descriptions(
result,
generator, # NOSONAR
request, # NOSONAR
public, # NOSONAR
):
"""
Inject x-ai-description into operations from the overlay file.
Many endpoints have human-readable AI descriptions that were added
downstream but not backported as @extend_schema_if_available decorators.
This hook merges them from a JSON file keyed by operationId.
"""
overlay_path = os.path.join(os.path.dirname(__file__), 'openapi_ai_descriptions.json')
try:
with open(overlay_path) as f:
descriptions = json.load(f)
except (FileNotFoundError, json.JSONDecodeError):
return result
for path_item in result.get('paths', {}).values():
for operation in path_item.values():
if not isinstance(operation, dict):
continue
op_id = operation.get('operationId')
if op_id and op_id in descriptions and 'x-ai-description' not in operation:
operation['x-ai-description'] = descriptions[op_id]
return result
class CustomAutoSchema(AutoSchema):
"""Custom AutoSchema to add swagger_topic to tags and handle deprecated endpoints."""

View File

@@ -120,8 +120,7 @@ from awx.main.utils.named_url_graph import reset_counters
from awx.main.utils.inventory_vars import update_group_variables
from awx.main.scheduler.task_manager_models import TaskManagerModels
from awx.main.redact import UriCleaner, REPLACE_STR
from awx.main.signals import update_inventory_computed_fields
from awx.main.tasks.system import update_inventory_computed_fields
from awx.main.validators import vars_validate_or_raise
@@ -175,8 +174,8 @@ SUMMARIZABLE_FK_FIELDS = {
'workflow_approval': DEFAULT_SUMMARY_FIELDS + ('timeout',),
'schedule': DEFAULT_SUMMARY_FIELDS + ('next_run',),
'unified_job_template': DEFAULT_SUMMARY_FIELDS + ('unified_job_type',),
'last_job': DEFAULT_SUMMARY_FIELDS + ('finished', 'status', 'failed', 'license_error', 'canceled_on'),
'last_job_host_summary': DEFAULT_SUMMARY_FIELDS + ('failed',),
# last_job and last_job_host_summary are derived from JobHostSummary in HostSerializer,
# not from the stale FK fields on Host.
'last_update': DEFAULT_SUMMARY_FIELDS + ('status', 'failed', 'license_error'),
'current_update': DEFAULT_SUMMARY_FIELDS + ('status', 'failed', 'license_error'),
'current_job': DEFAULT_SUMMARY_FIELDS + ('status', 'failed', 'license_error'),
@@ -962,14 +961,27 @@ class UnifiedJobSerializer(BaseSerializer):
class UnifiedJobListSerializer(UnifiedJobSerializer):
OPTIONAL_EXCLUDE_FIELDS = frozenset({'artifacts', 'extra_vars'})
_ALWAYS_STRIPPED_FIELDS = frozenset({'job_args', 'job_cwd', 'job_env', 'result_traceback', 'event_processing_finished'})
class Meta:
fields = ('*', '-job_args', '-job_cwd', '-job_env', '-result_traceback', '-event_processing_finished', '-artifacts')
fields = ('*', '-job_args', '-job_cwd', '-job_env', '-result_traceback', '-event_processing_finished')
def _requested_excludes(self):
request = self.context.get('request')
if request is None:
return frozenset()
raw = request.query_params.get('exclude', '')
requested = {name.strip() for name in raw.split(',') if name.strip()}
return frozenset(requested) & self.OPTIONAL_EXCLUDE_FIELDS
def get_field_names(self, declared_fields, info):
field_names = super(UnifiedJobListSerializer, self).get_field_names(declared_fields, info)
# Meta multiple inheritance and -field_name options don't seem to be
# taking effect above, so remove the undesired fields here.
return tuple(x for x in field_names if x not in ('job_args', 'job_cwd', 'job_env', 'result_traceback', 'event_processing_finished', 'artifacts'))
strip = self._ALWAYS_STRIPPED_FIELDS | self._requested_excludes()
return tuple(x for x in field_names if x not in strip)
def get_types(self):
if type(self) is UnifiedJobListSerializer:
@@ -1022,7 +1034,7 @@ class UnifiedJobStdoutSerializer(UnifiedJobSerializer):
class UserSerializer(BaseSerializer):
password = serializers.CharField(required=False, default='', help_text=_('Field used to change the password.'))
password = serializers.CharField(required=False, default='', allow_blank=True, help_text=_('Field used to change the password.'))
is_system_auditor = serializers.BooleanField(default=False)
show_capabilities = ['edit', 'delete']
@@ -1838,19 +1850,35 @@ class HostSerializer(BaseSerializerWithVariables):
res['ansible_facts'] = self.reverse('api:host_ansible_facts_detail', kwargs={'pk': obj.instance_id})
if obj.inventory:
res['inventory'] = self.reverse('api:inventory_detail', kwargs={'pk': obj.inventory.pk})
if obj.last_job:
res['last_job'] = self.reverse('api:job_detail', kwargs={'pk': obj.last_job.pk})
if obj.last_job_host_summary:
res['last_job_host_summary'] = self.reverse('api:job_host_summary_detail', kwargs={'pk': obj.last_job_host_summary.pk})
last_summary = obj.latest_summary
if last_summary:
res['last_job_host_summary'] = self.reverse('api:job_host_summary_detail', kwargs={'pk': last_summary.pk})
if last_summary.job_id:
res['last_job'] = self.reverse('api:job_detail', kwargs={'pk': last_summary.job_id})
return res
def get_summary_fields(self, obj):
d = super(HostSerializer, self).get_summary_fields(obj)
try:
d['last_job']['job_template_id'] = obj.last_job.job_template.id
d['last_job']['job_template_name'] = obj.last_job.job_template.name
except (KeyError, AttributeError):
pass
last_summary = obj.latest_summary
if last_summary:
d['last_job_host_summary'] = OrderedDict()
d['last_job_host_summary']['id'] = last_summary.id
d['last_job_host_summary']['failed'] = last_summary.failed
try:
last_job = last_summary.job
d['last_job'] = OrderedDict()
for field in DEFAULT_SUMMARY_FIELDS + ('finished', 'status', 'failed', 'canceled_on'):
fval = getattr(last_job, field, None)
if fval is not None:
d['last_job'][field] = fval
if last_job.job_template:
d['last_job']['job_template_id'] = last_job.job_template.id
d['last_job']['job_template_name'] = last_job.job_template.name
except ObjectDoesNotExist:
pass
else:
d.pop('last_job', None)
d.pop('last_job_host_summary', None)
if has_model_field_prefetched(obj, 'groups'):
group_list = sorted([{'id': g.id, 'name': g.name} for g in obj.groups.all()], key=lambda x: x['id'])[:5]
else:
@@ -1925,14 +1953,16 @@ class HostSerializer(BaseSerializerWithVariables):
return ret
if 'inventory' in ret and not obj.inventory:
ret['inventory'] = None
if 'last_job' in ret and not obj.last_job:
ret['last_job'] = None
if 'last_job_host_summary' in ret and not obj.last_job_host_summary:
ret['last_job_host_summary'] = None
last_summary = obj.latest_summary
if 'last_job' in ret:
ret['last_job'] = last_summary.job_id if last_summary else None
if 'last_job_host_summary' in ret:
ret['last_job_host_summary'] = last_summary.pk if last_summary else None
return ret
def get_has_active_failures(self, obj):
return bool(obj.last_job_host_summary and obj.last_job_host_summary.failed)
last_summary = obj.latest_summary
return bool(last_summary and last_summary.failed)
def get_has_inventory_sources(self, obj):
return obj.inventory_sources.exists()
@@ -2079,9 +2109,17 @@ class BulkHostCreateSerializer(serializers.Serializer):
if request and not request.user.is_superuser:
if request.user not in inv.admin_role:
raise serializers.ValidationError(_(f'Inventory with id {inv.id} not found or lack permissions to add hosts.'))
current_hostnames = set(inv.hosts.values_list('name', flat=True))
# Performance optimization (AAP-67978): Instead of loading ALL host names from
# the inventory, only check if the specific new names already exist in the database.
new_names = [host['name'] for host in attrs['hosts']]
duplicate_new_names = [n for n in new_names if n in current_hostnames or new_names.count(n) > 1]
new_name_counts = Counter(new_names)
duplicates_in_new = [name for name, count in new_name_counts.items() if count > 1]
unique_new_names = list(new_name_counts.keys())
existing_duplicates = list(Host.objects.filter(inventory=inv, name__in=unique_new_names).values_list('name', flat=True))
duplicate_new_names = list(set(duplicates_in_new + existing_duplicates))
if duplicate_new_names:
raise serializers.ValidationError(_(f'Hostnames must be unique in an inventory. Duplicates found: {duplicate_new_names}'))
@@ -2932,6 +2970,19 @@ class CredentialTypeSerializer(BaseSerializer):
field['label'] = _(field['label'])
if 'help_text' in field:
field['help_text'] = _(field['help_text'])
# Deep copy inputs to avoid modifying the original model data
inputs = value.get('inputs')
if not isinstance(inputs, dict):
inputs = {}
value['inputs'] = copy.deepcopy(inputs)
fields = value['inputs'].get('fields', [])
if not isinstance(fields, list):
fields = []
# Normalize fields and filter out internal fields
value['inputs']['fields'] = [f for f in fields if not f.get('internal')]
return value
def filter_field_metadata(self, fields, method):
@@ -4122,9 +4173,28 @@ class LaunchConfigurationBaseSerializer(BaseSerializer):
attrs['extra_data'][key] = db_extra_data[key]
# Build unsaved version of this config, use it to detect prompts errors
# Capture keys before _build_mock_obj pops pseudo-fields from attrs
incoming_attr_keys = set(attrs.keys())
mock_obj = self._build_mock_obj(attrs)
if set(list(ujt.get_ask_mapping().keys()) + ['extra_data']) & set(attrs.keys()):
accepted, rejected, errors = ujt._accept_or_ignore_job_kwargs(_exclude_errors=self.exclude_errors, **mock_obj.prompts_dict())
ask_mapping_keys = set(ujt.get_ask_mapping().keys())
requested_prompt_fields = incoming_attr_keys & ask_mapping_keys
if 'extra_data' in incoming_attr_keys:
requested_prompt_fields.add('extra_vars')
requested_prompt_fields.add('survey_passwords')
# prompts_dict() pulls persisted M2M state (labels, credentials,
# instance_groups) via the instance pk. Only re-validate the full prompt
# state when the caller is switching the underlying template; otherwise
# restrict validation to the fields the request explicitly provided.
if 'unified_job_template' in attrs:
prompts_to_validate = mock_obj.prompts_dict()
elif requested_prompt_fields:
prompts_to_validate = {k: v for k, v in mock_obj.prompts_dict().items() if k in requested_prompt_fields}
else:
prompts_to_validate = None
if prompts_to_validate is not None:
accepted, rejected, errors = ujt._accept_or_ignore_job_kwargs(_exclude_errors=self.exclude_errors, **prompts_to_validate)
else:
# Only perform validation of prompts if prompts fields are provided
errors = {}
@@ -5393,7 +5463,11 @@ class SchedulePreviewSerializer(BaseSerializer):
for a_rule in match_multiple_rrule:
if 'interval' not in a_rule.lower():
errors.append("{0}: {1}".format(_('INTERVAL required in rrule'), a_rule))
elif 'secondly' in a_rule.lower():
else:
match_interval = re.match(r".*?INTERVAL=([0-9]+)", a_rule)
if match_interval and int(match_interval.group(1)) < 1:
errors.append("{0}: {1}".format(_("INTERVAL must be a positive integer"), a_rule))
if 'secondly' in a_rule.lower():
errors.append("{0}: {1}".format(_('SECONDLY is not supported'), a_rule))
if re.match(by_day_with_numeric_prefix, a_rule):
errors.append("{0}: {1}".format(_("BYDAY with numeric prefix not supported"), a_rule))

View File

@@ -1,4 +1,4 @@
---
collections:
- name: ansible.receptor
version: 2.0.6
version: 2.0.8

View File

View File

@@ -14,13 +14,14 @@ import sys
import time
from base64 import b64encode
from collections import OrderedDict
from jwt import decode as _jwt_decode
from urllib3.exceptions import ConnectTimeoutError
# Django
from django.conf import settings
from django.core.exceptions import FieldError, ObjectDoesNotExist
from django.db.models import Q, Sum, Count
from django.db.models import Q, Sum, Count, Subquery, OuterRef
from django.db import IntegrityError, ProgrammingError, transaction, connection
from django.db.models.fields.related import ManyToManyField, ForeignKey
from django.db.models.functions import Trunc
@@ -55,10 +56,16 @@ from wsgiref.util import FileWrapper
from drf_spectacular.utils import extend_schema_view, extend_schema
# django-ansible-base
from ansible_base.lib.utils.requests import get_remote_hosts
from ansible_base.rbac.models import RoleEvaluation
from ansible_base.lib.utils.schema import extend_schema_if_available
from ansible_base.lib.workload_identity.controller import AutomationControllerJobScope
# flags
from flags.state import flag_enabled
# AWX
from awx.main.utils.workload_identity import retrieve_workload_identity_jwt_with_claims
from awx.main.tasks.system import send_notifications, update_inventory_computed_fields
from awx.main.access import get_user_queryset
from awx.api.generics import (
@@ -97,7 +104,6 @@ from awx.main.utils import (
from awx.main.utils.encryption import encrypt_value
from awx.main.utils.filters import SmartFilter
from awx.main.utils.plugins import compute_cloud_inventory_sources
from awx.main.utils.proxy import get_first_remote_host_from_headers
from awx.main.utils.common import memoize
from awx.main.redact import UriCleaner
from awx.api.permissions import (
@@ -121,8 +127,9 @@ from awx.api.views.mixin import (
RelatedJobsPreventDeleteMixin,
UnifiedJobDeletionMixin,
NoTruncateMixin,
UnifiedJobExcludeMixin,
)
from awx.api.pagination import UnifiedJobEventPagination
from awx.api.pagination import ActivityStreamPagination, UnifiedJobEventPagination, UnifiedJobPagination
from awx.main.utils import set_environ
logger = logging.getLogger('awx.api.views')
@@ -203,11 +210,12 @@ class DashboardView(APIView):
groups_inventory_failed = models.Group.objects.filter(inventory_sources__last_job_failed=True).count()
data['groups'] = {'url': reverse('api:group_list', request=request), 'total': user_groups.count(), 'inventory_failed': groups_inventory_failed}
user_hosts = get_user_queryset(request.user, models.Host)
user_hosts_failed = user_hosts.filter(last_job_host_summary__failed=True)
user_hosts = get_user_queryset(request.user, models.Host).exclude(inventory__kind='constructed')
latest_summary_failed = Subquery(models.JobHostSummary.objects.filter(host_id=OuterRef('pk')).order_by('-id').values('failed')[:1])
user_hosts_failed = user_hosts.annotate(_latest_failed=latest_summary_failed).filter(_latest_failed=True)
data['hosts'] = {
'url': reverse('api:host_list', request=request),
'failures_url': reverse('api:host_list', request=request) + "?last_job_host_summary__failed=True",
'total': user_hosts.count(),
'failed': user_hosts_failed.count(),
}
@@ -794,22 +802,11 @@ class TeamRolesList(SubListAttachDetachAPIView):
data = dict(msg=_("You cannot grant system-level permissions to a team."))
return Response(data, status=status.HTTP_400_BAD_REQUEST)
team = get_object_or_404(models.Team, pk=self.kwargs['pk'])
credential_content_type = ContentType.objects.get_for_model(models.Credential)
if role.content_type == credential_content_type:
if not role.content_object.organization:
data = dict(
msg=_("You cannot grant access to a credential that is not assigned to an organization (private credentials cannot be assigned to teams)")
)
return Response(data, status=status.HTTP_400_BAD_REQUEST)
elif role.content_object.organization.id != team.organization.id:
if not request.user.is_superuser:
data = dict(
msg=_(
"You cannot grant a team access to a credential in a different organization. Only superusers can grant cross-organization credential access to teams"
)
)
return Response(data, status=status.HTTP_400_BAD_REQUEST)
if not request.data.get('disassociate'):
team = get_object_or_404(models.Team, pk=self.kwargs['pk'])
content_object = role.content_object
if hasattr(content_object, 'validate_role_assignment'):
content_object.validate_role_assignment(team, role_definition=None, requesting_user=request.user)
return super(TeamRolesList, self).post(request, *args, **kwargs)
@@ -1268,19 +1265,12 @@ class UserRolesList(SubListAttachDetachAPIView):
if not sub_id:
return super(UserRolesList, self).post(request)
user = get_object_or_400(models.User, pk=self.kwargs['pk'])
role = get_object_or_400(models.Role, pk=sub_id)
content_types = ContentType.objects.get_for_models(models.Organization, models.Team, models.Credential) # dict of {model: content_type}
credential_content_type = content_types[models.Credential]
if role.content_type == credential_content_type:
if 'disassociate' not in request.data and role.content_object.organization and user not in role.content_object.organization.member_role:
data = dict(msg=_("You cannot grant credential access to a user not in the credentials' organization"))
return Response(data, status=status.HTTP_400_BAD_REQUEST)
if not role.content_object.organization and not request.user.is_superuser:
data = dict(msg=_("You cannot grant private credential access to another user"))
return Response(data, status=status.HTTP_400_BAD_REQUEST)
if not request.data.get('disassociate'):
role = get_object_or_400(models.Role, pk=sub_id)
user = get_object_or_400(models.User, pk=self.kwargs['pk'])
content_object = role.content_object
if hasattr(content_object, 'validate_role_assignment'):
content_object.validate_role_assignment(user, role_definition=None, requesting_user=request.user)
return super(UserRolesList, self).post(request, *args, **kwargs)
@@ -1595,7 +1585,175 @@ class CredentialCopy(CopyAPIView):
resource_purpose = 'copy of a credential'
class CredentialExternalTest(SubDetailAPIView):
class OIDCCredentialTestMixin:
"""
Mixin to add OIDC workload identity token support to credential test endpoints.
This mixin provides methods to handle OIDC-enabled external credentials that use
workload identity tokens for authentication.
"""
@staticmethod
def _get_workload_identity_token(job_template: models.JobTemplate, audience: str) -> str:
"""Generate a workload identity token for a job template.
Args:
job_template: The JobTemplate instance to generate claims for
audience: The JWT audience claim value
Returns:
str: The generated JWT token
"""
claims = {
AutomationControllerJobScope.CLAIM_ORGANIZATION_NAME: job_template.organization.name,
AutomationControllerJobScope.CLAIM_ORGANIZATION_ID: job_template.organization.id,
AutomationControllerJobScope.CLAIM_PROJECT_NAME: job_template.project.name,
AutomationControllerJobScope.CLAIM_PROJECT_ID: job_template.project.id,
AutomationControllerJobScope.CLAIM_JOB_TEMPLATE_NAME: job_template.name,
AutomationControllerJobScope.CLAIM_JOB_TEMPLATE_ID: job_template.id,
AutomationControllerJobScope.CLAIM_PLAYBOOK_NAME: job_template.playbook,
}
return retrieve_workload_identity_jwt_with_claims(
claims=claims,
audience=audience,
scope=AutomationControllerJobScope.name,
)
@staticmethod
def _decode_jwt_payload_for_display(jwt_token):
"""Decode JWT payload for display purposes only (signature not verified).
This is safe because the JWT was just created by AWX and is only decoded
to show the user what claims are being sent to the external system.
The external system will perform proper signature verification.
Args:
jwt_token: The JWT token to decode
Returns:
dict: The decoded JWT payload
"""
return _jwt_decode(jwt_token, algorithms=["RS256"], options={"verify_signature": False}) # NOSONAR python:S5659
def _has_workload_identity_token(self, credential_type_inputs):
"""Check if credential type has an internal workload_identity_token field.
Args:
credential_type_inputs: The inputs dict from a credential type
Returns:
bool: True if the credential type has a workload_identity_token field marked as internal
"""
fields = credential_type_inputs.get('fields', []) if isinstance(credential_type_inputs, dict) else []
return any(field.get('internal') and field.get('id') == 'workload_identity_token' for field in fields)
def _validate_and_get_job_template(self, job_template_id):
"""Validate job template ID and return the JobTemplate instance.
Args:
job_template_id: The job template ID from metadata
Returns:
JobTemplate instance
Raises:
ParseError: If job_template_id is invalid or not found
"""
if job_template_id is None:
raise ParseError(_('Job template ID is required.'))
try:
return models.JobTemplate.objects.get(id=int(job_template_id))
except ValueError:
raise ParseError(_('Job template ID must be an integer.'))
except models.JobTemplate.DoesNotExist:
raise ParseError(_('Job template with ID %(id)s does not exist.') % {'id': job_template_id})
def _handle_oidc_credential_test(self, backend_kwargs):
"""
Handle OIDC workload identity token generation for external credential test endpoints.
This method should only be called when FEATURE_OIDC_WORKLOAD_IDENTITY_ENABLED is enabled
and the credential type has a workload_identity_token field.
Args:
backend_kwargs: The kwargs dict to pass to the backend (will be modified in place)
Returns:
dict: Response body containing details with the sent JWT payload
Raises:
PermissionDenied: If user lacks access to the job template (re-raised for 403 response)
All other exceptions are caught and converted to 400 responses with error details.
Modifies backend_kwargs in place to add workload_identity_token.
"""
# Validate job template
job_template_id = backend_kwargs.pop('job_template_id', None)
job_template = self._validate_and_get_job_template(job_template_id)
# Check user access
if not self.request.user.can_access(models.JobTemplate, 'start', job_template):
raise PermissionDenied(_('You do not have access to job template with id: %(id)s.') % {'id': job_template.id})
# Generate workload identity token
jwt_token = self._get_workload_identity_token(job_template, backend_kwargs.get('url'))
backend_kwargs['workload_identity_token'] = jwt_token
return {'details': {'sent_jwt_payload': self._decode_jwt_payload_for_display(jwt_token)}}
def _call_backend_with_error_handling(self, plugin, backend_kwargs, response_body):
"""Call credential backend and handle errors."""
try:
with set_environ(**settings.AWX_TASK_ENV):
plugin.backend(**backend_kwargs)
return Response(response_body, status=status.HTTP_202_ACCEPTED)
except requests.exceptions.HTTPError as exc:
message = self._extract_http_error_message(exc)
self._add_error_to_response(response_body, message)
return Response(response_body, status=status.HTTP_400_BAD_REQUEST)
except Exception as exc:
message = self._extract_generic_error_message(exc)
self._add_error_to_response(response_body, message)
return Response(response_body, status=status.HTTP_400_BAD_REQUEST)
@staticmethod
def _extract_http_error_message(exc):
"""Extract error message from HTTPError, checking response JSON and text."""
message = str(exc)
if not hasattr(exc, 'response') or exc.response is None:
return message
try:
error_data = exc.response.json()
if 'errors' in error_data and error_data['errors']:
return ', '.join(error_data['errors'])
if 'error' in error_data:
return error_data['error']
except (ValueError, KeyError):
if exc.response.text:
return exc.response.text
return message
@staticmethod
def _extract_generic_error_message(exc):
"""Extract error message from exception, handling ConnectTimeoutError specially."""
message = str(exc) if str(exc) else exc.__class__.__name__
for arg in getattr(exc, 'args', []):
if isinstance(getattr(arg, 'reason', None), ConnectTimeoutError):
return str(arg.reason)
return message
@staticmethod
def _add_error_to_response(response_body, message):
"""Add error message to both 'detail' and 'details.error_message' fields."""
response_body['detail'] = message
if 'details' in response_body:
response_body['details']['error_message'] = message
class CredentialExternalTest(OIDCCredentialTestMixin, SubDetailAPIView):
"""
Test updates to the input values and metadata of an external credential
before saving them.
@@ -1615,6 +1773,8 @@ class CredentialExternalTest(SubDetailAPIView):
It does not support standard credential types such as Machine, SCM, and Cloud."""})
def post(self, request, *args, **kwargs):
obj = self.get_object()
if obj.credential_type.kind != 'external':
raise ParseError(_('Credential is not testable.'))
backend_kwargs = {}
for field_name, value in obj.inputs.items():
backend_kwargs[field_name] = obj.get_input(field_name)
@@ -1622,23 +1782,22 @@ class CredentialExternalTest(SubDetailAPIView):
if value != '$encrypted$':
backend_kwargs[field_name] = value
backend_kwargs.update(request.data.get('metadata', {}))
try:
with set_environ(**settings.AWX_TASK_ENV):
obj.credential_type.plugin.backend(**backend_kwargs)
return Response({}, status=status.HTTP_202_ACCEPTED)
except requests.exceptions.HTTPError:
message = """Test operation is not supported for credential type {}.
This endpoint only supports credentials that connect to
external secret management systems such as CyberArk, HashiCorp
Vault, or cloud-based secret managers.""".format(obj.credential_type.kind)
return Response({'detail': message}, status=status.HTTP_400_BAD_REQUEST)
except Exception as exc:
message = exc.__class__.__name__
exc_args = getattr(exc, 'args', [])
for a in exc_args:
if isinstance(getattr(a, 'reason', None), ConnectTimeoutError):
message = str(a.reason)
return Response({'inputs': message}, status=status.HTTP_400_BAD_REQUEST)
# Handle OIDC workload identity token generation if enabled
response_body = {}
if flag_enabled('FEATURE_OIDC_WORKLOAD_IDENTITY_ENABLED') and self._has_workload_identity_token(obj.credential_type.inputs):
try:
oidc_response_body = self._handle_oidc_credential_test(backend_kwargs)
response_body.update(oidc_response_body)
except PermissionDenied:
raise
except Exception as exc:
error_message = str(exc.detail) if hasattr(exc, 'detail') else str(exc)
response_body['detail'] = error_message
response_body['details'] = {'error_message': error_message}
return Response(response_body, status=status.HTTP_400_BAD_REQUEST)
return self._call_backend_with_error_handling(obj.credential_type.plugin, backend_kwargs, response_body)
class CredentialInputSourceDetail(RetrieveUpdateDestroyAPIView):
@@ -1668,7 +1827,7 @@ class CredentialInputSourceSubList(SubListCreateAPIView):
parent_key = 'target_credential'
class CredentialTypeExternalTest(SubDetailAPIView):
class CredentialTypeExternalTest(OIDCCredentialTestMixin, SubDetailAPIView):
"""
Test a complete set of input values for an external credential before
saving it.
@@ -1683,21 +1842,26 @@ class CredentialTypeExternalTest(SubDetailAPIView):
@extend_schema_if_available(extensions={"x-ai-description": "Test a complete set of input values for an external credential"})
def post(self, request, *args, **kwargs):
obj = self.get_object()
if obj.kind != 'external':
raise ParseError(_('Credential type is not testable.'))
backend_kwargs = request.data.get('inputs', {})
backend_kwargs.update(request.data.get('metadata', {}))
try:
obj.plugin.backend(**backend_kwargs)
return Response({}, status=status.HTTP_202_ACCEPTED)
except requests.exceptions.HTTPError as exc:
message = 'HTTP {}'.format(exc.response.status_code)
return Response({'inputs': message}, status=status.HTTP_400_BAD_REQUEST)
except Exception as exc:
message = exc.__class__.__name__
args_exc = getattr(exc, 'args', [])
for a in args_exc:
if isinstance(getattr(a, 'reason', None), ConnectTimeoutError):
message = str(a.reason)
return Response({'inputs': message}, status=status.HTTP_400_BAD_REQUEST)
# Handle OIDC workload identity token generation if enabled
response_body = {}
if flag_enabled('FEATURE_OIDC_WORKLOAD_IDENTITY_ENABLED') and self._has_workload_identity_token(obj.inputs):
try:
oidc_response_body = self._handle_oidc_credential_test(backend_kwargs)
response_body.update(oidc_response_body)
except PermissionDenied:
raise
except Exception as exc:
error_message = str(exc.detail) if hasattr(exc, 'detail') else str(exc)
response_body['detail'] = error_message
response_body['details'] = {'error_message': error_message}
return Response(response_body, status=status.HTTP_400_BAD_REQUEST)
return self._call_backend_with_error_handling(obj.plugin, backend_kwargs, response_body)
class HostRelatedSearchMixin(object):
@@ -1763,7 +1927,8 @@ class HostList(HostRelatedSearchMixin, ListCreateAPIView):
if filter_string:
filter_qs = SmartFilter.query_from_string(filter_string)
qs &= filter_qs
return qs.distinct()
qs = qs.distinct()
return qs.with_latest_summary_id()
def list(self, *args, **kwargs):
try:
@@ -1778,6 +1943,9 @@ class HostDetail(RelatedJobsPreventDeleteMixin, RetrieveUpdateDestroyAPIView):
serializer_class = serializers.HostSerializer
resource_purpose = 'host detail'
def get_queryset(self):
return super().get_queryset().with_latest_summary_id()
@extend_schema_if_available(extensions={"x-ai-description": "Delete a host"})
def delete(self, request, *args, **kwargs):
if self.get_object().inventory.pending_deletion:
@@ -1811,6 +1979,9 @@ class InventoryHostsList(HostRelatedSearchMixin, SubListCreateAttachDetachAPIVie
filter_read_permission = False
resource_purpose = 'hosts of an inventory'
def get_queryset(self):
return super().get_queryset().with_latest_summary_id()
class HostGroupsList(SubListCreateAttachDetachAPIView):
'''the list of groups a host is directly a member of'''
@@ -1994,6 +2165,9 @@ class GroupHostsList(HostRelatedSearchMixin, SubListCreateAttachDetachAPIView):
relationship = 'hosts'
resource_purpose = 'hosts of a group'
def get_queryset(self):
return super().get_queryset().with_latest_summary_id()
def update_raw_data(self, data):
data.pop('inventory', None)
return super(GroupHostsList, self).update_raw_data(data)
@@ -2025,7 +2199,7 @@ class GroupAllHostsList(HostRelatedSearchMixin, SubListAPIView):
self.check_parent_access(parent)
qs = self.request.user.get_queryset(self.model).distinct() # need distinct for '&' operator
sublist_qs = parent.all_hosts.distinct()
return qs & sublist_qs
return (qs & sublist_qs).with_latest_summary_id()
class GroupInventorySourcesList(SubListAPIView):
@@ -2318,6 +2492,9 @@ class InventorySourceHostsList(HostRelatedSearchMixin, SubListDestroyAPIView):
check_sub_obj_permission = False
resource_purpose = 'hosts of an inventory source'
def get_queryset(self):
return super().get_queryset().with_latest_summary_id()
def perform_list_destroy(self, instance_list):
inv_source = self.get_parent_object()
with ignore_inventory_computed_fields():
@@ -2877,8 +3054,7 @@ class JobTemplateCallback(GenericAPIView):
host for the current request.
"""
# Find the list of remote host names/IPs to check.
# Only consider the first entry from each header (for comma-separated values like X-Forwarded-For)
remote_hosts = get_first_remote_host_from_headers(self.request, settings.REMOTE_HOST_HEADERS)
remote_hosts = set(get_remote_hosts(self.request))
# Add the reverse lookup of IP addresses.
for rh in list(remote_hosts):
try:
@@ -3676,7 +3852,7 @@ class SystemJobTemplateNotificationTemplatesSuccessList(SystemJobTemplateNotific
resource_purpose = 'notification templates triggered on system job success'
class JobList(ListAPIView):
class JobList(UnifiedJobExcludeMixin, ListAPIView):
model = models.Job
serializer_class = serializers.JobListSerializer
resource_purpose = 'jobs'
@@ -4393,10 +4569,11 @@ class UnifiedJobTemplateList(ListAPIView):
resource_purpose = 'unified job templates'
class UnifiedJobList(ListAPIView):
class UnifiedJobList(UnifiedJobExcludeMixin, ListAPIView):
model = models.UnifiedJob
serializer_class = serializers.UnifiedJobListSerializer
search_fields = ('description', 'name', 'job__playbook')
pagination_class = UnifiedJobPagination
resource_purpose = 'unified jobs'
@@ -4642,6 +4819,7 @@ class ActivityStreamList(SimpleListAPIView):
model = models.ActivityStream
serializer_class = serializers.ActivityStreamSerializer
search_fields = ('changes',)
pagination_class = ActivityStreamPagination
resource_purpose = 'audit trail entries for tracking system changes'
@extend_schema_if_available(
@@ -4696,19 +4874,12 @@ class RoleUsersList(SubListAttachDetachAPIView):
if not sub_id:
return super(RoleUsersList, self).post(request)
user = get_object_or_400(models.User, pk=sub_id)
role = self.get_parent_object()
content_types = ContentType.objects.get_for_models(models.Organization, models.Team, models.Credential) # dict of {model: content_type}
credential_content_type = content_types[models.Credential]
if role.content_type == credential_content_type:
if 'disassociate' not in request.data and role.content_object.organization and user not in role.content_object.organization.member_role:
data = dict(msg=_("You cannot grant credential access to a user not in the credentials' organization"))
return Response(data, status=status.HTTP_400_BAD_REQUEST)
if not role.content_object.organization and not request.user.is_superuser:
data = dict(msg=_("You cannot grant private credential access to another user"))
return Response(data, status=status.HTTP_400_BAD_REQUEST)
if not request.data.get('disassociate'):
user = get_object_or_400(models.User, pk=sub_id)
role = self.get_parent_object()
content_object = role.content_object
if hasattr(content_object, 'validate_role_assignment'):
content_object.validate_role_assignment(user, role_definition=None, requesting_user=request.user)
return super(RoleUsersList, self).post(request, *args, **kwargs)
@@ -4741,24 +4912,6 @@ class RoleTeamsList(SubListAttachDetachAPIView):
data = dict(msg=_("You cannot assign an Organization participation role as a child role for a Team."))
return Response(data, status=status.HTTP_400_BAD_REQUEST)
credential_content_type = ContentType.objects.get_for_model(models.Credential)
if role.content_type == credential_content_type:
# Private credentials (no organization) are never allowed for teams
if not role.content_object.organization:
data = dict(
msg=_("You cannot grant access to a credential that is not assigned to an organization (private credentials cannot be assigned to teams)")
)
return Response(data, status=status.HTTP_400_BAD_REQUEST)
# Cross-organization credentials are only allowed for superusers
elif role.content_object.organization.id != team.organization.id:
if not request.user.is_superuser:
data = dict(
msg=_(
"You cannot grant a team access to a credential in a different organization. Only superusers can grant cross-organization credential access to teams"
)
)
return Response(data, status=status.HTTP_400_BAD_REQUEST)
action = 'attach'
if request.data.get('disassociate', None):
action = 'unattach'
@@ -4767,6 +4920,11 @@ class RoleTeamsList(SubListAttachDetachAPIView):
data = dict(msg=_("You cannot grant system-level permissions to a team."))
return Response(data, status=status.HTTP_400_BAD_REQUEST)
if action == 'attach':
content_object = role.content_object
if hasattr(content_object, 'validate_role_assignment'):
content_object.validate_role_assignment(team, role_definition=None, requesting_user=request.user)
if not request.user.can_access(self.parent_model, action, role, team, self.relationship, request.data, skip_sub_obj_read_check=False):
raise PermissionDenied()
if request.data.get('disassociate', None):

View File

@@ -49,7 +49,6 @@ class GetNotAllowedMixin(object):
class AnalyticsRootView(APIView):
permission_classes = (AnalyticsPermission,)
name = _('Automation Analytics')
swagger_topic = 'Automation Analytics'
resource_purpose = 'automation analytics endpoints'
@extend_schema_if_available(extensions={"x-ai-description": "A list of additional API endpoints related to analytics"})
@@ -306,7 +305,6 @@ class AnalyticsAuthorizedView(AnalyticsGenericListView):
class AnalyticsReportsList(GetNotAllowedMixin, AnalyticsGenericListView):
name = _("Reports")
swagger_topic = "Automation Analytics"
resource_purpose = 'automation analytics reports'

View File

@@ -4,7 +4,8 @@
import dateutil
import logging
from django.db.models import Count
from django.db.models import Count, Q, TextField
from django.db.models.functions import Cast
from django.db import transaction
from django.shortcuts import get_object_or_404
from django.utils.timezone import now
@@ -15,6 +16,7 @@ from rest_framework.response import Response
from rest_framework import status
from awx.main.constants import ACTIVE_STATES
from ansible_base.rbac.models import RoleDefinition, RoleUserAssignment
from awx.main.models import Organization
from awx.main.utils import get_object_or_400
from awx.main.models.ha import Instance, InstanceGroup, schedule_policy_task
@@ -177,29 +179,37 @@ class OrganizationCountsMixin(object):
db_results['projects'] = project_qs.values('organization').annotate(Count('organization')).order_by('organization')
# Other members and admins of organization are always viewable
db_results['users'] = org_qs.annotate(users=Count('member_role__members', distinct=True), admins=Count('admin_role__members', distinct=True)).values(
'id', 'users', 'admins'
)
count_context = {}
for org in org_id_list:
org_id = org['id']
count_context[org_id] = {'inventories': 0, 'teams': 0, 'users': 0, 'job_templates': 0, 'admins': 0, 'projects': 0}
for res, count_qs in db_results.items():
if res == 'users':
org_reference = 'id'
else:
org_reference = 'organization'
for entry in count_qs:
org_id = entry[org_reference]
org_id = entry['organization']
if org_id in count_context:
if res == 'users':
count_context[org_id]['admins'] = entry['admins']
count_context[org_id]['users'] = entry['users']
continue
count_context[org_id][res] = entry['%s__count' % org_reference]
count_context[org_id][res] = entry['organization__count']
member_rd = RoleDefinition.objects.filter(name='Organization Member').first()
admin_rd = RoleDefinition.objects.filter(name='Organization Admin').first()
if member_rd and admin_rd:
user_admin_counts = (
RoleUserAssignment.objects.filter(
role_definition__in=[member_rd, admin_rd],
object_id__in=org_qs.annotate(text_pk=Cast('pk', TextField())).values('text_pk'),
)
.values('object_id')
.annotate(
users=Count('pk', filter=Q(role_definition=member_rd)),
admins=Count('pk', filter=Q(role_definition=admin_rd)),
)
)
for entry in user_admin_counts:
org_id = int(entry['object_id'])
if org_id in count_context:
count_context[org_id]['users'] = entry['users']
count_context[org_id]['admins'] = entry['admins']
full_context['related_field_counts'] = count_context
@@ -212,3 +222,9 @@ class NoTruncateMixin(object):
if self.request.query_params.get('no_truncate'):
context.update(no_truncate=True)
return context
class UnifiedJobExcludeMixin(object):
# Reserve the name 'exclude' so we can use it as a query param. Otherwise, the rest-filters backend
# would treat it as a model field lookup.
rest_filters_reserved_names = ('exclude',)

View File

@@ -5,11 +5,12 @@
import logging
# Django
from django.db.models import Count
from django.db.models import Count, Q
from django.contrib.contenttypes.models import ContentType
from django.utils.translation import gettext_lazy as _
# AWX
from ansible_base.rbac.models import RoleDefinition, RoleUserAssignment
from awx.main.models import (
ActivityStream,
Inventory,
@@ -77,16 +78,21 @@ class OrganizationDetail(RelatedJobsPreventDeleteMixin, RetrieveUpdateDestroyAPI
org_counts = {}
access_kwargs = {'accessor': self.request.user, 'role_field': 'read_role'}
direct_counts = (
Organization.objects.filter(id=org_id)
.annotate(users=Count('member_role__members', distinct=True), admins=Count('admin_role__members', distinct=True))
.values('users', 'admins')
)
member_rd = RoleDefinition.objects.filter(name='Organization Member').first()
admin_rd = RoleDefinition.objects.filter(name='Organization Admin').first()
if not direct_counts:
return full_context
if member_rd and admin_rd:
counts = RoleUserAssignment.objects.filter(
role_definition__in=[member_rd, admin_rd],
object_id=str(org_id),
).aggregate(
users=Count('pk', filter=Q(role_definition=member_rd)),
admins=Count('pk', filter=Q(role_definition=admin_rd)),
)
org_counts.update(counts)
else:
org_counts.update({'users': 0, 'admins': 0})
org_counts = direct_counts[0]
org_counts['inventories'] = Inventory.accessible_objects(**access_kwargs).filter(organization__id=org_id).count()
org_counts['teams'] = Team.accessible_objects(**access_kwargs).filter(organization__id=org_id).count()
org_counts['projects'] = Project.accessible_objects(**access_kwargs).filter(organization__id=org_id).count()

View File

@@ -344,13 +344,22 @@ class ApiV2ConfigView(APIView):
become_methods=PRIVILEGE_ESCALATION_METHODS,
)
if (
request.user.is_superuser
or request.user.is_system_auditor
or Organization.accessible_objects(request.user, 'admin_role').exists()
or Organization.accessible_objects(request.user, 'auditor_role').exists()
or Organization.accessible_objects(request.user, 'project_admin_role').exists()
):
# Check superuser/auditor first
if request.user.is_superuser or request.user.is_system_auditor:
has_org_access = True
else:
# Single query checking all three organization role types at once
has_org_access = (
(
Organization.access_qs(request.user, 'change')
| Organization.access_qs(request.user, 'audit')
| Organization.access_qs(request.user, 'add_project')
)
.distinct()
.exists()
)
if has_org_access:
data.update(
dict(
project_base_dir=settings.PROJECTS_ROOT,
@@ -358,8 +367,10 @@ class ApiV2ConfigView(APIView):
custom_virtualenvs=get_custom_venv_choices(),
)
)
elif JobTemplate.accessible_objects(request.user, 'admin_role').exists():
data['custom_virtualenvs'] = get_custom_venv_choices()
else:
# Only check JobTemplate access if org check failed
if JobTemplate.accessible_objects(request.user, 'admin_role').exists():
data['custom_virtualenvs'] = get_custom_venv_choices()
return Response(data)

View File

@@ -17,7 +17,7 @@ from awx.api import serializers
from awx.api.generics import APIView, GenericAPIView
from awx.api.permissions import WebhookKeyPermission
from awx.main.models import Job, JobTemplate, WorkflowJob, WorkflowJobTemplate
from awx.main.constants import JOB_VARIABLE_PREFIXES
from awx.main.utils.common import get_job_variable_prefixes
logger = logging.getLogger('awx.api.views.webhooks')
@@ -166,7 +166,7 @@ class WebhookReceiverBase(APIView):
'extra_vars': {},
}
for name in JOB_VARIABLE_PREFIXES:
for name in get_job_variable_prefixes():
kwargs['extra_vars']['{}_webhook_event_type'.format(name)] = event_type
kwargs['extra_vars']['{}_webhook_event_guid'.format(name)] = event_guid
kwargs['extra_vars']['{}_webhook_event_ref'.format(name)] = event_ref

View File

@@ -20,6 +20,7 @@ from rest_framework.exceptions import ParseError, PermissionDenied
# django-ansible-base
from ansible_base.lib.utils.validation import to_python_boolean
from ansible_base.rbac.models import RoleEvaluation
from ansible_base.rbac.policies import visible_users
from ansible_base.rbac import permission_registry
# AWX
@@ -643,6 +644,8 @@ class UserAccess(BaseAccess):
Organization.access_qs(self.user, 'change').exists() or Organization.access_qs(self.user, 'audit').exists()
):
qs = User.objects.all()
elif settings.ANSIBLE_BASE_ROLE_SYSTEM_ACTIVATED:
qs = visible_users(self.user)
else:
qs = (
User.objects.filter(pk__in=Organization.access_qs(self.user, 'view').values('member_role__members'))
@@ -706,12 +709,13 @@ class UserAccess(BaseAccess):
# in these cases only superusers can modify orphan users
return False
if settings.ANSIBLE_BASE_ROLE_SYSTEM_ACTIVATED:
# Permission granted if the user has all permissions that the target user has
target_perms = set(
RoleEvaluation.objects.filter(role__in=obj.has_roles.all()).values_list('object_id', 'content_type_id', 'codename').distinct()
RoleEvaluation.objects.filter(**RoleEvaluation._actor_role_filter(obj)).values_list('object_id', 'content_type_id', 'codename').distinct()
)
user_perms = set(
RoleEvaluation.objects.filter(role__in=self.user.has_roles.all()).values_list('object_id', 'content_type_id', 'codename').distinct()
RoleEvaluation.objects.filter(**RoleEvaluation._actor_role_filter(self.user))
.values_list('object_id', 'content_type_id', 'codename')
.distinct()
)
return not (target_perms - user_perms)
return not obj.roles.all().exclude(ancestors__in=self.user.roles.all()).exists()
@@ -897,8 +901,6 @@ class HostAccess(BaseAccess):
'created_by',
'modified_by',
'inventory',
'last_job__job_template',
'last_job_host_summary__job',
)
prefetch_related = ('groups', 'inventory_sources')
@@ -1230,9 +1232,11 @@ class TeamAccess(BaseAccess):
Organization.access_qs(self.user, 'change').exists() or Organization.access_qs(self.user, 'audit').exists()
):
return self.model.objects.all()
return self.model.objects.filter(
Q(organization__in=Organization.accessible_pk_qs(self.user, 'member_role')) | Q(pk__in=self.model.accessible_pk_qs(self.user, 'read_role'))
org_member_teams = (
self.model.objects.filter(organization__in=Organization.accessible_pk_qs(self.user, 'member_role')).order_by().values_list('pk', flat=True)
)
direct_read_teams = self.model.objects.filter(pk__in=self.model.accessible_pk_qs(self.user, 'read_role')).order_by().values_list('pk', flat=True)
return self.model.objects.filter(pk__in=org_member_teams.union(direct_read_teams))
@check_superuser
def can_add(self, data):
@@ -1667,11 +1671,11 @@ class JobAccess(BaseAccess):
def filtered_queryset(self):
qs = self.model.objects
qs_jt = qs.filter(job_template__in=JobTemplate.access_qs(self.user, 'view'))
org_access_qs = Organization.objects.filter(Q(admin_role__members=self.user) | Q(auditor_role__members=self.user))
org_access_qs = Organization.objects.filter(
Q(pk__in=Organization.access_ids_qs(self.user, 'change')) | Q(pk__in=Organization.access_ids_qs(self.user, 'audit_organization'))
)
if not org_access_qs.exists():
return qs_jt
return qs.filter(job_template__in=JobTemplate.access_qs(self.user, 'view'))
return qs.filter(Q(job_template__in=JobTemplate.access_qs(self.user, 'view')) | Q(organization__in=org_access_qs)).distinct()
@@ -2311,7 +2315,7 @@ class JobHostSummaryAccess(BaseAccess):
class JobEventAccess(BaseAccess):
"""
I can see job event records whenever I can read both job and host.
I can see job event records whenever I can read the job or the host.
"""
model = JobEvent
@@ -2322,8 +2326,8 @@ class JobEventAccess(BaseAccess):
def filtered_queryset(self):
return self.model.objects.filter(
Q(host__inventory__in=Inventory.accessible_pk_qs(self.user, 'read_role'))
| Q(job__job_template__in=JobTemplate.accessible_pk_qs(self.user, 'read_role'))
Q(host_id__in=Host.objects.filter(inventory__in=Inventory.access_ids_qs(self.user, 'view')).values('pk'))
| Q(job_id__in=Job.objects.filter(job_template__in=JobTemplate.access_ids_qs(self.user, 'view')).values('pk'))
)
def can_add(self, data):
@@ -2453,7 +2457,11 @@ class UnifiedJobTemplateAccess(BaseAccess):
def filtered_queryset(self):
return self.model.objects.filter(
Q(pk__in=self.model.accessible_pk_qs(self.user, 'read_role'))
| Q(inventorysource__inventory__id__in=Inventory._accessible_pk_qs(Inventory, self.user, 'read_role'))
| Q(
pk__in=InventorySource.objects.filter(
inventory__id__in=Inventory.access_ids_qs(self.user, 'view'),
).values('unifiedjobtemplate_ptr_id')
)
)
def can_start(self, obj, validate_license=True):
@@ -2499,14 +2507,39 @@ class UnifiedJobAccess(BaseAccess):
# )
def filtered_queryset(self):
inv_pk_qs = Inventory._accessible_pk_qs(Inventory, self.user, 'read_role')
qs = self.model.objects.filter(
Q(unified_job_template_id__in=UnifiedJobTemplate.accessible_pk_qs(self.user, 'read_role'))
| Q(inventoryupdate__inventory_source__inventory__id__in=inv_pk_qs)
| Q(adhoccommand__inventory__id__in=inv_pk_qs)
| Q(organization__in=Organization.accessible_pk_qs(self.user, 'auditor_role'))
inv_pk_qs = Inventory.access_ids_qs(self.user, 'view')
by_template = (
self.model.objects.filter(unified_job_template_id__in=UnifiedJobTemplate.accessible_pk_qs(self.user, 'read_role'))
.order_by()
.values_list('pk', flat=True)
)
return qs
by_inventory_update = (
InventoryUpdate.objects.filter(
inventory_source__inventory__id__in=inv_pk_qs,
)
.order_by()
.values_list('pk', flat=True)
)
by_adhoc = (
AdHocCommand.objects.filter(
inventory__id__in=inv_pk_qs,
)
.order_by()
.values_list('pk', flat=True)
)
by_org_auditor = (
self.model.objects.filter(
organization__in=Organization.access_ids_qs(self.user, 'audit_organization'),
)
.order_by()
.values_list('pk', flat=True)
)
return self.model.objects.filter(pk__in=by_template.union(by_inventory_update, by_adhoc, by_org_auditor))
def get_queryset(self):
return super(UnifiedJobAccess, self).get_queryset().filter(workflowapproval__isnull=True)
@@ -2624,9 +2657,13 @@ class LabelAccess(BaseAccess):
def filtered_queryset(self):
return self.model.objects.filter(
Q(organization__in=Organization.accessible_pk_qs(self.user, 'read_role'))
| Q(unifiedjobtemplate_labels__in=UnifiedJobTemplate.accessible_pk_qs(self.user, 'read_role'))
).distinct()
Q(organization__in=Organization.access_ids_qs(self.user, 'view'))
| Q(
pk__in=UnifiedJobTemplate.labels.through.objects.filter(
unifiedjobtemplate_id__in=UnifiedJobTemplate.accessible_pk_qs(self.user, 'read_role'),
).values('label_id')
)
)
@check_superuser
def can_add(self, data):
@@ -2700,54 +2737,73 @@ class ActivityStreamAccess(BaseAccess):
# 'job_template', 'job', 'project', 'project_update', 'workflow_job',
# 'inventory_source', 'workflow_job_template'
q = Q(user=self.user)
inventory_set = Inventory.accessible_pk_qs(self.user, 'read_role')
if inventory_set:
AS = ActivityStream
q = Q(pk__in=AS.user.through.objects.filter(user=self.user).values('activitystream_id'))
inventory_set = Inventory.access_ids_qs(self.user, 'view')
if inventory_set.exists():
q |= (
Q(ad_hoc_command__inventory__in=inventory_set)
| Q(inventory__in=inventory_set)
| Q(host__inventory__in=inventory_set)
| Q(group__inventory__in=inventory_set)
| Q(inventory_source__inventory__in=inventory_set)
| Q(inventory_update__inventory_source__inventory__in=inventory_set)
Q(pk__in=AS.ad_hoc_command.through.objects.filter(adhoccommand__inventory__in=inventory_set).values('activitystream_id'))
| Q(pk__in=AS.inventory.through.objects.filter(inventory__in=inventory_set).values('activitystream_id'))
| Q(pk__in=AS.host.through.objects.filter(host__inventory__in=inventory_set).values('activitystream_id'))
| Q(pk__in=AS.group.through.objects.filter(group__inventory__in=inventory_set).values('activitystream_id'))
| Q(pk__in=AS.inventory_source.through.objects.filter(inventorysource__inventory__in=inventory_set).values('activitystream_id'))
| Q(
pk__in=AS.inventory_update.through.objects.filter(inventoryupdate__inventory_source__inventory__in=inventory_set).values(
'activitystream_id'
)
)
)
credential_set = Credential.accessible_pk_qs(self.user, 'read_role')
if credential_set:
q |= Q(credential__in=credential_set)
credential_set = Credential.access_ids_qs(self.user, 'view')
if credential_set.exists():
q |= Q(pk__in=AS.credential.through.objects.filter(credential__in=credential_set).values('activitystream_id'))
auditing_orgs = (Organization.access_qs(self.user, 'change') | Organization.access_qs(self.user, 'audit')).distinct().values_list('id', flat=True)
if auditing_orgs:
if auditing_orgs.exists():
q |= (
Q(user__in=auditing_orgs.values('member_role__members'))
| Q(organization__in=auditing_orgs)
| Q(notification_template__organization__in=auditing_orgs)
| Q(notification__notification_template__organization__in=auditing_orgs)
| Q(label__organization__in=auditing_orgs)
| Q(role__in=Role.visible_roles(self.user) if auditing_orgs else [])
Q(pk__in=AS.user.through.objects.filter(user__in=auditing_orgs.values('member_role__members')).values('activitystream_id'))
| Q(pk__in=AS.organization.through.objects.filter(organization__in=auditing_orgs).values('activitystream_id'))
| Q(pk__in=AS.notification_template.through.objects.filter(notificationtemplate__organization__in=auditing_orgs).values('activitystream_id'))
| Q(
pk__in=AS.notification.through.objects.filter(notification__notification_template__organization__in=auditing_orgs).values(
'activitystream_id'
)
)
| Q(pk__in=AS.label.through.objects.filter(label__organization__in=auditing_orgs).values('activitystream_id'))
| Q(pk__in=AS.role.through.objects.filter(role__in=Role.visible_roles(self.user)).values('activitystream_id'))
)
project_set = Project.accessible_pk_qs(self.user, 'read_role')
if project_set:
q |= Q(project__in=project_set) | Q(project_update__project__in=project_set)
jt_set = JobTemplate.accessible_pk_qs(self.user, 'read_role')
if jt_set:
q |= Q(job_template__in=jt_set) | Q(job__job_template__in=jt_set)
wfjt_set = WorkflowJobTemplate.accessible_pk_qs(self.user, 'read_role')
if wfjt_set:
q |= (
Q(workflow_job_template__in=wfjt_set)
| Q(workflow_job_template_node__workflow_job_template__in=wfjt_set)
| Q(workflow_job__workflow_job_template__in=wfjt_set)
project_set = Project.access_ids_qs(self.user, 'view')
if project_set.exists():
q |= Q(pk__in=AS.project.through.objects.filter(project__in=project_set).values('activitystream_id')) | Q(
pk__in=AS.project_update.through.objects.filter(projectupdate__project__in=project_set).values('activitystream_id')
)
team_set = Team.accessible_pk_qs(self.user, 'read_role')
if team_set:
q |= Q(team__in=team_set)
jt_set = JobTemplate.access_ids_qs(self.user, 'view')
if jt_set.exists():
q |= Q(pk__in=AS.job_template.through.objects.filter(jobtemplate__in=jt_set).values('activitystream_id')) | Q(
pk__in=AS.job.through.objects.filter(job__job_template__in=jt_set).values('activitystream_id')
)
return qs.filter(q).distinct()
wfjt_set = WorkflowJobTemplate.access_ids_qs(self.user, 'view')
if wfjt_set.exists():
q |= (
Q(pk__in=AS.workflow_job_template.through.objects.filter(workflowjobtemplate__in=wfjt_set).values('activitystream_id'))
| Q(
pk__in=AS.workflow_job_template_node.through.objects.filter(workflowjobtemplatenode__workflow_job_template__in=wfjt_set).values(
'activitystream_id'
)
)
| Q(pk__in=AS.workflow_job.through.objects.filter(workflowjob__workflow_job_template__in=wfjt_set).values('activitystream_id'))
)
team_set = Team.access_ids_qs(self.user, 'view')
if team_set.exists():
q |= Q(pk__in=AS.team.through.objects.filter(team__in=team_set).values('activitystream_id'))
return qs.filter(q)
def can_add(self, data):
return False

View File

@@ -8,6 +8,7 @@ import pathlib
import shutil
import tarfile
import tempfile
from urllib.parse import urlparse, urlunparse
from django.conf import settings
from django.core.serializers.json import DjangoJSONEncoder
@@ -23,6 +24,8 @@ from awx.main.models import Job
from awx.main.access import access_registry
from awx.main.utils import get_awx_http_client_headers, set_environ, datetime_hook
from awx.main.utils.analytics_proxy import OIDCClient
from awx.main.utils.candlepin import get_or_generate_candlepin_certificate
from awx.main.utils.candlepin.client import _temp_cert_files
__all__ = ['register', 'gather', 'ship']
@@ -41,6 +44,76 @@ def _valid_license():
return True
def _get_cert_upload_url(url):
"""
Convert analytics URL to use 'cert.' subdomain for mTLS uploads.
Some analytics services use different hostnames for different auth methods:
- cert.example.com - for mTLS (certificate-based) uploads
- example.com - for OIDC (token-based) uploads
Args:
url: Original analytics URL
Returns:
URL with 'cert.' prepended to hostname if not already present
"""
try:
parsed = urlparse(url)
hostname = parsed.hostname
# Only modify if hostname doesn't already start with 'cert.'
if hostname and not hostname.startswith('cert.'):
new_hostname = f'cert.{hostname}'
# Reconstruct URL with new hostname
netloc = new_hostname
if parsed.port:
netloc = f'{new_hostname}:{parsed.port}'
new_parsed = parsed._replace(netloc=netloc)
return urlunparse(new_parsed)
return url
except Exception as e:
logger.warning(f'Could not modify URL for cert upload: {e}, using original URL')
return url
def _get_analytics_credentials():
"""
Get Red Hat Insights credentials from settings.
Attempts to retrieve credentials in the following priority order:
1. REDHAT_USERNAME / REDHAT_PASSWORD
2. SUBSCRIPTIONS_USERNAME / SUBSCRIPTIONS_PASSWORD
3. SUBSCRIPTIONS_CLIENT_ID / SUBSCRIPTIONS_CLIENT_SECRET
Returns:
tuple: (username, password) if credentials are found, (None, None) otherwise
"""
rh_id = getattr(settings, 'REDHAT_USERNAME', None)
rh_secret = getattr(settings, 'REDHAT_PASSWORD', None)
if rh_id and rh_secret:
return rh_id, rh_secret
# Try SUBSCRIPTIONS_USERNAME / SUBSCRIPTIONS_PASSWORD
rh_id = getattr(settings, 'SUBSCRIPTIONS_USERNAME', None)
rh_secret = getattr(settings, 'SUBSCRIPTIONS_PASSWORD', None)
if rh_id and rh_secret:
return rh_id, rh_secret
# Try SUBSCRIPTIONS_CLIENT_ID / SUBSCRIPTIONS_CLIENT_SECRET
rh_id = getattr(settings, 'SUBSCRIPTIONS_CLIENT_ID', None)
rh_secret = getattr(settings, 'SUBSCRIPTIONS_CLIENT_SECRET', None)
if rh_id and rh_secret:
return rh_id, rh_secret
return None, None
def all_collectors():
from awx.main.analytics import collectors
@@ -184,10 +257,8 @@ def gather(dest=None, module=None, subset=None, since=None, until=None, collecti
logger.log(log_level, "Automation Analytics not enabled. Use --dry-run to gather locally without sending.")
return None
if not (
settings.AUTOMATION_ANALYTICS_URL
and ((settings.REDHAT_USERNAME and settings.REDHAT_PASSWORD) or (settings.SUBSCRIPTIONS_CLIENT_ID and settings.SUBSCRIPTIONS_CLIENT_SECRET))
):
rh_id, rh_secret = _get_analytics_credentials()
if not (settings.AUTOMATION_ANALYTICS_URL and rh_id and rh_secret):
logger.log(log_level, "Not gathering analytics, configuration is invalid. Use --dry-run to gather locally without sending.")
return None
@@ -349,6 +420,18 @@ def gather(dest=None, module=None, subset=None, since=None, until=None, collecti
return tarfiles
def _log_shipping_response(response, path):
filename = os.path.basename(path)
try:
data = response.json()
request_id = data.get('request_id', 'unknown')
account_number = data.get('account_number', 'unknown')
org_id = data.get('org_id', 'unknown')
logger.info(f"Analytics upload successful: file={filename} request_id={request_id} account_number={account_number} org_id={org_id}")
except Exception:
logger.info(f"Analytics upload successful: file={filename} status={response.status_code}")
def ship(path):
"""
Ship gathered metrics to the Insights API
@@ -368,19 +451,14 @@ def ship(path):
logger.error('AUTOMATION_ANALYTICS_URL is not set')
return False
rh_id = getattr(settings, 'REDHAT_USERNAME', None)
rh_secret = getattr(settings, 'REDHAT_PASSWORD', None)
if not (rh_id and rh_secret):
rh_id = getattr(settings, 'SUBSCRIPTIONS_CLIENT_ID', None)
rh_secret = getattr(settings, 'SUBSCRIPTIONS_CLIENT_SECRET', None)
rh_id, rh_secret = _get_analytics_credentials()
if not rh_id:
logger.error('Neither REDHAT_USERNAME nor SUBSCRIPTIONS_CLIENT_ID are set')
logger.error('No valid username found. Tried: REDHAT_USERNAME, SUBSCRIPTIONS_USERNAME, SUBSCRIPTIONS_CLIENT_ID')
return False
if not rh_secret:
logger.error('Neither REDHAT_PASSWORD nor SUBSCRIPTIONS_CLIENT_SECRET are set')
logger.error('No valid password found. Tried: REDHAT_PASSWORD, SUBSCRIPTIONS_PASSWORD, SUBSCRIPTIONS_CLIENT_SECRET')
return False
with open(path, 'rb') as f:
@@ -388,17 +466,42 @@ def ship(path):
s = requests.Session()
s.headers = get_awx_http_client_headers()
s.headers.pop('Content-Type')
with set_environ(**settings.AWX_TASK_ENV):
# Try Certificate-based mTLS authentication (zero-touch)
cert_pem, key_pem = get_or_generate_candlepin_certificate()
if cert_pem and key_pem:
# Use cert. subdomain for mTLS uploads
cert_url = _get_cert_upload_url(url)
logger.debug("Attempting certificate-based authentication for analytics upload")
try:
with _temp_cert_files(cert_pem, key_pem) as (cert_path, key_path):
response = s.post(
cert_url, files=files, cert=(cert_path, key_path), verify=settings.INSIGHTS_CERT_PATH, headers=s.headers, timeout=(31, 31)
)
if response.status_code < 300:
_log_shipping_response(response, path)
return True
else:
logger.warning(
f'Certificate-based authentication failed with status {response.status_code}, {response.text}. Falling back to OIDC auth'
)
except Exception as e:
logger.warning(f"Certificate-based authentication failed: {e}, falling back to OIDC auth")
# Try OIDC authentication
logger.debug("Attempting OIDC authentication for analytics upload")
f.seek(0) # requests POST may read from the handler, so seek to beginning of file for the next POST attempt
try:
client = OIDCClient(rh_id, rh_secret)
response = client.make_request("POST", url, headers=s.headers, files=files, verify=settings.INSIGHTS_CERT_PATH, timeout=(31, 31))
except requests.RequestException:
logger.error("Automation Analytics API request failed, trying base auth method")
response = s.post(url, files=files, verify=settings.INSIGHTS_CERT_PATH, auth=(rh_id, rh_secret), headers=s.headers, timeout=(31, 31))
# Accept 2XX status_codes
if response.status_code >= 300:
logger.error('Upload failed with status {}, {}'.format(response.status_code, response.text))
return False
return True
if response.status_code < 300:
_log_shipping_response(response, path)
return True
else:
logger.error(f'OIDC authentication failed with status {response.status_code}, {response.text}')
return False
except requests.RequestException as e:
logger.error(f"OIDC authentication failed: {e}")
return False

View File

@@ -1,22 +1,25 @@
import os
from dispatcherd.config import setup as dispatcher_setup
from django.apps import AppConfig
from django.db import connection
from django.utils.translation import gettext_lazy as _
from awx.main.utils.common import bypass_in_test, load_all_entry_points_for
from awx.main.utils.migration import is_database_synchronized
from awx.main.utils.named_url_graph import _customize_graph, generate_graph
from awx.conf import register, fields
from django.core.management.base import CommandError
from django.db.models.signals import pre_migrate
from awx_plugins.interfaces._temporary_private_licensing_api import detect_server_product_name
from awx.main.utils.named_url_graph import _customize_graph, generate_graph
from awx.main.utils.db import db_requirement_violations
from awx.conf import register, fields
class MainConfig(AppConfig):
name = 'awx.main'
verbose_name = _('Main')
def check_db_requirement(self, *args, **kwargs):
violations = db_requirement_violations()
if violations:
raise CommandError(violations)
def load_named_url_feature(self):
models = [m for m in self.get_models() if hasattr(m, 'get_absolute_url')]
generate_graph(models)
@@ -43,42 +46,6 @@ class MainConfig(AppConfig):
category_slug='named-url',
)
def _load_credential_types_feature(self):
"""
Create CredentialType records for any discovered credentials.
Note that Django docs advise _against_ interacting with the database using
the ORM models in the ready() path. Specifically, during testing.
However, we explicitly use the @bypass_in_test decorator to avoid calling this
method during testing.
Django also advises against running pattern because it runs everywhere i.e.
every management command. We use an advisory lock to ensure correctness and
we will deal performance if it becomes an issue.
"""
from awx.main.models.credential import CredentialType
if is_database_synchronized():
CredentialType.setup_tower_managed_defaults(app_config=self)
@bypass_in_test
def load_credential_types_feature(self):
from awx.main.models.credential import load_credentials
load_credentials()
return self._load_credential_types_feature()
def load_inventory_plugins(self):
from awx.main.models.inventory import InventorySourceOptions
is_awx = detect_server_product_name() == 'AWX'
extra_entry_point_groups = () if is_awx else ('inventory.supported',)
entry_points = load_all_entry_points_for(['inventory', *extra_entry_point_groups])
for entry_point_name, entry_point in entry_points.items():
cls = entry_point.load()
InventorySourceOptions.injectors[entry_point_name] = cls
def configure_dispatcherd(self):
"""This implements the default configuration for dispatcherd
@@ -101,12 +68,27 @@ class MainConfig(AppConfig):
self.configure_dispatcherd()
"""
Credential loading triggers database operations. There are cases we want to call
awx-manage collectstatic without a database. All management commands invoke the ready() code
path. Using settings.AWX_SKIP_CREDENTIAL_TYPES_DISCOVER _could_ invoke a database operation.
"""
if not os.environ.get('AWX_SKIP_CREDENTIAL_TYPES_DISCOVER', None):
self.load_credential_types_feature()
from ansible_base.rbac.triggers import dab_post_migrate
dab_post_migrate.connect(self._sync_managed_role_definitions, dispatch_uid='awx-sync-managed-role-definitions')
self.load_named_url_feature()
self.load_inventory_plugins()
pre_migrate.connect(self.check_db_requirement, sender=self)
@staticmethod
def _sync_managed_role_definitions(sender, **kwargs):
from django.apps import apps as global_apps
from ansible_base.resource_registry.signals.handlers import no_reverse_sync
# NOTE: setup_managed_role_definitions lives in the migrations module because
# it is also called from migration 0192. Ideally this would be extracted to a
# shared non-migration module, but doing so requires updating the migration
# import, which is a broader refactor (see also models/rbac.py imports).
from awx.main.migrations._dab_rbac import setup_managed_role_definitions
# During post-migrate the resource server (gateway) may not be ready
# (e.g. migrate_service_data still holds a 423 lock). Disable reverse
# sync for this call — gateway reconciles via migrate_service_data.
with no_reverse_sync():
setup_managed_role_definitions(global_apps, None)

View File

@@ -1,87 +0,0 @@
import functools
from django.conf import settings
from django.core.cache.backends.base import DEFAULT_TIMEOUT
from django.core.cache.backends.redis import RedisCache
from redis.exceptions import ConnectionError, ResponseError, TimeoutError
import socket
# This list comes from what django-redis ignores and the behavior we are trying
# to retain while dropping the dependency on django-redis.
IGNORED_EXCEPTIONS = (TimeoutError, ResponseError, ConnectionError, socket.timeout)
CONNECTION_INTERRUPTED_SENTINEL = object()
def optionally_ignore_exceptions(func=None, return_value=None):
if func is None:
return functools.partial(optionally_ignore_exceptions, return_value=return_value)
@functools.wraps(func)
def wrapper(*args, **kwargs):
try:
return func(*args, **kwargs)
except IGNORED_EXCEPTIONS as e:
if settings.DJANGO_REDIS_IGNORE_EXCEPTIONS:
return return_value
raise e.__cause__ or e
return wrapper
class AWXRedisCache(RedisCache):
"""
We just want to wrap the upstream RedisCache class so that we can ignore
the exceptions that it raises when the cache is unavailable.
"""
@optionally_ignore_exceptions
def add(self, key, value, timeout=DEFAULT_TIMEOUT, version=None):
return super().add(key, value, timeout, version)
@optionally_ignore_exceptions(return_value=CONNECTION_INTERRUPTED_SENTINEL)
def _get(self, key, default=None, version=None):
return super().get(key, default, version)
def get(self, key, default=None, version=None):
value = self._get(key, default, version)
if value is CONNECTION_INTERRUPTED_SENTINEL:
return default
return value
@optionally_ignore_exceptions
def set(self, key, value, timeout=DEFAULT_TIMEOUT, version=None):
return super().set(key, value, timeout, version)
@optionally_ignore_exceptions
def touch(self, key, timeout=DEFAULT_TIMEOUT, version=None):
return super().touch(key, timeout, version)
@optionally_ignore_exceptions
def delete(self, key, version=None):
return super().delete(key, version)
@optionally_ignore_exceptions
def get_many(self, keys, version=None):
return super().get_many(keys, version)
@optionally_ignore_exceptions
def has_key(self, key, version=None):
return super().has_key(key, version)
@optionally_ignore_exceptions
def incr(self, key, delta=1, version=None):
return super().incr(key, delta, version)
@optionally_ignore_exceptions
def set_many(self, data, timeout=DEFAULT_TIMEOUT, version=None):
return super().set_many(data, timeout, version)
@optionally_ignore_exceptions
def delete_many(self, keys, version=None):
return super().delete_many(keys, version)
@optionally_ignore_exceptions
def clear(self):
return super().clear()

View File

@@ -213,6 +213,40 @@ register(
category_slug='system',
)
register(
'AWX_ANALYTICS_CANDLEPIN_CA',
field_class=fields.CharField,
default='/etc/rhsm/ca/redhat-uep.pem',
allow_blank=True,
label=_('Candlepin CA Certificate Path'),
help_text=_('Path to the CA certificate file for verifying TLS connections to Candlepin. Leave blank to use system certificates.'),
category=_('System'),
category_slug='system',
)
register(
'AWX_ANALYTICS_CANDLEPIN_RENEWAL_THRESHOLD_DAYS',
field_class=fields.IntegerField,
default=90,
min_value=1,
label=_('Candlepin Certificate Renewal Threshold'),
help_text=_('Number of days before certificate expiry to trigger automatic renewal of Candlepin identity certificates.'),
category=_('System'),
category_slug='system',
unit=_('days'),
)
register(
'AWX_ANALYTICS_CANDLEPIN_PROXY_URL',
field_class=fields.CharField,
default='',
allow_blank=True,
label=_('Candlepin Proxy URL'),
help_text=_('HTTP/HTTPS proxy URL for Candlepin API requests (e.g., http://proxy.example.com:8080). Leave blank for no proxy.'),
category=_('System'),
category_slug='system',
)
register(
'INSTALL_UUID',
field_class=fields.CharField,
@@ -291,6 +325,22 @@ register(
category_slug='jobs',
)
register(
'INCLUDE_DEPRECATED_AWX_VAR_PREFIX',
field_class=fields.BooleanField,
default=True,
label=_('Include Deprecated AWX Variable Prefix'),
help_text=_(
'When enabled (default), auto-generated job variables are emitted '
'with both the tower_ prefix and the deprecated awx_ prefix for '
'backward compatibility. Disable to emit only tower_ prefixed '
'variables and eliminate duplicates. The awx_ prefix is deprecated '
'and this setting will default to False in a future release.'
),
category=_('Jobs'),
category_slug='jobs',
)
register(
'AWX_ISOLATION_BASE_PATH',
field_class=fields.CharField,
@@ -824,6 +874,58 @@ register(
unit=_('seconds'),
)
register(
'CANDLEPIN_CONSUMER_UUID',
field_class=fields.CharField,
default='',
allow_blank=True,
encrypted=False,
label=_('Candlepin Consumer UUID'),
help_text=_('UUID of the registered Candlepin consumer for this AAP instance.'),
category=_('System'),
category_slug='system',
hidden=True,
)
register(
'CANDLEPIN_CERT_PEM',
field_class=fields.CharField,
default='',
allow_blank=True,
encrypted=True,
label=_('Candlepin Identity Certificate'),
help_text=_('PEM-encoded Candlepin identity certificate for mTLS authentication.'),
category=_('System'),
category_slug='system',
hidden=True,
)
register(
'CANDLEPIN_KEY_PEM',
field_class=fields.CharField,
default='',
allow_blank=True,
encrypted=True,
label=_('Candlepin Identity Key'),
help_text=_('PEM-encoded private key for Candlepin identity certificate.'),
category=_('System'),
category_slug='system',
hidden=True,
)
register(
'CANDLEPIN_SERIAL_NUMBER',
field_class=fields.CharField,
default='',
allow_blank=True,
encrypted=False,
label=_('Candlepin Certificate Serial Number'),
help_text=_('Serial number of the Candlepin identity certificate for tracking.'),
category=_('System'),
category_slug='system',
hidden=True,
)
register(
'IS_K8S',
field_class=fields.BooleanField,

View File

@@ -11,6 +11,7 @@ __all__ = [
'CAN_CANCEL',
'ACTIVE_STATES',
'STANDARD_INVENTORY_UPDATE_ENV',
'OIDC_CREDENTIAL_TYPE_NAMESPACES',
]
PRIVILEGE_ESCALATION_METHODS = [
@@ -99,10 +100,6 @@ MAX_ISOLATED_PATH_COLON_DELIMITER = 2
SURVEY_TYPE_MAPPING = {'text': str, 'textarea': str, 'password': str, 'multiplechoice': str, 'multiselect': str, 'integer': int, 'float': (float, int)}
JOB_VARIABLE_PREFIXES = [
'awx',
'tower',
]
# Note, the \u001b[... are ansi color codes. We don't currenly import any of the python modules which define the codes.
# Importing a library just for this message seemed like overkill
@@ -140,3 +137,6 @@ org_role_to_permission = {
'execution_environment_admin_role': 'add_executionenvironment',
'auditor_role': 'view_project', # TODO: also doesnt really work
}
# OIDC credential type namespaces for feature flag filtering
OIDC_CREDENTIAL_TYPE_NAMESPACES = ['hashivault-kv-oidc', 'hashivault-ssh-oidc']

View File

@@ -68,7 +68,7 @@ class RecordedQueryLog(object):
progname = match
break
else:
progname = os.path.basename(sys.argv[0])
progname = 'unknown'
filepath = os.path.join(self.dest, '{}.sqlite'.format(progname))
version = _get_version('awx')
log = sqlite3.connect(filepath, timeout=3)

View File

@@ -25,8 +25,13 @@ def get_dispatcherd_config(for_service: bool = False, mock_publish: bool = False
"version": 2,
"service": {
"pool_kwargs": {
"min_workers": settings.JOB_EVENT_WORKERS,
"min_workers": settings.DISPATCHER_MIN_WORKERS,
"max_workers": max_workers,
# This must be less than max_workers to make sense, which is usually 4
# With reserve of 1, after a burst of tasks, load needs to down to 4-1=3
# before we return to min_workers
"scaledown_reserve": 1,
"worker_max_lifetime_seconds": settings.WORKER_MAX_LIFETIME_SECONDS,
},
"main_kwargs": {"node_id": settings.CLUSTER_HOST_ID},
"process_manager_cls": "ForkServerManager",

View File

@@ -77,13 +77,13 @@ class CallbackBrokerWorker:
MAX_RETRIES = 2
INDIVIDUAL_EVENT_RETRIES = 3
last_stats = time.time()
last_flush = time.time()
total = 0
last_event = ''
prof = None
def __init__(self):
self.last_stats = time.time()
self.last_flush = time.time()
self.buff = {}
self.redis = get_redis_client()
self.subsystem_metrics = s_metrics.CallbackReceiverMetrics(auto_pipe_execute=False)

View File

@@ -428,6 +428,9 @@ class CredentialInputField(JSONSchemaField):
# determine the defined fields for the associated credential type
properties = {}
for field in model_instance.credential_type.inputs.get('fields', []):
# Prevent users from providing values for internally resolved fields
if 'internal' in field:
continue
field = field.copy()
properties[field['id']] = field
if field.get('choices', []):
@@ -566,6 +569,7 @@ class CredentialTypeInputField(JSONSchemaField):
},
'label': {'type': 'string'},
'help_text': {'type': 'string'},
'internal': {'type': 'boolean'},
'multiline': {'type': 'boolean'},
'secret': {'type': 'boolean'},
'ask_at_runtime': {'type': 'boolean'},

View File

@@ -0,0 +1,330 @@
import sys
from argparse import RawDescriptionHelpFormatter
from django.core.management.base import BaseCommand
from awx.main.utils.candlepin.client import CandlepinClient
from awx.main.utils.candlepin.lifecycle import (
get_candlepin_ca,
get_candlepin_url,
get_proxy_url,
get_renewal_days,
needs_renewal,
parse_cert,
)
from awx.main.utils.candlepin import (
_fetch_candlepin_cert_from_db,
_save_candlepin_cert_to_db,
_save_candlepin_registration_to_db,
resolve_registration_credentials,
)
class Command(BaseCommand):
"""
Manage Candlepin consumer registration and certificate lifecycle.
Subcommands:
register Register this AAP instance as a Candlepin consumer and obtain an
identity certificate for mTLS analytics uploads.
renew Perform a manual check-in and, if needed, renew the stored identity
certificate.
"""
help = 'Manage Candlepin consumer registration and certificate lifecycle'
def create_parser(self, prog_name, subcommand, **kwargs):
return super().create_parser(
prog_name,
subcommand,
formatter_class=RawDescriptionHelpFormatter,
epilog='\n'.join(
[
'SUBCOMMANDS',
'',
' register Register this instance as a Candlepin consumer.',
' Credentials are read from AWX database by default',
' (REDHAT_USERNAME, REDHAT_PASSWORD). The organization is',
' discovered automatically from the Candlepin account.',
' Pass --username / --password-stdin / --org to override.',
' Example: echo "password" | awx-manage candlepin_cert register --username user --password-stdin',
'',
' renew Perform a manual check-in and proactive cert renewal.',
' Reads the stored cert/key/UUID from database.',
' Use --force to renew even if the cert is not near expiry.',
'',
'CONFIGURATION',
'',
' Settings can be configured via Django settings (awx/settings/defaults.py):',
'',
' AWX_ANALYTICS_CANDLEPIN_URL Candlepin base URL',
' (default: https://subscription.example.com/candlepin)',
' AWX_ANALYTICS_CANDLEPIN_CA Path to Candlepin CA cert for TLS verification',
' AWX_ANALYTICS_CANDLEPIN_RENEWAL_THRESHOLD_DAYS Days before expiry to trigger renewal (default: 90)',
' AWX_ANALYTICS_CANDLEPIN_PROXY_URL HTTP/HTTPS proxy for Candlepin API calls',
]
),
**kwargs,
)
def add_arguments(self, parser):
subparsers = parser.add_subparsers(dest='subcommand', metavar='subcommand')
subparsers.required = True
# --- register ---
reg = subparsers.add_parser(
'register',
help='Register this instance as a Candlepin consumer',
formatter_class=RawDescriptionHelpFormatter,
)
reg.add_argument('--username', help='Red Hat subscription username (overrides REDHAT_USERNAME from database)')
reg.add_argument(
'--password-stdin', dest='password_stdin', action='store_true', help='Read password from stdin (overrides REDHAT_PASSWORD from database)'
)
reg.add_argument('--org', help='Candlepin owner/org key (overrides auto-discovered organization)')
reg.add_argument('--candlepin-url', dest='candlepin_url', help='Candlepin base URL (overrides AWX_ANALYTICS_CANDLEPIN_URL setting)')
reg.add_argument(
'--candlepin-ca', dest='candlepin_ca', help='Path to Candlepin CA cert for TLS verification (overrides AWX_ANALYTICS_CANDLEPIN_CA setting)'
)
reg.add_argument('--proxy', help='HTTP/HTTPS proxy URL (overrides AWX_ANALYTICS_CANDLEPIN_PROXY_URL setting)')
reg.add_argument('--no-verify-tls', dest='no_verify_tls', action='store_true', help='Disable TLS certificate verification for Candlepin API calls')
reg.add_argument('--force', action='store_true', help='Re-register even if a certificate already exists in database')
reg.add_argument('--dry-run', dest='dry_run', action='store_true', help='Perform registration but do not save the result to database')
# --- renew ---
ren = subparsers.add_parser(
'renew',
help='Check in and renew the Candlepin identity certificate',
formatter_class=RawDescriptionHelpFormatter,
)
ren.add_argument('--candlepin-url', dest='candlepin_url', help='Candlepin base URL (overrides AWX_ANALYTICS_CANDLEPIN_URL setting)')
ren.add_argument(
'--candlepin-ca', dest='candlepin_ca', help='Path to Candlepin CA cert for TLS verification (overrides AWX_ANALYTICS_CANDLEPIN_CA setting)'
)
ren.add_argument('--proxy', help='HTTP/HTTPS proxy URL (overrides AWX_ANALYTICS_CANDLEPIN_PROXY_URL setting)')
ren.add_argument('--no-verify-tls', dest='no_verify_tls', action='store_true', help='Disable TLS certificate verification for Candlepin API calls')
ren.add_argument('--force', action='store_true', help='Renew the certificate even if it is not near expiry')
ren.add_argument('--dry-run', dest='dry_run', action='store_true', help='Perform check-in and renewal but do not save the result to database')
def handle(self, *args, **options):
subcommand = options['subcommand']
if subcommand == 'register':
ok = self._handle_register(options)
elif subcommand == 'renew':
ok = self._handle_renew(options)
else:
self.stderr.write(f'Unknown subcommand: {subcommand}')
sys.exit(1)
if not ok:
sys.exit(1)
# ------------------------------------------------------------------
# register
# ------------------------------------------------------------------
def _resolve_and_validate_credentials(self, options):
"""Merge CLI options with DB values and validate all required fields are present.
Returns ``(username, password, org, db_install_uuid)`` on success, or ``None``
if any required field is missing (errors are written to ``self.stderr``).
"""
username_override = options.get('username')
org_override = options.get('org')
verify_tls = not options.get('no_verify_tls', False)
# Read password from stdin if --password-stdin is set
if options.get('password_stdin'):
password_override = sys.stdin.read().strip()
if not password_override:
self.stderr.write('--password-stdin specified but no password provided on stdin')
return None
else:
password_override = None
# Use shared resolution and validation function
username, password, org, install_uuid, errors = resolve_registration_credentials(
username_override=username_override, password_override=password_override, org_override=org_override, verify_tls=verify_tls
)
if errors:
for error in errors:
self.stderr.write(f'Missing required value: {error}')
return None
return username, password, org, install_uuid
def _handle_register(self, options):
dry_run = options['dry_run']
force = options['force']
# Check whether a cert is already stored unless --force.
existing_cert, existing_key, _ = _fetch_candlepin_cert_from_db()
if existing_cert and existing_key and not force:
self.stdout.write('A Candlepin identity certificate is already stored in database. Use --force to re-register and replace it.')
return True
# Resolve credentials: CLI flags take precedence over database.
resolved = self._resolve_and_validate_credentials(options)
if resolved is None:
return False
username, password, org, db_install_uuid = resolved
candlepin_url = options.get('candlepin_url') or get_candlepin_url()
candlepin_ca = options.get('candlepin_ca') or get_candlepin_ca()
proxy = options.get('proxy') or get_proxy_url()
verify_tls = not options.get('no_verify_tls', False)
# If dry-run, display what would happen and exit early before any Candlepin operations
if dry_run:
self.stdout.write('[dry-run] Would register with Candlepin:')
self.stdout.write(f' URL : {candlepin_url}')
self.stdout.write(f' Organization : {org}')
self.stdout.write(f' Username : {username}')
self.stdout.write(f' Install UUID : {db_install_uuid}')
if candlepin_ca:
self.stdout.write(f' CA cert : {candlepin_ca}')
if proxy:
self.stdout.write(f' Proxy : {proxy}')
self.stdout.write(f' Verify TLS : {verify_tls}')
self.stdout.write('[dry-run] No Candlepin operations performed.')
return True
client = CandlepinClient(base_url=candlepin_url, candlepin_ca=candlepin_ca, proxy=proxy, verify_tls=verify_tls)
self.stdout.write(f'Registering with Candlepin at {candlepin_url} (org={org}) ...')
try:
cert_pem, key_pem, consumer_uuid = client.register_consumer(username, password, org, install_uuid=db_install_uuid)
except Exception as e:
self.stderr.write(f'Registration failed: {e}')
return False
self.stdout.write('Registered successfully.')
self.stdout.write(f' Consumer UUID : {consumer_uuid}')
# Save to database
if _save_candlepin_registration_to_db(cert_pem, key_pem, consumer_uuid):
self.stdout.write('Certificate, key, and consumer UUID saved to database.')
else:
self.stderr.write('Failed to save registration to database.')
return False
# Best-effort certificate metadata display
try:
info = parse_cert(cert_pem)
self.stdout.write(f' Cert serial : {info["serial"]}')
self.stdout.write(f' Cert CN : {info["cn"]}')
self.stdout.write(f' Valid until : {info["not_after"]} ({info["days_remaining"]} days remaining)')
except ValueError as e:
self.stdout.write(f'Certificate metadata unavailable: {e}')
return True
# ------------------------------------------------------------------
# renew
# ------------------------------------------------------------------
def _handle_renew(self, options):
dry_run = options['dry_run']
force = options['force']
cert_pem, key_pem, consumer_uuid = _fetch_candlepin_cert_from_db()
if not cert_pem or not key_pem:
self.stderr.write('No Candlepin identity certificate found in database. Run the register subcommand first.')
return False
if not consumer_uuid:
self.stderr.write('CANDLEPIN_CONSUMER_UUID is not set. Run the register subcommand first.')
return False
try:
info = parse_cert(cert_pem)
self.stdout.write('Current certificate:')
self.stdout.write(f' Serial : {info["serial"]}')
self.stdout.write(f' CN : {info["cn"]}')
self.stdout.write(f' Valid until : {info["not_after"]} ({info["days_remaining"]} days remaining)')
except ValueError as e:
self.stdout.write('Current certificate:')
self.stdout.write(f' Certificate metadata unavailable: {e}')
info = None
candlepin_url = options.get('candlepin_url') or get_candlepin_url()
candlepin_ca = options.get('candlepin_ca') or get_candlepin_ca()
proxy = options.get('proxy') or get_proxy_url()
verify_tls = not options.get('no_verify_tls', False)
renewal_days = get_renewal_days()
# Check if renewal is needed (without force, just check cert expiry locally)
renewal_needed = force or needs_renewal(cert_pem, renewal_days)
# If dry-run, display what would happen and exit early before any Candlepin operations
if dry_run:
self.stdout.write('[dry-run] Would perform the following operations:')
self.stdout.write(f' URL : {candlepin_url}')
self.stdout.write(f' Consumer UUID : {consumer_uuid}')
if candlepin_ca:
self.stdout.write(f' CA cert : {candlepin_ca}')
if proxy:
self.stdout.write(f' Proxy : {proxy}')
self.stdout.write(f' Verify TLS : {verify_tls}')
self.stdout.write(' 1. Check in with Candlepin')
if renewal_needed:
reason = 'forced via --force' if force else f'expiry within {renewal_days} days'
self.stdout.write(f' 2. Renew certificate ({reason})')
else:
if info:
self.stdout.write(f' 2. No renewal needed ({info["days_remaining"]} days remaining, threshold: {renewal_days} days)')
else:
self.stdout.write(f' 2. No renewal needed (threshold: {renewal_days} days)')
self.stdout.write('[dry-run] No Candlepin operations performed.')
return True
client = CandlepinClient(base_url=candlepin_url, candlepin_ca=candlepin_ca, proxy=proxy, verify_tls=verify_tls)
self.stdout.write(f'Checking in with Candlepin at {candlepin_url} (consumer={consumer_uuid}) ...')
checkin_success = client.checkin(consumer_uuid, cert_pem, key_pem)
if not checkin_success:
self.stderr.write('Check-in with Candlepin failed. Unable to verify certificate status.')
self.stderr.write('Certificate renewal may still be needed. Use --force to renew anyway, or check logs for details.')
return False
self.stdout.write('Check-in successful.')
if not renewal_needed:
if info:
self.stdout.write(f'Certificate has {info["days_remaining"]} days remaining (renewal threshold: {renewal_days} days). No renewal needed.')
else:
self.stdout.write(f'Certificate renewal threshold is {renewal_days} days. No renewal needed.')
return True
reason = 'forced via --force' if force else f'expiry within {renewal_days} days'
self.stdout.write(f'Renewing certificate ({reason}) ...')
try:
new_cert_pem, new_key_pem = client.regenerate_cert(consumer_uuid, cert_pem, key_pem)
except Exception as e:
self.stderr.write(f'Certificate renewal failed: {e}')
return False
self.stdout.write('Certificate renewed successfully.')
# Save to database
if _save_candlepin_cert_to_db(new_cert_pem, new_key_pem):
self.stdout.write('Renewed certificate and key saved to database.')
else:
self.stderr.write('Failed to save renewed certificate to database.')
return False
# Best-effort certificate metadata display
try:
new_info = parse_cert(new_cert_pem)
if info:
self.stdout.write(f' Old serial : {info["serial"]}')
self.stdout.write(f' New serial : {new_info["serial"]}')
self.stdout.write(f' Valid until : {new_info["not_after"]} ({new_info["days_remaining"]} days remaining)')
except ValueError as e:
self.stdout.write(f'Certificate metadata unavailable: {e}')
return True

View File

@@ -1,9 +1,11 @@
# Copyright (c) 2015 Ansible, Inc.
# All Rights Reserved
from django.core.management.base import BaseCommand
from django.core.management.base import BaseCommand, CommandError
from django.db import connection
from awx.main.utils.db import db_requirement_violations
class Command(BaseCommand):
"""Checks connection to the database, and prints out connection info if not connected"""
@@ -13,4 +15,8 @@ class Command(BaseCommand):
cursor.execute("SELECT version()")
version = str(cursor.fetchone()[0])
violations = db_requirement_violations()
if violations:
raise CommandError(violations)
return "Database Version: {}".format(version)

View File

@@ -49,6 +49,41 @@ def dt_to_partition_name(tbl_name, dt):
return f"{tbl_name}_{dt.strftime('%Y%m%d_%H')}"
JHS_CHUNK_SIZE = 1000
def _pre_delete_job_host_summaries(job_pks, logger=None):
"""Pre-delete JobHostSummary rows and clear Host FK references in batches.
Django's cascade collector materializes all JHS IDs into a single
UPDATE ... IN (...) to SET_NULL on Host.last_job_host_summary.
With many jobs x hosts this exceeds PostgreSQL's 1GB alloc limit.
Doing it in chunks with raw SQL avoids that.
"""
if not job_pks:
return
# ANY(%s) is PostgreSQL-specific; AWX only supports PostgreSQL
with connection.cursor() as cursor:
for i in range(0, len(job_pks), JHS_CHUNK_SIZE):
chunk = list(job_pks[i : i + JHS_CHUNK_SIZE])
cursor.execute(
"UPDATE main_host SET last_job_host_summary_id = NULL"
" WHERE last_job_host_summary_id IN"
" (SELECT id FROM main_jobhostsummary WHERE job_id = ANY(%s))",
[chunk],
)
cursor.execute(
"DELETE FROM main_jobhostsummary WHERE job_id = ANY(%s)",
[chunk],
)
if logger:
logger.debug("Pre-deleted JobHostSummary chunk %d-%d of %d job PKs", i, i + len(chunk), len(job_pks))
class DeleteMeta:
def __init__(self, logger, job_class, cutoff, dry_run):
self.logger = logger
@@ -91,6 +126,8 @@ class DeleteMeta:
def delete_jobs(self):
if not self.dry_run:
if self.job_class is Job:
_pre_delete_job_host_summaries(self.jobs_pk_list, self.logger)
self.job_class.objects.filter(pk__in=self.jobs_pk_list).delete()
def find_partitions_to_drop(self):
@@ -265,8 +302,9 @@ class Command(BaseCommand):
if info['min'] is not None:
for start in range(info['min'], info['max'] + 1, self.batch_size):
qs_batch = qs.filter(id__gte=start, id__lte=start + self.batch_size)
pk_list = qs_batch.values_list('id', flat=True)
pk_list = list(qs_batch.values_list('id', flat=True))
_pre_delete_job_host_summaries(pk_list, self.logger)
_, results = qs_batch.delete()
deleted += results['main.Job']
# Avoid dropping the job event table in case we have interacted with it already

View File

@@ -52,7 +52,11 @@ class Command(BaseCommand):
ssh_type = CredentialType.objects.filter(namespace='ssh').first()
c, _ = Credential.objects.get_or_create(
credential_type=ssh_type, name='Demo Credential', inputs={'username': getattr(superuser, 'username', 'null')}, created_by=superuser
credential_type=ssh_type,
name='Demo Credential',
inputs={'username': getattr(superuser, 'username', 'null')},
created_by=superuser,
organization=o,
)
if superuser:

View File

@@ -409,10 +409,12 @@ class Command(BaseCommand):
del_child_group_pks = list(set(db_children_name_pk_map.values()))
for offset in range(0, len(del_child_group_pks), self._batch_size):
child_group_pks = del_child_group_pks[offset : (offset + self._batch_size)]
for db_child in db_children.filter(pk__in=child_group_pks):
group_group_count += 1
db_group.children.remove(db_child)
logger.debug('Group "%s" removed from group "%s"', db_child.name, db_group.name)
children_to_remove = list(db_children.filter(pk__in=child_group_pks))
if children_to_remove:
group_group_count += len(children_to_remove)
db_group.children.remove(*children_to_remove)
for db_child in children_to_remove:
logger.debug('Group "%s" removed from group "%s"', db_child.name, db_group.name)
# FIXME: Inventory source group relationships
# Delete group/host relationships not present in imported data.
db_hosts = db_group.hosts
@@ -441,12 +443,12 @@ class Command(BaseCommand):
del_host_pks = list(del_host_pks)
for offset in range(0, len(del_host_pks), self._batch_size):
del_pks = del_host_pks[offset : (offset + self._batch_size)]
for db_host in db_hosts.filter(pk__in=del_pks):
group_host_count += 1
if db_host not in db_group.hosts.all():
continue
db_group.hosts.remove(db_host)
logger.debug('Host "%s" removed from group "%s"', db_host.name, db_group.name)
hosts_to_remove = list(db_hosts.filter(pk__in=del_pks))
if hosts_to_remove:
group_host_count += len(hosts_to_remove)
db_group.hosts.remove(*hosts_to_remove)
for db_host in hosts_to_remove:
logger.debug('Host "%s" removed from group "%s"', db_host.name, db_group.name)
if settings.SQL_DEBUG:
logger.warning(
'group-group and group-host deletions took %d queries for %d relationships',

View File

@@ -5,10 +5,12 @@ import logging
import uuid
from django.db import models
from django.conf import settings
from django.db.models import OuterRef, Subquery
from django.db.models.functions import Lower
from ansible_base.lib.utils.db import advisory_lock
from awx.main.utils.common import memoize
from awx.main.utils.filters import SmartFilter
from awx.main.constants import RECEPTOR_PENDING
@@ -23,46 +25,113 @@ class DeferJobCreatedManager(models.Manager):
return super(DeferJobCreatedManager, self).get_queryset().defer('job_created')
class HostManager(models.Manager):
class HostLatestSummaryQuerySet(models.QuerySet):
"""Queryset that annotates and bulk-attaches the latest JobHostSummary
at queryset evaluation time, similar to prefetch_related().
Why not use Django's Prefetch?
Django's Prefetch with [:1] slicing fetches 1 record globally, not per-host
(Django ticket #26780). Window-function workarounds require Django 4.2+ and
are more complex. Prefetching all summaries then filtering in Python wastes
memory for hosts with many job runs. The approach here — annotate the latest
ID via Subquery, then in_bulk() only those IDs — is the same 2-query pattern
prefetch_related uses internally, customized for "latest per group."
Not streaming-safe: relies on _result_cache existing after _fetch_all().
"""
_awx_latest_summary_attached = False
def _clone(self):
clone = super()._clone()
clone._awx_latest_summary_attached = self._awx_latest_summary_attached
return clone
def with_latest_summary_id(self):
from awx.main.models.jobs import JobHostSummary
latest_summary = JobHostSummary.objects.filter(host_id=OuterRef('pk')).order_by('-id')
return self.annotate(
_latest_summary_id=Subquery(latest_summary.values('id')[:1]),
)
def _fetch_all(self):
super()._fetch_all()
if self._awx_latest_summary_attached or not self._result_cache:
return
# Only bulk-attach if the queryset was annotated via with_latest_summary_id().
# Without this guard, we'd set _latest_summary_cache=None on every host,
# masking the per-object fallback query in Host.latest_summary.
if not hasattr(self._result_cache[0], '_latest_summary_id'):
return
from awx.main.models.jobs import JobHostSummary
latest_summary_ids = [host._latest_summary_id for host in self._result_cache if host._latest_summary_id is not None]
if latest_summary_ids:
summaries_by_id = JobHostSummary.objects.select_related('job', 'job__job_template').in_bulk(latest_summary_ids)
else:
summaries_by_id = {}
for host in self._result_cache:
latest_summary_id = getattr(host, '_latest_summary_id', None)
host._latest_summary_cache = summaries_by_id.get(latest_summary_id)
self._awx_latest_summary_attached = True
class HostManager(models.Manager.from_queryset(HostLatestSummaryQuerySet)):
"""Custom manager class for Hosts model."""
@memoize(ttl=60, cache_key='host_active_count')
def active_count(self):
"""Return count of active, unique hosts for licensing.
Construction of query involves:
- remove any ordering specified in model's Meta
- Exclude hosts sourced from another Tower
- Exclude hosts in constructed inventories (these are shadow rows of source-inventory hosts)
- Restrict the query to only return the name column
- Only consider results that are unique
- Return the count of this query
"""
return self.order_by().exclude(inventory_sources__source='controller').values(name_lower=Lower('name')).distinct().count()
return (
self.order_by()
.exclude(inventory_sources__source='controller')
.exclude(inventory__kind='constructed')
.values(name_lower=Lower('name'))
.distinct()
.count()
)
def org_active_count(self, org_id):
"""Return count of active, unique hosts used by an organization.
Construction of query involves:
- remove any ordering specified in model's Meta
- Exclude hosts sourced from another Tower
- Exclude hosts in constructed inventories (these are shadow rows of source-inventory hosts)
- Consider only hosts where the canonical inventory is owned by the organization
- Restrict the query to only return the name column
- Only consider results that are unique
- Return the count of this query
"""
return self.order_by().exclude(inventory_sources__source='controller').filter(inventory__organization=org_id).values('name').distinct().count()
return (
self.order_by()
.exclude(inventory_sources__source='controller')
.exclude(inventory__kind='constructed')
.filter(inventory__organization=org_id)
.values('name')
.distinct()
.count()
)
def get_queryset(self):
"""When the parent instance of the host query set has a `kind=smart` and a `host_filter`
set. Use the `host_filter` to generate the queryset for the hosts.
"""
qs = (
super(HostManager, self)
.get_queryset()
.defer(
'last_job__extra_vars',
'last_job_host_summary__job__extra_vars',
'last_job__artifacts',
'last_job_host_summary__job__artifacts',
)
)
qs = super().get_queryset().defer('ansible_facts')
if hasattr(self, 'instance') and hasattr(self.instance, 'host_filter') and hasattr(self.instance, 'kind'):
if self.instance.kind == 'smart' and self.instance.host_filter is not None:

View File

@@ -0,0 +1,29 @@
# Generated by Django 5.2.8 on 2026-02-20 03:39
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
('main', '0204_squashed_deletions'),
]
operations = [
migrations.AlterModelOptions(
name='instancegroup',
options={
'default_permissions': ('change', 'delete', 'view'),
'ordering': ('pk',),
'permissions': [('use_instancegroup', 'Can use instance group in a preference list of a resource')],
},
),
migrations.AlterModelOptions(
name='workflowjobnode',
options={'ordering': ('pk',)},
),
migrations.AlterModelOptions(
name='workflowjobtemplatenode',
options={'ordering': ('pk',)},
),
]

View File

@@ -0,0 +1,17 @@
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('main', '0205_add_ordering_to_instancegroup_and_workflow_nodes'),
]
operations = [
migrations.AddIndex(
model_name='jobhostsummary',
index=models.Index(
fields=['host', '-id'],
name='main_jobhostsumm_host_id_desc',
),
),
]

View File

@@ -0,0 +1,23 @@
# Generated by Django 5.2.8 on 2026-07-20 11:07
from django.db import migrations, models
class Migration(migrations.Migration):
dependencies = [
('main', '0206_jobhostsummary_host_id_idx'),
]
operations = [
migrations.AlterField(
model_name='job',
name='skip_tags',
field=models.TextField(blank=True, default=''),
),
migrations.AlterField(
model_name='jobtemplate',
name='skip_tags',
field=models.TextField(blank=True, default=''),
),
]

View File

@@ -148,6 +148,12 @@ def get_permissions_for_role(role_field, children_map, apps):
if role_field.name == 'auditor_role':
perm_list.append(Permission.objects.get(codename='view_notificationtemplate'))
# organization child admin roles need member_organization for create operations
if role_field.model._meta.model_name == 'organization' and role_field.name.endswith('_admin_role') and role_field.name != 'admin_role':
member_perm = Permission.objects.get(codename='member_organization')
if member_perm not in perm_list:
perm_list.append(member_perm)
return perm_list
@@ -339,6 +345,7 @@ def setup_managed_role_definitions(apps, schema_editor):
if 'org_children' in to_create and (cls_name not in ('organization', 'instancegroup', 'team')):
org_child_perms = object_perms.copy()
org_child_perms.add(Permission.objects.get(codename='view_organization'))
org_child_perms.add(Permission.objects.get(codename='member_organization'))
managed_role_definitions.append(
get_or_create_managed(

View File

@@ -211,7 +211,7 @@ class AdHocCommand(UnifiedJob, JobNotificationMixin):
return AdHocCommand.objects.create(**data)
def save(self, *args, **kwargs):
update_fields = kwargs.get('update_fields', [])
update_fields = kwargs.get('update_fields') or []
def add_to_update_fields(name):
if name not in update_fields:

View File

@@ -177,7 +177,7 @@ class CreatedModifiedModel(BaseModel):
)
def save(self, *args, **kwargs):
update_fields = list(kwargs.get('update_fields', []))
update_fields = list(kwargs.get('update_fields') or [])
# Manually perform auto_now_add and auto_now logic.
if not self.pk and not self.created:
self.created = now()
@@ -207,7 +207,7 @@ class PasswordFieldsModel(BaseModel):
new_instance = not bool(self.pk)
# If update_fields has been specified, add our field names to it,
# if it hasn't been specified, then we're just doing a normal save.
update_fields = kwargs.get('update_fields', [])
update_fields = kwargs.get('update_fields') or []
# When first saving to the database, don't store any password field
# values, but instead save them until after the instance is created.
# Otherwise, store encrypted values to the database.
@@ -322,7 +322,7 @@ class PrimordialModel(HasEditsMixin, CreatedModifiedModel):
self._prior_values_store = {}
def save(self, *args, **kwargs):
update_fields = kwargs.get('update_fields', [])
update_fields = kwargs.get('update_fields') or []
user = get_current_user()
if user and not user.id:
user = None

View File

@@ -28,6 +28,7 @@ from rest_framework.serializers import ValidationError as DRFValidationError
from ansible_base.lib.utils.db import advisory_lock
# AWX
from awx.main.constants import OIDC_CREDENTIAL_TYPE_NAMESPACES
from awx.api.versioning import reverse
from awx.main.fields import (
ImplicitRoleField,
@@ -46,12 +47,9 @@ from awx.main.models.rbac import (
)
from awx.main.models import Team, Organization
from awx.main.utils import encrypt_field
from awx.main.utils.lazy_registry import LazyLoadDict
from awx_plugins.interfaces._temporary_private_licensing_api import detect_server_product_name
# DAB
from ansible_base.resource_registry.tasks.sync import get_resource_server_client
from ansible_base.resource_registry.utils.settings import resource_server_defined
__all__ = ['Credential', 'CredentialType', 'CredentialInputSource', 'build_safe_env']
logger = logging.getLogger('awx.main.models.credential')
@@ -79,46 +77,6 @@ def build_safe_env(env):
return safe_env
def check_resource_server_for_user_in_organization(user, organization, requesting_user):
if not resource_server_defined():
return False
if not requesting_user:
return False
client = get_resource_server_client(settings.RESOURCE_SERVICE_PATH, jwt_user_id=str(requesting_user.resource.ansible_id), raise_if_bad_request=False)
# need to get the organization object_id in resource server, by querying with ansible_id
response = client._make_request(path=f'resources/?ansible_id={str(organization.resource.ansible_id)}', method='GET')
response_json = response.json()
if response.status_code != 200:
logger.error(f'Failed to get organization object_id in resource server: {response_json.get("detail", "")}')
return False
if response_json.get('count', 0) == 0:
return False
org_id_in_resource_server = response_json['results'][0]['object_id']
client.base_url = client.base_url.replace('/api/gateway/v1/service-index/', '/api/gateway/v1/')
# find role assignments with:
# - roles Organization Member or Organization Admin
# - user ansible id
# - organization object id
response = client._make_request(
path=f'role_user_assignments/?role_definition__name__in=Organization Member,Organization Admin&user__resource__ansible_id={str(user.resource.ansible_id)}&object_id={org_id_in_resource_server}',
method='GET',
)
response_json = response.json()
if response.status_code != 200:
logger.error(f'Failed to get role user assignments in resource server: {response_json.get("detail", "")}')
return False
if response_json.get('count', 0) > 0:
return True
return False
class Credential(PasswordFieldsModel, CommonModelNameNotUnique, ResourceMixin):
"""
A credential contains information about how to talk to a remote resource
@@ -242,6 +200,29 @@ class Credential(PasswordFieldsModel, CommonModelNameNotUnique, ResourceMixin):
needed.append('vault_password')
return needed
@functools.cached_property
def context(self):
"""
Property for storing runtime context during credential resolution.
The context is a dict keyed by CredentialInputSource PK, where each value
is a dict of runtime fields for that input source. Example::
{
<input_source_pk>: {
"workload_identity_token": "<jwt_token>"
},
<another_input_source_pk>: {
"workload_identity_token": "<different_jwt_token>"
},
}
This structure allows each input source to have its own set of runtime
values, avoiding conflicts when a credential has multiple input sources
with different configurations (e.g., different JWT audiences).
"""
return {}
@cached_property
def dynamic_input_fields(self):
# if the credential is not yet saved we can't access the input_sources
@@ -367,21 +348,20 @@ class Credential(PasswordFieldsModel, CommonModelNameNotUnique, ResourceMixin):
def _get_dynamic_input(self, field_name):
for input_source in self.input_sources.all():
if input_source.input_field_name == field_name:
return input_source.get_input_value()
return input_source.get_input_value(context=self.context)
else:
raise ValueError('{} is not a dynamic input field'.format(field_name))
def validate_role_assignment(self, actor, role_definition, **kwargs):
requesting_user = kwargs.get('requesting_user', None)
if requesting_user and requesting_user.is_superuser:
return
if self.organization:
if isinstance(actor, User):
if actor.is_superuser:
return
if Organization.access_qs(actor, 'member').filter(id=self.organization.id).exists():
return
requesting_user = kwargs.get('requesting_user', None)
if check_resource_server_for_user_in_organization(actor, self.organization, requesting_user):
return
if isinstance(actor, Team):
if actor.organization == self.organization:
return
@@ -435,13 +415,15 @@ class CredentialType(CommonModelNameNotUnique):
def from_db(cls, db, field_names, values):
instance = super(CredentialType, cls).from_db(db, field_names, values)
if instance.managed and instance.namespace and instance.kind != "external":
native = ManagedCredentialType.registry[instance.namespace]
instance.inputs = native.inputs
instance.injectors = native.injectors
instance.custom_injectors = getattr(native, 'custom_injectors', None)
native = ManagedCredentialType.registry.get(instance.namespace)
if native:
instance.inputs = native.inputs
instance.injectors = native.injectors
instance.custom_injectors = getattr(native, 'custom_injectors', None)
elif instance.namespace and instance.kind == "external":
native = ManagedCredentialType.registry[instance.namespace]
instance.inputs = native.inputs
native = ManagedCredentialType.registry.get(instance.namespace)
if native:
instance.inputs = native.inputs
return instance
@@ -505,6 +487,7 @@ class CredentialType(CommonModelNameNotUnique):
existing = ct_class.objects.filter(name=default.name, kind=default.kind).first()
if existing is not None:
existing.namespace = default.namespace
existing.description = getattr(default, 'description', '')
existing.inputs = {}
existing.injectors = {}
existing.save()
@@ -544,7 +527,14 @@ class CredentialType(CommonModelNameNotUnique):
@classmethod
def load_plugin(cls, ns, plugin):
# TODO: User "side-loaded" credential custom_injectors isn't supported
ManagedCredentialType.registry[ns] = SimpleNamespace(namespace=ns, name=plugin.name, kind='external', inputs=plugin.inputs, backend=plugin.backend)
ManagedCredentialType.registry[ns] = SimpleNamespace(
namespace=ns,
name=plugin.name,
kind='external',
inputs=plugin.inputs,
backend=plugin.backend,
description=getattr(plugin, 'plugin_description', ''),
)
def inject_credential(self, credential, env, safe_env, args, private_data_dir, container_root=None):
from awx_plugins.interfaces._temporary_private_inject_api import inject_credential
@@ -556,7 +546,13 @@ class CredentialTypeHelper:
@classmethod
def get_creation_params(cls, cred_type):
if cred_type.kind == 'external':
return dict(namespace=cred_type.namespace, kind=cred_type.kind, name=cred_type.name, managed=True)
return {
'namespace': cred_type.namespace,
'kind': cred_type.kind,
'name': cred_type.name,
'managed': True,
'description': getattr(cred_type, 'description', ''),
}
return dict(
namespace=cred_type.namespace,
kind=cred_type.kind,
@@ -574,7 +570,7 @@ class CredentialTypeHelper:
class ManagedCredentialType(SimpleNamespace):
registry = {}
registry = None # initialized as LazyLoadDict after load_credentials is defined
class CredentialInputSource(PrimordialModel):
@@ -622,7 +618,15 @@ class CredentialInputSource(PrimordialModel):
raise ValidationError(_('Input field must be defined on target credential (options are {}).'.format(', '.join(sorted(defined_fields)))))
return self.input_field_name
def get_input_value(self):
def get_input_value(self, context: dict | None = None):
"""
Retrieve the value from the external credential backend.
Args:
context: Optional runtime context dict passed from the target credential.
"""
if context is None:
context = {}
backend = self.source_credential.credential_type.plugin.backend
backend_kwargs = {}
for field_name, value in self.source_credential.inputs.items():
@@ -633,6 +637,17 @@ class CredentialInputSource(PrimordialModel):
backend_kwargs.update(self.metadata)
# Resolve internal fields from the per-input-source context.
# The context dict is keyed by input source PK, e.g.:
# {42: {"workload_identity_token": "eyJ..."}, 43: {"workload_identity_token": "eyX..."}}
# This allows each input source to carry its own runtime values.
input_source_context = context.get(self.pk, {})
for field in self.source_credential.credential_type.inputs.get('fields', []):
if field.get('internal'):
value = input_source_context.get(field['id'])
if value is not None:
backend_kwargs[field['id']] = value
with set_environ(**settings.AWX_TASK_ENV):
return backend(**backend_kwargs)
@@ -641,13 +656,22 @@ class CredentialInputSource(PrimordialModel):
return reverse(view_name, kwargs={'pk': self.pk}, request=request)
def _is_oidc_namespace_disabled(ns):
"""Check if a credential namespace should be skipped based on the OIDC feature flag."""
return ns in OIDC_CREDENTIAL_TYPE_NAMESPACES and not getattr(settings, 'FEATURE_OIDC_WORKLOAD_IDENTITY_ENABLED', False)
def load_credentials():
ManagedCredentialType.registry.clear()
awx_entry_points = {ep.name: ep for ep in entry_points(group='awx_plugins.managed_credentials')}
supported_entry_points = {ep.name: ep for ep in entry_points(group='awx_plugins.managed_credentials.supported')}
plugin_entry_points = awx_entry_points if detect_server_product_name() == 'AWX' else {**awx_entry_points, **supported_entry_points}
for ns, ep in plugin_entry_points.items():
if _is_oidc_namespace_disabled(ns):
continue
cred_plugin = ep.load()
if not hasattr(cred_plugin, 'inputs'):
setattr(cred_plugin, 'inputs', {})
@@ -666,5 +690,13 @@ def load_credentials():
credential_plugins = {}
for ns, ep in credential_plugins.items():
if _is_oidc_namespace_disabled(ns):
continue
plugin = ep.load()
CredentialType.load_plugin(ns, plugin)
# load_credentials writes directly into this dict via registry[ns] = ...,
# LazyLoadDict just ensures it runs once before the first read access
ManagedCredentialType.registry = LazyLoadDict(load_credentials)

View File

@@ -24,7 +24,6 @@ from awx.main.managers import DeferJobCreatedManager
from awx.main.constants import MINIMAL_EVENTS
from awx.main.models.base import CreatedModifiedModel
from awx.main.utils import ignore_inventory_computed_fields, camelcase_to_underscore
from awx.main.utils.db import bulk_update_sorted_by_id
analytics_logger = logging.getLogger('awx.analytics.job_events')
@@ -590,20 +589,8 @@ class JobEvent(BasePlaybookEvent):
JobHostSummary.objects.bulk_create(summaries.values())
# update the last_job_id and last_job_host_summary_id
# in single queries
host_mapping = dict((summary['host_id'], summary['id']) for summary in JobHostSummary.objects.filter(job_id=job.id).values('id', 'host_id'))
updated_hosts = set()
for h in all_hosts:
# if the hostname *shows up* in the playbook_on_stats event
if h.name in hostnames:
h.last_job_id = job.id
updated_hosts.add(h)
if h.id in host_mapping:
h.last_job_host_summary_id = host_mapping[h.id]
updated_hosts.add(h)
bulk_update_sorted_by_id(Host, updated_hosts, ['last_job_id', 'last_job_host_summary_id'])
# last_job and last_job_host_summary are now derived via
# JobHostSummary.latest_for_host / latest_job_for_host
# Create/update Host Metrics
self._update_host_metrics(updated_hosts_list)

View File

@@ -58,8 +58,6 @@ class ExecutionEnvironment(CommonModel):
return reverse('api:execution_environment_detail', kwargs={'pk': self.pk}, request=request)
def validate_role_assignment(self, actor, role_definition, **kwargs):
from awx.main.models.credential import check_resource_server_for_user_in_organization
if self.managed:
raise ValidationError({'object_id': _('Can not assign object roles to managed Execution Environments')})
if self.organization_id is None:
@@ -69,8 +67,4 @@ class ExecutionEnvironment(CommonModel):
if actor.has_obj_perm(self.organization, 'view'):
return
requesting_user = kwargs.get('requesting_user', None)
if check_resource_server_for_user_in_organization(actor, self.organization, requesting_user):
return
raise ValidationError({'user': _('User must have view permission to Execution Environment organization')})

View File

@@ -374,6 +374,11 @@ class Instance(HasPolicyEditsMixin, BaseModel):
self.memory = new_memory
update_fields.append('memory')
# Do not mark nodes READY if cpu or memory is zero
if not errors and self.node_type != Instance.Types.HOP and (not new_cpu or not new_memory):
errors = _('Health check for {} reported invalid values: cpu={}, memory={}').format(self.hostname, new_cpu, new_memory)
logger.warning(errors)
if not errors:
self.refresh_capacity_fields()
self.errors = ''
@@ -485,6 +490,7 @@ class InstanceGroup(HasPolicyEditsMixin, BaseModel, RelatedJobsMixin, ResourceMi
class Meta:
app_label = 'main'
ordering = ('pk',)
permissions = [('use_instancegroup', 'Can use instance group in a preference list of a resource')]
# Since this has no direct organization field only superuser can add, so remove add permission
default_permissions = ('change', 'delete', 'view')

View File

@@ -18,7 +18,7 @@ from django.db import transaction
from django.core.exceptions import ValidationError
from django.urls import resolve
from django.utils.timezone import now
from django.db.models import Q
from django.db.models import Q, Subquery, OuterRef
# REST Framework
from rest_framework.exceptions import ParseError
@@ -27,7 +27,10 @@ from ansible_base.lib.utils.models import prevent_search
# AWX
from awx.api.versioning import reverse
from awx.main.utils.common import load_all_entry_points_for
from awx.main.utils.lazy_registry import LazyLoadDict
from awx.main.utils.plugins import discover_available_cloud_provider_plugin_names, compute_cloud_inventory_sources
from awx_plugins.interfaces._temporary_private_licensing_api import detect_server_product_name
from awx.main.consumers import emit_channel_notification
from awx.main.fields import (
ImplicitRoleField,
@@ -386,7 +389,10 @@ class Inventory(CommonModelNameNotUnique, ResourceMixin, RelatedJobsMixin, OpaQu
logger.debug("Going to update inventory computed fields, pk={0}".format(self.pk))
start_time = time.time()
active_hosts = self.hosts
failed_hosts = active_hosts.filter(last_job_host_summary__failed=True)
from awx.main.models.jobs import JobHostSummary # circular import: inventory.py loads before jobs.py
latest_summary_failed = Subquery(JobHostSummary.objects.filter(host_id=OuterRef('pk')).order_by('-id').values('failed')[:1])
failed_hosts = active_hosts.annotate(_latest_failed=latest_summary_failed).filter(_latest_failed=True)
active_groups = self.groups
if self.kind == 'smart':
active_groups = active_groups.none()
@@ -582,6 +588,23 @@ class Host(CommonModelNameNotUnique, RelatedJobsMixin):
objects = HostManager()
@property
def latest_summary(self):
if hasattr(self, '_latest_summary_cache'):
return self._latest_summary_cache
from awx.main.models.jobs import JobHostSummary
summary = JobHostSummary.objects.filter(host_id=self.pk).order_by('-id').select_related('job', 'job__job_template').first()
self._latest_summary_cache = summary
return summary
@property
def latest_job(self):
summary = self.latest_summary
if summary is None:
return None
return summary.job
def get_absolute_url(self, request=None):
return reverse('api:host_detail', kwargs={'pk': self.pk}, request=request)
@@ -906,12 +929,22 @@ class HostMetricSummaryMonthly(models.Model):
indirectly_managed_hosts = models.IntegerField(default=0, help_text=("Manually entered number indirectly managed hosts for a certain month"))
def _load_inventory_plugins():
is_awx = detect_server_product_name() == 'AWX'
extra_entry_point_groups = () if is_awx else ('inventory.supported',)
all_entry_points = load_all_entry_points_for(['inventory', *extra_entry_point_groups])
for entry_point_name, entry_point in all_entry_points.items():
cls = entry_point.load()
InventorySourceOptions.injectors[entry_point_name] = cls
class InventorySourceOptions(BaseModel):
"""
Common fields for InventorySource and InventoryUpdate.
"""
injectors = dict()
injectors = LazyLoadDict(_load_inventory_plugins)
# From the options of the Django management base command
INVENTORY_UPDATE_VERBOSITY_CHOICES = [
@@ -1129,7 +1162,7 @@ class InventorySource(UnifiedJobTemplate, InventorySourceOptions, CustomVirtualE
# If update_fields has been specified, add our field names to it,
# if it hasn't been specified, then we're just doing a normal save.
update_fields = kwargs.get('update_fields', [])
update_fields = kwargs.get('update_fields') or []
is_new_instance = not bool(self.pk)
# Set name automatically. Include PK (or placeholder) to make sure the names are always unique.

View File

@@ -52,7 +52,7 @@ from awx.main.models.mixins import (
WebhookTemplateMixin,
OpaQueryPathMixin,
)
from awx.main.constants import JOB_VARIABLE_PREFIXES
from awx.main.utils.common import get_job_variable_prefixes
logger = logging.getLogger('awx.main.models.jobs')
@@ -132,8 +132,7 @@ class JobOptions(BaseModel):
blank=True,
default=False,
)
skip_tags = models.CharField(
max_length=1024,
skip_tags = models.TextField(
blank=True,
default='',
)
@@ -347,7 +346,7 @@ class JobTemplate(
return actual_slice_count
def save(self, *args, **kwargs):
update_fields = kwargs.get('update_fields', [])
update_fields = kwargs.get('update_fields') or []
# if project is deleted for some reason, then keep the old organization
# to retain ownership for organization admins
if self.project and self.project.organization_id != self.organization_id:
@@ -817,19 +816,20 @@ class Job(UnifiedJob, JobOptions, SurveyJobMixin, JobNotificationMixin, TaskMana
def awx_meta_vars(self):
r = super(Job, self).awx_meta_vars()
prefixes = get_job_variable_prefixes()
if self.project:
for name in JOB_VARIABLE_PREFIXES:
for name in prefixes:
r['{}_project_revision'.format(name)] = self.project.scm_revision
r['{}_project_scm_branch'.format(name)] = self.project.scm_branch
if self.scm_branch:
for name in JOB_VARIABLE_PREFIXES:
for name in prefixes:
r['{}_job_scm_branch'.format(name)] = self.scm_branch
if self.job_template:
for name in JOB_VARIABLE_PREFIXES:
for name in prefixes:
r['{}_job_template_id'.format(name)] = self.job_template.pk
r['{}_job_template_name'.format(name)] = self.job_template.name
if self.execution_node:
for name in JOB_VARIABLE_PREFIXES:
for name in prefixes:
r['{}_execution_node'.format(name)] = self.execution_node
return r
@@ -845,6 +845,21 @@ class Job(UnifiedJob, JobOptions, SurveyJobMixin, JobNotificationMixin, TaskMana
def get_notification_friendly_name(self):
return "Job"
def get_source_hosts_for_constructed_inventory(self):
"""Return a QuerySet of the source (input inventory) hosts for a constructed inventory.
Constructed inventory hosts have an instance_id pointing to the real
host in the input inventory. This resolves those references and returns
a proper QuerySet (never a list), suitable for use with finish_fact_cache.
"""
Host = JobHostSummary._meta.get_field('host').related_model
if not self.inventory_id:
return Host.objects.none()
id_field = Host._meta.get_field('id')
return Host.objects.filter(id__in=self.inventory.hosts.exclude(instance_id='').values_list(Cast('instance_id', output_field=id_field))).only(
*HOST_FACTS_FIELDS
)
def get_hosts_for_fact_cache(self):
"""
Builds the queryset to use for writing or finalizing the fact cache
@@ -852,17 +867,15 @@ class Job(UnifiedJob, JobOptions, SurveyJobMixin, JobNotificationMixin, TaskMana
For constructed inventories, that means the original (input inventory) hosts
when slicing, that means only returning hosts in that slice
"""
Host = JobHostSummary._meta.get_field('host').related_model
if not self.inventory_id:
Host = JobHostSummary._meta.get_field('host').related_model
return Host.objects.none()
if self.inventory.kind == 'constructed':
id_field = Host._meta.get_field('id')
host_qs = Host.objects.filter(id__in=self.inventory.hosts.exclude(instance_id='').values_list(Cast('instance_id', output_field=id_field)))
host_qs = self.get_source_hosts_for_constructed_inventory()
else:
host_qs = self.inventory.hosts
host_qs = self.inventory.hosts.only(*HOST_FACTS_FIELDS)
host_qs = host_qs.only(*HOST_FACTS_FIELDS)
host_qs = self.inventory.get_sliced_hosts(host_qs, self.job_slice_number, self.job_slice_count)
return host_qs
@@ -1078,6 +1091,9 @@ class JobHostSummary(CreatedModifiedModel):
unique_together = [('job', 'host_name')]
verbose_name_plural = _('job host summaries')
ordering = ('-pk',)
indexes = [
models.Index(fields=['host', '-id'], name='main_jobhostsumm_host_id_desc'),
]
job = models.ForeignKey(
'Job',
@@ -1127,6 +1143,22 @@ class JobHostSummary(CreatedModifiedModel):
self.skipped,
)
@classmethod
def latest_for_host(cls, host_id):
"""Return the most recent JobHostSummary for a given host, or None."""
return cls.objects.filter(host_id=host_id).order_by('-id').first()
@classmethod
def latest_job_for_host(cls, host_id):
"""Return the Job from the most recent JobHostSummary for a host, or None."""
summary = cls.latest_for_host(host_id)
if summary:
try:
return summary.job
except cls.job.field.related_model.DoesNotExist:
return None
return None
def get_absolute_url(self, request=None):
return reverse('api:job_host_summary_detail', kwargs={'pk': self.pk}, request=request)
@@ -1135,7 +1167,7 @@ class JobHostSummary(CreatedModifiedModel):
# if it hasn't been specified, then we're just doing a normal save.
if self.host is not None:
self.host_name = self.host.name
update_fields = kwargs.get('update_fields', [])
update_fields = kwargs.get('update_fields') or []
self.failed = bool(self.dark or self.failures)
update_fields.append('failed')
super(JobHostSummary, self).save(*args, **kwargs)

View File

@@ -188,6 +188,16 @@ class SurveyJobTemplateMixin(models.Model):
runtime_extra_vars.pop(variable_key)
if default is not None:
# do not add variables that contain an empty string, are not required and are not present in extra_vars
# password fields must be skipped, because default values have special behaviour
if (
default == ''
and not survey_element.get('required')
and survey_element.get('type') != 'password'
and variable_key not in runtime_extra_vars
):
continue
decrypted_default = default
if survey_element['type'] == "password" and isinstance(decrypted_default, str) and decrypted_default.startswith('$encrypted$'):
decrypted_default = decrypt_value(get_encryption_key('value', pk=None), decrypted_default)

View File

@@ -99,7 +99,7 @@ class NotificationTemplate(CommonModelNameNotUnique):
def save(self, *args, **kwargs):
new_instance = not bool(self.pk)
update_fields = kwargs.get('update_fields', [])
update_fields = kwargs.get('update_fields') or []
# preserve existing notification messages if not overwritten by new messages
if not new_instance:

View File

@@ -367,7 +367,7 @@ class Project(UnifiedJobTemplate, ProjectOptions, ResourceMixin, CustomVirtualEn
pre_save_vals = getattr(self, '_prior_values_store', {})
# If update_fields has been specified, add our field names to it,
# if it hasn't been specified, then we're just doing a normal save.
update_fields = kwargs.get('update_fields', [])
update_fields = kwargs.get('update_fields') or []
self._skip_update = bool(kwargs.pop('skip_update', False))
# Create auto-generated local path if project uses SCM.
if self.pk and self.scm_type and not self.local_path.startswith('_'):
@@ -450,13 +450,14 @@ class Project(UnifiedJobTemplate, ProjectOptions, ResourceMixin, CustomVirtualEn
@property
def cache_id(self):
"""This gives the folder name where collections and roles will be saved to so it does not re-download
# Prefer scm_revision as the cache key if available. This guarantees that project changes are tracked correctly
# even over multiple nodes. The scm_revision is a hex string and thus safe to be used as a directory name.
if self.scm_revision:
return self.scm_revision
Normally we want this to track with the last update, because every update should pull new content.
This does not count sync jobs, but sync jobs do not update last_job or current_job anyway.
If cleanup_jobs deletes the last jobs, then we can fallback to using any given heuristic related
to the last job ran.
"""
# If no scm_revision is available (e.g. non-scm projects), use these. current_job_id and last_job_id are global
# IDs in the database. This means that when a project sync runs on one node, all other nodes become outdated,
# resulting in unnecessary re-syncs.
if self.current_job_id:
return str(self.current_job_id)
elif self.last_job_id:
@@ -638,7 +639,7 @@ class ProjectUpdate(UnifiedJob, ProjectOptions, JobNotificationMixin, TaskManage
@property
def cache_id(self):
if self.branch_override or self.job_type == 'check' or (not self.project):
if self.branch_override or (not self.project):
return str(self.id) # causes it to not use the cache, basically
return self.project.cache_id

View File

@@ -613,7 +613,7 @@ def get_role_from_object_role(object_role):
model_name, role_name = rd.name.split()
role_name = role_name.lower()
role_name += '_role'
return getattr(object_role.content_object, role_name)
return getattr(object_role.content_object, role_name, None)
def give_or_remove_permission(role, actor, giving=True, rd=None):
@@ -649,6 +649,8 @@ def give_creator_permissions(user, obj):
if assignment:
with disable_rbac_sync():
old_role = get_role_from_object_role(assignment.object_role)
if old_role is None:
return
old_role.members.add(user)
@@ -802,7 +804,17 @@ def _sync_assignments_to_old_rbac(instance, delete=True):
@receiver(post_delete, sender=RoleUserAssignment)
@receiver(post_delete, sender=RoleTeamAssignment)
def sync_assignments_to_old_rbac_delete(instance, **kwargs):
def sync_assignments_to_old_rbac_delete(instance, origin=None, **kwargs):
# Skip cascade deletes from non-assignment origins — sync is redundant:
# - Model origin with app_label != dab_rbac: a parent object (e.g.
# Organization) is being deleted and old Role M2M tables cascade from
# the same parent.
# - QuerySet of a different model (e.g. ObjectRole): bulk RBAC cleanup
# such as defer_rbac_computations flush — parent objects already gone.
if isinstance(origin, models.Model) and origin._meta.app_label != 'dab_rbac':
return
if isinstance(origin, models.QuerySet) and origin.model is not type(instance):
return
_sync_assignments_to_old_rbac(instance, delete=True)

View File

@@ -72,10 +72,10 @@ def _fast_forward_rrule(rrule, ref_dt=None):
if ref_dt is None:
ref_dt = now()
ref_dt = ref_dt.astimezone(datetime.timezone.utc)
dtstart_tz = rrule._dtstart.tzinfo
ref_dt = ref_dt.astimezone(dtstart_tz)
rrule_dtstart_utc = rrule._dtstart.astimezone(datetime.timezone.utc)
if rrule_dtstart_utc > ref_dt:
if rrule._dtstart > ref_dt:
return rrule
interval = rrule._interval if rrule._interval else 1
@@ -84,20 +84,14 @@ def _fast_forward_rrule(rrule, ref_dt=None):
elif rrule._freq == dateutil.rrule.MINUTELY:
interval *= 60
# if after converting to seconds the interval is still a fraction,
# just return original rrule
if isinstance(interval, float) and not interval.is_integer():
return rrule
seconds_since_dtstart = (ref_dt - rrule_dtstart_utc).total_seconds()
seconds_since_dtstart = (ref_dt - rrule._dtstart).total_seconds()
# it is important to fast forward by a number that is divisible by
# interval. For example, if interval is 7 hours, we fast forward by 7, 14, 21, etc. hours.
# Otherwise, the occurrences after the fast forward might not match the ones before.
# x // y is integer division, lopping off any remainder, so that we get the outcome we want.
interval_aligned_offset = datetime.timedelta(seconds=(seconds_since_dtstart // interval) * interval)
new_start = rrule_dtstart_utc + interval_aligned_offset
new_rrule = rrule.replace(dtstart=new_start.astimezone(rrule._dtstart.tzinfo))
new_start = rrule._dtstart + interval_aligned_offset
new_rrule = rrule.replace(dtstart=new_start)
return new_rrule

View File

@@ -20,6 +20,9 @@ from dispatcherd.factories import get_control_from_settings
# Django
from django.conf import settings
from django.db import models, connection, transaction
# psycopg
from psycopg import sql
from django.db.models.constraints import UniqueConstraint
from django.core.exceptions import NON_FIELD_ERRORS
from django.utils.translation import gettext_lazy as _
@@ -58,7 +61,8 @@ from awx.main.utils.common import (
)
from awx.main.utils.encryption import encrypt_dict, decrypt_field
from awx.main.utils import polymorphic
from awx.main.constants import ACTIVE_STATES, CAN_CANCEL, JOB_VARIABLE_PREFIXES
from awx.main.constants import ACTIVE_STATES, CAN_CANCEL
from awx.main.utils.common import get_job_variable_prefixes
from awx.main.redact import UriCleaner, REPLACE_STR
from awx.main.consumers import emit_channel_notification
from awx.main.fields import AskForField, OrderedManyToManyField
@@ -233,7 +237,11 @@ class UnifiedJobTemplate(PolymorphicModel, CommonModelNameNotUnique, ExecutionEn
dab_role_cts = permission_registry.content_type_model.objects.get_for_models(*role_subclasses).values()
return (
RoleEvaluation.objects.filter(role__in=accessor.has_roles.all(), codename__in=all_codenames, content_type_id__in=[ct.id for ct in dab_role_cts])
RoleEvaluation.objects.filter(
**RoleEvaluation._actor_role_filter(accessor),
codename__in=all_codenames,
content_type_id__in=[ct.id for ct in dab_role_cts],
)
.values_list('object_id')
.distinct()
)
@@ -304,7 +312,7 @@ class UnifiedJobTemplate(PolymorphicModel, CommonModelNameNotUnique, ExecutionEn
def save(self, *args, **kwargs):
# If update_fields has been specified, add our field names to it,
# if it hasn't been specified, then we're just doing a normal save.
update_fields = kwargs.get('update_fields', [])
update_fields = kwargs.get('update_fields') or []
# Update status and last_updated fields.
if not getattr(_inventory_updates, 'is_updating', False):
updated_fields = self._set_status_and_last_job_run(save=False)
@@ -876,7 +884,7 @@ class UnifiedJob(
"""
# If update_fields has been specified, add our field names to it,
# if it hasn't been specified, then we're just doing a normal save.
update_fields = kwargs.get('update_fields', [])
update_fields = kwargs.get('update_fields') or []
# Get status before save...
status_before = self.status or 'new'
@@ -1174,17 +1182,23 @@ class UnifiedJob(
raise StdoutMaxBytesExceeded(total, max_supported)
tbl = self._meta.db_table + 'event'
created_by_cond = ''
where_parts = [
sql.SQL('{} = {}').format(sql.Identifier(self.event_parent_key), sql.Literal(self.id)),
sql.SQL("stdout != ''"),
]
if self.has_unpartitioned_events:
tbl = f'_unpartitioned_{tbl}'
tbl = '_unpartitioned_' + tbl
else:
created_by_cond = f"job_created='{self.created.isoformat()}' AND "
where_parts.insert(0, sql.SQL('job_created = {}').format(sql.Literal(self.created)))
sql = f"copy (select stdout from {tbl} where {created_by_cond}{self.event_parent_key}={self.id} and stdout != '' order by start_line) to stdout" # nosql
copy_sql = sql.SQL('COPY (SELECT stdout FROM {} WHERE {} ORDER BY start_line) TO STDOUT').format(
sql.Identifier(tbl),
sql.SQL(' AND ').join(where_parts),
)
# psycopg3's copy writes bytes, but callers of this
# function assume a str-based fd will be returned; decode
# .write() calls on the fly to maintain this interface
with cursor.copy(sql) as copy:
with cursor.copy(copy_sql) as copy:
while data := copy.read():
fd.write(smart_str(bytes(data)))
@@ -1568,7 +1582,8 @@ class UnifiedJob(
by AWX, for purposes of client playbook hooks
"""
r = {}
for name in JOB_VARIABLE_PREFIXES:
prefixes = get_job_variable_prefixes()
for name in prefixes:
r['{}_job_id'.format(name)] = self.pk
r['{}_job_launch_type'.format(name)] = self.launch_type
@@ -1577,7 +1592,7 @@ class UnifiedJob(
wj = self.get_workflow_job()
if wj:
schedule = getattr_dne(wj, 'schedule')
for name in JOB_VARIABLE_PREFIXES:
for name in prefixes:
r['{}_workflow_job_id'.format(name)] = wj.pk
r['{}_workflow_job_name'.format(name)] = wj.name
r['{}_workflow_job_launch_type'.format(name)] = wj.launch_type
@@ -1588,12 +1603,12 @@ class UnifiedJob(
if not created_by:
schedule = getattr_dne(self, 'schedule')
if schedule:
for name in JOB_VARIABLE_PREFIXES:
for name in prefixes:
r['{}_schedule_id'.format(name)] = schedule.pk
r['{}_schedule_name'.format(name)] = schedule.name
if created_by:
for name in JOB_VARIABLE_PREFIXES:
for name in prefixes:
r['{}_user_id'.format(name)] = created_by.pk
r['{}_user_name'.format(name)] = created_by.username
r['{}_user_email'.format(name)] = created_by.email
@@ -1602,7 +1617,7 @@ class UnifiedJob(
inventory = getattr_dne(self, 'inventory')
if inventory:
for name in JOB_VARIABLE_PREFIXES:
for name in prefixes:
r['{}_inventory_id'.format(name)] = inventory.pk
r['{}_inventory_name'.format(name)] = inventory.name

View File

@@ -200,6 +200,7 @@ class WorkflowJobTemplateNode(WorkflowNodeBase):
indexes = [
models.Index(fields=['identifier']),
]
ordering = ('pk',)
def get_absolute_url(self, request=None):
return reverse('api:workflow_job_template_node_detail', kwargs={'pk': self.pk}, request=request)
@@ -286,6 +287,7 @@ class WorkflowJobNode(WorkflowNodeBase):
models.Index(fields=["identifier", "workflow_job"]),
models.Index(fields=['identifier']),
]
ordering = ('pk',)
@property
def event_processing_finished(self):
@@ -343,7 +345,11 @@ class WorkflowJobNode(WorkflowNodeBase):
)
data.update(accepted_fields) # missing fields are handled in the scheduler
# build ancestor artifacts, save them to node model for later
aa_dict = {}
# initialize from pre-seeded ancestor_artifacts (set on root nodes of
# child workflows via seed_root_ancestor_artifacts to carry artifacts
# from the parent workflow); exclude job_slice which is internal
# metadata handled separately below
aa_dict = {k: v for k, v in self.ancestor_artifacts.items() if k != 'job_slice'} if self.ancestor_artifacts else {}
is_root_node = True
for parent_node in self.get_parent_nodes():
is_root_node = False
@@ -364,11 +370,13 @@ class WorkflowJobNode(WorkflowNodeBase):
data['survey_passwords'] = password_dict
# process extra_vars
extra_vars = data.get('extra_vars', {})
if ujt_obj and isinstance(ujt_obj, (JobTemplate, WorkflowJobTemplate)):
if ujt_obj and isinstance(ujt_obj, JobTemplate):
if aa_dict:
functional_aa_dict = copy(aa_dict)
functional_aa_dict.pop('_ansible_no_log', None)
extra_vars.update(functional_aa_dict)
elif ujt_obj and isinstance(ujt_obj, WorkflowJobTemplate):
pass # artifacts are applied via seed_root_ancestor_artifacts in the task manager
# Workflow Job extra_vars higher precedence than ancestor artifacts
extra_vars.update(wj_special_vars)
@@ -732,6 +740,18 @@ class WorkflowJob(UnifiedJob, WorkflowJobOptions, SurveyJobMixin, JobNotificatio
wj = wj.get_workflow_job()
return ancestors
def seed_root_ancestor_artifacts(self, artifacts):
"""Apply parent workflow artifacts to root nodes so they propagate
through the normal ancestor_artifacts channel instead of being
baked into this workflow's extra_vars."""
self.workflow_job_nodes.exclude(
workflowjobnodes_success__isnull=False,
).exclude(
workflowjobnodes_failure__isnull=False,
).exclude(
workflowjobnodes_always__isnull=False,
).update(ancestor_artifacts=artifacts)
def get_effective_artifacts(self, **kwargs):
"""
For downstream jobs of a workflow nested inside of a workflow,
@@ -880,7 +900,7 @@ class WorkflowApproval(UnifiedJob, JobNotificationMixin):
return 'workflow_approval_template'
def save(self, *args, **kwargs):
update_fields = list(kwargs.get('update_fields', []))
update_fields = list(kwargs.get('update_fields') or [])
if self.timeout != 0 and ((not self.pk) or (not update_fields) or ('timeout' in update_fields)):
if not self.created: # on creation, created will be set by parent class, so we fudge it here
created = now()
@@ -916,6 +936,17 @@ class WorkflowApproval(UnifiedJob, JobNotificationMixin):
ScheduleWorkflowManager().schedule()
return reverse('api:workflow_approval_deny', kwargs={'pk': self.pk}, request=request)
def cancel(self, job_explanation=None, is_chain=False):
# WorkflowApprovals have no dispatcher process (they wait for human
# input) and are excluded from TaskManager processing, so the base
# cancel() would only set cancel_flag without ever transitioning the
# status. We call super() for the flag, then transition directly.
has_already_canceled = bool(self.status == 'canceled')
super().cancel(job_explanation=job_explanation, is_chain=is_chain)
if self.status != 'canceled' and not has_already_canceled:
self.status = 'canceled'
self.save(update_fields=['status'])
def signal_start(self, **kwargs):
can_start = super(WorkflowApproval, self).signal_start(**kwargs)
self.started = self.created

View File

@@ -19,13 +19,8 @@ class ActivityStreamRegistrar(object):
pre_delete.connect(activity_stream_delete, sender=model, dispatch_uid=str(self.__class__) + str(model) + "_delete")
for m2mfield in model._meta.many_to_many:
try:
m2m_attr = getattr(model, m2mfield.name)
m2m_changed.connect(
activity_stream_associate, sender=m2m_attr.through, dispatch_uid=str(self.__class__) + str(m2m_attr.through) + "_associate"
)
except AttributeError:
pass
m2m_attr = getattr(model, m2mfield.name)
m2m_changed.connect(activity_stream_associate, sender=m2m_attr.through, dispatch_uid=str(self.__class__) + str(m2m_attr.through) + "_associate")
def disconnect(self, model):
if model in self.models:

View File

@@ -48,11 +48,6 @@ class SimpleDAG(object):
'''
self.node_to_edges_by_label = dict()
def __contains__(self, obj):
if self.node['node_object'] in self.node_obj_to_node_index:
return True
return False
def __len__(self):
return len(self.nodes)

View File

@@ -122,8 +122,11 @@ class WorkflowDAG(SimpleDAG):
if not job:
continue
elif job.can_cancel:
cancel_finished = False
job.cancel()
# If the job is not yet in a terminal state after .cancel(),
# the TaskManager still needs to process it.
if job.status not in ('successful', 'failed', 'canceled', 'error'):
cancel_finished = False
return cancel_finished
def is_workflow_done(self):

View File

@@ -196,6 +196,10 @@ class WorkflowManager(TaskBase):
workflow_job.start_args = '' # blank field to remove encrypted passwords
workflow_job.save(update_fields=['status', 'start_args'])
status_changed = True
else:
# Speed-up: schedule the task manager so it can process the
# canceled pending jobs without waiting for the next cycle.
ScheduleTaskManager().schedule()
else:
dnr_nodes = dag.mark_dnr_nodes()
WorkflowJobNode.objects.bulk_update(dnr_nodes, ['do_not_run'])
@@ -237,6 +241,8 @@ class WorkflowManager(TaskBase):
job = spawn_node.unified_job_template.create_unified_job(**kv)
spawn_node.job = job
spawn_node.save()
if spawn_node.ancestor_artifacts and isinstance(spawn_node.unified_job_template, WorkflowJobTemplate):
job.seed_root_ancestor_artifacts(spawn_node.ancestor_artifacts)
logger.debug('Spawned %s in %s for node %s', job.log_format, workflow_job.log_format, spawn_node.pk)
can_start = True
if isinstance(spawn_node.unified_job_template, WorkflowJobTemplate):
@@ -443,17 +449,29 @@ class TaskManager(TaskBase):
self.controlplane_ig = self.tm_models.instance_groups.controlplane_ig
def process_job_dep_failures(self, task):
"""If job depends on a job that has failed, mark as failed and handle misc stuff."""
"""If job depends on a job that has failed or been canceled, mark as failed.
Returns True if a dep failure was found, False otherwise.
"""
for dep in task.dependent_jobs.all():
# if we detect a failed or error dependency, go ahead and fail this task.
if dep.status in ("error", "failed"):
# if we detect a failed, error, or canceled dependency, go ahead and fail this task.
if dep.status in ("error", "failed", "canceled"):
task.status = 'failed'
logger.warning(f'Previous task failed task: {task.id} dep: {dep.id} task manager')
task.job_explanation = 'Previous Task Failed: {"job_type": "%s", "job_name": "%s", "job_id": "%s"}' % (
get_type_for_model(type(dep)),
dep.name,
dep.id,
)
if dep.status == 'canceled':
logger.warning(f'Previous task canceled, failing task: {task.id} dep: {dep.id} task manager')
task.job_explanation = 'Previous Task Canceled: {"job_type": "%s", "job_name": "%s", "job_id": "%s"}' % (
get_type_for_model(type(dep)),
dep.name,
dep.id,
)
ScheduleWorkflowManager().schedule() # speedup for dependency chains in workflow, on workflow cancel
else:
logger.warning(f'Previous task failed, failing task: {task.id} dep: {dep.id} task manager')
task.job_explanation = 'Previous Task Failed: {"job_type": "%s", "job_name": "%s", "job_id": "%s"}' % (
get_type_for_model(type(dep)),
dep.name,
dep.id,
)
task.save(update_fields=['status', 'job_explanation'])
task.websocket_emit_status('failed')
self.pre_start_failed.append(task.id)
@@ -545,8 +563,17 @@ class TaskManager(TaskBase):
logger.warning("Task manager has reached time out while processing pending jobs, exiting loop early")
break
has_failed = self.process_job_dep_failures(task)
if has_failed:
if task.cancel_flag:
logger.debug(f"Canceling pending task {task.log_format} because cancel_flag is set")
task.status = 'canceled'
task.job_explanation = gettext_noop("This job was canceled before it started.")
task.save(update_fields=['status', 'job_explanation'])
task.websocket_emit_status('canceled')
self.pre_start_failed.append(task.id)
ScheduleWorkflowManager().schedule()
continue
if self.process_job_dep_failures(task):
continue
blocked_by = self.job_blocked_by(task)
@@ -661,6 +688,17 @@ class TaskManager(TaskBase):
logger.error(f'{j.execution_node} is not a registered instance; reaping {j.log_format}')
reap_job(j, 'failed')
# Reset waiting jobs whose controller_node was deprovisioned (e.g. K8s pod replaced).
# These jobs will never be picked up because no live node is listening for them.
registered_control_nodes = Instance.objects.filter(node_type__in=('control', 'hybrid')).values_list('hostname', flat=True)
orphaned_waiting = UnifiedJob.objects.filter(status='waiting').exclude(controller_node__in=registered_control_nodes)
for j in orphaned_waiting:
logger.warning(f'{j.controller_node} is not a registered instance; resetting {j.log_format} to pending')
j.status = 'pending'
j.controller_node = ''
j.execution_node = ''
j.save(update_fields=['status', 'controller_node', 'execution_node'])
def process_tasks(self):
# maintain a list of jobs that went to an early failure state,
# meaning the dispatcher never got these jobs,

View File

@@ -146,12 +146,19 @@ class TaskManagerInstances:
self.instances_by_hostname[instance.hostname] = TaskManagerInstance(instance, **kwargs)
def consume_capacity(self, task):
"""Subtract a task's capacity from its execution and control instances.
For the control instance, jobs_running is only incremented when the controller
differs from the execution node to avoid double-counting on hybrid nodes.
"""
control_instance = self.instances_by_hostname.get(task.controller_node, '')
execution_instance = self.instances_by_hostname.get(task.execution_node, '')
if execution_instance and execution_instance.node_type in ('hybrid', 'execution'):
self.instances_by_hostname[task.execution_node].consume_capacity(task.task_impact, job_impact=True)
if control_instance and control_instance.node_type in ('hybrid', 'control'):
self.instances_by_hostname[task.controller_node].consume_capacity(self.control_task_impact)
# Track jobs_running on the controller unless it was already counted as the execution node
count_as_job = control_instance != execution_instance
self.instances_by_hostname[task.controller_node].consume_capacity(self.control_task_impact, job_impact=count_as_job)
def __getitem__(self, hostname):
return self.instances_by_hostname.get(hostname)
@@ -207,6 +214,11 @@ class TaskManagerInstanceGroups:
return self.instance_groups[group_name].instances
def fit_task_to_most_remaining_capacity_instance(self, task, instance_group_name, impact=None, capacity_type=None, add_hybrid_control_cost=False):
"""Select the instance with the most remaining capacity after absorbing the task.
When two or more instances would have equal remaining capacity, prefer the
instance with fewer jobs_running to balance controller load during bursts.
"""
impact = impact if impact else task.task_impact
capacity_type = capacity_type if capacity_type else task.capacity_type
instance_most_capacity = None
@@ -220,7 +232,11 @@ class TaskManagerInstanceGroups:
# hybrid nodes _always_ control their own tasks
if add_hybrid_control_cost and i.node_type == 'hybrid':
would_be_remaining -= self.control_task_impact
if would_be_remaining >= 0 and (instance_most_capacity is None or would_be_remaining > most_remaining_capacity):
if would_be_remaining >= 0 and (
instance_most_capacity is None
or would_be_remaining > most_remaining_capacity
or (would_be_remaining == most_remaining_capacity and i.jobs_running < instance_most_capacity.jobs_running)
):
instance_most_capacity = i
most_remaining_capacity = would_be_remaining
return instance_most_capacity

View File

@@ -36,7 +36,6 @@ from awx.main.models import (
Inventory,
InventorySource,
Job,
JobHostSummary,
Organization,
Project,
Role,
@@ -251,45 +250,9 @@ def migrate_children_from_deleted_group_to_parent_groups(sender, **kwargs):
pass
# Update host pointers to last_job and last_job_host_summary when a job is deleted
def _update_host_last_jhs(host):
jhs_qs = JobHostSummary.objects.filter(host__pk=host.pk)
try:
jhs = jhs_qs.order_by('-job__pk')[0]
except IndexError:
jhs = None
update_fields = []
try:
last_job = jhs.job if jhs else None
except Job.DoesNotExist:
# The job (and its summaries) have already been/are currently being
# deleted, so there's no need to update the host w/ a reference to it
return
if host.last_job != last_job:
host.last_job = last_job
update_fields.append('last_job')
if host.last_job_host_summary != jhs:
host.last_job_host_summary = jhs
update_fields.append('last_job_host_summary')
if update_fields:
host.save(update_fields=update_fields)
@receiver(pre_delete, sender=Job)
def save_host_pks_before_job_delete(sender, **kwargs):
instance = kwargs['instance']
hosts_qs = Host.objects.filter(last_job__pk=instance.pk)
instance._saved_hosts_pks = set(hosts_qs.values_list('pk', flat=True))
@receiver(post_delete, sender=Job)
def update_host_last_job_after_job_deleted(sender, **kwargs):
instance = kwargs['instance']
hosts_pks = getattr(instance, '_saved_hosts_pks', [])
for host in Host.objects.filter(pk__in=hosts_pks):
_update_host_last_jhs(host)
# Host.last_job and Host.last_job_host_summary are now derived from
# JobHostSummary.latest_for_host / latest_job_for_host.
# No signal handlers needed to maintain these denormalized FKs.
# Set via ActivityStreamRegistrar to record activity stream events

View File

@@ -54,9 +54,6 @@ def try_load_query_file(artifact_dir) -> Tuple[bool, Optional[dict]]:
returns the contents of ansible_data.json if present
"""
if not flag_enabled("FEATURE_INDIRECT_NODE_COUNTING_ENABLED"):
return False, None
queries_path = os.path.join(artifact_dir, COLLECTION_FILENAME)
if not os.path.isfile(queries_path):
logger.info(f"no query file found: {queries_path}")
@@ -277,20 +274,6 @@ class RunnerCallback:
def artifacts_handler(self, artifact_dir):
success, query_file_contents = try_load_query_file(artifact_dir)
if success:
self.delay_update(event_queries_processed=False)
collections_info = collect_queries(query_file_contents)
for collection, data in collections_info.items():
version = data['version']
event_query = data['host_query']
instance = EventQuery(fqcn=collection, collection_version=version, event_query=event_query)
try:
instance.validate_unique()
instance.save()
logger.info(f"eventy query for collection {collection}, version {version} created")
except ValidationError as e:
logger.info(e)
if 'installed_collections' in query_file_contents:
self.delay_update(installed_collections=query_file_contents['installed_collections'])
else:
@@ -301,6 +284,21 @@ class RunnerCallback:
else:
logger.warning(f'The file {COLLECTION_FILENAME} unexpectedly did not contain ansible_version')
if flag_enabled("FEATURE_INDIRECT_NODE_COUNTING_ENABLED"):
self.delay_update(event_queries_processed=False)
collections_info = collect_queries(query_file_contents)
for collection, data in collections_info.items():
version = data['version']
event_query = data['host_query']
instance = EventQuery(fqcn=collection, collection_version=version, event_query=event_query)
try:
instance.validate_unique()
instance.save()
logger.info(f"event query for collection {collection}, version {version} created")
except ValidationError as e:
logger.info(e)
self.artifacts_processed = True

View File

@@ -25,7 +25,8 @@ def start_fact_cache(hosts, artifacts_dir, timeout=None, inventory_id=None, log_
log_data = log_data or {}
log_data['inventory_id'] = inventory_id
log_data['written_ct'] = 0
hosts_cached = []
# Dict mapping host name -> bool (True if a fact file was written)
hosts_cached = {}
# Create the fact_cache directory inside artifacts_dir
fact_cache_dir = os.path.join(artifacts_dir, 'fact_cache')
@@ -37,13 +38,14 @@ def start_fact_cache(hosts, artifacts_dir, timeout=None, inventory_id=None, log_
last_write_time = None
for host in hosts:
hosts_cached.append(host.name)
if not host.ansible_facts_modified or (timeout and host.ansible_facts_modified < now() - datetime.timedelta(seconds=timeout)):
hosts_cached[host.name] = False
continue # facts are expired - do not write them
filepath = os.path.join(fact_cache_dir, host.name)
if not os.path.realpath(filepath).startswith(fact_cache_dir):
logger.error(f'facts for host {smart_str(host.name)} could not be cached')
hosts_cached[host.name] = False
continue
try:
@@ -51,9 +53,18 @@ def start_fact_cache(hosts, artifacts_dir, timeout=None, inventory_id=None, log_
os.chmod(f.name, 0o600)
json.dump(host.ansible_facts, f)
log_data['written_ct'] += 1
last_write_time = os.path.getmtime(filepath)
# Backdate the file by 2 seconds so finish_fact_cache can reliably
# distinguish these reference files from files updated by ansible.
# This guarantees fact file mtime < summary file mtime even with
# zipfile's 2-second timestamp rounding during artifact transfer.
mtime = os.path.getmtime(filepath)
backdated = mtime - 2
os.utime(filepath, (backdated, backdated))
last_write_time = backdated
hosts_cached[host.name] = True
except IOError:
logger.error(f'facts for host {smart_str(host.name)} could not be cached')
hosts_cached[host.name] = False
continue
# Write summary file directly to the artifacts_dir
@@ -62,7 +73,6 @@ def start_fact_cache(hosts, artifacts_dir, timeout=None, inventory_id=None, log_
summary_data = {
'last_write_time': last_write_time,
'hosts_cached': hosts_cached,
'written_ct': log_data['written_ct'],
}
with open(summary_file, 'w', encoding='utf-8') as f:
json.dump(summary_data, f, indent=2)
@@ -74,7 +84,7 @@ def start_fact_cache(hosts, artifacts_dir, timeout=None, inventory_id=None, log_
msg='Inventory {inventory_id} host facts: updated {updated_ct}, cleared {cleared_ct}, unchanged {unmodified_ct}, took {delta:.3f} s',
add_log_data=True,
)
def finish_fact_cache(artifacts_dir, job_id=None, inventory_id=None, log_data=None):
def finish_fact_cache(host_qs, artifacts_dir, job_id=None, inventory_id=None, job_created=None, log_data=None):
log_data = log_data or {}
log_data['inventory_id'] = inventory_id
log_data['updated_ct'] = 0
@@ -89,63 +99,118 @@ def finish_fact_cache(artifacts_dir, job_id=None, inventory_id=None, log_data=No
try:
with open(summary_path, 'r', encoding='utf-8') as f:
summary = json.load(f)
facts_write_time = os.path.getmtime(summary_path) # After successful read
facts_write_time = os.path.getmtime(summary_path)
except (json.JSONDecodeError, OSError) as e:
logger.error(f'Error reading summary file at {summary_path}: {e}')
return
host_names = summary.get('hosts_cached', [])
hosts_cached = Host.objects.filter(name__in=host_names).order_by('id').iterator()
# Path where individual fact files were written
hosts_cached_map = summary.get('hosts_cached', {})
fact_cache_dir = os.path.join(artifacts_dir, 'fact_cache')
hosts_to_update = []
for host in hosts_cached:
filepath = os.path.join(fact_cache_dir, host.name)
if not os.path.realpath(filepath).startswith(fact_cache_dir):
logger.error(f'Invalid path for facts file: {filepath}')
continue
# Phase 1: Scan files on disk to discover which hosts have updated or missing facts
hosts_with_updates = set() # hostnames whose fact file was modified by Ansible
hosts_to_clear = [] # hostnames where Ansible removed the fact file
seen_in_dir = set() # hostnames we found as files on disk
if os.path.exists(filepath):
# If the file changed since we wrote the last facts file, pre-playbook run...
modified = os.path.getmtime(filepath)
if not facts_write_time or modified >= facts_write_time:
try:
with codecs.open(filepath, 'r', encoding='utf-8') as f:
ansible_facts = json.load(f)
except ValueError:
continue
if os.path.isdir(fact_cache_dir):
for filename in os.listdir(fact_cache_dir):
if filename not in hosts_cached_map:
continue # not an expected host for this job
if ansible_facts != host.ansible_facts:
host.ansible_facts = ansible_facts
host.ansible_facts_modified = now()
hosts_to_update.append(host)
logger.info(
f'New fact for inventory {smart_str(host.inventory.name)} host {smart_str(host.name)}',
extra=dict(
inventory_id=host.inventory.id,
host_name=host.name,
ansible_facts=host.ansible_facts,
ansible_facts_modified=host.ansible_facts_modified.isoformat(),
job_id=job_id,
),
)
log_data['updated_ct'] += 1
else:
log_data['unmodified_ct'] += 1
filepath = os.path.join(fact_cache_dir, filename)
if os.path.islink(filepath):
logger.error(f'Invalid path for facts file: {filepath}')
continue
if not os.path.isfile(filepath):
continue
seen_in_dir.add(filename)
try:
modified = os.path.getmtime(filepath)
except OSError as e:
logger.warning(f'Could not stat facts file {filepath}: {e}')
continue
if modified >= facts_write_time:
hosts_with_updates.add(filename)
else:
log_data['unmodified_ct'] += 1
# Check for files we wrote pre-job that are now missing (Ansible cleared facts)
for hostname, was_written in hosts_cached_map.items():
if hostname in seen_in_dir:
continue # already handled above
if was_written:
hosts_to_clear.append(hostname)
else:
# if the file goes missing, ansible removed it (likely via clear_facts)
# if the file goes missing, but the host has not started facts, then we should not clear the facts
host.ansible_facts = {}
host.ansible_facts_modified = now()
hosts_to_update.append(host)
logger.info(f'Facts cleared for inventory {smart_str(host.inventory.name)} host {smart_str(host.name)}')
log_data['cleared_ct'] += 1
log_data['unmodified_ct'] += 1
if len(hosts_to_update) >= 100:
bulk_update_sorted_by_id(Host, hosts_to_update, fields=['ansible_facts', 'ansible_facts_modified'])
hosts_to_update = []
# Phase 2: Stream updated facts to database in batches
if hosts_with_updates:
hosts_to_save = []
total_rows_updated = 0
for host in host_qs.filter(name__in=list(hosts_with_updates)).select_related('inventory').iterator():
filepath = os.path.join(fact_cache_dir, host.name)
try:
with codecs.open(filepath, 'r', encoding='utf-8') as f:
new_facts = json.load(f)
except (ValueError, OSError):
continue
bulk_update_sorted_by_id(Host, hosts_to_update, fields=['ansible_facts', 'ansible_facts_modified'])
if new_facts != host.ansible_facts:
host.ansible_facts = new_facts
host.ansible_facts_modified = now()
hosts_to_save.append(host)
logger.info(
f'New fact for inventory {smart_str(host.inventory.name)} host {smart_str(host.name)}',
extra=dict(
inventory_id=host.inventory.id,
host_name=host.name,
ansible_facts=host.ansible_facts,
ansible_facts_modified=host.ansible_facts_modified.isoformat(),
job_id=job_id,
),
)
log_data['updated_ct'] += 1
else:
log_data['unmodified_ct'] += 1
if len(hosts_to_save) >= 100:
total_rows_updated += bulk_update_sorted_by_id(Host, hosts_to_save, fields=['ansible_facts', 'ansible_facts_modified'])
hosts_to_save = []
if hosts_to_save:
total_rows_updated += bulk_update_sorted_by_id(Host, hosts_to_save, fields=['ansible_facts', 'ansible_facts_modified'])
# Mismatch means a concurrent process changed or deleted hosts between our read and bulk update
if total_rows_updated != log_data['updated_ct']:
logger.warning(
f'Fact update for inventory {inventory_id} job {job_id}: expected to update {log_data["updated_ct"]} hosts but {total_rows_updated} rows were changed'
)
# Phase 3: Clear facts for hosts whose files were removed by Ansible
if hosts_to_clear:
hosts = list(host_qs.filter(name__in=hosts_to_clear).select_related('inventory'))
clear_hosts = []
for host in hosts:
if job_created and host.ansible_facts_modified and host.ansible_facts_modified > job_created:
logger.warning(
f'Skipping fact clear for host {smart_str(host.name)} in job {job_id} '
f'inventory {inventory_id}: host ansible_facts_modified '
f'({host.ansible_facts_modified.isoformat()}) is after this job\'s '
f'created time ({job_created.isoformat()}). '
f'A concurrent job likely updated this host\'s facts while this job was running.'
)
log_data['unmodified_ct'] += 1
else:
host.ansible_facts = {}
host.ansible_facts_modified = now()
clear_hosts.append(host)
logger.info(f'Facts cleared for inventory {smart_str(host.inventory.name)} host {smart_str(host.name)}')
log_data['cleared_ct'] += 1
if clear_hosts:
rows = bulk_update_sorted_by_id(Host, clear_hosts, fields=['ansible_facts', 'ansible_facts_modified'])
if rows != len(clear_hosts):
logger.warning(f'Fact clear for inventory {inventory_id} job {job_id}: expected to clear {len(clear_hosts)} hosts but {rows} rows were changed')
logger.debug(f'Updated {log_data["updated_ct"]} host facts for inventory {inventory_id} in job {job_id}')

View File

@@ -17,7 +17,6 @@ import urllib.parse as urlparse
# Django
from django.conf import settings
from django.db import transaction
# Shared code for the AWX platform
from awx_plugins.interfaces._temporary_private_container_api import CONTAINER_ROOT, get_incontainer_path
@@ -95,6 +94,7 @@ from flags.state import flag_enabled
# Workload Identity
from ansible_base.lib.workload_identity.controller import AutomationControllerJobScope
from awx.main.utils.workload_identity import retrieve_workload_identity_jwt_with_claims
logger = logging.getLogger('awx.main.tasks.jobs')
@@ -104,11 +104,6 @@ def populate_claims_for_workload(unified_job) -> dict:
Extract JWT claims from a Controller workload for the aap_controller_automation_job scope.
"""
# Related objects in the UnifiedJob model, applies to all job types
organization = getattr_dne(unified_job, 'organization')
ujt = getattr_dne(unified_job, 'unified_job_template')
instance_group = getattr_dne(unified_job, 'instance_group')
claims = {
AutomationControllerJobScope.CLAIM_JOB_ID: unified_job.id,
AutomationControllerJobScope.CLAIM_JOB_NAME: unified_job.name,
@@ -163,6 +158,24 @@ def populate_claims_for_workload(unified_job) -> dict:
return claims
def retrieve_workload_identity_jwt(
unified_job: UnifiedJob,
audience: str,
scope: str,
workload_ttl_seconds: int | None = None,
) -> str:
"""Retrieve JWT token from workload claims.
Raises:
RuntimeError: if the workload identity client is not configured.
"""
return retrieve_workload_identity_jwt_with_claims(
populate_claims_for_workload(unified_job),
audience,
scope,
workload_ttl_seconds,
)
def with_path_cleanup(f):
@functools.wraps(f)
def _wrapped(self, *args, **kwargs):
@@ -189,6 +202,7 @@ def dispatch_waiting_jobs(binder):
if not kwargs:
kwargs = {}
binder.control('run', data={'task': serialize_task(uj._get_task_class()), 'args': [uj.id], 'kwargs': kwargs, 'uuid': uj.celery_task_id})
UnifiedJob.objects.filter(pk=uj.pk, status='waiting').update(status='running', start_args='')
class BaseTask(object):
@@ -203,6 +217,63 @@ class BaseTask(object):
self.update_attempts = int(getattr(settings, 'DISPATCHER_DB_DOWNTOWN_TOLLERANCE', settings.DISPATCHER_DB_DOWNTIME_TOLERANCE) / 5)
self.runner_callback = self.callback_class(model=self.model)
@functools.cached_property
def _credentials(self):
"""
Credentials for the task execution.
Fetches credentials once using build_credentials_list() and stores
them for the duration of the task to avoid redundant database queries.
"""
credentials_list = self.build_credentials_list(self.instance)
# Convert to list to prevent re-evaluation of QuerySet
return list(credentials_list)
def populate_workload_identity_tokens(self, additional_credentials=None):
"""
Populate credentials with workload identity tokens.
Sets the context on Credential objects that have input sources
using compatible external credential types.
"""
credentials = list(self._credentials)
if additional_credentials:
credentials.extend(additional_credentials)
credential_input_sources = (
(credential.context, src)
for credential in credentials
for src in credential.input_sources.all()
if any(
field.get('id') == 'workload_identity_token' and field.get('internal')
for field in src.source_credential.credential_type.inputs.get('fields', [])
)
)
for credential_ctx, input_src in credential_input_sources:
if flag_enabled("FEATURE_OIDC_WORKLOAD_IDENTITY_ENABLED"):
effective_timeout = self.get_instance_timeout(self.instance)
workload_ttl = effective_timeout if effective_timeout else None
try:
jwt = retrieve_workload_identity_jwt(
self.instance,
audience=input_src.source_credential.get_input('url'),
scope=AutomationControllerJobScope.name,
workload_ttl_seconds=workload_ttl,
)
# Store token keyed by input source PK, since a credential can have
# multiple input sources (one per field), each potentially with a different audience
credential_ctx[input_src.pk] = {"workload_identity_token": jwt}
except Exception as e:
self.instance.job_explanation = (
f'Could not generate workload identity token for credential {input_src.source_credential.name} used in this job. Error:\n{e}'
)
self.instance.status = 'error'
self.instance.save()
else:
self.instance.job_explanation = (
f'Flag FEATURE_OIDC_WORKLOAD_IDENTITY_ENABLED is not enabled, required for credential {input_src.source_credential.name} used in this job.'
)
self.instance.status = 'error'
self.instance.save()
def update_model(self, pk, _attempt=0, **updates):
return update_model(self.model, pk, _attempt=0, _max_attempts=self.update_attempts, **updates)
@@ -354,6 +425,19 @@ class BaseTask(object):
private_data_files['credentials'][credential] = self.write_private_data_file(private_data_dir, None, data, sub_dir='env')
for credential, data in private_data.get('certificates', {}).items():
self.write_private_data_file(private_data_dir, 'ssh_key_data-cert.pub', data, sub_dir=os.path.join('artifacts', str(self.instance.id)))
# Copy vendor collections to private_data_dir for indirect node counting
# This makes external query files available to the callback plugin in EEs
if flag_enabled("FEATURE_INDIRECT_NODE_COUNTING_ENABLED"):
vendor_src = '/var/lib/awx/vendor_collections'
vendor_dest = os.path.join(private_data_dir, 'vendor_collections')
if os.path.exists(vendor_src):
try:
shutil.copytree(vendor_src, vendor_dest)
logger.debug(f"Copied vendor collections from {vendor_src} to {vendor_dest}")
except Exception as e:
logger.warning(f"Failed to copy vendor collections: {e}")
return private_data_files, ssh_key_data
def build_passwords(self, instance, runtime_passwords):
@@ -427,6 +511,7 @@ class BaseTask(object):
return []
def get_instance_timeout(self, instance):
"""Return the effective job timeout in seconds."""
global_timeout_setting_name = instance._global_timeout_setting()
if global_timeout_setting_name:
global_timeout = getattr(settings, global_timeout_setting_name, 0)
@@ -535,48 +620,32 @@ class BaseTask(object):
def should_use_fact_cache(self):
return False
def transition_status(self, pk: int) -> bool:
"""Atomically transition status to running, if False returned, another process got it"""
with transaction.atomic():
# Explanation of parts for the fetch:
# .values - avoid loading a full object, this is known to lead to deadlocks due to signals
# the signals load other related rows which another process may be locking, and happens in practice
# of=('self',) - keeps FK tables out of the lock list, another way deadlocks can happen
# .get - just load the single job
instance_data = UnifiedJob.objects.select_for_update(of=('self',)).values('status', 'cancel_flag').get(pk=pk)
# If status is not waiting (obtained under lock) then this process does not have clearence to run
if instance_data['status'] == 'waiting':
if instance_data['cancel_flag']:
updated_status = 'canceled'
else:
updated_status = 'running'
# Explanation of the update:
# .filter - again, do not load the full object
# .update - a bulk update on just that one row, avoid loading unintended data
UnifiedJob.objects.filter(pk=pk).update(status=updated_status, start_args='')
elif instance_data['status'] == 'running':
logger.info(f'Job {pk} is being ran by another process, exiting')
return False
return True
@with_path_cleanup
@with_signal_handling
def run(self, pk, **kwargs):
"""
Run the job/task and capture its output.
"""
if not self.instance: # Used to skip fetch for local runs
if not self.transition_status(pk):
logger.info(f'Job {pk} is being ran by another process, exiting')
return
# Load the instance
self.instance = self.update_model(pk)
if not self.instance: # Used to skip fetch for local runs
# Load the instance
self.instance = self.update_model(pk)
# status should be "running" from dispatch_waiting_jobs,
# but may still be "waiting" if the worker picked this up before the status update landed.
if self.instance.status == 'waiting':
UnifiedJob.objects.filter(pk=pk).update(status="running", start_args='')
self.instance.refresh_from_db()
if self.instance.status != 'running':
logger.error(f'Not starting {self.instance.status} task pk={pk} because its status "{self.instance.status}" is not expected')
return
if self.instance.cancel_flag:
self.instance = self.update_model(pk, status='canceled')
self.instance.websocket_emit_status('canceled')
return
self.instance.websocket_emit_status("running")
status, rc = 'error', None
self.runner_callback.event_ct = 0
@@ -615,6 +684,12 @@ class BaseTask(object):
if not os.path.exists(settings.AWX_ISOLATION_BASE_PATH):
raise RuntimeError('AWX_ISOLATION_BASE_PATH=%s does not exist' % settings.AWX_ISOLATION_BASE_PATH)
if flag_enabled("FEATURE_OIDC_WORKLOAD_IDENTITY_ENABLED"):
logger.info(f'Generating workload identity tokens for {self.instance.log_format}')
self.populate_workload_identity_tokens()
if self.instance.status == 'error':
raise RuntimeError('not starting %s task' % self.instance.status)
# May have to serialize the value
private_data_files, ssh_key_data = self.build_private_data_files(self.instance, private_data_dir)
passwords = self.build_passwords(self.instance, kwargs)
@@ -632,7 +707,7 @@ class BaseTask(object):
self.runner_callback.job_created = str(self.instance.created)
credentials = self.build_credentials_list(self.instance)
credentials = self._credentials
container_root = None
if settings.IS_K8S and isinstance(self.instance, ProjectUpdate):
@@ -810,7 +885,9 @@ class SourceControlMixin(BaseTask):
# Determine whether or not this project sync needs to populate the cache for Ansible content, roles and collections
has_cache = os.path.exists(os.path.join(project.get_cache_path(), project.cache_id))
# Galaxy requirements are not supported for manual projects
if project.scm_type and ((not has_cache) or branch_override):
# If a source update is scheduled, always include roles/collections because
# the new revision may have different requirements.
if project.scm_type and ((not has_cache) or branch_override or source_update_tag in sync_needs):
sync_needs.extend(['install_roles', 'install_collections'])
return sync_needs
@@ -927,6 +1004,29 @@ class RunJob(SourceControlMixin, BaseTask):
model = Job
event_model = JobEvent
def _extract_credentials_of_kind(self, kind: str):
return (cred for cred in self._credentials if cred.credential_type.kind == kind)
@property
def _machine_credential(self) -> object:
"""Get machine credential."""
return next(self._extract_credentials_of_kind('ssh'), None)
@property
def _vault_credentials(self) -> list[object]:
"""Get vault credentials."""
return list(self._extract_credentials_of_kind('vault'))
@property
def _network_credentials(self) -> list[object]:
"""Get network credentials."""
return list(self._extract_credentials_of_kind('net'))
@property
def _cloud_credentials(self) -> list[object]:
"""Get cloud credentials."""
return list(self._extract_credentials_of_kind('cloud'))
def build_private_data(self, job, private_data_dir):
"""
Returns a dict of the form
@@ -944,7 +1044,7 @@ class RunJob(SourceControlMixin, BaseTask):
}
"""
private_data = {'credentials': {}}
for credential in job.credentials.prefetch_related('input_sources__source_credential').all():
for credential in self._credentials:
# If we were sent SSH credentials, decrypt them and send them
# back (they will be written to a temporary file).
if credential.has_input('ssh_key_data'):
@@ -960,14 +1060,14 @@ class RunJob(SourceControlMixin, BaseTask):
and ansible-vault.
"""
passwords = super(RunJob, self).build_passwords(job, runtime_passwords)
cred = job.machine_credential
cred = self._machine_credential
if cred:
for field in ('ssh_key_unlock', 'ssh_password', 'become_password', 'vault_password'):
value = runtime_passwords.get(field, cred.get_input('password' if field == 'ssh_password' else field, default=''))
if value not in ('', 'ASK'):
passwords[field] = value
for cred in job.vault_credentials:
for cred in self._vault_credentials:
field = 'vault_password'
vault_id = cred.get_input('vault_id', default=None)
if vault_id:
@@ -983,7 +1083,7 @@ class RunJob(SourceControlMixin, BaseTask):
key unlock over network key unlock.
'''
if 'ssh_key_unlock' not in passwords:
for cred in job.network_credentials:
for cred in self._network_credentials:
if cred.inputs.get('ssh_key_unlock'):
passwords['ssh_key_unlock'] = runtime_passwords.get('ssh_key_unlock', cred.get_input('ssh_key_unlock', default=''))
break
@@ -1018,11 +1118,11 @@ class RunJob(SourceControlMixin, BaseTask):
# Set environment variables for cloud credentials.
cred_files = private_data_files.get('credentials', {})
for cloud_cred in job.cloud_credentials:
for cloud_cred in self._cloud_credentials:
if cloud_cred and cloud_cred.credential_type.namespace == 'openstack' and cred_files.get(cloud_cred, ''):
env['OS_CLIENT_CONFIG_FILE'] = get_incontainer_path(cred_files.get(cloud_cred, ''), private_data_dir)
for network_cred in job.network_credentials:
for network_cred in self._network_credentials:
env['ANSIBLE_NET_USERNAME'] = network_cred.get_input('username', default='')
env['ANSIBLE_NET_PASSWORD'] = network_cred.get_input('password', default='')
@@ -1040,12 +1140,11 @@ class RunJob(SourceControlMixin, BaseTask):
('ANSIBLE_COLLECTIONS_PATH', 'collections_path', 'requirements_collections', '~/.ansible/collections:/usr/share/ansible/collections'),
]
if flag_enabled("FEATURE_INDIRECT_NODE_COUNTING_ENABLED"):
path_vars.append(
('ANSIBLE_CALLBACK_PLUGINS', 'callback_plugins', 'plugins_path', '~/.ansible/plugins:/plugins/callback:/usr/share/ansible/plugins/callback'),
)
path_vars.append(
('ANSIBLE_CALLBACK_PLUGINS', 'callback_plugins', 'plugins_path', '~/.ansible/plugins:/plugins/callback:/usr/share/ansible/plugins/callback'),
)
config_values = read_ansible_config(os.path.join(private_data_dir, 'project'), list(map(lambda x: x[1], path_vars)))
config_values = read_ansible_config(os.path.join(private_data_dir, 'project'), list(map(lambda x: x[1], path_vars)) + ['callbacks_enabled'])
for env_key, config_setting, folder, default in path_vars:
paths = default.split(':')
@@ -1060,10 +1159,16 @@ class RunJob(SourceControlMixin, BaseTask):
paths = [os.path.join(CONTAINER_ROOT, folder)] + paths
env[env_key] = os.pathsep.join(paths)
env['ANSIBLE_CALLBACKS_ENABLED'] = 'indirect_instance_count'
if 'callbacks_enabled' in config_values:
env['ANSIBLE_CALLBACKS_ENABLED'] += ',' + config_values['callbacks_enabled']
if flag_enabled("FEATURE_INDIRECT_NODE_COUNTING_ENABLED"):
env['ANSIBLE_CALLBACKS_ENABLED'] = 'indirect_instance_count'
if 'callbacks_enabled' in config_values:
env['ANSIBLE_CALLBACKS_ENABLED'] += ':' + config_values['callbacks_enabled']
env['AWX_COLLECT_HOST_QUERIES'] = '1'
# Add vendor collections path for external query file discovery
vendor_collections_path = os.path.join(CONTAINER_ROOT, 'vendor_collections')
env['ANSIBLE_COLLECTIONS_PATH'] = f"{vendor_collections_path}:{env['ANSIBLE_COLLECTIONS_PATH']}"
logger.debug(f"ANSIBLE_COLLECTIONS_PATH updated for vendor collections: {env['ANSIBLE_COLLECTIONS_PATH']}")
return env
@@ -1072,7 +1177,7 @@ class RunJob(SourceControlMixin, BaseTask):
Build command line argument list for running ansible-playbook,
optionally using ssh-agent for public/private key authentication.
"""
creds = job.machine_credential
creds = self._machine_credential
ssh_username, become_username, become_method = '', '', ''
if creds:
@@ -1224,10 +1329,17 @@ class RunJob(SourceControlMixin, BaseTask):
return
if self.should_use_fact_cache() and self.runner_callback.artifacts_processed:
job.log_lifecycle("finish_job_fact_cache")
if job.inventory.kind == 'constructed':
hosts_qs = job.get_source_hosts_for_constructed_inventory()
else:
hosts_qs = job.inventory.hosts
hosts_qs = hosts_qs.only(*HOST_FACTS_FIELDS)
finish_fact_cache(
hosts_qs,
artifacts_dir=os.path.join(private_data_dir, 'artifacts', str(job.id)),
job_id=job.id,
inventory_id=job.inventory_id,
job_created=job.created,
)
def final_run_hook(self, job, status, private_data_dir):
@@ -1502,16 +1614,14 @@ class RunProjectUpdate(BaseTask):
shutil.copytree(cache_subpath, dest_subpath, symlinks=True)
logger.debug('{0} {1} prepared {2} from cache'.format(type(project).__name__, project.pk, dest_subpath))
if flag_enabled("FEATURE_INDIRECT_NODE_COUNTING_ENABLED"):
# copy the special callback (not stdout type) plugin to get list of collections
pdd_plugins_path = os.path.join(job_private_data_dir, 'plugins_path')
if not os.path.exists(pdd_plugins_path):
os.mkdir(pdd_plugins_path)
from awx.playbooks import library
pdd_plugins_path = os.path.join(job_private_data_dir, 'plugins_path')
if not os.path.exists(pdd_plugins_path):
os.mkdir(pdd_plugins_path)
from awx.playbooks import library
plugin_file_source = os.path.join(library.__path__._path[0], 'indirect_instance_count.py')
plugin_file_dest = os.path.join(pdd_plugins_path, 'indirect_instance_count.py')
shutil.copyfile(plugin_file_source, plugin_file_dest)
plugin_file_source = os.path.join(library.__path__[0], 'indirect_instance_count.py')
plugin_file_dest = os.path.join(pdd_plugins_path, 'indirect_instance_count.py')
shutil.copyfile(plugin_file_source, plugin_file_dest)
def post_run_hook(self, instance, status):
super(RunProjectUpdate, self).post_run_hook(instance, status)
@@ -1574,7 +1684,7 @@ class RunProjectUpdate(BaseTask):
return params
def build_credentials_list(self, project_update):
if project_update.scm_type == 'insights' and project_update.credential:
if project_update.credential:
return [project_update.credential]
return []
@@ -1757,6 +1867,24 @@ class RunInventoryUpdate(SourceControlMixin, BaseTask):
# All credentials not used by inventory source injector
return inventory_update.get_extra_credentials()
def populate_workload_identity_tokens(self, additional_credentials=None):
"""Also generate OIDC tokens for the cloud credential.
The cloud credential is not in _credentials (it is handled by the
inventory source injector), but it may still need a workload identity
token generated for it.
"""
cloud_cred = self.instance.get_cloud_credential()
creds = list(additional_credentials or [])
if cloud_cred:
creds.append(cloud_cred)
super().populate_workload_identity_tokens(additional_credentials=creds or None)
# Override get_cloud_credential on this instance so the injector
# uses the credential with OIDC context instead of doing a fresh
# DB fetch that would lose it.
if cloud_cred and cloud_cred.context:
self.instance.get_cloud_credential = lambda: cloud_cred
def build_project_dir(self, inventory_update, private_data_dir):
source_project = None
if inventory_update.inventory_source:

View File

@@ -393,9 +393,9 @@ def evaluate_policy(instance):
raise PolicyEvaluationError(_('Following certificate settings are missing for OPA_AUTH_TYPE=Certificate: {}').format(cert_settings_missing))
query_paths = [
('Organization', instance.organization.opa_query_path),
('Inventory', instance.inventory.opa_query_path),
('Job template', instance.job_template.opa_query_path),
('Organization', instance.organization.opa_query_path if instance.organization else None),
('Inventory', instance.inventory.opa_query_path if instance.inventory else None),
('Job template', instance.job_template.opa_query_path if instance.job_template else None),
]
violations = dict()
errors = dict()

View File

@@ -19,6 +19,7 @@ from dispatcherd.publish import task
# Runner
import ansible_runner.cleanup
import psycopg
from ansible_base.lib.cache.tasks import clear_cache as dab_clear_cache
from ansible_base.lib.utils.db import advisory_lock
# django-ansible-base
@@ -68,10 +69,12 @@ from awx.main.models import (
UnifiedJob,
convert_jsonfields,
)
from awx.main.models.credential import CredentialType
from awx.main.tasks.helpers import is_run_threshold_reached
from awx.main.tasks.host_indirect import save_indirect_host_entries
from awx.main.tasks.receptor import administrative_workunit_reaper, get_receptor_ctl, worker_cleanup, worker_info, write_receptor_config
from awx.main.utils.common import ignore_inventory_computed_fields, ignore_inventory_group_removal
from awx.main.utils.migration import is_database_synchronized
from awx.main.utils.reload import stop_local_services
logger = logging.getLogger('awx.main.tasks.system')
@@ -83,6 +86,16 @@ Try upgrading OpenSSH or providing your private key in an different format. \
'''
def _sync_credential_types_to_db():
"""Ensure CredentialType DB rows match the installed plugins.
The in-memory registry is populated lazily on first access via LazyLoadDict.
This function only handles the DB sync step.
"""
if is_database_synchronized():
CredentialType.setup_tower_managed_defaults()
def _run_dispatch_startup_common():
"""
Execute the common startup initialization steps.
@@ -98,6 +111,11 @@ def _run_dispatch_startup_common():
except Exception:
logger.exception("Failed to write receptor config, skipping.")
try:
_sync_credential_types_to_db()
except Exception:
logger.exception("Failed to sync credential types to DB, skipping.")
try:
convert_jsonfields()
except Exception:
@@ -240,12 +258,17 @@ def apply_cluster_membership_policies():
# Process policy instance list first, these will represent manually managed memberships
instance_hostnames_map = {inst.hostname: inst for inst in all_instances}
for ig in all_groups:
# we don't want to allow execution nodes in the control plane
exclude_type = 'execution' if ig.name == settings.DEFAULT_CONTROL_PLANE_QUEUE_NAME else 'control'
group_actual = Group(obj=ig, instances=[], prior_instances=[instance.pk for instance in ig.instances.all()]) # obtained in prefetch
for hostname in ig.policy_instance_list:
if hostname not in instance_hostnames_map:
logger.info("Unknown instance {} in {} policy list".format(hostname, ig.name))
continue
inst = instance_hostnames_map[hostname]
if inst.node_type == exclude_type:
logger.info("Instance {} is excluded in {} policy list".format(hostname, ig.name))
continue
group_actual.instances.append(inst.id)
# NOTE: arguable behavior: policy-list-group is not added to
# instance's group count for consideration in minimum-policy rules
@@ -326,24 +349,22 @@ def apply_cluster_membership_policies():
logger.debug('Cluster policy computation finished in {} seconds'.format(time.time() - started_compute))
@task(queue='tower_settings_change', timeout=600)
def clear_setting_cache(setting_keys):
# log that cache is being cleared
logger.info(f"clear_setting_cache of keys {setting_keys}")
orig_len = len(setting_keys)
for i in range(orig_len):
for dependent_key in settings_registry.get_dependent_settings(setting_keys[i]):
setting_keys.append(dependent_key)
cache_keys = set(setting_keys)
logger.debug('cache delete_many(%r)', cache_keys)
cache.delete_many(cache_keys)
def _resolve_setting_dependents(key):
return settings_registry.get_dependent_settings(key)
if 'LOG_AGGREGATOR_LEVEL' in setting_keys:
def _post_setting_invalidation(invalidated_keys):
if 'LOG_AGGREGATOR_LEVEL' in invalidated_keys:
ctl = get_control_from_settings()
ctl.queuename = get_task_queuename()
ctl.control('set_log_level', data={'level': settings.LOG_AGGREGATOR_LEVEL})
@task(queue='tower_settings_change', timeout=600)
def clear_setting_cache(setting_keys):
dab_clear_cache(setting_keys, _resolve_setting_dependents, _post_setting_invalidation)
@task(queue='tower_broadcast_all', timeout=600)
def delete_project_files(project_path):
# TODO: possibly implement some retry logic
@@ -612,7 +633,7 @@ def inspect_execution_and_hop_nodes(instance_list):
# check
logger.warning(f'Execution node attempting to rejoin as instance {hostname}.')
execution_node_health_check.apply_async([hostname])
elif instance.capacity == 0 and instance.enabled:
elif (instance.capacity == 0 or (instance.cpu == 0 and instance.memory == 0)) and instance.enabled:
# nodes with proven connection but need remediation run health checks are reduced frequency
if not instance.last_health_check or (nowtime - instance.last_health_check).total_seconds() >= settings.EXECUTION_NODE_REMEDIATION_CHECKS:
# Periodically re-run the health check of errored nodes, in case someone fixed it
@@ -1001,6 +1022,34 @@ def update_host_smart_inventory_memberships():
smart_inventory.update_computed_fields()
def _batched_delete_inventory(inventory, batch_size=500):
"""Delete inventory hosts in batches to avoid high memory usage.
With ansible facts, loading thousands of hosts at once can use a lot of memory. To avoid
this, we delete them in batches (of 500).
Safe to retry after a crash because inventory.pending_deletion
is already set and each batch is its own transaction.
"""
from awx.main.models.inventory import Host
# first delete all hosts in batches
total_deleted = 0
while True:
pks = list(Host.objects.filter(inventory_id=inventory.id).values_list('pk', flat=True)[:batch_size])
if not pks:
break
with transaction.atomic():
deleted_count, _ = Host.objects.filter(pk__in=pks).delete()
total_deleted += deleted_count
logger.debug('Batch-deleted %d hosts from inventory %d (%d total so far)', len(pks), inventory.id, total_deleted)
# then delete the inventory itself
inv_id = inventory.id
inventory.delete()
logger.info('Batched deletion of inventory %d complete (%d hosts removed)', inv_id, total_deleted)
@task(queue=get_task_queuename, timeout=3600 * 5)
def delete_inventory(inventory_id, user_id, retries=5):
# Delete inventory as user
@@ -1013,11 +1062,12 @@ def delete_inventory(inventory_id, user_id, retries=5):
user = None
with ignore_inventory_computed_fields(), ignore_inventory_group_removal(), impersonate(user):
try:
Inventory.objects.get(id=inventory_id).delete()
inv = Inventory.objects.get(id=inventory_id)
_batched_delete_inventory(inv)
emit_channel_notification('inventories-status_changed', {'group_name': 'inventories', 'inventory_id': inventory_id, 'status': 'deleted'})
logger.debug('Deleted inventory {} as user {}.'.format(inventory_id, user_id))
except Inventory.DoesNotExist:
logger.exception("Delete Inventory failed due to missing inventory: " + str(inventory_id))
logger.warning("Delete Inventory failed due to missing inventory: " + str(inventory_id))
return
except DatabaseError:
logger.exception('Database error deleting inventory {}, but will retry.'.format(inventory_id))

View File

@@ -0,0 +1,19 @@
---
authors:
- AWX Project Contributors <awx-project@googlegroups.com>
dependencies: {}
description: External query testing collection. No embedded query file. Not for use in production.
documentation: https://github.com/ansible/awx
homepage: https://github.com/ansible/awx
issues: https://github.com/ansible/awx
license:
- GPL-3.0-or-later
name: external
namespace: demo
readme: README.md
repository: https://github.com/ansible/awx
tags:
- demo
- testing
- external_query
version: 1.0.0

View File

@@ -0,0 +1,78 @@
#!/usr/bin/python
# Same licensing as AWX
from __future__ import absolute_import, division, print_function
__metaclass__ = type
DOCUMENTATION = r'''
---
module: example
short_description: Module for specific live tests
version_added: "2.0.0"
description: This module is part of a test collection in local source. Used for external query testing.
options:
host_name:
description: Name to return as the host name.
required: false
type: str
author:
- AWX Live Tests
'''
EXAMPLES = r'''
- name: Test with defaults
demo.external.example:
- name: Test with custom host name
demo.external.example:
host_name: foo_host
'''
RETURN = r'''
direct_host_name:
description: The name of the host, this will be collected with the feature.
type: str
returned: always
sample: 'foo_host'
'''
from ansible.module_utils.basic import AnsibleModule
def run_module():
module_args = dict(
host_name=dict(type='str', required=False, default='foo_host_default'),
)
result = dict(
changed=False,
other_data='sample_string',
)
module = AnsibleModule(argument_spec=module_args, supports_check_mode=True)
if module.check_mode:
module.exit_json(**result)
result['direct_host_name'] = module.params['host_name']
result['nested_host_name'] = {'host_name': module.params['host_name']}
result['name'] = 'vm-foo'
# non-cononical facts
result['device_type'] = 'Fake Host'
module.exit_json(**result)
def main():
run_module()
if __name__ == '__main__':
main()

View File

@@ -0,0 +1,19 @@
---
authors:
- AWX Project Contributors <awx-project@googlegroups.com>
dependencies: {}
description: External query testing collection v1.5.0. No embedded query file. Not for use in production.
documentation: https://github.com/ansible/awx
homepage: https://github.com/ansible/awx
issues: https://github.com/ansible/awx
license:
- GPL-3.0-or-later
name: external
namespace: demo
readme: README.md
repository: https://github.com/ansible/awx
tags:
- demo
- testing
- external_query
version: 1.5.0

View File

@@ -0,0 +1,78 @@
#!/usr/bin/python
# Same licensing as AWX
from __future__ import absolute_import, division, print_function
__metaclass__ = type
DOCUMENTATION = r'''
---
module: example
short_description: Module for specific live tests
version_added: "2.0.0"
description: This module is part of a test collection in local source. Used for external query testing.
options:
host_name:
description: Name to return as the host name.
required: false
type: str
author:
- AWX Live Tests
'''
EXAMPLES = r'''
- name: Test with defaults
demo.external.example:
- name: Test with custom host name
demo.external.example:
host_name: foo_host
'''
RETURN = r'''
direct_host_name:
description: The name of the host, this will be collected with the feature.
type: str
returned: always
sample: 'foo_host'
'''
from ansible.module_utils.basic import AnsibleModule
def run_module():
module_args = dict(
host_name=dict(type='str', required=False, default='foo_host_default'),
)
result = dict(
changed=False,
other_data='sample_string',
)
module = AnsibleModule(argument_spec=module_args, supports_check_mode=True)
if module.check_mode:
module.exit_json(**result)
result['direct_host_name'] = module.params['host_name']
result['nested_host_name'] = {'host_name': module.params['host_name']}
result['name'] = 'vm-foo'
# non-cononical facts
result['device_type'] = 'Fake Host'
module.exit_json(**result)
def main():
run_module()
if __name__ == '__main__':
main()

View File

@@ -0,0 +1,19 @@
---
authors:
- AWX Project Contributors <awx-project@googlegroups.com>
dependencies: {}
description: External query testing collection v3.0.0. No embedded query file. Not for use in production.
documentation: https://github.com/ansible/awx
homepage: https://github.com/ansible/awx
issues: https://github.com/ansible/awx
license:
- GPL-3.0-or-later
name: external
namespace: demo
readme: README.md
repository: https://github.com/ansible/awx
tags:
- demo
- testing
- external_query
version: 3.0.0

View File

@@ -0,0 +1,78 @@
#!/usr/bin/python
# Same licensing as AWX
from __future__ import absolute_import, division, print_function
__metaclass__ = type
DOCUMENTATION = r'''
---
module: example
short_description: Module for specific live tests
version_added: "2.0.0"
description: This module is part of a test collection in local source. Used for external query testing.
options:
host_name:
description: Name to return as the host name.
required: false
type: str
author:
- AWX Live Tests
'''
EXAMPLES = r'''
- name: Test with defaults
demo.external.example:
- name: Test with custom host name
demo.external.example:
host_name: foo_host
'''
RETURN = r'''
direct_host_name:
description: The name of the host, this will be collected with the feature.
type: str
returned: always
sample: 'foo_host'
'''
from ansible.module_utils.basic import AnsibleModule
def run_module():
module_args = dict(
host_name=dict(type='str', required=False, default='foo_host_default'),
)
result = dict(
changed=False,
other_data='sample_string',
)
module = AnsibleModule(argument_spec=module_args, supports_check_mode=True)
if module.check_mode:
module.exit_json(**result)
result['direct_host_name'] = module.params['host_name']
result['nested_host_name'] = {'host_name': module.params['host_name']}
result['name'] = 'vm-foo'
# non-cononical facts
result['device_type'] = 'Fake Host'
module.exit_json(**result)
def main():
run_module()
if __name__ == '__main__':
main()

View File

@@ -0,0 +1,11 @@
---
- hosts: all
gather_facts: false
connection: local
tasks:
- name: Set artifacts via set_stats
ansible.builtin.set_stats:
data: "{{ stats_data }}"
per_host: false
aggregate: false
when: stats_data is defined

View File

@@ -0,0 +1,21 @@
---
# Generated by Claude Opus 4.6 (claude-opus-4-6).
- hosts: all
vars:
extra_value: ""
gather_facts: false
connection: local
tasks:
- name: set a custom fact
set_fact:
foo: "bar{{ extra_value }}"
bar:
a:
b:
- "c"
- "d"
cacheable: true
- name: sleep to create overlap window for concurrent job testing
wait_for:
timeout: 2

View File

@@ -0,0 +1,5 @@
---
collections:
- name: 'file:///tmp/live_tests/host_query_external_v1_0_0'
type: git
version: devel

View File

@@ -0,0 +1,8 @@
---
- hosts: all
gather_facts: false
connection: local
tasks:
- demo.external.example:
register: result
- debug: var=result

View File

@@ -0,0 +1,5 @@
---
collections:
- name: 'file:///tmp/live_tests/host_query_external_v1_5_0'
type: git
version: devel

View File

@@ -0,0 +1,8 @@
---
- hosts: all
gather_facts: false
connection: local
tasks:
- demo.external.example:
register: result
- debug: var=result

View File

@@ -0,0 +1,5 @@
---
collections:
- name: 'file:///tmp/live_tests/host_query_external_v3_0_0'
type: git
version: devel

View File

@@ -0,0 +1,8 @@
---
- hosts: all
gather_facts: false
connection: local
tasks:
- demo.external.example:
register: result
- debug: var=result

View File

@@ -74,9 +74,9 @@ def temp_analytic_tar():
@pytest.fixture
def mock_analytic_post():
# Patch the Session.post method to return a mock response with status_code 200
with mock.patch('awx.main.analytics.core.requests.Session.post', return_value=mock.Mock(status_code=200)) as mock_post:
yield mock_post
# Patch get_or_generate_candlepin_certificate to skip mTLS path
with mock.patch('awx.main.analytics.core.get_or_generate_candlepin_certificate', return_value=(None, None)):
yield
@pytest.mark.parametrize(
@@ -141,15 +141,22 @@ def mock_analytic_post():
)
@pytest.mark.django_db
def test_ship_credential(setting_map, expected_result, expected_auth, temp_analytic_tar, mock_analytic_post):
with override_settings(**setting_map):
result = ship(temp_analytic_tar)
with override_settings(**setting_map, AUTOMATION_ANALYTICS_URL='https://example.com/api'):
with mock.patch('awx.main.analytics.core.OIDCClient') as mock_oidc:
mock_oidc_instance = mock.Mock()
mock_oidc_instance.make_request.return_value = mock.Mock(status_code=200)
mock_oidc.return_value = mock_oidc_instance
assert result == expected_result
if expected_auth:
mock_analytic_post.assert_called_once()
assert mock_analytic_post.call_args[1]['auth'] == expected_auth
else:
mock_analytic_post.assert_not_called()
result = ship(temp_analytic_tar)
assert result == expected_result
if expected_auth:
# Verify OIDC client was instantiated with correct credentials
mock_oidc.assert_called_once_with(expected_auth[0], expected_auth[1])
mock_oidc_instance.make_request.assert_called_once()
else:
# When credentials are missing, OIDCClient should not be called
mock_oidc.assert_not_called()
@pytest.mark.django_db

View File

@@ -139,6 +139,29 @@ def test_stream_queryset_hides_shows_items(
assert access.can_read(activity_stream_entry)
@pytest.mark.django_db
def test_activity_stream_pagination_uses_unfiltered_count(get, organization, project, user, settings):
"""The pagination count should reflect total activity stream rows, not
the RBAC-filtered subset. The RBAC-filtered COUNT is catastrophically
slow on large tables (AAP-83773); an approximate over-count from an
unfiltered SELECT COUNT(*) is acceptable for pagination UI."""
settings.ACTIVITY_STREAM_ENABLED = True
no_access_user = user('no-access-user', False)
total_entries = ActivityStream.objects.count()
assert total_entries > 0
url = reverse('api:activity_stream_list')
response = get(url, no_access_user)
assert response.status_code == 200
visible_results = len(response.data['results'])
pagination_count = response.data['count']
assert pagination_count == total_entries
assert visible_results < pagination_count
@pytest.mark.django_db
def test_stream_user_direct_role_updates(get, post, organization_factory):
objects = organization_factory('test_org', superusers=['admin'], users=['test'], inventories=['inv1'])

View File

@@ -0,0 +1,84 @@
import pytest
from awx.api.versioning import reverse
from rest_framework import status
from awx.main.models.jobs import JobTemplate
@pytest.mark.django_db
class TestConfigEndpointFields:
def test_base_fields_all_users(self, get, rando):
url = reverse('api:api_v2_config_view')
response = get(url, rando, expect=200)
assert 'time_zone' in response.data
assert 'license_info' in response.data
assert 'version' in response.data
assert 'eula' in response.data
assert 'analytics_status' in response.data
assert 'analytics_collectors' in response.data
assert 'become_methods' in response.data
@pytest.mark.parametrize(
"role_type",
[
"superuser",
"system_auditor",
"org_admin",
"org_auditor",
"org_project_admin",
],
)
def test_privileged_users_conditional_fields(self, get, user, organization, admin, role_type):
url = reverse('api:api_v2_config_view')
if role_type == "superuser":
test_user = admin
elif role_type == "system_auditor":
test_user = user('system-auditor', is_superuser=False)
test_user.is_system_auditor = True
test_user.save()
elif role_type == "org_admin":
test_user = user('org-admin', is_superuser=False)
organization.admin_role.members.add(test_user)
elif role_type == "org_auditor":
test_user = user('org-auditor', is_superuser=False)
organization.auditor_role.members.add(test_user)
elif role_type == "org_project_admin":
test_user = user('org-project-admin', is_superuser=False)
organization.project_admin_role.members.add(test_user)
response = get(url, test_user, expect=200)
assert 'project_base_dir' in response.data
assert 'project_local_paths' in response.data
assert 'custom_virtualenvs' in response.data
def test_job_template_admin_gets_venvs_only(self, get, user, organization, project, inventory):
"""Test that JobTemplate admin without org access gets only custom_virtualenvs"""
jt_admin = user('jt-admin', is_superuser=False)
jt = JobTemplate.objects.create(name='test-jt', organization=organization, project=project, inventory=inventory)
jt.admin_role.members.add(jt_admin)
url = reverse('api:api_v2_config_view')
response = get(url, jt_admin, expect=200)
assert 'custom_virtualenvs' in response.data
assert 'project_base_dir' not in response.data
assert 'project_local_paths' not in response.data
def test_normal_user_no_conditional_fields(self, get, rando):
url = reverse('api:api_v2_config_view')
response = get(url, rando, expect=200)
assert 'project_base_dir' not in response.data
assert 'project_local_paths' not in response.data
assert 'custom_virtualenvs' not in response.data
def test_unauthenticated_denied(self, get):
"""Test that unauthenticated requests are denied"""
url = reverse('api:api_v2_config_view')
response = get(url, None, expect=401)
assert response.status_code == status.HTTP_401_UNAUTHORIZED

View File

@@ -200,6 +200,7 @@ def test_grant_org_credential_to_org_user_through_user_roles(post, credential, o
@pytest.mark.django_db
def test_grant_org_credential_to_non_org_user_through_role_users(post, credential, organization, org_admin, alice):
# NOTE: this endpoint is going away soon
credential.organization = organization
credential.save()
response = post(reverse('api:role_users_list', kwargs={'pk': credential.use_role.id}), {'id': alice.id}, org_admin)
@@ -208,6 +209,7 @@ def test_grant_org_credential_to_non_org_user_through_role_users(post, credentia
@pytest.mark.django_db
def test_grant_org_credential_to_non_org_user_through_user_roles(post, credential, organization, org_admin, alice):
# NOTE: this endpoint is going away soon
credential.organization = organization
credential.save()
response = post(reverse('api:user_roles_list', kwargs={'pk': alice.id}), {'id': credential.use_role.id}, org_admin)
@@ -216,18 +218,18 @@ def test_grant_org_credential_to_non_org_user_through_user_roles(post, credentia
@pytest.mark.django_db
def test_grant_private_credential_to_user_through_role_users(post, credential, alice, bob):
# normal users can't do this
# NOTE: this endpoint is going away soon
credential.admin_role.members.add(alice)
response = post(reverse('api:role_users_list', kwargs={'pk': credential.use_role.id}), {'id': bob.id}, alice)
assert response.status_code == 400
assert response.status_code == 403
@pytest.mark.django_db
def test_grant_private_credential_to_org_user_through_role_users(post, credential, org_admin, org_member):
# org admins can't either
# NOTE: this endpoint is going away soon
credential.admin_role.members.add(org_admin)
response = post(reverse('api:role_users_list', kwargs={'pk': credential.use_role.id}), {'id': org_member.id}, org_admin)
assert response.status_code == 400
assert response.status_code == 204
@pytest.mark.django_db
@@ -239,18 +241,18 @@ def test_sa_grant_private_credential_to_user_through_role_users(post, credential
@pytest.mark.django_db
def test_grant_private_credential_to_user_through_user_roles(post, credential, alice, bob):
# normal users can't do this
# NOTE: this endpoint is going away soon
credential.admin_role.members.add(alice)
response = post(reverse('api:user_roles_list', kwargs={'pk': bob.id}), {'id': credential.use_role.id}, alice)
assert response.status_code == 400
assert response.status_code == 403
@pytest.mark.django_db
def test_grant_private_credential_to_org_user_through_user_roles(post, credential, org_admin, org_member):
# org admins can't either
# NOTE: this endpoint is going away soon
credential.admin_role.members.add(org_admin)
response = post(reverse('api:user_roles_list', kwargs={'pk': org_member.id}), {'id': credential.use_role.id}, org_admin)
assert response.status_code == 400
assert response.status_code == 204
@pytest.mark.django_db
@@ -282,14 +284,14 @@ def test_grant_org_credential_to_team_through_team_roles(post, credential, organ
@pytest.mark.django_db
def test_sa_grant_private_credential_to_team_through_role_teams(post, credential, admin, team):
# not even a system admin can grant a private cred to a team though
# NOTE: this endpoint is going away soon
response = post(reverse('api:role_teams_list', kwargs={'pk': credential.use_role.id}), {'id': team.id}, admin)
assert response.status_code == 400
assert response.status_code == 204
@pytest.mark.django_db
def test_grant_credential_to_team_different_organization_through_role_teams(post, get, credential, organizations, admin, org_admin, team, team_member):
# # Test that credential from different org can be assigned to team by a superuser through role_teams_list endpoint
# NOTE: this endpoint is going away soon
orgs = organizations(2)
credential.organization = orgs[0]
credential.save()
@@ -299,10 +301,7 @@ def test_grant_credential_to_team_different_organization_through_role_teams(post
# Non-superuser (org_admin) trying cross-org assignment should be denied
response = post(reverse('api:role_teams_list', kwargs={'pk': credential.use_role.id}), {'id': team.id}, org_admin)
assert response.status_code == 400
assert (
"You cannot grant a team access to a credential in a different organization. Only superusers can grant cross-organization credential access to teams"
in response.data['msg']
)
assert "You cannot grant credential access to a Team not in the credentials' organization" in str(response.data['detail'])
# Superuser (admin) can do cross-org assignment
response = post(reverse('api:role_teams_list', kwargs={'pk': credential.use_role.id}), {'id': team.id}, admin)
@@ -316,20 +315,17 @@ def test_grant_credential_to_team_different_organization_through_role_teams(post
@pytest.mark.django_db
def test_grant_credential_to_team_different_organization(post, get, credential, organizations, admin, org_admin, team, team_member):
# Test that credential from different org can be assigned to team by a superuser
# NOTE: this endpoint is going away soon
orgs = organizations(2)
credential.organization = orgs[0]
credential.save()
team.organization = orgs[1]
team.save()
# Non-superuser (org_admin, ...) trying cross-org assignment should be denied
# Non-superuser (org_admin) trying cross-org assignment should be denied
response = post(reverse('api:team_roles_list', kwargs={'pk': team.id}), {'id': credential.use_role.id}, org_admin)
assert response.status_code == 400
assert (
"You cannot grant a team access to a credential in a different organization. Only superusers can grant cross-organization credential access to teams"
in response.data['msg']
)
assert "You cannot grant credential access to a Team not in the credentials' organization" in str(response.data['detail'])
# Superuser (system admin) can do cross-org assignment
response = post(reverse('api:team_roles_list', kwargs={'pk': team.id}), {'id': credential.use_role.id}, admin)

View File

@@ -1,5 +1,7 @@
import pytest
from ansible_base.lib.testing.util import feature_flag_enabled, feature_flag_disabled
from awx.main.models import CredentialInputSource
from awx.api.versioning import reverse
@@ -316,3 +318,60 @@ def test_create_credential_input_source_with_already_used_input_returns_400(post
]
all_responses = [post(list_url, params, admin) for params in all_params]
assert all_responses.pop().status_code == 400
@pytest.mark.django_db
def test_credential_input_source_passes_workload_identity_token_when_flag_enabled(vault_credential, external_credential, mocker):
"""Test that workload_identity_token is passed to backend when flag is enabled."""
with feature_flag_enabled('FEATURE_OIDC_WORKLOAD_IDENTITY_ENABLED'):
# Add workload_identity_token as an internal field on the external credential type
# so get_input_value resolves it from the per-input-source context
external_credential.credential_type.inputs['fields'].append(
{'id': 'workload_identity_token', 'label': 'Workload Identity Token', 'type': 'string', 'internal': True}
)
# Create an input source
input_source = CredentialInputSource.objects.create(
target_credential=vault_credential,
source_credential=external_credential,
input_field_name='vault_password',
metadata={'key': 'test_key'},
)
# Mock the credential plugin backend
mock_backend = mocker.patch.object(external_credential.credential_type.plugin, 'backend', autospec=True, return_value='test_value')
# Call with context keyed by input source PK
test_context = {input_source.pk: {'workload_identity_token': 'jwt_token_here'}}
result = input_source.get_input_value(context=test_context)
# Verify backend was called with workload_identity_token
assert result == 'test_value'
call_kwargs = mock_backend.call_args[1]
assert call_kwargs['workload_identity_token'] == 'jwt_token_here'
assert call_kwargs['key'] == 'test_key'
@pytest.mark.django_db
def test_credential_input_source_skips_workload_identity_token_when_flag_disabled(vault_credential, external_credential, mocker):
"""Test that workload_identity_token is NOT passed when flag is disabled."""
with feature_flag_disabled('FEATURE_OIDC_WORKLOAD_IDENTITY_ENABLED'):
# Create an input source
input_source = CredentialInputSource.objects.create(
target_credential=vault_credential,
source_credential=external_credential,
input_field_name='vault_password',
metadata={'key': 'test_key'},
)
# Mock the credential plugin backend
mock_backend = mocker.patch.object(external_credential.credential_type.plugin, 'backend', autospec=True, return_value='test_value')
# Call with context containing workload_identity_token but NO internal field defined,
# simulating a flag-disabled scenario where tokens are not generated upstream
test_context = {input_source.pk: {'workload_identity_token': 'jwt_token_here'}}
result = input_source.get_input_value(context=test_context)
# Verify backend was called WITHOUT workload_identity_token since the credential type
# does not define it as an internal field (flag-disabled path doesn't register it)
assert result == 'test_value'
call_kwargs = mock_backend.call_args[1]
assert 'workload_identity_token' not in call_kwargs
assert call_kwargs['key'] == 'test_key'

View File

@@ -2,6 +2,7 @@ import json
import pytest
from ansible_base.lib.testing.util import feature_flag_enabled
from awx.main.models.credential import CredentialType, Credential
from awx.api.versioning import reverse
@@ -159,7 +160,8 @@ def test_create_as_admin(get, post, admin):
response = get(reverse('api:credential_type_list'), admin)
assert response.data['count'] == 1
assert response.data['results'][0]['name'] == 'Custom Credential Type'
assert response.data['results'][0]['inputs'] == {}
# Serializer normalizes empty inputs to {'fields': []}
assert response.data['results'][0]['inputs'] == {'fields': []}
assert response.data['results'][0]['injectors'] == {}
assert response.data['results'][0]['managed'] is False
@@ -474,3 +476,98 @@ def test_credential_type_rbac_external_test(post, alice, admin, credentialtype_e
data = {'inputs': {}, 'metadata': {}}
assert post(url, data, admin).status_code == 202
assert post(url, data, alice).status_code == 403
# --- Tests for internal field filtering with None/invalid inputs ---
@pytest.mark.django_db
def test_credential_type_with_none_inputs(get, admin):
"""Test that credential type with empty inputs dict works correctly."""
# Create a credential type with empty dict
ct = CredentialType.objects.create(
kind='cloud',
name='Test Type',
managed=False,
inputs={}, # Empty dict, not None (DB has NOT NULL constraint)
)
url = reverse('api:credential_type_detail', kwargs={'pk': ct.pk})
response = get(url, admin)
assert response.status_code == 200
# Should have normalized inputs to empty dict
assert 'inputs' in response.data
assert isinstance(response.data['inputs'], dict)
assert response.data['inputs']['fields'] == []
@pytest.mark.django_db
def test_credential_type_with_invalid_inputs_type(get, admin):
"""Test that credential type with non-dict inputs doesn't cause errors."""
# Create a credential type with invalid inputs type
ct = CredentialType.objects.create(kind='cloud', name='Test Type', managed=False, inputs={'fields': 'not-a-list'})
url = reverse('api:credential_type_detail', kwargs={'pk': ct.pk})
response = get(url, admin)
assert response.status_code == 200
# Should gracefully handle invalid fields type
assert 'inputs' in response.data
assert response.data['inputs']['fields'] == []
@pytest.mark.django_db
def test_credential_type_filters_internal_fields(get, admin):
"""Test that internal fields are filtered from API responses."""
ct = CredentialType.objects.create(
kind='cloud',
name='Test OIDC Type',
managed=False,
inputs={
'fields': [
{'id': 'url', 'label': 'URL', 'type': 'string'},
{'id': 'token', 'label': 'Token', 'type': 'string', 'secret': True, 'internal': True},
{'id': 'public_field', 'label': 'Public', 'type': 'string'},
]
},
)
url = reverse('api:credential_type_detail', kwargs={'pk': ct.pk})
with feature_flag_enabled('FEATURE_OIDC_WORKLOAD_IDENTITY_ENABLED'):
response = get(url, admin)
assert response.status_code == 200
field_ids = [f['id'] for f in response.data['inputs']['fields']]
# Internal field should be filtered out
assert 'token' not in field_ids
assert 'url' in field_ids
assert 'public_field' in field_ids
@pytest.mark.django_db
def test_credential_type_list_filters_internal_fields(get, admin):
"""Test that internal fields are filtered in list view."""
CredentialType.objects.create(
kind='cloud',
name='Test OIDC Type',
managed=False,
inputs={
'fields': [
{'id': 'url', 'label': 'URL', 'type': 'string'},
{'id': 'workload_identity_token', 'label': 'Token', 'type': 'string', 'secret': True, 'internal': True},
]
},
)
url = reverse('api:credential_type_list')
with feature_flag_enabled('FEATURE_OIDC_WORKLOAD_IDENTITY_ENABLED'):
response = get(url, admin)
assert response.status_code == 200
# Find our credential type in the results
test_ct = next((ct for ct in response.data['results'] if ct['name'] == 'Test OIDC Type'), None)
assert test_ct is not None
field_ids = [f['id'] for f in test_ct['inputs']['fields']]
# Internal field should be filtered out
assert 'workload_identity_token' not in field_ids
assert 'url' in field_ids

View File

@@ -0,0 +1,34 @@
import pytest
from awx.api.versioning import reverse
from awx.main.models import Host, Inventory
@pytest.mark.django_db
def test_dashboard_hosts_total_excludes_constructed(get, admin_user, organization):
"""
Constructed inventory hosts are not counted in the dashboard
"""
source_inv = Inventory.objects.create(name='source-inv', organization=organization)
source_host = source_inv.hosts.create(name='host1')
constructed = Inventory.objects.create(name='constructed-inv', kind='constructed', organization=organization)
Host.objects.create(name='host1', inventory=constructed, instance_id=str(source_host.pk))
response = get(reverse('api:dashboard_view'), user=admin_user, expect=200)
assert response.data['hosts']['total'] == 1
@pytest.mark.django_db
def test_host_list_still_returns_constructed(get, admin_user, organization):
"""
Constructed inventory hosts are still visible through the API
"""
source_inv = Inventory.objects.create(name='source-inv', organization=organization)
source_host = source_inv.hosts.create(name='host1')
constructed = Inventory.objects.create(name='constructed-inv', kind='constructed', organization=organization)
Host.objects.create(name='host1', inventory=constructed, instance_id=str(source_host.pk))
response = get(reverse('api:host_list'), user=admin_user, expect=200)
assert response.data['count'] == 2

View File

@@ -1,5 +1,3 @@
# TODO: As of writing this our only concern is ensuring that the fact feature is reflected in the Host endpoint.
# Other host tests should live here to make this test suite more complete.
import pytest
import urllib.parse
@@ -20,6 +18,48 @@ def inventory_structure():
Group.objects.create(name="g3", inventory=inv)
@pytest.fixture
def host_filter_inventory():
"""Inventory with hosts and groups matching the tower-qa test_host_filter structure.
Groups: groupA (contains groupAA as child), groupAA, groupB
Hosts: hostA (in groupA), hostAA (in groupAA), hostB (in groupB), hostDup (in all 3 groups)
"""
org = Organization.objects.create(name="hf-org")
inv = Inventory.objects.create(name="hf-inv", organization=org)
groupA = Group.objects.create(name="groupA", inventory=inv)
groupAA = Group.objects.create(name="groupAA", inventory=inv)
groupB = Group.objects.create(name="groupB", inventory=inv)
hostA = Host.objects.create(name="hostA", inventory=inv)
hostAA = Host.objects.create(name="hostAA", inventory=inv)
hostB = Host.objects.create(name="hostB", inventory=inv)
hostDup = Host.objects.create(name="hostDup", inventory=inv)
groupA.hosts.add(hostA, hostDup)
groupAA.hosts.add(hostAA, hostDup)
groupB.hosts.add(hostB, hostDup)
groupA.children.add(groupAA)
return {
'org': org,
'inv': inv,
'hosts': {'hostA': hostA, 'hostAA': hostAA, 'hostB': hostB, 'hostDup': hostDup},
'groups': {'groupA': groupA, 'groupAA': groupAA, 'groupB': groupB},
}
def get_host_names(response):
return sorted(h['name'] for h in response.data['results'])
def host_filter_get(get, user, host_filter):
url = reverse('api:host_list')
params = "?host_filter=%s" % urllib.parse.quote(host_filter, safe='')
return get(url + params, user)
@pytest.mark.django_db
def test_q1(inventory_structure, get, user):
def evaluate_query(query, expected_hosts):
@@ -50,3 +90,184 @@ def test_q1(inventory_structure, get, user):
# The following test verifies if the search in host_filter is case insensitive.
query = 'search="HOST1"'
evaluate_query(query, [hosts[0]])
# --- Host filter query tests (migrated from tower-qa test_host_filter.py) ---
@pytest.mark.django_db
@pytest.mark.parametrize(
"host_filter, expected",
[
("name=hostA", ["hostA"]),
("name=not_found", []),
("name=hostDup", ["hostDup"]),
],
)
def test_basic_host_name_search(host_filter_inventory, get, admin_user, host_filter, expected):
response = host_filter_get(get, admin_user, host_filter)
assert response.status_code == 200
assert get_host_names(response) == sorted(expected)
@pytest.mark.django_db
@pytest.mark.parametrize(
"host_filter, expected",
[
("name=hostA or name=hostB", ["hostA", "hostB"]),
("name=hostA or name=not_found", ["hostA"]),
("name=not_found or name=not_found", []),
("name=hostA or name=hostA", ["hostA"]),
("name=hostDup or name=hostDup", ["hostDup"]),
("name=hostA or name=hostAA or name=not_found", ["hostA", "hostAA"]),
],
)
def test_host_name_search_with_or(host_filter_inventory, get, admin_user, host_filter, expected):
response = host_filter_get(get, admin_user, host_filter)
assert response.status_code == 200
assert get_host_names(response) == sorted(expected)
@pytest.mark.django_db
@pytest.mark.parametrize(
"host_filter, expected",
[
("name=hostA and name=hostB", []),
("name=hostA and name=hostA", ["hostA"]),
("name=not_found and name=not_found", []),
("name=hostDup and name=hostDup", ["hostDup"]),
("name=hostA and name=hostB and name=not_found", []),
],
)
def test_host_name_search_with_and(host_filter_inventory, get, admin_user, host_filter, expected):
response = host_filter_get(get, admin_user, host_filter)
assert response.status_code == 200
assert get_host_names(response) == sorted(expected)
@pytest.mark.django_db
@pytest.mark.parametrize(
"host_filter, expected",
[
("groups__name=groupA", ["hostA", "hostDup"]),
("groups__name=groupAA", ["hostAA", "hostDup"]),
("groups__name=not_found", []),
],
)
def test_basic_group_search(host_filter_inventory, get, admin_user, host_filter, expected):
response = host_filter_get(get, admin_user, host_filter)
assert response.status_code == 200
assert get_host_names(response) == sorted(expected)
@pytest.mark.django_db
@pytest.mark.parametrize(
"host_filter, expected",
[
("groups__name=groupA or groups__name=groupB", ["hostA", "hostB", "hostDup"]),
("groups__name=groupA or groups__name=not_found", ["hostA", "hostDup"]),
("groups__name=not_found or groups__name=not_found", []),
("groups__name=groupA or groups__name=groupA", ["hostA", "hostDup"]),
(
"groups__name=groupA or groups__name=groupAA or groups__name=not_found",
["hostA", "hostAA", "hostDup"],
),
],
)
def test_group_search_with_or(host_filter_inventory, get, admin_user, host_filter, expected):
response = host_filter_get(get, admin_user, host_filter)
assert response.status_code == 200
assert get_host_names(response) == sorted(expected)
@pytest.mark.django_db
@pytest.mark.parametrize(
"host_filter, expected",
[
("groups__name=groupA and groups__name=groupB", ["hostDup"]),
("groups__name=groupA and groups__name=groupA", ["hostA", "hostDup"]),
("groups__name=not_found and groups__name=not_found", []),
("groups__name=groupA and groups__name=groupB and groups__name=not_found", []),
],
)
def test_group_search_with_and(host_filter_inventory, get, admin_user, host_filter, expected):
response = host_filter_get(get, admin_user, host_filter)
assert response.status_code == 200
assert get_host_names(response) == sorted(expected)
@pytest.mark.django_db
@pytest.mark.parametrize(
"host_filter, expected",
[
("name=hostA or groups__name=groupB", ["hostA", "hostB", "hostDup"]),
("name=hostA and groups__name=groupA", ["hostA"]),
("name=hostA and groups__name=not_found", []),
("name=not_found and groups__name=not_found", []),
("name=hostDup and groups__name=groupA", ["hostDup"]),
("name=hostDup and groups__name=groupB", ["hostDup"]),
],
)
def test_basic_hybrid_search(host_filter_inventory, get, admin_user, host_filter, expected):
response = host_filter_get(get, admin_user, host_filter)
assert response.status_code == 200
assert get_host_names(response) == sorted(expected)
@pytest.mark.django_db
def test_smart_search(get, admin_user):
org = Organization.objects.create(name="search-org")
inv = Inventory.objects.create(name="search-inv", organization=org)
host = Host.objects.create(name="unique_search_target", description="findme_description", inventory=inv)
for search_term in ["unique_search_target", "findme_description"]:
response = host_filter_get(get, admin_user, "search=%s" % search_term)
assert response.status_code == 200
names = get_host_names(response)
assert host.name in names
@pytest.mark.django_db
def test_password_field_filter_blocked(get, admin_user):
url = reverse('api:host_list')
filters = [
"created_by__password__icontains=pas3w3rd",
"search=foo or created_by__password__icontains=pas3w3rd",
"created_by__password__icontains=passw3rd or search=foo",
]
for f in filters:
params = "?host_filter=%s" % urllib.parse.quote(f, safe='')
response = get(url + params, admin_user)
assert response.status_code == 400, f"Expected 400 for filter: {f}"
@pytest.mark.django_db
def test_unicode_host_filter(get, admin_user):
org = Organization.objects.create(name="unicode-org")
inv = Inventory.objects.create(name="unicode-inv", organization=org)
host = Host.objects.create(name="ホスト", inventory=inv)
group = Group.objects.create(name="グループ", inventory=inv)
group.hosts.add(host)
response = host_filter_get(get, admin_user, "name=ホスト")
assert response.status_code == 200
assert len(response.data['results']) == 1
assert response.data['results'][0]['id'] == host.id
response = host_filter_get(get, admin_user, "groups__name=グループ")
assert response.status_code == 200
assert len(response.data['results']) == 1
assert response.data['results'][0]['id'] == host.id
@pytest.mark.django_db
@pytest.mark.parametrize(
"invalid_filter",
["string_without_equals", "1", "1.0", "true"],
ids=["bare_string", "integer", "float", "bool"],
)
def test_invalid_host_filter(get, admin_user, invalid_filter):
url = reverse('api:host_list')
params = "?host_filter=%s" % urllib.parse.quote(invalid_filter, safe='')
response = get(url + params, admin_user)
assert response.status_code == 400

View File

@@ -7,7 +7,7 @@ from django.core.exceptions import ValidationError
from awx.api.versioning import reverse
from awx.main.models import InventorySource, Inventory, ActivityStream
from awx.main.models import InventorySource, Inventory, ActivityStream, Organization
from awx.main.utils.inventory_vars import update_group_variables
@@ -963,3 +963,45 @@ class TestInventoryAllVariables:
# Test step 6: Value of var x from source A reappears, because the
# latest update from source B did not contain var x.
self.update_and_verify(inv_src_c, {}, expect={"x": 1}, teststep=6)
@pytest.mark.django_db
def test_inventory_names_unique_per_organization(post, admin_user):
"""Validate that two inventories can have the same name if they belong to different organizations."""
org1 = Organization.objects.create(name='org-inv-1')
org2 = Organization.objects.create(name='org-inv-2')
inv_name = 'SharedInventoryName'
# Create inventory with same name in org1
resp1 = post(
reverse('api:inventory_list'),
{'name': inv_name, 'organization': org1.id},
admin_user,
expect=201,
)
inv1_id = resp1.data['id']
# Create inventory with same name in org2 - should succeed
resp2 = post(
reverse('api:inventory_list'),
{'name': inv_name, 'organization': org2.id},
admin_user,
expect=201,
)
inv2_id = resp2.data['id']
assert inv1_id != inv2_id
inv1 = Inventory.objects.get(id=inv1_id)
inv2 = Inventory.objects.get(id=inv2_id)
assert inv1.name == inv2.name == inv_name
assert inv1.organization.id == org1.id
assert inv2.organization.id == org2.id
# Attempt to create another inventory with same name in org1 - should fail
resp3 = post(
reverse('api:inventory_list'),
{'name': inv_name, 'organization': org1.id},
admin_user,
expect=400,
)
assert 'Inventory with this Name and Organization already exists' in json.dumps(resp3.data)

View File

@@ -191,6 +191,21 @@ def test_job_accept_empty_tags(job_template_prompts, post, admin_user, mocker):
mock_job.signal_start.assert_called_once()
@pytest.mark.django_db
@pytest.mark.job_runtime_vars
def test_job_accept_long_skip_tags(job_template_prompts, post, admin_user, mocker):
job_template = job_template_prompts(True)
long_skip_tags = ','.join(f'tag{i}' for i in range(500))
assert len(long_skip_tags) > 1024
mock_job = mocker.MagicMock(spec=Job, id=968)
mocker.patch.object(JobTemplate, 'create_unified_job', return_value=mock_job)
mocker.patch('awx.api.serializers.JobSerializer.to_representation')
post(reverse('api:job_template_launch', kwargs={'pk': job_template.pk}), {'skip_tags': long_skip_tags}, admin_user, expect=201)
JobTemplate.create_unified_job.assert_called_once_with(skip_tags=long_skip_tags)
@pytest.mark.django_db
@pytest.mark.job_runtime_vars
def test_slice_timeout_forks_need_int(job_template_prompts, post, admin_user, mocker):

View File

@@ -485,47 +485,3 @@ class TestJobTemplateCallbackProxyIntegration:
expect=400,
**headers
)
@override_settings(REMOTE_HOST_HEADERS=['HTTP_X_FROM_THE_LOAD_BALANCER', 'REMOTE_ADDR', 'REMOTE_HOST'], PROXY_IP_ALLOWED_LIST=[])
def test_only_first_entry_in_comma_separated_header_is_considered(self, job_template, admin_user, post):
"""
Test that only the first entry in a comma-separated header value is used for host matching.
This is important for X-Forwarded-For style headers where the format is "client, proxy1, proxy2".
Only the original client (first entry) should be matched against inventory hosts.
"""
# Create host that matches the SECOND entry in the comma-separated list
job_template.inventory.hosts.create(name='second-host.example.com')
headers = {
# First entry is 'first-host.example.com', second is 'second-host.example.com'
# Only the first should be considered, so this should NOT match
'HTTP_X_FROM_THE_LOAD_BALANCER': 'first-host.example.com, second-host.example.com',
'REMOTE_ADDR': 'unrelated-addr',
'REMOTE_HOST': 'unrelated-host',
}
# Should return 400 because only 'first-host.example.com' is considered,
# and that host is NOT in the inventory
r = post(
url=reverse('api:job_template_callback', kwargs={'pk': job_template.pk}), data={'host_config_key': 'abcd'}, user=admin_user, expect=400, **headers
)
assert r.data['msg'] == 'No matching host could be found!'
@override_settings(REMOTE_HOST_HEADERS=['HTTP_X_FROM_THE_LOAD_BALANCER', 'REMOTE_ADDR', 'REMOTE_HOST'], PROXY_IP_ALLOWED_LIST=[])
def test_first_entry_in_comma_separated_header_matches(self, job_template, admin_user, post):
"""
Test that the first entry in a comma-separated header value correctly matches an inventory host.
"""
# Create host that matches the FIRST entry in the comma-separated list
job_template.inventory.hosts.create(name='first-host.example.com')
headers = {
# First entry is 'first-host.example.com', second is 'second-host.example.com'
# The first entry matches the inventory host
'HTTP_X_FROM_THE_LOAD_BALANCER': 'first-host.example.com, second-host.example.com',
'REMOTE_ADDR': 'unrelated-addr',
'REMOTE_HOST': 'unrelated-host',
}
# Should return 201 because 'first-host.example.com' is the first entry and matches
post(url=reverse('api:job_template_callback', kwargs={'pk': job_template.pk}), data={'host_config_key': 'abcd'}, user=admin_user, expect=201, **headers)

View File

@@ -0,0 +1,92 @@
# -*- coding: utf-8 -*-
import pytest
from awx.api.versioning import reverse
from awx.main.models import NotificationTemplate, Organization
@pytest.mark.django_db
def test_notification_template_names_unique_per_organization(post, admin_user):
"""
Validate that notification templates must have unique names within an organization,
but can have the same name across different organizations.
"""
org1 = Organization.objects.create(name='org-notif-1')
org2 = Organization.objects.create(name='org-notif-2')
template_name = 'SharedNotificationName'
# Create notification template in org1
resp1 = post(
reverse('api:notification_template_list'),
{
'name': template_name,
'organization': org1.id,
'notification_type': 'email',
'notification_configuration': {
'username': 'user@example.com',
'password': 'pass',
'sender': 'sender@example.com',
'recipients': ['recipient@example.com'],
'host': 'smtp.example.com',
'port': 25,
'use_tls': False,
'use_ssl': False,
},
},
admin_user,
expect=201,
)
template1_id = resp1.data['id']
# Create notification template with same name in org2 - should succeed
resp2 = post(
reverse('api:notification_template_list'),
{
'name': template_name,
'organization': org2.id,
'notification_type': 'email',
'notification_configuration': {
'username': 'user@example.com',
'password': 'pass',
'sender': 'sender@example.com',
'recipients': ['recipient@example.com'],
'host': 'smtp.example.com',
'port': 25,
'use_tls': False,
'use_ssl': False,
},
},
admin_user,
expect=201,
)
template2_id = resp2.data['id']
assert template1_id != template2_id
template1 = NotificationTemplate.objects.get(id=template1_id)
template2 = NotificationTemplate.objects.get(id=template2_id)
assert template1.name == template2.name == template_name
assert template1.organization.id == org1.id
assert template2.organization.id == org2.id
# Attempt to create another notification template with same name in org1 - should fail
resp3 = post(
reverse('api:notification_template_list'),
{
'name': template_name,
'organization': org1.id,
'notification_type': 'email',
'notification_configuration': {
'username': 'user@example.com',
'password': 'pass',
'sender': 'sender@example.com',
'recipients': ['recipient@example.com'],
'host': 'smtp.example.com',
'port': 25,
'use_tls': False,
'use_ssl': False,
},
},
admin_user,
expect=400,
)
assert 'Notification template with this Organization and Name already exists' in str(resp3.data)

View File

@@ -0,0 +1,163 @@
"""
Tests for OIDC workload identity credential type feature flag.
The FEATURE_OIDC_WORKLOAD_IDENTITY_ENABLED flag is an install-time flag that
controls whether OIDC credential types are loaded into the registry at startup.
When disabled, OIDC credential types are not loaded and do not exist in the database.
"""
import pytest
from unittest import mock
from django.test import override_settings
from awx.main.constants import OIDC_CREDENTIAL_TYPE_NAMESPACES
from awx.main.models.credential import CredentialType, ManagedCredentialType, load_credentials
from awx.api.versioning import reverse
@pytest.fixture
def reload_credentials_with_flag(django_db_setup, django_db_blocker):
"""
Fixture that reloads credentials with a specific flag state.
This simulates what happens at application startup.
"""
# Save original registry state
original_registry = ManagedCredentialType.registry.copy()
def _reload(flag_enabled):
with django_db_blocker.unblock():
# Clear the entire registry before reloading
ManagedCredentialType.registry.clear()
# Reload credentials with the specified flag state
with override_settings(FEATURE_OIDC_WORKLOAD_IDENTITY_ENABLED=flag_enabled):
with mock.patch('awx.main.models.credential.detect_server_product_name', return_value='NOT_AWX'):
load_credentials()
# Sync to database
CredentialType.setup_tower_managed_defaults(lock=False)
# In tests, the session fixture pre-loads all credential types into the DB.
# Remove OIDC types when testing the disabled state so the API test is accurate.
if not flag_enabled:
CredentialType.objects.filter(namespace__in=OIDC_CREDENTIAL_TYPE_NAMESPACES).delete()
yield _reload
# Restore original registry state after tests
ManagedCredentialType.registry.clear()
ManagedCredentialType.registry.update(original_registry)
@pytest.fixture
def isolated_registry():
"""Save and restore the ManagedCredentialType registry, with full isolation via mocked entry_points."""
original_registry = ManagedCredentialType.registry.copy()
ManagedCredentialType.registry.clear()
yield
ManagedCredentialType.registry.clear()
ManagedCredentialType.registry.update(original_registry)
def _make_mock_entry_point(name):
"""Create a mock entry point that mimics a credential plugin."""
ep = mock.MagicMock()
ep.name = name
ep.value = f'test_plugin:{name}'
plugin = mock.MagicMock(spec=[])
ep.load.return_value = plugin
return ep
def _mock_entry_points_factory(managed_names, supported_names):
"""Return a side_effect function for mocking entry_points() with controlled plugins."""
managed = [_make_mock_entry_point(n) for n in managed_names]
supported = [_make_mock_entry_point(n) for n in supported_names]
def _entry_points(group):
if group == 'awx_plugins.managed_credentials':
return managed
elif group == 'awx_plugins.managed_credentials.supported':
return supported
return []
return _entry_points
# --- Unit tests for load_credentials() registry behavior ---
def test_oidc_types_in_registry_when_flag_enabled(isolated_registry):
"""Test that OIDC credential types are added to the registry when flag is enabled."""
mock_eps = _mock_entry_points_factory(
managed_names=['ssh', 'vault'],
supported_names=['hashivault-kv-oidc', 'hashivault-ssh-oidc'],
)
with override_settings(FEATURE_OIDC_WORKLOAD_IDENTITY_ENABLED=True):
with mock.patch('awx.main.models.credential.detect_server_product_name', return_value='NOT_AWX'):
with mock.patch('awx.main.models.credential.entry_points', side_effect=mock_eps):
load_credentials()
for ns in OIDC_CREDENTIAL_TYPE_NAMESPACES:
assert ns in ManagedCredentialType.registry, f"{ns} should be in registry when flag is enabled"
assert 'ssh' in ManagedCredentialType.registry
assert 'vault' in ManagedCredentialType.registry
def test_oidc_types_not_in_registry_when_flag_disabled(isolated_registry):
"""Test that OIDC credential types are excluded from the registry when flag is disabled."""
mock_eps = _mock_entry_points_factory(
managed_names=['ssh', 'vault'],
supported_names=['hashivault-kv-oidc', 'hashivault-ssh-oidc'],
)
with override_settings(FEATURE_OIDC_WORKLOAD_IDENTITY_ENABLED=False):
with mock.patch('awx.main.models.credential.detect_server_product_name', return_value='NOT_AWX'):
with mock.patch('awx.main.models.credential.entry_points', side_effect=mock_eps):
load_credentials()
for ns in OIDC_CREDENTIAL_TYPE_NAMESPACES:
assert ns not in ManagedCredentialType.registry, f"{ns} should not be in registry when flag is disabled"
# Non-OIDC types should still be loaded
assert 'ssh' in ManagedCredentialType.registry
assert 'vault' in ManagedCredentialType.registry
def test_oidc_namespaces_constant():
"""Test that OIDC_CREDENTIAL_TYPE_NAMESPACES contains the expected namespaces."""
assert 'hashivault-kv-oidc' in OIDC_CREDENTIAL_TYPE_NAMESPACES
assert 'hashivault-ssh-oidc' in OIDC_CREDENTIAL_TYPE_NAMESPACES
assert len(OIDC_CREDENTIAL_TYPE_NAMESPACES) == 2
# --- Functional API tests ---
@pytest.mark.django_db
def test_oidc_types_loaded_when_flag_enabled(get, admin, reload_credentials_with_flag):
"""Test that OIDC credential types are visible in the API when flag is enabled."""
reload_credentials_with_flag(flag_enabled=True)
response = get(reverse('api:credential_type_list'), admin)
assert response.status_code == 200
namespaces = [ct['namespace'] for ct in response.data['results']]
assert 'hashivault-kv-oidc' in namespaces
assert 'hashivault-ssh-oidc' in namespaces
@pytest.mark.django_db
def test_oidc_types_not_loaded_when_flag_disabled(get, admin, reload_credentials_with_flag):
"""Test that OIDC credential types are not visible in the API when flag is disabled."""
reload_credentials_with_flag(flag_enabled=False)
response = get(reverse('api:credential_type_list'), admin)
assert response.status_code == 200
namespaces = [ct['namespace'] for ct in response.data['results']]
assert 'hashivault-kv-oidc' not in namespaces
assert 'hashivault-ssh-oidc' not in namespaces
# Verify they're also not in the database
assert not CredentialType.objects.filter(namespace='hashivault-kv-oidc').exists()
assert not CredentialType.objects.filter(namespace='hashivault-ssh-oidc').exists()

View File

@@ -0,0 +1,311 @@
"""
Tests for OIDC workload identity credential test endpoints.
Tests the /api/v2/credentials/<id>/test/ and /api/v2/credential_types/<id>/test/
endpoints when used with OIDC-enabled credential types.
"""
import pytest
from unittest import mock
from django.test import override_settings
from awx.main.models import Credential, CredentialType, JobTemplate
from awx.api.versioning import reverse
@pytest.fixture
def job_template(organization, project):
"""Job template with organization and project for OIDC JWT generation."""
return JobTemplate.objects.create(name='test-jt', organization=organization, project=project, playbook='helloworld.yml')
@pytest.fixture
def oidc_credentialtype():
"""Create a credential type with workload_identity_token internal field."""
oidc_type_inputs = {
'fields': [
{'id': 'url', 'label': 'Vault URL', 'type': 'string', 'help_text': 'The Vault server URL.'},
{'id': 'auth_path', 'label': 'Auth Path', 'type': 'string', 'help_text': 'JWT auth mount path.'},
{'id': 'role_id', 'label': 'Role ID', 'type': 'string', 'help_text': 'Vault role.'},
{'id': 'workload_identity_token', 'label': 'Workload Identity Token', 'type': 'string', 'secret': True, 'internal': True},
],
'metadata': [
{'id': 'secret_path', 'label': 'Secret Path', 'type': 'string'},
{'id': 'job_template_id', 'label': 'Job Template ID', 'type': 'string'},
],
'required': ['url', 'auth_path', 'role_id'],
}
class MockPlugin(object):
def backend(self, **kwargs):
# Simulate successful backend call
return 'secret'
with mock.patch('awx.main.models.credential.CredentialType.plugin', new_callable=mock.PropertyMock) as mock_plugin:
mock_plugin.return_value = MockPlugin()
oidc_type = CredentialType(kind='external', managed=True, namespace='hashivault-kv-oidc', name='HashiCorp Vault KV (OIDC)', inputs=oidc_type_inputs)
oidc_type.save()
yield oidc_type
@pytest.fixture
def oidc_credential(oidc_credentialtype):
"""Create a credential using the OIDC credential type."""
return Credential.objects.create(
credential_type=oidc_credentialtype,
name='oidc-vault-cred',
inputs={'url': 'http://vault.example.com:8200', 'auth_path': 'jwt', 'role_id': 'test-role'},
)
@pytest.fixture
def mock_oidc_backend():
"""Fixture that mocks OIDC JWT generation and credential backend."""
with mock.patch('awx.api.views.retrieve_workload_identity_jwt_with_claims') as mock_jwt, mock.patch('awx.api.views._jwt_decode') as mock_decode, mock.patch(
'awx.main.models.credential.CredentialType.plugin', new_callable=mock.PropertyMock
) as mock_plugin:
# Set default return values
mock_jwt.return_value = 'fake.jwt.token'
mock_decode.return_value = {'iss': 'http://gateway/o', 'aud': 'vault'}
# Create mock backend
mock_backend = mock.MagicMock()
mock_backend.backend.return_value = 'secret'
mock_plugin.return_value = mock_backend
# Yield all mocks for test customization
yield {
'jwt': mock_jwt,
'decode': mock_decode,
'plugin': mock_plugin,
'backend': mock_backend,
}
# --- Tests for CredentialExternalTest endpoint ---
@pytest.mark.django_db
@override_settings(FEATURE_OIDC_WORKLOAD_IDENTITY_ENABLED=False)
def test_credential_test_without_oidc_feature_flag(post, admin, oidc_credential):
"""Test that credential test works without OIDC feature flag enabled."""
url = reverse('api:credential_external_test', kwargs={'pk': oidc_credential.pk})
data = {'metadata': {'secret_path': 'test/secret', 'job_template_id': '1'}}
with mock.patch('awx.main.models.credential.CredentialType.plugin', new_callable=mock.PropertyMock) as mock_plugin:
mock_backend = mock.MagicMock()
mock_backend.backend.return_value = 'secret'
mock_plugin.return_value = mock_backend
response = post(url, data, admin)
assert response.status_code == 202
# Should not contain JWT payload when feature flag is disabled
assert 'details' not in response.data or 'sent_jwt_payload' not in response.data.get('details', {})
@pytest.mark.django_db
@mock.patch('awx.api.views.flag_enabled', return_value=True)
@pytest.mark.parametrize(
'job_template_id, expected_error',
[
(None, 'Job template ID is required'),
('not-an-integer', 'must be an integer'),
('99999', 'does not exist'),
],
ids=['missing_job_template_id', 'invalid_job_template_id_type', 'nonexistent_job_template_id'],
)
def test_credential_test_job_template_validation(mock_flag, post, admin, oidc_credential, job_template_id, expected_error):
"""Test that invalid job_template_id values return 400 with appropriate error messages."""
url = reverse('api:credential_external_test', kwargs={'pk': oidc_credential.pk})
data = {'metadata': {'secret_path': 'test/secret'}}
if job_template_id is not None:
data['metadata']['job_template_id'] = job_template_id
response = post(url, data, admin)
assert response.status_code == 400
assert 'details' in response.data
assert 'error_message' in response.data['details']
assert expected_error in response.data['details']['error_message']
@pytest.mark.django_db
@mock.patch('awx.api.views.flag_enabled', return_value=True)
def test_credential_test_no_access_to_job_template(mock_flag, post, alice, oidc_credential, job_template):
"""Test that user without access to job template gets 403."""
url = reverse('api:credential_external_test', kwargs={'pk': oidc_credential.pk})
data = {'metadata': {'secret_path': 'test/secret', 'job_template_id': str(job_template.id)}}
# Give alice use permission on credential but not on job template
oidc_credential.use_role.members.add(alice)
response = post(url, data, alice)
assert response.status_code == 403
assert 'You do not have access to job template' in str(response.data)
@pytest.mark.django_db
@mock.patch('awx.api.views.flag_enabled', return_value=True)
def test_credential_test_success_returns_jwt_payload(mock_flag, post, admin, oidc_credential, job_template, mock_oidc_backend):
"""Test that successful test returns JWT payload in response."""
url = reverse('api:credential_external_test', kwargs={'pk': oidc_credential.pk})
data = {'metadata': {'secret_path': 'test/secret', 'job_template_id': str(job_template.id)}}
# Customize mock for this test
mock_oidc_backend['decode'].return_value = {
'iss': 'http://gateway/o',
'sub': 'system:serviceaccount:default:awx-operator',
'aud': 'vault',
'job_template_id': job_template.id,
}
response = post(url, data, admin)
assert response.status_code == 202
assert 'details' in response.data
assert 'sent_jwt_payload' in response.data['details']
assert response.data['details']['sent_jwt_payload']['job_template_id'] == job_template.id
@pytest.mark.django_db
@mock.patch('awx.api.views.flag_enabled', return_value=True)
def test_credential_test_response_does_not_contain_secret_value(mock_flag, post, admin, oidc_credential, job_template, mock_oidc_backend):
"""
the OIDC credential test endpoint must not echo the resolved Vault secret back to the caller.
"""
url = reverse('api:credential_external_test', kwargs={'pk': oidc_credential.pk})
data = {'metadata': {'secret_path': 'test/secret', 'job_template_id': str(job_template.id)}}
credential_secret_value = 'CREDENTIAL_SECRET'
mock_oidc_backend['backend'].backend.return_value = credential_secret_value
response = post(url, data, admin)
assert response.status_code == 202
assert 'details' in response.data
assert 'sent_jwt_payload' in response.data['details']
assert 'secret_value' not in response.data['details']
assert credential_secret_value not in str(response.data)
@pytest.mark.django_db
@mock.patch('awx.api.views.flag_enabled', return_value=True)
def test_credential_test_backend_failure_returns_jwt_and_error(mock_flag, post, admin, oidc_credential, job_template, mock_oidc_backend):
"""Test that backend failure still returns JWT payload along with error message."""
url = reverse('api:credential_external_test', kwargs={'pk': oidc_credential.pk})
data = {'metadata': {'secret_path': 'test/secret', 'job_template_id': str(job_template.id)}}
# Make backend fail
mock_oidc_backend['backend'].backend.side_effect = RuntimeError('Connection failed')
response = post(url, data, admin)
assert response.status_code == 400
assert 'details' in response.data
# Both JWT payload and error message should be present
assert 'sent_jwt_payload' in response.data['details']
assert 'error_message' in response.data['details']
assert 'Connection failed' in response.data['details']['error_message']
@pytest.mark.django_db
@mock.patch('awx.api.views.flag_enabled', return_value=True)
def test_credential_test_jwt_generation_failure(mock_flag, post, admin, oidc_credential, job_template):
"""Test that JWT generation failure returns error without JWT payload."""
url = reverse('api:credential_external_test', kwargs={'pk': oidc_credential.pk})
data = {'metadata': {'secret_path': 'test/secret', 'job_template_id': str(job_template.id)}}
with mock.patch('awx.api.views.OIDCCredentialTestMixin._get_workload_identity_token') as mock_jwt:
mock_jwt.side_effect = RuntimeError('Failed to generate JWT')
response = post(url, data, admin)
assert response.status_code == 400
assert 'details' in response.data
assert 'error_message' in response.data['details']
assert 'Failed to generate JWT' in response.data['details']['error_message']
# No JWT payload when generation fails
assert 'sent_jwt_payload' not in response.data['details']
@pytest.mark.django_db
@mock.patch('awx.api.views.flag_enabled', return_value=True)
def test_credential_test_job_template_id_not_passed_to_backend(mock_flag, post, admin, oidc_credential, job_template, mock_oidc_backend):
"""Test that job_template_id is removed from backend_kwargs."""
url = reverse('api:credential_external_test', kwargs={'pk': oidc_credential.pk})
data = {'metadata': {'secret_path': 'test/secret', 'job_template_id': str(job_template.id)}}
response = post(url, data, admin)
assert response.status_code == 202
# Check that backend was called without job_template_id but with url and workload_identity_token
call_kwargs = mock_oidc_backend['backend'].backend.call_args[1]
assert 'job_template_id' not in call_kwargs
assert 'url' in call_kwargs
assert 'workload_identity_token' in call_kwargs
# --- Tests for CredentialTypeExternalTest endpoint ---
@pytest.mark.django_db
@mock.patch('awx.api.views.flag_enabled', return_value=True)
def test_credential_type_test_response_does_not_contain_secret_value(mock_flag, post, admin, oidc_credentialtype, job_template, mock_oidc_backend):
"""
the credential-type variant of the test endpoint should not return the secret value
"""
url = reverse('api:credential_type_external_test', kwargs={'pk': oidc_credentialtype.pk})
data = {
'inputs': {'url': 'http://vault.example.com:8200', 'auth_path': 'jwt', 'role_id': 'test-role', 'jwt_aud': 'vault'},
'metadata': {'secret_path': 'test/secret', 'job_template_id': str(job_template.id)},
}
credential_type_seret_value = 'CREDENTIAL_TYPE_SECRET'
mock_oidc_backend['backend'].backend.return_value = credential_type_seret_value
response = post(url, data, admin)
assert response.status_code == 202
assert 'details' in response.data
assert 'sent_jwt_payload' in response.data['details']
assert 'secret_value' not in response.data['details']
assert credential_type_seret_value not in str(response.data)
@pytest.mark.django_db
@mock.patch('awx.api.views.flag_enabled', return_value=True)
def test_credential_type_test_missing_job_template_id(mock_flag, post, admin, oidc_credentialtype):
"""Test that missing job_template_id returns 400 for credential type test endpoint."""
url = reverse('api:credential_type_external_test', kwargs={'pk': oidc_credentialtype.pk})
data = {
'inputs': {'url': 'http://vault.example.com:8200', 'auth_path': 'jwt', 'role_id': 'test-role'},
'metadata': {'secret_path': 'test/secret'},
}
response = post(url, data, admin)
assert response.status_code == 400
assert 'details' in response.data
assert 'error_message' in response.data['details']
assert 'Job template ID is required' in response.data['details']['error_message']
@pytest.mark.django_db
@mock.patch('awx.api.views.flag_enabled', return_value=True)
def test_credential_type_test_success_returns_jwt_payload(mock_flag, post, admin, oidc_credentialtype, job_template, mock_oidc_backend):
"""Test that successful credential type test returns JWT payload."""
url = reverse('api:credential_type_external_test', kwargs={'pk': oidc_credentialtype.pk})
data = {
'inputs': {'url': 'http://vault.example.com:8200', 'auth_path': 'jwt', 'role_id': 'test-role'},
'metadata': {'secret_path': 'test/secret', 'job_template_id': str(job_template.id)},
}
response = post(url, data, admin)
assert response.status_code == 202
assert 'details' in response.data
assert 'sent_jwt_payload' in response.data['details']
@pytest.mark.django_db
def test_credential_external_test_returns_400_for_non_external_credential(post, admin, credential):
# credential fixture creates a non-external credential (e.g. SSH/vault kind)
url = reverse('api:credential_external_test', kwargs={'pk': credential.pk})
response = post(url, {'metadata': {}}, admin)
assert response.status_code == 400
assert 'not testable' in response.data.get('detail', '').lower()

View File

@@ -1,20 +1,26 @@
import pytest
from django.test.utils import CaptureQueriesContext
from django.db import connection
from ansible_base.rbac.models import RoleDefinition
from awx.api.versioning import reverse
from awx.main.models import Project, Host
@pytest.fixture
def organization_resource_creator(organization, user):
def organization_resource_creator(organization, user, setup_managed_roles):
def rf(users, admins, job_templates, projects, inventories, teams):
member_rd = RoleDefinition.objects.get(name='Organization Member')
admin_rd = RoleDefinition.objects.get(name='Organization Admin')
# Associate one resource of every type with the organization
for i in range(users):
member_user = user('org-member %s' % i)
organization.member_role.members.add(member_user)
member_rd.give_permission(member_user, organization)
for i in range(admins):
admin_user = user('org-admin %s' % i)
organization.admin_role.members.add(admin_user)
admin_rd.give_permission(admin_user, organization)
for i in range(teams):
organization.teams.create(name='org-team %s' % i)
for i in range(inventories):
@@ -210,3 +216,24 @@ def test_JT_not_double_counted(resourced_organization, user, get):
assert 'hosts' in counts
counts.pop('hosts')
assert counts == counts_dict
@pytest.mark.django_db
def test_org_list_user_admin_query_count(organization_resource_creator, organizations, user, get):
"""User/admin counts use O(1) queries against roleuserassignment, not O(N) correlated subqueries."""
admin_user = user('admin', True)
extra_orgs = organizations(4)
member_rd = RoleDefinition.objects.get(name='Organization Member')
admin_rd = RoleDefinition.objects.get(name='Organization Admin')
for org in extra_orgs:
for i in range(3):
member_rd.give_permission(user(f'member-{org.pk}-{i}'), org)
admin_rd.give_permission(user(f'admin-{org.pk}'), org)
with CaptureQueriesContext(connection) as ctx:
response = get(reverse('api:organization_list'), admin_user)
assert response.status_code == 200
rua_queries = [q for q in ctx.captured_queries if 'dab_rbac_roleuserassignment' in q['sql']]
assert len(rua_queries) <= 2, f"Expected at most 2 roleuserassignment queries, got {len(rua_queries)}"

View File

@@ -56,7 +56,7 @@ def test_organization_list_access_tests(options, head, get, admin, alice):
@pytest.mark.django_db
def test_organization_access_tests(organization, get, admin, alice, bob):
def test_organization_access_tests(organization, get, admin, alice, bob, setup_managed_roles):
organization.member_role.members.add(alice)
get(reverse('api:organization_detail', kwargs={'pk': organization.id}), user=admin, expect=200)
get(reverse('api:organization_detail', kwargs={'pk': organization.id}), user=alice, expect=200)
@@ -65,14 +65,14 @@ def test_organization_access_tests(organization, get, admin, alice, bob):
@pytest.mark.django_db
def test_organization_list_integrity(organization, get, admin, alice):
def test_organization_list_integrity(organization, get, admin, alice, setup_managed_roles):
res = get(reverse('api:organization_list'), user=admin)
for field in ['id', 'url', 'name', 'description', 'created']:
assert field in res.data['results'][0]
@pytest.mark.django_db
def test_organization_list_order_integrity(organizations, get, admin):
def test_organization_list_order_integrity(organizations, get, admin, setup_managed_roles):
# check that the order of the organization list retains integrity.
orgs = organizations(4)
org_ids = {}
@@ -83,7 +83,7 @@ def test_organization_list_order_integrity(organizations, get, admin):
@pytest.mark.django_db
def test_organization_list_visibility(organizations, get, admin, alice):
def test_organization_list_visibility(organizations, get, admin, alice, setup_managed_roles):
orgs = organizations(2)
res = get(reverse('api:organization_list'), user=admin)
@@ -151,7 +151,7 @@ def test_organization_inventory_list_order_integrity(organization, admin, invent
@pytest.mark.django_db
def test_create_organization(post, admin, alice):
def test_create_organization(post, admin, alice, setup_managed_roles):
new_org = {'name': 'new org', 'description': 'my description'}
res = post(reverse('api:organization_list'), new_org, user=admin, expect=201)
assert res.data['name'] == new_org['name']
@@ -197,7 +197,7 @@ def test_add_admin_to_organization_xfail(post, organization, alice, bob):
@pytest.mark.django_db
def test_update_organization(get, put, organization, alice, bob):
def test_update_organization(get, put, organization, alice, bob, setup_managed_roles):
organization.admin_role.members.add(alice)
data = get(reverse('api:organization_detail', kwargs={'pk': organization.id}), user=alice, expect=200).data
data['description'] = 'hi'
@@ -209,7 +209,7 @@ def test_update_organization(get, put, organization, alice, bob):
@pytest.mark.django_db
def test_update_organization_max_hosts(get, put, organization, admin, alice, bob):
def test_update_organization_max_hosts(get, put, organization, admin, alice, bob, setup_managed_roles):
# Admin users can get and update max_hosts
data = get(reverse('api:organization_detail', kwargs={'pk': organization.id}), user=admin, expect=200).data
assert organization.max_hosts == 0

View File

@@ -139,6 +139,7 @@ def test_survey_password_default(post, patch, admin_user, project, inventory, su
("DTSTART:20300308T050000Z", "One or more rule required in rrule"),
("DTSTART:20300308T050000Z RRULE:FREQ=MONTHLY;INTERVAL=1; EXDATE:20220401", "EXDATE not allowed in rrule"),
("DTSTART:20300308T050000Z RRULE:FREQ=MONTHLY;INTERVAL=1; RDATE:20220401", "RDATE not allowed in rrule"),
("DTSTART:20300308T050000Z RRULE:FREQ=YEARLY;INTERVAL=0;BYDAY=MO", "INTERVAL must be a positive integer"),
("DTSTART:20300308T050000Z RRULE:FREQ=SECONDLY;INTERVAL=5;COUNT=6", "SECONDLY is not supported"),
# Individual rule test
("DTSTART:20300308T050000Z RRULE:NONSENSE", "INTERVAL required in rrule"),
@@ -202,6 +203,7 @@ def test_multiple_invalid_rrules(post, admin_user, project, inventory):
"rrule": [
"Multiple DTSTART is not supported.",
"INTERVAL required in rrule: RULE:FREQ=SECONDLY",
"SECONDLY is not supported: RULE:FREQ=SECONDLY",
"RRULE may not contain both COUNT and UNTIL: RULE:FREQ=MINUTELY;INTERVAL=10;COUNT=5;UNTIL=20220101",
"rrule parsing failed validation: 'NoneType' object has no attribute 'group'",
]

Some files were not shown because too many files have changed in this diff Show More