Move plugin loading to lazy-on-first-access, DB sync to dispatcher
Remove credential type and inventory plugin loading from Django's
app.ready() path. In-memory registries (ManagedCredentialType.registry
and InventorySourceOptions.injectors) are now populated lazily on first
access via LazyLoadDict, a dict subclass that calls a loader function
on the first read operation. This ensures web workers, dispatcher
workers, and management commands all get the registries populated
exactly when needed, without eager loading at startup.
The DB sync (CredentialType.setup_tower_managed_defaults) is moved to
the dispatcher's startup task, where it only needs to run once per
deployment rather than in every Django process.
Co-Authored-By: Alan Rominger <arominge@redhat.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* perf: stop eagerly updating Host.last_job_host_summary on every job completion
The playbook_on_stats wrapup path bulk-updates last_job_host_summary_id
on every host touched by a job. In the Q4CY25 scale lab this query had
a median execution time of 75 seconds due to index churn on main_host.
Replace all reads of the denormalized FK with a new classmethod
JobHostSummary.latest_for_host(host_id) that queries for the most
recent summary on demand. This eliminates the write-side bulk_update
of last_job_host_summary_id entirely.
Changes:
- Add JobHostSummary.latest_for_host() classmethod
- Serializer: use latest_for_host() instead of obj.last_job_host_summary
- Dashboard view: use subquery instead of FK traversal for failed hosts
- Inventory.update_computed_fields: use subquery for failed host count
- events.py: remove last_job_host_summary_id from bulk_update
- signals.py: simplify _update_host_last_jhs to only update last_job
- access.py/managers.py: remove select_related/defer through the FK
The FK field on Host is left in place for now (removal requires a
migration) but is no longer written to.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix .pk AttributeError, add job_template annotations, annotate host sublists
- Add 'pk' to AnnotatedSummary dynamic type (fixes AttributeError in get_related)
- Add job_template_id and job_template_name to subquery annotations so list
views include these fields in summary_fields.last_job (matching detail views)
- Traverse job__ FK from JobHostSummary instead of using separate UnifiedJob
subquery with OuterRef on another annotation (cleaner SQL, avoids alias issue)
- Annotate all host sublist views (InventoryHostsList, GroupHostsList,
GroupAllHostsList, InventorySourceHostsList) to prevent N+1 queries
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Update test_events to use JobHostSummary.latest_for_host instead of stale FKs
Tests were asserting host.last_job_id and host.last_job_host_summary_id
which are no longer updated. Use JobHostSummary.latest_for_host() to
derive the same data, matching the new read-time derivation approach.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Remove stale failures_url from deprecated DashboardView
The failures_url linked to ?last_job_host_summary__failed=True which
filters on the now-stale FK. The dashboard count itself was already
fixed to use a subquery annotation. Since DashboardView is deprecated
and has_active_failures is a SerializerMethodField (not filterable),
remove the failures_url entirely rather than creating a custom filter.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Apply black formatting to changed files
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Refactor: replace 10 subquery annotations with bulk prefetch
Instead of annotating every host queryset with 10 correlated subqueries
(summary + job + job_template fields), annotate only _latest_summary_id
and bulk-fetch the full JobHostSummary objects after pagination via
select_related('job', 'job__job_template').
This reduces the SQL from 10 correlated subqueries to 1 subquery + 1 IN
query, addressing review feedback about annotation overhead on host list
views.
- _annotate_host_latest_summary: only annotates _latest_summary_id
- _prefetch_latest_summaries: bulk-fetches and attaches to host objects
- HostSummaryPrefetchMixin: hooks into list() after pagination
- Serializer uses real JobHostSummary objects (no more AnnotatedSummary)
- to_representation always overwrites stale FK values
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Refactor: move latest summary to QuerySet._fetch_all + Host.latest_summary
Per review feedback, replace the view-level HostSummaryPrefetchMixin
with a custom QuerySet that bulk-attaches summaries at evaluation time
(like prefetch_related), and a Host.latest_summary property as the
single access point.
- HostLatestSummaryQuerySet: overrides _fetch_all() to bulk-fetch
JobHostSummary objects with select_related after queryset evaluation
- HostManager now inherits from the custom queryset via from_queryset()
- Host.latest_summary property: uses cache if available, falls back to
individual query
- Remove _annotate_host_latest_summary, _prefetch_latest_summaries,
HostSummaryPrefetchMixin from views — no more list() override needed
- Remove last_job/last_job_host_summary from SUMMARIZABLE_FK_FIELDS
- Serializer uses obj.latest_summary and DEFAULT_SUMMARY_FIELDS loop
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix: scope annotation to views, restore license_error/canceled_on
- Remove with_latest_summary_id() from HostManager.get_queryset() to
avoid applying the correlated subquery to every Host query globally
(count, exists, internal relations)
- Apply with_latest_summary_id() in get_queryset() of the 6
host-serving views only
- Restore license_error and canceled_on to last_job summary fields
to avoid breaking API change
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Guard _fetch_all() to skip bulk-attach on non-annotated querysets
Without this guard, _fetch_all() would set _latest_summary_cache=None
on every host in non-annotated querysets (e.g. Host.objects.filter()),
masking the per-object fallback query in Host.latest_summary.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Remove name from last_job_host_summary and canceled_on from last_job summary
Per reviewer feedback: these fields were not in the original API contract
via SUMMARIZABLE_FK_FIELDS and their addition would be an API change.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add functional tests for HostLatestSummaryQuerySet and Host.latest_summary
Tests cover:
- with_latest_summary_id() annotation and most-recent selection
- _fetch_all() bulk-attach behavior on annotated querysets
- _fetch_all() skips non-annotated querysets (preserves fallback)
- .count() and .exists() do NOT trigger _fetch_all
- Host.latest_summary cache hits (zero queries) and fallback
- Host.latest_job property
- select_related on bulk-attached summaries (no N+1)
- Chaining preserves annotation
- Multiple jobs / partial host coverage
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Apply black formatting to test_host_queryset.py
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ben Thomasson <bthomass@redhat.com>
* Fix flake8 F841: remove unused job1/job2 variables in tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Ben Thomasson <bthomass@redhat.com>
* Add comment explaining why Prefetch was not used for host latest summary
Django Prefetch cannot handle latest per group -- [:1] slicing fetches
1 record globally, not per host (Django ticket #26780). The custom
_fetch_all override uses the same 2-query pattern as prefetch_related
internally, customized for this use case.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Fix null handling to keep old behavior
---------
Signed-off-by: Ben Thomasson <bthomass@redhat.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: AlanCoding <arominge@redhat.com>
* AAP-41742: Fix workflow node update failing when JT has unprompted labels
PATCH extra_data on a workflow node fails with
{"labels":["Field is not configured to prompt on launch."]}
when the node has labels associated but the JT has
ask_labels_on_launch=False.
The serializer was passing all persisted M2M state from prompts_dict()
to _accept_or_ignore_job_kwargs() on every PATCH, re-validating
unchanged fields. Fix scopes validation to only the fields in the
request; full re-validation still occurs when unified_job_template
is being changed.
* Capture attrs keys before _build_mock_obj mutates them
_build_mock_obj() pops pseudo-fields (limit, scm_branch, job_tags,
etc.) from attrs. Computing requested_prompt_fields after the pop
would miss those fields and skip their ask_on_launch validation.
* Include survey_passwords when validating extra_vars prompts
prompts_dict() emits survey_passwords alongside extra_vars.
_accept_or_ignore_job_kwargs uses it to decrypt encrypted survey
values before validation. Without it, encrypted password blobs
are validated as-is against the survey spec.
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Adds support for testing external credentials that use OIDC workload identity tokens.
When FEATURE_OIDC_WORKLOAD_IDENTITY_ENABLED is enabled, the /test endpoints return
JWT payload details alongside test results.
- Add OIDC credential test endpoints with job template selection
- Return JWT payload and secret value in test response
- Maintain backward compatibility (detail field for errors)
- Add comprehensive unit and functional tests
- Refactor shared error handling logic
Co-authored-by: Daniel Finca <dfinca@redhat.com>
Co-authored-by: melissalkelly <melissalkelly1@gmail.com>
Remove Controller specific roles
Removes
- Controller Organization Admin
- Controller Organization Member
- Controller Team Admin
- Controller Team Member
- Controller System Auditor
Going forward the platform role definitions
will be used, e.g. Organization Member
The migration will take care of any assignments
with those controller specific roles and use
the platform roles instead.
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
* Delete existing all-group vars on inventory sync (with overwrite-vars=True) instead of merging them.
* Implementation of inv var handling with file as db.
* Improve serialization to file of inv vars for src update
* Include inventory-level variable editing into inventory source update handling
* Add group vars to inventory source update handling
* Add support for overwrite_vars to new inventory source handling
* Persist inventory var history in the database instead of a file.
* Remove logging which was needed during development.
* Remove further debugging code and improve comments
* Move special handling for user edits of variables into serializers
* Relate the inventory variable history model to its inventory
* Allow for inventory variables to have the value 'None'
* Fix KeyError in new inventory variable handling
* Add unique-together constraint for new model InventoryGroupVariablesWithHistory
* Use only one special invsrc_id for initial update and manual updates
* Fix internal server error when creating a new inventory
* Print the empty string for a variable with value 'None'
* Fix comment which incorrectly states old behaviour
* Fix inventory_group_variables_update tests which did not take the new handling of None into account
* Allow any type for Ansible-core variable values
* Refactor misleading method names
* Fix internal server error when savig vars from group form
* Remove superfluous json conversion in front of JSONField
* Call variable update from create/update instead from validate
* Use group_id instead of group_name in model InventoryGroupVariablesWithHistory
* Disable new variable update handling for all regular (non-'all') groups
* Add live test to verify AAP-17690 (inv var deleted from source)
* Add functional tests to verify inventory variables update logic
* Fix migration which was corrupted by a rebase
* Add a more complex live test and resolve linter complaints
* Force overwrite_vars=False for updates from source on all-group
* Change behavior with respect to overwrite_vars
* Delete existing all-group vars on inventory sync (with overwrite-vars=True) instead of merging them.
* Implementation of inv var handling with file as db.
* Improve serialization to file of inv vars for src update
* Include inventory-level variable editing into inventory source update handling
* Add group vars to inventory source update handling
* Add support for overwrite_vars to new inventory source handling
* Persist inventory var history in the database instead of a file.
* Remove logging which was needed during development.
* Remove further debugging code and improve comments
* Move special handling for user edits of variables into serializers
* Relate the inventory variable history model to its inventory
* Allow for inventory variables to have the value 'None'
* Fix KeyError in new inventory variable handling
* Add unique-together constraint for new model InventoryGroupVariablesWithHistory
* Use only one special invsrc_id for initial update and manual updates
* Fix internal server error when creating a new inventory
* Print the empty string for a variable with value 'None'
* Fix comment which incorrectly states old behaviour
* Fix inventory_group_variables_update tests which did not take the new handling of None into account
* Allow any type for Ansible-core variable values
* Refactor misleading method names
* Fix internal server error when savig vars from group form
* Remove superfluous json conversion in front of JSONField
* Call variable update from create/update instead from validate
* Use group_id instead of group_name in model InventoryGroupVariablesWithHistory
* Disable new variable update handling for all regular (non-'all') groups
* Add live test to verify AAP-17690 (inv var deleted from source)
* Add functional tests to verify inventory variables update logic
* Fix migration which was corrupted by a rebase
* Add a more complex live test and resolve linter complaints
* Force overwrite_vars=False for updates from source on all-group
* Change behavior with respect to overwrite_vars
* Prevent job pod from mounting serviceaccount token
* Add serializer validation for cg pod_spec_override
Prevent automountServiceAccountToken to be set to true and provide an error message when automountServiceAccountToken is being set to true
* Move call to django_validate_password to the correct method were the user object is available.
* Added tests for the Django password validation functionality.
* Move call to django_validate_password to the correct method were the user object is available.
* Added tests for the Django password validation functionality.
* Add `opa_query_path field` for Inventory, Organization and JobTemplate models (#6850)
Add `opa_query_path` model field to Inventory, Organizatio and JobTemplate. Add migration file and expose opa_query_path field in the related API serializers.
* Gather and evaluate `opa_query_path` fields and raise violation exceptions (#6864)
gather and evaluate all opa query related to a job execution during policy evaluation phase
* Add OPA_AUTH_CUSTOM_HEADERS support (#6863)
* Extend policy input data serializers (#6890)
* Extend policy input data serializers
* Update help text for PaC related fields (#6891)
* Remove encrypted from OPA_AUTH_CUSTOMER_HEADER
Unable to encrypt a dict field
---------
Co-authored-by: Jiří Jeřábek (Jiri Jerabek) <Jerabekjirka@email.cz>
Co-authored-by: Alexander Saprykin <cutwatercore@gmail.com>
Co-authored-by: Tina Tien <98424339+tiyiprh@users.noreply.github.com>
* Remove oauth provider
This removes the oauth provider functionality from awx. The
oauth2_provider app and all references to it have been removed.
Migrations to delete the two tables that locally overwrote
oauth2_provider tables are included. This change does not include
migrations to delete the tables provided by the oauth2_provider app.
Also not included here are changes to awxkit, awx_collection or the ui.
* Fix linters
* Update migrations after rebase
* Update collection tests for auth changes
The changes in https://github.com/ansible/awx/pull/15554 will cause a
few collection tests to fail, depending on what the test configuration
is. This changes the tests to look for a specific warning rather than
counting the number of warnings emitted.
* Update migration
* Removed unused oauth_scopes references
---------
Co-authored-by: Mike Graves <mgraves@redhat.com>
Co-authored-by: Alan Rominger <arominge@redhat.com>
* Add descriptions for plugin names
* Update serializers to display plugin and plugin description
* Add function to extract plugin name descriptions
* Add description for scm
* Conditionalize scm and file descriptions
This is to emphasize that this role is specific
to controller component. That is, not an auditor
for the entire AAP platform.
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
Fixes bug where creating a new user will
request a new awx_sessionid cookie, invalidating
the previous session.
Do not refresh session if updating or
creating a password for a different user.
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
* Prevent job pod from mounting serviceaccount token
* Add serializer validation for cg pod_spec_override
Prevent automountServiceAccountToken to be set to true and provide an error message when automountServiceAccountToken is being set to true
This is to emphasize that this role is specific
to controller component. That is, not an auditor
for the entire AAP platform.
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
Fixes bug where creating a new user will
request a new awx_sessionid cookie, invalidating
the previous session.
Do not refresh session if updating or
creating a password for a different user.
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
Support for AWS SNS notifications. SNS is a widespread service that is used to integrate with other AWS services(EG lambdas). This support would unlock use cases like triggering lambda functions, especially when AWX is deployed on EKS.
Decisions:
Data Structure
- I preferred using the same structure as Webhook for message body data because it contains all job details. For now, I directly linked to Webhook to avoid duplication, but I am open to suggestions.
AWS authentication
- To support non-AWS native environments, I added configuration options for AWS secret key, ID, and session tokens. When entered, these values are supplied to the underlining boto3 SNS client. If not entered, it falls back to the default authentication chain to support the native AWS environment. Properly configured EKS pods are created with temporary credentials that the default authentication chain can pick automatically.
---------
Signed-off-by: Ethem Cem Ozkan <ethemcem.ozkan@gmail.com>
* Add new enablement settings from DAB RBAC
* Initial implementation of system auditor as role without testing
* Fix system auditor role, remove duplicate assignments
* Make the system auditor role managed
* Flake8 fix
* Remove another thing from old solution
* Fix a few test failures
* Add extra setting to disable custom system roles via API
* Add test for custom role prohibition
Develop ability to list permissions for existing roles
Create a model registry for RBAC-tracked models
Write the data migration logic for creating
the preloaded role definitions
Write migration to migrate old Role into ObjectRole model
This loops over the old Role model, knowing it is unique
on object and role_field
Most of the logic is concerned with identifying the
needed permissions, and then corresponding role definition
As needed, object roles are created and users then teams
are assigned
Write re-computation of cache logic for teams
and then for object role permissions
Migrate new RBAC internals to ansible_base
Migrate tests to ansible_base
Implement solution for visible_roles
Expose URLs for DAB RBAC
As we do for control nodes, disable the
install_bundle endpoint for ingress nodes.
This can be done by checking if instance managed
is True.
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
Make protocol be blank on instance if there
is no canonical address for this instance.
It was defaulting to "tcp" before.
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
In receptor address post-save method:
- Fixed detecting if address was missing
a link from control nodes
- Use InstanceLink create_or_update to prevent
adding duplicate InstanceLink source and target
peers
In instance serializer create_or_update,
delete receptor addresses first before doing
instance create or update. This ensures that we don't
trigger unnecessary post-save methods that might
attempt to manipulate receptor addresses that
will just be removed later.
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
