* Introduced in PR https://github.com/ansible/awx/pull/16058/changes
then a later large merge from AAP back into devel removed the changes
* This PR re-introduces the github app lookup migration rename tests
with the migration names updated and the kind to namespace correction
* Multiple credentialtype's have the same kind and kind values look
like: cloud, network, machine, etc.
* namespace is the field that we want to rename
Upgrade to Django 5.2 LTS with compatibility fixes across fields, migrations, dispatch config, tests, and dev deps.
Dependencies:
- Upgrade django to 5.2.8 and relax requirements.in to >=5.2,<5.3.
- Bump django-debug-toolbar to >=6.0 for compatibility.
Backend:
- awx/conf/fields.py: switch URL TLD regex to use DomainNameValidator.ul in custom URLField.
- awx/main/management/commands/gather_analytics.py: use datetime.timezone.utc for naïve datetime handling.
- awx/main/dispatch/config.py: add mock_publish option; avoid DB access for test runs, set default max_workers, and support a noop broker.
Migrations (SQLite/Postgres compatibility):
- Add awx/main/migrations/_sqlite_helper.py with db-aware AlterIndexTogether/RenameIndex wrappers; consume in 0144_event_partitions.py and 0184_django_indexes.py.
- Update 0187_hop_nodes.py to use CheckConstraint(condition=...).
- Add 0205_alter_instance_peers_alter_job_hosts_and_more.py adjusting through_fields/relations on instance.peers, job.hosts, and role.ancestors.
- _dab_rbac.py: iterate roles with chunk_size=1000 for migration performance.
Tests:
Include hcp_terraform in default credential types in test_credential.py.
---------
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Alan Rominger <arominge@redhat.com>
Moved the AddField operation before the RunPython operations for 'rename_jts' and 'rename_projects' in migration 0200_template_name_constraint.py. This ensures the new 'org_unique' field exists before related data migrations are executed.
Fix
```
django.db.utils.ProgrammingError: column main_unifiedjobtemplate.org_unique does not exist
```
while applying migration 0200_template_name_constraint.py
when there's a job template or poject with duplicate name in the same org
* We had race conditions with the system_administrator role being
created just-in-time. Instead of fixing the race condition(s), dodge
them by ensuring the role always exists
* Disconnect logic to fill in role parents
Get tests passing hopefully
Whatever SonarCloud
* remove role parents/children endpoints and related views
* remove duplicate get_queryset method from RoleTeamsList
---------
Co-authored-by: Peter Braun <pbraun@redhat.com>
Bump migrations and delete some files
Resolve remaining conflicts
Fix requirements
Flake8 fixes
Prefer devel changes for schema
Use correct versions
Remove sso connected stuff
Update to modern actions and collection fixes
Remove unwated alias
Version problems in actions
Fix more versioning problems
Update warning string
Messed it up again
Shorten exception
More removals
Remove pbr license
Remove tests deleted in devel
Remove unexpected files
Remove some content missed in the rebase
Use sleep_task from devel
Restore devel live conftest file
Add in settings that got missed
Prefer devel version of collection test
Finish repairing .github path
Remove unintended test file duplication
Undo more unintended file additions
* Working branch for testing DAB RBAC changes
* AAP-48392 Handle DAB RBAC either before or after new type model (for merge) (#16045)
* Handle DAB RBAC either before or after new type model
* Translate CT to DAB CT
* Fix for rearrangement of post_migration methods
* Directly include RBAC service URLs
* Add a run before remote permission additions
* Sync old rbac to remote rbac (#7025)
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
* Set DAB requirement back to devel
---------
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
Co-authored-by: Seth Foster <fosterseth@users.noreply.github.com>
* migrate team on team users
add setting to prevent team on team cases. remove tests that should fail now
* adjust tests for disallowing team on teams
* use RoleUserAssignment to retrieve users
* assign users with RoleUserAssignment instead
* fix broken test
* move methods out to utils file. add tests
* add missed positional arg
* test old rbac system also consolidates
* fix test
Remove Controller specific roles
Removes
- Controller Organization Admin
- Controller Organization Member
- Controller Team Admin
- Controller Team Member
- Controller System Auditor
Going forward the platform role definitions
will be used, e.g. Organization Member
The migration will take care of any assignments
with those controller specific roles and use
the platform roles instead.
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
* Bug fix for AAP-47771 this data migration updates existing CredentialType entries
in the database and changes the kind from github_app to github_app_lookup
* Combine migration 0203 into 0202
* Add test to ensure reconciliation issue has been resolved
* wfjt migration to catch renaming
* Added rename_wfjt function to template constraint migration
* Add test to add duplicate names and verify that the duplicates are renamed
* move object creation
* add missing rename_wfjt operation
* fix linter issues
* fix tox issues
* test manually and move operation
* added back credential type validation code
* Handle DAB RBAC either before or after new type model
* Translate CT to DAB CT
* Fixes for content type switch
* Use more compatible coding pattern
* Deeper purge of content_type_id
* revert, turns out that did not work
* More content type replacements
* Revert changes to serializer
* Revert another content_type change
* Fix for rearrangement of post_migration methods
* Remove thing I am not going to do
* Revert branch pin that was temporary
* Make the JT name uniqueness enforced at the database level
* Forgot demo project fixture
* New approach, done by adding a new field
* Update for linters and failures
* Fix logical error in migration test
* Revert some test changes based on review comment
* Do not rename first template, add test
* Avoid name-too-long rename errors
* Insert migration into place
* Move existing files with git
* Bump migrations of existing
* Update migration test
* Awkward bump
* Fix migration file link
* update test reference again
* Delete existing all-group vars on inventory sync (with overwrite-vars=True) instead of merging them.
* Implementation of inv var handling with file as db.
* Improve serialization to file of inv vars for src update
* Include inventory-level variable editing into inventory source update handling
* Add group vars to inventory source update handling
* Add support for overwrite_vars to new inventory source handling
* Persist inventory var history in the database instead of a file.
* Remove logging which was needed during development.
* Remove further debugging code and improve comments
* Move special handling for user edits of variables into serializers
* Relate the inventory variable history model to its inventory
* Allow for inventory variables to have the value 'None'
* Fix KeyError in new inventory variable handling
* Add unique-together constraint for new model InventoryGroupVariablesWithHistory
* Use only one special invsrc_id for initial update and manual updates
* Fix internal server error when creating a new inventory
* Print the empty string for a variable with value 'None'
* Fix comment which incorrectly states old behaviour
* Fix inventory_group_variables_update tests which did not take the new handling of None into account
* Allow any type for Ansible-core variable values
* Refactor misleading method names
* Fix internal server error when savig vars from group form
* Remove superfluous json conversion in front of JSONField
* Call variable update from create/update instead from validate
* Use group_id instead of group_name in model InventoryGroupVariablesWithHistory
* Disable new variable update handling for all regular (non-'all') groups
* Add live test to verify AAP-17690 (inv var deleted from source)
* Add functional tests to verify inventory variables update logic
* Fix migration which was corrupted by a rebase
* Add a more complex live test and resolve linter complaints
* Force overwrite_vars=False for updates from source on all-group
* Change behavior with respect to overwrite_vars
* Make the JT name uniqueness enforced at the database level
* Forgot demo project fixture
* New approach, done by adding a new field
* Update for linters and failures
* Fix logical error in migration test
* Revert some test changes based on review comment
* Do not rename first template, add test
* Avoid name-too-long rename errors
* Insert migration into place
* Move existing files with git
* Bump migrations of existing
* Update migration test
* Awkward bump
* Fix migration file link
* update test reference again
* Delete existing all-group vars on inventory sync (with overwrite-vars=True) instead of merging them.
* Implementation of inv var handling with file as db.
* Improve serialization to file of inv vars for src update
* Include inventory-level variable editing into inventory source update handling
* Add group vars to inventory source update handling
* Add support for overwrite_vars to new inventory source handling
* Persist inventory var history in the database instead of a file.
* Remove logging which was needed during development.
* Remove further debugging code and improve comments
* Move special handling for user edits of variables into serializers
* Relate the inventory variable history model to its inventory
* Allow for inventory variables to have the value 'None'
* Fix KeyError in new inventory variable handling
* Add unique-together constraint for new model InventoryGroupVariablesWithHistory
* Use only one special invsrc_id for initial update and manual updates
* Fix internal server error when creating a new inventory
* Print the empty string for a variable with value 'None'
* Fix comment which incorrectly states old behaviour
* Fix inventory_group_variables_update tests which did not take the new handling of None into account
* Allow any type for Ansible-core variable values
* Refactor misleading method names
* Fix internal server error when savig vars from group form
* Remove superfluous json conversion in front of JSONField
* Call variable update from create/update instead from validate
* Use group_id instead of group_name in model InventoryGroupVariablesWithHistory
* Disable new variable update handling for all regular (non-'all') groups
* Add live test to verify AAP-17690 (inv var deleted from source)
* Add functional tests to verify inventory variables update logic
* Fix migration which was corrupted by a rebase
* Add a more complex live test and resolve linter complaints
* Force overwrite_vars=False for updates from source on all-group
* Change behavior with respect to overwrite_vars
* Add in ESXI plugin as a choice
* added in vmware esxi as an inventory source
* made a migration that may not be needed but will need to circle back
* black formatting
* linter fixes that I missed in the first commit, squash
* Update esxi to use_fqcn to true
* added use_fqcn on the esxi cred to true to correctly lay down
collection name
* add fqcn true
* updated vmware esxi to use true for fqcn
* Update defaults and re-order migrations
* updated defaults to add correct env var to get empty
* re-ordered migrations to be in line with others
* Add condition to replace vmware_esxi cred
* replace direct name match with vmware cred since source supports old
cred
* add skeleton test
* quick pass, needs more
* squash this
* Add tests for creating inventory ESXI source
* add test case to test creating an inventory with different cred type
to source name
* update test and linting
* added correct cred return since esxi uses same cred
* assert on status code
* assert that we received a 204
* Added new folder for vmware_exsi and empty json file.
* Corrected the misspelling of folder name to 'esxi'
* fixed misspelling for `vmware_`
---------
Co-authored-by: Thanhnguyet Vo <thavo@redhat.com>
* Add `opa_query_path field` for Inventory, Organization and JobTemplate models (#6850)
Add `opa_query_path` model field to Inventory, Organizatio and JobTemplate. Add migration file and expose opa_query_path field in the related API serializers.
* Gather and evaluate `opa_query_path` fields and raise violation exceptions (#6864)
gather and evaluate all opa query related to a job execution during policy evaluation phase
* Add OPA_AUTH_CUSTOM_HEADERS support (#6863)
* Extend policy input data serializers (#6890)
* Extend policy input data serializers
* Update help text for PaC related fields (#6891)
* Remove encrypted from OPA_AUTH_CUSTOMER_HEADER
Unable to encrypt a dict field
---------
Co-authored-by: Jiří Jeřábek (Jiri Jerabek) <Jerabekjirka@email.cz>
Co-authored-by: Alexander Saprykin <cutwatercore@gmail.com>
Co-authored-by: Tina Tien <98424339+tiyiprh@users.noreply.github.com>
* Feature indirect host counting (#15802)
* AAP-37282 Add parse JQ data and test it for a `job` object in isolation (#15774)
* Add jq dependency
* Add file in progress
* Add license for jq
* Write test and get it passing
* Successfully test collection of `event_query.yml` data (#15761)
* Callback plugin method from cmeyers adapted to global collection list
Get tests passing
Mild rebranding
Put behind feature flag, flip true in dev
Add noqa flag
* Add missing wait_for_events
* feat: try grabbing query files from artifacts directory (#15776)
* Contract changes for the event_query collection callback plugin (#15785)
* Minor import changes to collection processing in callback plugin
* Move agreed location of event_query file
* feat: remaining schema changes for indirect host audits (#15787)
* Re-organize test file and move artifacts processing logic to callback (#15784)
* Rename the indirect host counting test file
* Combine artifacts saving logic
* Connect host audit model to jq logic via new task
* Add unit tests for indirect host counting (#15792)
* Do not get django flags from database (#15794)
* Document, implement, and test remaining indirect host audit fields (#15796)
* Document, implement, and test remaining indirect host audit fields
* Fix hashing
* AAP-39559 Wait for all event processing to finish, add fallback task (#15798)
* Wait for all event processing to finish, add fallback task
* Add flag check to periodic task
* feat: cleanup of old indirect host audit records (#15800)
* By default, do not count indirect hosts (#15801)
* By default, do not count indirect hosts
* Fix copy paste goof
* Fix linter issue from base branch
* prevent multiple tasks from processing the same job events, prevent p… (#15805)
prevent multiple tasks from processing the same job events, prevent periodic task from spawning another task per job
* Fix typos and other bugs found by Pablo review
* fix: rely on resolved_action instead of task, adapt to proposed query… (#15815)
* fix: rely on resolved_action instead of task, adapt to proposed query structure
* tests: update indirect host tests
* update remaining queries to new format
* update live test
* Remove polling loop for job finishing event processing (#15811)
* Remove polling loop for job finishing event processing
* Make awx/main/tests/live dramatically faster (#15780)
* AAP-37282 Add parse JQ data and test it for a `job` object in isolation (#15774)
* Add jq dependency
* Add file in progress
* Add license for jq
* Write test and get it passing
* Successfully test collection of `event_query.yml` data (#15761)
* Callback plugin method from cmeyers adapted to global collection list
Get tests passing
Mild rebranding
Put behind feature flag, flip true in dev
Add noqa flag
* Add missing wait_for_events
* feat: try grabbing query files from artifacts directory (#15776)
* Contract changes for the event_query collection callback plugin (#15785)
* Minor import changes to collection processing in callback plugin
* Move agreed location of event_query file
* feat: remaining schema changes for indirect host audits (#15787)
* Re-organize test file and move artifacts processing logic to callback (#15784)
* Rename the indirect host counting test file
* Combine artifacts saving logic
* Connect host audit model to jq logic via new task
* Document, implement, and test remaining indirect host audit fields (#15796)
* AAP-39559 Wait for all event processing to finish, add fallback task (#15798)
* Wait for all event processing to finish, add fallback task
* Add flag check to periodic task
* feat: cleanup of old indirect host audit records (#15800)
* prevent multiple tasks from processing the same job events, prevent p… (#15805)
prevent multiple tasks from processing the same job events, prevent periodic task from spawning another task per job
* Remove polling loop for job finishing event processing (#15811)
* Make awx/main/tests/live dramatically faster (#15780)
* reorder migrations to allow indirect instances backport
* cleanup for rebase and merge into devel
---------
Co-authored-by: Peter Braun <pbraun@redhat.com>
Co-authored-by: jessicamack <jmack@redhat.com>
Co-authored-by: Peter Braun <pbranu@redhat.com>
* AAP-37282 Add parse JQ data and test it for a `job` object in isolation (#15774)
* Add jq dependency
* Add file in progress
* Add license for jq
* Write test and get it passing
* Successfully test collection of `event_query.yml` data (#15761)
* Callback plugin method from cmeyers adapted to global collection list
Get tests passing
Mild rebranding
Put behind feature flag, flip true in dev
Add noqa flag
* Add missing wait_for_events
* feat: try grabbing query files from artifacts directory (#15776)
* Contract changes for the event_query collection callback plugin (#15785)
* Minor import changes to collection processing in callback plugin
* Move agreed location of event_query file
* feat: remaining schema changes for indirect host audits (#15787)
* Re-organize test file and move artifacts processing logic to callback (#15784)
* Rename the indirect host counting test file
* Combine artifacts saving logic
* Connect host audit model to jq logic via new task
* Add unit tests for indirect host counting (#15792)
* Do not get django flags from database (#15794)
* Document, implement, and test remaining indirect host audit fields (#15796)
* Document, implement, and test remaining indirect host audit fields
* Fix hashing
* AAP-39559 Wait for all event processing to finish, add fallback task (#15798)
* Wait for all event processing to finish, add fallback task
* Add flag check to periodic task
* feat: cleanup of old indirect host audit records (#15800)
* By default, do not count indirect hosts (#15801)
* By default, do not count indirect hosts
* Fix copy paste goof
* Fix linter issue from base branch
* prevent multiple tasks from processing the same job events, prevent p… (#15805)
prevent multiple tasks from processing the same job events, prevent periodic task from spawning another task per job
* Fix typos and other bugs found by Pablo review
* fix: rely on resolved_action instead of task, adapt to proposed query… (#15815)
* fix: rely on resolved_action instead of task, adapt to proposed query structure
* tests: update indirect host tests
* update remaining queries to new format
* update live test
* Remove polling loop for job finishing event processing (#15811)
* Remove polling loop for job finishing event processing
* Make awx/main/tests/live dramatically faster (#15780)
* AAP-37282 Add parse JQ data and test it for a `job` object in isolation (#15774)
* Add jq dependency
* Add file in progress
* Add license for jq
* Write test and get it passing
* Successfully test collection of `event_query.yml` data (#15761)
* Callback plugin method from cmeyers adapted to global collection list
Get tests passing
Mild rebranding
Put behind feature flag, flip true in dev
Add noqa flag
* Add missing wait_for_events
* feat: try grabbing query files from artifacts directory (#15776)
* Contract changes for the event_query collection callback plugin (#15785)
* Minor import changes to collection processing in callback plugin
* Move agreed location of event_query file
* feat: remaining schema changes for indirect host audits (#15787)
* Re-organize test file and move artifacts processing logic to callback (#15784)
* Rename the indirect host counting test file
* Combine artifacts saving logic
* Connect host audit model to jq logic via new task
* Document, implement, and test remaining indirect host audit fields (#15796)
* Document, implement, and test remaining indirect host audit fields
* Fix hashing
* AAP-39559 Wait for all event processing to finish, add fallback task (#15798)
* Wait for all event processing to finish, add fallback task
* Add flag check to periodic task
* feat: cleanup of old indirect host audit records (#15800)
* prevent multiple tasks from processing the same job events, prevent p… (#15805)
prevent multiple tasks from processing the same job events, prevent periodic task from spawning another task per job
* Remove polling loop for job finishing event processing (#15811)
* Remove polling loop for job finishing event processing
* Make awx/main/tests/live dramatically faster (#15780)
* temp
* remove test
* reorder migrations to allow indirect instances backport
* cleanup for rebase and merge into devel
---------
Co-authored-by: Peter Braun <pbraun@redhat.com>
Co-authored-by: jessicamack <jmack@redhat.com>
Co-authored-by: Peter Braun <pbranu@redhat.com>
* Remove oauth provider
This removes the oauth provider functionality from awx. The
oauth2_provider app and all references to it have been removed.
Migrations to delete the two tables that locally overwrote
oauth2_provider tables are included. This change does not include
migrations to delete the tables provided by the oauth2_provider app.
Also not included here are changes to awxkit, awx_collection or the ui.
* Fix linters
* Update migrations after rebase
* Update collection tests for auth changes
The changes in https://github.com/ansible/awx/pull/15554 will cause a
few collection tests to fail, depending on what the test configuration
is. This changes the tests to look for a specific warning rather than
counting the number of warnings emitted.
* Update migration
* Removed unused oauth_scopes references
---------
Co-authored-by: Mike Graves <mgraves@redhat.com>
Co-authored-by: Alan Rominger <arominge@redhat.com>
This is to emphasize that this role is specific
to controller component. That is, not an auditor
for the entire AAP platform.
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
This is to emphasize that this role is specific
to controller component. That is, not an auditor
for the entire AAP platform.
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
Adds the following managed Role Definitions
Controller Team Admin
Controller Team Member
Controller Organization Admin
Controller Organization Member
These have the same permission set as the
platform roles (without the Controller prefix)
Adding members to teams and orgs via the legacy RBAC system
will use these role definitions.
Other changes:
- Bump DAB to 2024.08.22
- Set ALLOW_LOCAL_ASSIGNING_JWT_ROLES to False in defaults.py.
This setting prevents assignments to the platform roles (e.g. Team Member).
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
Adds the following managed Role Definitions
Controller Team Admin
Controller Team Member
Controller Organization Admin
Controller Organization Member
These have the same permission set as the
platform roles (without the Controller prefix)
Adding members to teams and orgs via the legacy RBAC system
will use these role definitions.
Other changes:
- Bump DAB to 2024.08.22
- Set ALLOW_LOCAL_ASSIGNING_JWT_ROLES to False in defaults.py.
This setting prevents assignments to the platform roles (e.g. Team Member).
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
* Add test that we got all permissions right for every role
* Fix missing Org execute role and missing adhoc role permission
* Add in missing Organization Approval Role as well
* Remove Role from role names
* Add initial test for deletion of stale permission
* Delete existing EE view permission
* Hypothetically complete update of EE model permissions setup
* Tests passing locally
* Issue with user_capabilities was a test bug, fixed
* Add migration testing for certain managed roles
* Fix managed role bugs
* Add more tests
* Fix another bug with org workflow admin role reference
* Add test because another issue is fixed
* Mark reason for test
* Remove internal markers
* Reword failure message
Co-authored-by: Seth Foster <fosterseth@users.noreply.github.com>
---------
Co-authored-by: Seth Foster <fosterseth@users.noreply.github.com>
