* migrate team on team users
add setting to prevent team on team cases. remove tests that should fail now
* adjust tests for disallowing team on teams
* use RoleUserAssignment to retrieve users
* assign users with RoleUserAssignment instead
* fix broken test
* move methods out to utils file. add tests
* add missed positional arg
* test old rbac system also consolidates
* fix test
Remove Controller specific roles
Removes
- Controller Organization Admin
- Controller Organization Member
- Controller Team Admin
- Controller Team Member
- Controller System Auditor
Going forward the platform role definitions
will be used, e.g. Organization Member
The migration will take care of any assignments
with those controller specific roles and use
the platform roles instead.
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
* wfjt migration to catch renaming
* Added rename_wfjt function to template constraint migration
* Add test to add duplicate names and verify that the duplicates are renamed
* move object creation
* add missing rename_wfjt operation
* fix linter issues
* fix tox issues
* test manually and move operation
* added back credential type validation code
* Make the JT name uniqueness enforced at the database level
* Forgot demo project fixture
* New approach, done by adding a new field
* Update for linters and failures
* Fix logical error in migration test
* Revert some test changes based on review comment
* Do not rename first template, add test
* Avoid name-too-long rename errors
* Insert migration into place
* Move existing files with git
* Bump migrations of existing
* Update migration test
* Awkward bump
* Fix migration file link
* update test reference again
* Delete existing all-group vars on inventory sync (with overwrite-vars=True) instead of merging them.
* Implementation of inv var handling with file as db.
* Improve serialization to file of inv vars for src update
* Include inventory-level variable editing into inventory source update handling
* Add group vars to inventory source update handling
* Add support for overwrite_vars to new inventory source handling
* Persist inventory var history in the database instead of a file.
* Remove logging which was needed during development.
* Remove further debugging code and improve comments
* Move special handling for user edits of variables into serializers
* Relate the inventory variable history model to its inventory
* Allow for inventory variables to have the value 'None'
* Fix KeyError in new inventory variable handling
* Add unique-together constraint for new model InventoryGroupVariablesWithHistory
* Use only one special invsrc_id for initial update and manual updates
* Fix internal server error when creating a new inventory
* Print the empty string for a variable with value 'None'
* Fix comment which incorrectly states old behaviour
* Fix inventory_group_variables_update tests which did not take the new handling of None into account
* Allow any type for Ansible-core variable values
* Refactor misleading method names
* Fix internal server error when savig vars from group form
* Remove superfluous json conversion in front of JSONField
* Call variable update from create/update instead from validate
* Use group_id instead of group_name in model InventoryGroupVariablesWithHistory
* Disable new variable update handling for all regular (non-'all') groups
* Add live test to verify AAP-17690 (inv var deleted from source)
* Add functional tests to verify inventory variables update logic
* Fix migration which was corrupted by a rebase
* Add a more complex live test and resolve linter complaints
* Force overwrite_vars=False for updates from source on all-group
* Change behavior with respect to overwrite_vars
* Add in ESXI plugin as a choice
* added in vmware esxi as an inventory source
* made a migration that may not be needed but will need to circle back
* black formatting
* linter fixes that I missed in the first commit, squash
* Update esxi to use_fqcn to true
* added use_fqcn on the esxi cred to true to correctly lay down
collection name
* add fqcn true
* updated vmware esxi to use true for fqcn
* Update defaults and re-order migrations
* updated defaults to add correct env var to get empty
* re-ordered migrations to be in line with others
* Add condition to replace vmware_esxi cred
* replace direct name match with vmware cred since source supports old
cred
* add skeleton test
* quick pass, needs more
* squash this
* Add tests for creating inventory ESXI source
* add test case to test creating an inventory with different cred type
to source name
* update test and linting
* added correct cred return since esxi uses same cred
* assert on status code
* assert that we received a 204
* Added new folder for vmware_exsi and empty json file.
* Corrected the misspelling of folder name to 'esxi'
* fixed misspelling for `vmware_`
---------
Co-authored-by: Thanhnguyet Vo <thavo@redhat.com>
* Add `opa_query_path field` for Inventory, Organization and JobTemplate models (#6850)
Add `opa_query_path` model field to Inventory, Organizatio and JobTemplate. Add migration file and expose opa_query_path field in the related API serializers.
* Gather and evaluate `opa_query_path` fields and raise violation exceptions (#6864)
gather and evaluate all opa query related to a job execution during policy evaluation phase
* Add OPA_AUTH_CUSTOM_HEADERS support (#6863)
* Extend policy input data serializers (#6890)
* Extend policy input data serializers
* Update help text for PaC related fields (#6891)
* Remove encrypted from OPA_AUTH_CUSTOMER_HEADER
Unable to encrypt a dict field
---------
Co-authored-by: Jiří Jeřábek (Jiri Jerabek) <Jerabekjirka@email.cz>
Co-authored-by: Alexander Saprykin <cutwatercore@gmail.com>
Co-authored-by: Tina Tien <98424339+tiyiprh@users.noreply.github.com>
* Feature indirect host counting (#15802)
* AAP-37282 Add parse JQ data and test it for a `job` object in isolation (#15774)
* Add jq dependency
* Add file in progress
* Add license for jq
* Write test and get it passing
* Successfully test collection of `event_query.yml` data (#15761)
* Callback plugin method from cmeyers adapted to global collection list
Get tests passing
Mild rebranding
Put behind feature flag, flip true in dev
Add noqa flag
* Add missing wait_for_events
* feat: try grabbing query files from artifacts directory (#15776)
* Contract changes for the event_query collection callback plugin (#15785)
* Minor import changes to collection processing in callback plugin
* Move agreed location of event_query file
* feat: remaining schema changes for indirect host audits (#15787)
* Re-organize test file and move artifacts processing logic to callback (#15784)
* Rename the indirect host counting test file
* Combine artifacts saving logic
* Connect host audit model to jq logic via new task
* Add unit tests for indirect host counting (#15792)
* Do not get django flags from database (#15794)
* Document, implement, and test remaining indirect host audit fields (#15796)
* Document, implement, and test remaining indirect host audit fields
* Fix hashing
* AAP-39559 Wait for all event processing to finish, add fallback task (#15798)
* Wait for all event processing to finish, add fallback task
* Add flag check to periodic task
* feat: cleanup of old indirect host audit records (#15800)
* By default, do not count indirect hosts (#15801)
* By default, do not count indirect hosts
* Fix copy paste goof
* Fix linter issue from base branch
* prevent multiple tasks from processing the same job events, prevent p… (#15805)
prevent multiple tasks from processing the same job events, prevent periodic task from spawning another task per job
* Fix typos and other bugs found by Pablo review
* fix: rely on resolved_action instead of task, adapt to proposed query… (#15815)
* fix: rely on resolved_action instead of task, adapt to proposed query structure
* tests: update indirect host tests
* update remaining queries to new format
* update live test
* Remove polling loop for job finishing event processing (#15811)
* Remove polling loop for job finishing event processing
* Make awx/main/tests/live dramatically faster (#15780)
* AAP-37282 Add parse JQ data and test it for a `job` object in isolation (#15774)
* Add jq dependency
* Add file in progress
* Add license for jq
* Write test and get it passing
* Successfully test collection of `event_query.yml` data (#15761)
* Callback plugin method from cmeyers adapted to global collection list
Get tests passing
Mild rebranding
Put behind feature flag, flip true in dev
Add noqa flag
* Add missing wait_for_events
* feat: try grabbing query files from artifacts directory (#15776)
* Contract changes for the event_query collection callback plugin (#15785)
* Minor import changes to collection processing in callback plugin
* Move agreed location of event_query file
* feat: remaining schema changes for indirect host audits (#15787)
* Re-organize test file and move artifacts processing logic to callback (#15784)
* Rename the indirect host counting test file
* Combine artifacts saving logic
* Connect host audit model to jq logic via new task
* Document, implement, and test remaining indirect host audit fields (#15796)
* AAP-39559 Wait for all event processing to finish, add fallback task (#15798)
* Wait for all event processing to finish, add fallback task
* Add flag check to periodic task
* feat: cleanup of old indirect host audit records (#15800)
* prevent multiple tasks from processing the same job events, prevent p… (#15805)
prevent multiple tasks from processing the same job events, prevent periodic task from spawning another task per job
* Remove polling loop for job finishing event processing (#15811)
* Make awx/main/tests/live dramatically faster (#15780)
* reorder migrations to allow indirect instances backport
* cleanup for rebase and merge into devel
---------
Co-authored-by: Peter Braun <pbraun@redhat.com>
Co-authored-by: jessicamack <jmack@redhat.com>
Co-authored-by: Peter Braun <pbranu@redhat.com>
This is to emphasize that this role is specific
to controller component. That is, not an auditor
for the entire AAP platform.
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
Adds the following managed Role Definitions
Controller Team Admin
Controller Team Member
Controller Organization Admin
Controller Organization Member
These have the same permission set as the
platform roles (without the Controller prefix)
Adding members to teams and orgs via the legacy RBAC system
will use these role definitions.
Other changes:
- Bump DAB to 2024.08.22
- Set ALLOW_LOCAL_ASSIGNING_JWT_ROLES to False in defaults.py.
This setting prevents assignments to the platform roles (e.g. Team Member).
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
* Add test that we got all permissions right for every role
* Fix missing Org execute role and missing adhoc role permission
* Add in missing Organization Approval Role as well
* Remove Role from role names
* Add initial test for deletion of stale permission
* Delete existing EE view permission
* Hypothetically complete update of EE model permissions setup
* Tests passing locally
* Issue with user_capabilities was a test bug, fixed
* Add migration testing for certain managed roles
* Fix managed role bugs
* Add more tests
* Fix another bug with org workflow admin role reference
* Add test because another issue is fixed
* Mark reason for test
* Remove internal markers
* Reword failure message
Co-authored-by: Seth Foster <fosterseth@users.noreply.github.com>
---------
Co-authored-by: Seth Foster <fosterseth@users.noreply.github.com>
Support for AWS SNS notifications. SNS is a widespread service that is used to integrate with other AWS services(EG lambdas). This support would unlock use cases like triggering lambda functions, especially when AWX is deployed on EKS.
Decisions:
Data Structure
- I preferred using the same structure as Webhook for message body data because it contains all job details. For now, I directly linked to Webhook to avoid duplication, but I am open to suggestions.
AWS authentication
- To support non-AWS native environments, I added configuration options for AWS secret key, ID, and session tokens. When entered, these values are supplied to the underlining boto3 SNS client. If not entered, it falls back to the default authentication chain to support the native AWS environment. Properly configured EKS pods are created with temporary credentials that the default authentication chain can pick automatically.
---------
Signed-off-by: Ethem Cem Ozkan <ethemcem.ozkan@gmail.com>
* Adding CSRF Validation for schemas
* Changing retrieve of scheme to avoid importing new library
* check if CSRF_TRUSTED_ORIGINS exists before accessing it
---------
Signed-off-by: Bruno Sanchez <brsanche@redhat.com>
* Add new enablement settings from DAB RBAC
* Initial implementation of system auditor as role without testing
* Fix system auditor role, remove duplicate assignments
* Make the system auditor role managed
* Flake8 fix
* Remove another thing from old solution
* Fix a few test failures
* Add extra setting to disable custom system roles via API
* Add test for custom role prohibition
Develop ability to list permissions for existing roles
Create a model registry for RBAC-tracked models
Write the data migration logic for creating
the preloaded role definitions
Write migration to migrate old Role into ObjectRole model
This loops over the old Role model, knowing it is unique
on object and role_field
Most of the logic is concerned with identifying the
needed permissions, and then corresponding role definition
As needed, object roles are created and users then teams
are assigned
Write re-computation of cache logic for teams
and then for object role permissions
Migrate new RBAC internals to ansible_base
Migrate tests to ansible_base
Implement solution for visible_roles
Expose URLs for DAB RBAC
InstanceLink target should not be null.
Should be safe to set to null=False, because we have
a custom RunPython method to explicitly set
target to a proper key.
Also, add new test to test_migrations
which ensures data integrity after migrating
the receptor address model changes.
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
- Add forwards method to create a receptor address
for any existing Instance that has listener_port defined
- Add forwards method to modify each InstanceLink object
that changes target to the newly created receptor addresses
This migration was implemented as follows:
1. Add a target_new to InstanceLink which is a foreign key
to ReceptorAddress
2. create receptor addresses
3. link to these receptor addresses using the target_new field
4. rename target_new to target
5. drop listener_port and peers_from_control_nodes from Instance
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
Removes ability to directly create and delete
receptor addresses for a given node.
Instead, receptor addresses are created automatically
if listener_port is set on the Instance.
For example patching "hop" instance
with {"listener_port": 6667}
will create a canonical receptor address with port
6667.
Likewise, peers_from_control_nodes on the instance
sets the peers_from_control_nodes on the canonical
address (if listener port is also set).
protocol is a read-only field that simply reflects
the canonical address protocol.
Other Changes:
- rename k8s_routable to is_internal
- add protocol to ReceptorAddress
- remove peers_from_control_nodes and listener_port
from Instance model
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
Creates a non-deletable address that acts as
the "main" address for this instance.
All other addresses for that instance must
be non-canonical.
When listener_port on an instance is set, automatically
create a canonical receptor address where:
- address is hostname of instance
- port is listener_port
- canonical is True
Additionally, protocol field is added to instance to
denote the receptor listener protocol to use (ws, tcp).
The receptor config listener information is derived from
the listener_port and protocol information. Having a
canonical address that mirrors the listener_port ensures that
an address exists that matches the receptor config information.
Other changes:
- Add managed field to receptor address.
If managed is True, no fields on on this address can be edited
via the API.
If canonical is True, only the address cannot be edited.
- Add managed field to instance. If managed is True, users
cannot set node_state to deprovisioning (i.e. cannot delete node)
This change to our mechanism to prevent users from deleting
the mesh ingress hop node.
- Field is_internal is now renamed to k8s_routable
- Add reverse_peers on instance which is a list of instance IDs
that peer to this instance (via an address)
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
Make receptoraddress list views
searchable by "address"
Other changes:
- Add help text to source and target of the
InstanceLink model
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
- Add receptor_address module which allows
users to create addresses for instances
- Update awx_collection functional and integration
tests to support new peering design
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
- cannot peer to self
- cannot peer to instance that is already peered to self
Other changes:
- ReceptorAddress protocol field restricted to choices: tcp, ws, wss
- fix awx-manage list_instances when instance.last_seen is None
- InstanceLink make source and target unique together
- Add help text to the ReceptorAddress fields
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
- websocket_path can only be set if protocol is ws
- is_internal must be False
- only 1 address per instance can have
peers_from_control_nodes set to True
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
register_peers has inputs:
source: source instance
peers: list of instances the source should peer to
InstanceLink "target" is now expected to be a ReceptorAddress
For each peer, we can just use the first receptor address. If
multiple receptor addresses exist, throw a command error.
Currently this command is only used on VM-deployments, where
there is only a single receptor address per instance, so this
should work fine.
Other changes:
drop listener_port field from Instance. Listener port is now just
"port" on ReceptorAddress
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
- write_receptor_config peers to ReceptorAddress entries
that have peers_from_control_nodes enabled
- peers_from_control_nodes and listener_port removed from Instance model
- peers_from_control_nodes added to ReceptorAddress model
- ReceptorAddress is now unique by address and protocol combination
- Write receptor config task is dispatched upon ReceptorAddress creation
or deletion, and when control node is first created
- InstanceLinkSerializer adds a target_address field and has logic
to grab the instance hostname associated with the peered ReceptorAddress
Signed-off-by: Seth Foster <fosterbseth@gmail.com>
- Add database contraints to make sure addresses
are unique
If port is defined:
address, port, protocol, websocket_path are unique together
if port is not defined:
address, protocol, websocket_path are unique together
- Allow deleting address via API
- Add ReceptorAddressAccess to determine permissions
- awx-manage add_receptor_address returns changed: True
if successful
Add support for receiving webhooks from Bitbucket Data Center, and add support for posting build statuses back
Note that this is very explicitly only for Bitbucket Data Center.
The entire webhook format and API is entirely different for Bitbucket Cloud.
* allow pytest --migrations to succeed
* We actually subvert migrations from running in test via pytest.ini
--no-migrations option. This has led to bit rot for the sqlite
migrations happy path. This changeset pays off that tech debt and
allows for an sqlite migration happy path.
* This paves the way for programatic invocation of individual migrations
and weaving of the creation of resources (i.e. Instance, Job Template,
etc). With this, a developer can instantiate various database states,
trigger a migration, assert the state of the db, and then have pytest
rollback all of that.
* I will note that in practice, running these migrations is dog shit
slow BUT this work also opens up the possibility of saving and
re-using sqlite3 database files. Normally, caching is not THE answer
and causes more harm than good. But in this case, our migrations are
mostly write-once (I say mostly because this change set violates
that :) so cache invalidation isn't a major issue.
* functional test for migrations on sqlite
* We commonly subvert running migrations in test land. Test land uses
sqlite. By not constantly exercising this code path it atrophies. The
smoke test here is to continuously exercise that code path.
* Add ci test to run migration tests separately, they take =~ 2-3
minutes each on my laptop.
* The smoke tests also serves as an example of how to write migration
tests.
* run migration tests in ci
