* Fix race with heartbeat and reaper logic
* Fix tests to fail when over drift over heartbeat time
* replaced modified with started time for reap() code and added test
* fixed logic bug and cleaned up tests
* Added comments to tests to call out reasoning
The class that contained these tests wasn't named Test*, so the tests in
it weren't running. Fix that and fix the tests in it so that they pass.
Signed-off-by: Rick Elrod <rick@elrod.me>
* Adds support for a pseudolocalization query param to check to see whether a string has been marked for translation
Adds support for a pseudolocalization query param to check to see whether a string has been marked for translation
* Adds support for passing a lang param to force rendering in a particular language
* Remove unused import
* adding roles to instance groups
added ResourceMixin to Instancegroup and changed the filtered_queryset
* added necessary changes to rebuild relationship between IG and roles
* added description to InstanceGroupAccess
* preliminary ui plug for demo purposes
* preliminary ui plug for demo purposes
added inventory special logic for use_role to allow attaching instance groups
added more tests to handle those cases
* Add access_list to InstanceGroup
* scratch branch to test migration work
* refactored to shorten logic
* Added migration and am removing logic that enabled Org admin permissions
* Add Obj admin role to JT, Inv, Org
* Changed tests to reflect new permissions
* refactored some of the tests
* cleaned up more tests and reworded help on InstanceGroupAccess
* Removed unnecessary delete of Route for instance group perms change
* Fix UI tests and migration
* fixed permissions on prompt for InstanceGroups
* added related object roles endpoint
* added ui/api function for options instance_groups
* separate the migrations in order to avoid issues with migrations not being finished
* changed migrations parent class to disable the activity stream error in migrations
* Added logging to migration as activitystream is disabled
* added clarifying comment to jobtemlateaccess and linted UI addition
* renamed migrations to avoid collisions
* Rename migrations to avoid collisions
MOVE the config template v1 to v2
delete other v1 views since v1 is deleted
the host fact gather collection over time was removed
also the job start view was removed
Insights integration was changed and the host insights
view no longer exists
Slightly modernize config help
- periodically ping postres on port 5432 and only start
migrations if successful.
- prevents crash loop when attempting migrations before
postgres is ready.
linting
linting again
Use the correct role on org permission check
Co-authored-by: Elijah DeLee <kdelee@redhat.com>
Update docs/bulk_api.md
Co-authored-by: Elijah DeLee <kdelee@redhat.com>
Update docs/bulk_api.md
Co-authored-by: Elijah DeLee <kdelee@redhat.com>
Update awx/main/access.py
Co-authored-by: Elijah DeLee <kdelee@redhat.com>
Update awx/main/access.py
Co-authored-by: Elijah DeLee <kdelee@redhat.com>
Update docs/bulk_api.md
Co-authored-by: Alan Rominger <arominge@redhat.com>
fix collection test (#19)
improve readability of through model object creation (#18)
lower num jobs/hosts in tests (#20)
we can test query scaling at lower numbers, to reduce
load in tests. We suspect this was causing some flake
in the tests on PRs
adjust the num of queries
* Bulk launch serializer RBAC and code structure review
Use WJ node as base in bulk job launch child
remove fields we get for free this way
Minor translation marking
Consolidate bulk API permission methods
split out permission check for each UJT type
Code consolidation for org check method
add a save before starting the workflow job
Make the max host default 100. We are seeing with moderate number of hosts i.e. 500 hosts having a few host variable each runs into max size of nginx message and nginx rejects the request.
we are therefor keeping the value small so that it doesn't fail with decent number of host variables as well.
remove the 999 hosts test because the default max is 100
fix the credential check
fix the instance groups and execution env permission checks
* evaluate max bulk settings in validate...
instead of in class attribute. This makes them load at request time
instead of at app start up time, which fixes problems with test
as well as I think will be better user experience if admins
actually do change the setting it will apply without restarting
django app on each instance
* improve OPTIONS by not manually declaring feilds
alan pointed this out
* fix access problems and add Add bulk job max settings to api
filter workflow job nodes better
This will both improve performance by limiting the queryset for the node
sublists as well as fix our access problem.
override can_read instead of modify queryset in access.py
We do this because we are not going to expose bulk jobs to the list
views, which is complicatd and has poor performance implications.
Instead, we just care about individual Workflows that clients get linked
to not being broken.
fix comment
remove the get functions from the conf.py for bulk api max value
comment the api expose of the bulk job variables
reformt conf.py with make black
trailing space
add more assertion to the bulk host create test
check label permission and fix lint (#13)
* set created by and launch type correctly
This makes "launched_by" get computed right in the tests.
Mysteriously this seemed to work from API browser, but
this seems more correct to have it work this way, and makes
tests actually work.
For "manual" launch types the attribute used to populate "launched_by"
is "created_by". And we already have "is_bulk_job" to indicate that the
job is a bulk job. So lets just use this.
* check label is in an organization you can read
* add assertions around access to resulting job
there is a problem getting the job w/ the user that launched it
add more assertions to bulk tests (#11)
dig more into the results and assert on results
also, use a fixture that already implemented the "max queries" thing
fix ansible collection sanity tests (#12)
Various fixes
- Don't skip checking resource RBAC permissions for admins
Necessary to handle bad input, e.g. providing a
unified_job_template id that doesn't exit
- In awxkit, only "walk" if we get 'url' in the result
- Bulk host create should return url pointing to inventory,
not inventory/hosts
dont do org check for superuser
fix the api-lint
fix the api-lint
add the descrition to the bulk job launch module params
add the description for the description field
add the description for the description field
add docs for the bulk api
fix the models on the bulk api serializers
fix some of the issues highlighted in the code review
better use of role model
remove comments
better error message
revert the PrimaryKeyRelatedField for unified_job_template and inventory
Enabled the params bulk job
make black
make black again
Fixed inventory and organization input params for bulk modules
add collection integration tests
Fix cli return errors
fix test completeness
return more context for bulk host create
now return list of minimal info about host objects
[
{
"name": "lakjdsafoiaweirnladlk",
"enabled": true,
"instance_id": "",
"description": "",
"variables": "",
"id": 4593,
"url": "/api/v2/hosts/4593/",
"inventory": "/api/v2/inventories/1/"
}
]
Updated tests, but needed to work around some weird behavior with
sqlite. Apparently it behaves differently around assigning ID's to the
result of bulk_create and that is messed up my use of `reverse` to look
up the url of the hosts
make black changes
increase the number of queries to 30
fix the flake failure
add functional changes for bulk job launch and some minor fixes
pull changes
we needed to inherit from GenericAPIView to get the options to render
correctly
q!
add execution env support
add organization validation to the workflowjob
Update awx/api/serializers.py
Co-authored-by: Elijah DeLee <kdelee@redhat.com>
Update awx/api/serializers.py
Co-authored-by: Elijah DeLee <kdelee@redhat.com>
Provide a view that allows users to launch many jobs with one POST
request. Under the hood, this creates a workflow with a number of jobs
all in a "flat" structure -- much like a sliced job, but with arbitrary
"joblets".
For ~ 100 nodes looking at ~ 200 some queries, which is more than the
proof of concept, but still an order of magnitude better than individual
job launches.
Still more work to implement other m2m objects, and need to address what
Organization should be assigned to a WorkflowJob launched by a BulkJob.
They need this so they can step into the workflow_job_nodes and get the
status of all the containing jobs.
Also want to test when there are MANY job templates etc in the system
because the querires like
UnifiedJobTemplate.accessible_pk_qs(request.user, 'execute_role').all()
queries scare me, seems like it could be a lot of things.
use "many=True" instead of ListField
Still seeing identical number of queries when creatin 100 jobs, going to
investigate more
only validate type in nested serializer
then, we actually get the database object after we do the RBAC checks
This drops us down from hundreds of queries to launch 100 jobs,
to less than 100 queries to launch 100 jobs (I got around 24 queries to
launch 100 jobs with credentials)
pave way for more promptable things
add "limit" as possible prompt on launch to bulk jobs
re-organize how we add credentials to pave way for the other m2m items
not having to repeat too much code
add labels to the bulk job
add the other fields to the workflowjobnode
move urls around
allow system admins, org admins, and inventory admins to bulk create
hosts.
Testing on an "open" licensed awx as system admin, I created 1000 hosts with 6 queries in ~ 0.15 seconds
Testing on an "open" licensed awx as organization admin, I created 1000 hosts with 11 queries in ~ 0.15 seconds
fix org max host check
also only do permission denied if license is a trial
add /api/v2/bulk to list bulk apis available
add api description templates
One motiviation to not take a list of hosts with mixed inventories is to
keep things simple re: RBAC and keeping a constant number of queries.
If there is great clamor for accepting list of hosts to insert into
arbitrary different inventories, we could probably make it happen - we'd
need to pop the inventory off of each of the hosts, run the
HostSerializer validate, then in top level BulkHostCreateSerializer
fetch all the inventories/check permissions/org host limits for those
inventories/etc. But that makes this that much more complicated.
add test for rbac access
test also helped me find a bug in a query, fixed that
add test to assert num queries scales as expected
also move other test to dedicated file
also test with super user like I meant to
record activity stream for the inventory
this records that a certain number of hosts were added by a certain user
we could consider if there is any other additional information we want
to include
