related: https://github.com/ansible/awx/issues/6010
as noted in the comment removed from this diff, it's probably time
to stop calling this function on every dispatcher service restart
* Do not query the database for the set of Instance that belong to the
group for which we are trying to fit a job on, for each job.
* Instead, cache the set of instances per-instance group.
* We update the parent unified job template to point at new jobs
created. We also update a similar foreign key when the job finishes
running. This causes lock contention when the job template is
allow_simultaneous and there are a lot of jobs from that job template
running in parallel. I've seen as bad as 5 minutes waiting for the lock
when a job finishes.
* This change moves the parent->child update to OUTSIDE of the
transaction if the job is allow_simultaneous (inherited from the parent
unified job). We sacrafice a bit of correctness for performance. The
logic is, if you are launching 1,000 parallel jobs do you really care
that the job template contains a pointer to the last one you launched?
Probably not. If you do, you can always query jobs related to the job
template sorted by created time.
* Do not query the database for the set of Instance that belong to the
group for which we are trying to fit a job on, for each job.
* Instead, cache the set of instances per-instance group.
* We update the parent unified job template to point at new jobs
created. We also update a similar foreign key when the job finishes
running. This causes lock contention when the job template is
allow_simultaneous and there are a lot of jobs from that job template
running in parallel. I've seen as bad as 5 minutes waiting for the lock
when a job finishes.
* This change moves the parent->child update to OUTSIDE of the
transaction if the job is allow_simultaneous (inherited from the parent
unified job). We sacrafice a bit of correctness for performance. The
logic is, if you are launching 1,000 parallel jobs do you really care
that the job template contains a pointer to the last one you launched?
Probably not. If you do, you can always query jobs related to the job
template sorted by created time.
the bigint migration removed the foreign key constraints for:
- host_id
- job_id (and projectupdate_id, etc...)
because of this, we don't really need to check explicitly for a host_id
IntegrityError anymore (because it won't occur)
additionally, while it's possible to insert an event with a mismatched
job_id now (for example, you can totally start a long-running job, and
delete the job record in the background using the ORM or psql), doing
so results in DoesNotExist errors in the code that handles the
playbook_on_stats events
* Before, we were re-writing `plugin:` when users updated the
InventorySource via the API. Now, we just override at run-time. This
makes for a more sane API interaction
Revert the --force flags
use the update id as metric for role caching
Shift the movement of cache to job folder from rsync task to python
Only install roles and collections if needed
Deal with roles and collections for jobs without sync
Skip local copy if roles or collections turned off
update docs for content caching
Design pivot - use empty cache dir to indicate lack of content
Do not cache content if we did not install content
Test changes to allay concerns about reliability of local_path
Do not blow away cache for SCM inventory updates
Populate the cache the first time the job is run for a revision
that needs them, and for future runs for that revision just
copy it into the private directory.
Delete the cache on project deletion.
