Cache downloaded roles & collections

Populate the cache the first time the job is run for a revision
that needs them, and for future runs for that revision just
copy it into the private directory.

Delete the cache on project deletion.

Invalidate the cache on a new project revision

Also download roles/collections during the sync job

Since we're writing into a per-revision cache, we can do this easily now.

Don't try and install content if there aren't any requirements expecting it

Adjust pathing to the proper location.

Force install if doing a manual sync.

Requirements may be unversioned.

Remove the cache when delete-on-update is set

Integrate content caching with existing task logic

Revert the --force flags

use the update id as metric for role caching

Shift the movement of cache to job folder from rsync task to python

Only install roles and collections if needed

Deal with roles and collections for jobs without sync
Skip local copy if roles or collections turned off

update docs for content caching

Design pivot - use empty cache dir to indicate lack of content

Do not cache content if we did not install content

Test changes to allay concerns about reliability of local_path

Do not blow away cache for SCM inventory updates

Remove project update vars no longer used

Remove job pre-creation of content folders

code style edit, always use cache_id as property in tasks

Fix log message

docs/collections.md

@@ -4,15 +4,18 @@ AWX supports the use of Ansible Collections. This section will give ways to use
 
 ### Project Collections Requirements
 
-If you specify a Collections requirements file in SCM at `collections/requirements.yml`,
-then AWX will install Collections from that file in the implicit project sync
-before a job run. The invocation looks like:
+If you specify a collections requirements file in SCM at `collections/requirements.yml`,
+then AWX will install collections from that file to a special cache folder in project updates.
+Before a job runs, the roles and/or collections will be copied from the special
+cache folder to the job temporary folder.
+
+The invocation looks like:
 
 ```
-ansible-galaxy collection install -r requirements.yml -p <job tmp location>/requirements_collections
+ansible-galaxy collection install -r requirements.yml -p <project cache location>/requirements_collections
 ```
 
-Example of the resultant `tmp` directory where job is running:
+Example of the resultant job `tmp` directory where job is running:
 
 ```
 ├── project
@@ -20,7 +23,7 @@ Example of the resultant `tmp` directory where job is running:
 │   └── debug.yml
 ├── requirements_collections
 │   └── ansible_collections
-│       └── username
+│       └── collection_namespace
 │           └── collection_name
 │               ├── FILES.json
 │               ├── MANIFEST.json
@@ -53,6 +56,33 @@ Example of the resultant `tmp` directory where job is running:
 
 ```
 
+### Cache Folder Mechanics
+
+Every time a project is updated as a "check" job
+(via `/api/v2/projects/N/update/` or by a schedule, workflow, etc.),
+the roles and collections are downloaded and saved to the project's content cache.
+In other words, the cache is invalidated every time a project is updated.
+That means that the `ansible-galaxy` commands are ran to download content
+even if the project revision does not change in the course of the update.
+
+Project updates all initially target a staging directory at a path like:
+
+```
+/var/lib/awx/projects/.__awx_cache/_42__project_name/stage
+```
+
+After the update finishes, the task logic will decide what id to associate
+with the content downloaded.
+Then the folder will be renamed from "stage" to the cache id.
+For instance, if the cache id is determined to be 63:
+
+```
+/var/lib/awx/projects/.__awx_cache/_42__project_name/63
+```
+
+The cache may be updated by project syncs (the "run" type) which happen before
+job runs. It will populate the cache id set by the last "check" type update.
+
 ### Galaxy Server Selection
 
 Ansible core default settings will download collections from the public