JT-branch docs and code cleanup

bump migration

fine tune validation of project allow_override
  return highly custom error message

Restore branch after syncs to address bugs
  encountered after changing scm_refspec

remove unused code to determine scm_revision

Check Ansible version before project update and
  do not install collections if Ansible version too old

Add docs related to project branch override
  New file specific to branch override and refspec
Complete docs on collections to reflect current
  implementation and give a folder tree example
Update clustering docs related to project syncs

Fix bug where git depth was ignored during the
  local clone from project folder to run folder

Fix bug where submodules were not copied

docs/clustering.md

@@ -272,7 +272,7 @@ If an Instance Group is configured but all instances in that group are offline o
 
 #### Project synchronization behavior
 
-Project updates behave differently than they did before. Previously they were ordinary jobs that ran on a single instance. It's now important that they run successfully on any instance that could potentially run a job. Projects will sync themselves to the correct version on the instance immediately prior to running the job. If the needed revision is already locally available and galaxy or collections updates are not needed, then a sync may not be performed.
+Project updates behave differently than they did before. Previously they were ordinary jobs that ran on a single instance. It's now important that they run successfully on any instance that could potentially run a job. Projects will sync themselves to the correct version on the instance immediately prior to running the job. If the needed revision is already locally checked out and galaxy or collections updates are not needed, then a sync may not be performed.
 
 When the sync happens, it is recorded in the database as a project update with a `launch_type` of "sync" and a `job_type` of "run". Project syncs will not change the status or version of the project; instead, they will update the source tree _only_ on the instance where they run. The only exception to this behavior is when the project is in the "never updated" state (meaning that no project updates of any type have been run), in which case a sync should fill in the project's initial revision and status, and subsequent syncs should not make such changes.
 

docs/collections.md

@@ -1,6 +1,62 @@
-## Collections Support
+## Collections
 
-AWX supports Ansible collections by appending the directories specified in `AWX_ANSIBLE_COLLECTIONS_PATHS` 
+AWX supports using Ansible collections.
+This section will give ways to use collections in job runs.
+
+### Project Collections Requirements
+
+If you specify a collections requirements file in SCM at `collections/requirements.yml`,
+then AWX will install collections in that file in the implicit project sync
+before a job run. The invocation is:
+
+```
+ansible-galaxy collection install -r requirements.yml -p <job tmp location>
+```
+
+Example of tmp directory where job is running:
+
+```
+├── project
+│   ├── ansible.cfg
+│   └── debug.yml
+├── requirements_collections
+│   └── ansible_collections
+│       └── username
+│           └── collection_name
+│               ├── FILES.json
+│               ├── MANIFEST.json
+│               ├── README.md
+│               ├── roles
+│               │   ├── role_in_collection_name
+│               │   │   ├── defaults
+│               │   │   │   └── main.yml
+│               │   │   ├── tasks
+│               │   │   │   └── main.yml
+│               │   │   └── templates
+│               │   │       └── stuff.j2
+│               └── tests
+│                   └── main.yml
+├── requirements_roles
+│   └── username.role_name
+│       ├── defaults
+│       │   └── main.yml
+│       ├── meta
+│       │   └── main.yml
+│       ├── README.md
+│       ├── tasks
+│       │   ├── main.yml
+│       │   └── some_role.yml
+│       ├── templates
+│       │   └── stuff.j2
+│       └── vars
+│           └── Archlinux.yml
+└── tmp_6wod58k
+
+```
+
+### Global Collections Path
+
+AWX appends the directories specified in `AWX_ANSIBLE_COLLECTIONS_PATHS`
 to the environment variable `ANSIBLE_COLLECTIONS_PATHS`. The default value of `AWX_ANSIBLE_COLLECTIONS_PATHS`
-contains `/var/lib/awx/collections`. It is recommended that place your collections that you wish to call in 
+contains `/var/lib/awx/collections`. It is recommended that place your collections that you wish to call in
 your playbooks into this path.

docs/job_branch_override.md

@@ -0,0 +1,66 @@
+## Job Branch Specification
+
+Background: Projects specify the branch to use from source control
+in the `scm_branch` field.
+
+This feature allows project admins to delegate branch selection to
+admins of job templates that use that project (requiring only project
+`use_role`). Admins of job templates can further
+delegate that ability to users executing the job template
+(requiring only job template `execute_role`) by enabling
+`ask_scm_branch_on_launch` on the job template.
+
+### Source Tree Copy Behavior
+
+Every job run has its own private data directory. This folder is temporary,
+cleaned up at the end of the job run.
+
+This directory contains a copy of the project source tree for the given
+branch the job is running.
+
+A new copy is made for every job run.
+
+This folder will not contain the full git history for git project types.
+
+### Git Refspec
+
+The field `scm_refspec` has been added to projects. This is provided by
+the user or left blank.
+
+A non-blank `scm_refspec` field will cause project updates (of any type)
+to pass the `refspec` field when running the Ansible
+git module inside of the `project_update.yml` playbook. When the git module
+is provided with this field, it performs an extra `git fetch` command
+to pull that refspec from the remote.
+
+The refspec specifies what references the update will download from the remote.
+Examples:
+
+ - `refs/*:refs/remotes/origin/*`
+    This will fetch all references, including even remotes of the remote
+ - `refs/pull/*:refs/remotes/origin/pull/*`
+    Github-specific, this will fetch all refs for all pull requests
+ - `refs/pull/62/head:refs/remotes/origin/pull/62/head`
+    This will fetch only the ref for that one github pull request
+
+For large projects, users should consider performance when
+using the first or second examples here. For example, if a github project
+has over 40,000 pull requests, that refspec will fetch them all
+during a project update.
+
+This parameter affects availability of the project branch, and can allow
+access to references not otherwise available. For example, the third example
+will allow the user to use the branch `refs/pull/62/head`, which would
+not be possible without the refspec field.
+
+The Ansible git module always fetches `refs/heads/*`. It will do this
+whether or not a custom refspec is provided. This means that a project's
+branches and tags (and commit hashes therein) can be used as `scm_branch`
+no matter what is used for `scm_refspec`.
+
+The `scm_refspec` will affect which `scm_branch` fields can be used as overrides.
+For example, you could set up a project that allows branch override with a refspec
+of `refs/pull/*:refs/remotes/origin/pull/*`, then use this in a job template
+that prompts for `scm_branch`, then a client could launch the job template when
+a new pull request is created, providing the branch `refs/pull/N/head`,
+then the job template would run against the provided github pull request reference.