Merge pull request #10464 from AlanCoding/ee_docs

Update and expand docs/ folder for EEs

This has some new content about EE precedence, which I don't think we've documented anywhere else, thinking of @tvo318 here, content was developed by @jbradberry
(I think the numbers 2 and 3 in the global job default EE may still be subject to revision, just a warning)
@shanemcd #10324 is incorporated into this
This mentions @rebeccahhh's venv migration stuff, but I'm trying to write the absolute minimum possible while still mentioning migration.

Reviewed-by: Rebeccah Hunter <rhunter@redhat.com>
Reviewed-by: Alan Rominger <arominge@redhat.com>
Reviewed-by: Jeff Bradberry <None>

docs/clustering.md

@@ -181,13 +181,21 @@ If an Instance Group is configured but all instances in that group are offline o
 
 #### Project Synchronization Behavior
 
-Project updates behave differently than they did before. Previously they were ordinary jobs that ran on a single instance. It's now important that they run successfully on any instance that could potentially run a job. Projects will sync themselves to the correct version on the instance immediately prior to running the job. If the needed revision is already locally checked out and Galaxy or Collections updates are not needed, then a sync may not be performed.
+It is important that project updates run on the instance which prepares the ansible-runner private data directory.
+This is accomplished by a project sync which is done by the dispatcher control / launch process.
+The sync will update the source tree to the correct version on the instance immediately prior to transmitting the job.
+If the needed revision is already locally checked out and Galaxy or Collections updates are not needed, then a sync may not be performed.
 
 When the sync happens, it is recorded in the database as a project update with a `launch_type` of "sync" and a `job_type` of "run". Project syncs will not change the status or version of the project; instead, they will update the source tree _only_ on the instance where they run. The only exception to this behavior is when the project is in the "never updated" state (meaning that no project updates of any type have been run), in which case a sync should fill in the project's initial revision and status, and subsequent syncs should not make such changes.
 
+All project updates run with container isolation (like jobs) and volume mount to the persistent projects folder.
+
 #### Controlling Where a Particular Job Runs
 
-By default, a job will be submitted to the `tower` queue, meaning that it can be picked up by any of the workers.
+By default, a job will be submitted to the default queue (formerly the `tower` queue).
+To see the name of the queue, view the setting `DEFAULT_EXECUTION_QUEUE_NAME`.
+Administrative actions, like project updates, will run in the control plane queue.
+The name of the control plane queue is surfaced in the setting `DEFAULT_CONTROL_PLANE_QUEUE_NAME`.
 
 
 ##### How to Restrict the Instances a Job Will Run On

docs/execution_environments.md

@@ -0,0 +1,63 @@
+# Execution Environments
+
+All jobs use container isolation for environment consistency and security.
+Compliant images are referred to as Execution Environments (EE)s.
+For more information, see [ansible-runner EE docs](https://ansible-runner.readthedocs.io/en/latest/execution_environments.html)
+for information on how container isolation works, and see [ansible-builder docs](https://ansible-builder.readthedocs.io/en/latest/index.html)
+for instructions on how to build them.
+
+The Execution Environment model has an `image` field for the image identifier which will be used by jobs.
+The job details view will link to the execution environment that the job uses.
+
+## Creating and using EEs
+
+Users with an organization's `execution_environment_admin_role` can create new EEs in that organization.
+The RBAC rules follow standard rules for org-scoped resources.
+
+EEs without an organization (value is null in the API) are global EEs.
+Only superusers can create global EEs.
+These can become the global job default in certain circumstances.
+
+### Pre-created EEs
+
+Installers should run the `awx-manage register_default_execution_environments` command to pre-populate
+the system with some EEs defined in settings. This will create:
+
+ - the control plane EE - corresponding to the `CONTROL_PLANE_EXECUTION_ENVIRONMENT` setting
+ - global job EEs - all images enumerated in the `GLOBAL_JOB_EXECUTION_ENVIRONMENTS` setting
+
+These EEs are critical for system function, so this command must be ran for AWX to be functional.
+All EEs created by this command are global EEs.
+
+### Project Update EE Precedence
+
+Project updates will always use the control plane EE.
+
+### Job, Ad Hoc Commands, and inventory update EE Precedence
+
+Jobs will use the first available execution environment in this list:
+
+1. the `execution_environment` defined on the template (job template or inventory source) that spawned the job
+2. the `default_environment` defined on the project that the job uses
+3. the `default_environment` defined on the organization of the job
+4. the `default_environment` defined on the organization of the inventory the job uses
+5. the current `DEFAULT_EXECUTION_ENVIRONMENT` setting (configurable at `/api/v2/settings/jobs/`)
+6. Any image from the `GLOBAL_JOB_EXECUTION_ENVIRONMENTS` setting
+7. Any other global EE
+
+If more than one EE fits a criteria (applies for 6 and 7), then the most recently created one will be used.
+
+## Migrating from Custom Virtual Environments
+
+If you have installed dependencies inside of custom virtual environments in
+a prior release, then have a look at this series of commands for help migrating
+dependencies out of the venvs and into EEs.
+
+ - `awx-manage list_custom_venvs`
+ - `awx-manage custom_venv_associations`
+ - `awx-manage export_custom_venv`
+
+Follow those in order, and read the help text to see what arguments are necessary.
+
+Output from the `awx-manage export_custom_venv -q ..` command can
+be a starting point for writing an `ansible-builder` definition file.