External-Mirrors/awx
2
0
Fork 0
mirror of https://github.com/ansible/awx.git synced 2026-02-24 06:26:00 -03:30
Code Issues Packages Projects Releases Wiki Activity

More AWX docs edits

Browse Source
This commit is contained in:
beeankha
2019-09-17 15:49:01 -04:00
parent e2be392f31
commit 860715d088
26 changed files with 860 additions and 499 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
docs/inventory/README.md Normal file
View File

@@ -0,0 +1 @@
This folder contains documentation related to inventories in AWX / Ansible Tower.

89
docs/inventory/inventory_plugins.md Normal file
View File

@@ -0,0 +1,89 @@
# Transition to Ansible Inventory Plugins
Inventory updates have changed from using scripts, which are vendored as executable Python scripts, to using dynamically-generated YAML files which conform to the specifications of the `auto` inventory plugin. These are then parsed by their respective inventory plugin.
The major organizational change is that the inventory plugins are part of the Ansible core distribution, whereas the same logic used to be a part of AWX source.
## Prior Background for Transition
AWX used to maintain logic that parsed `.ini` inventory file contents, in addition to interpreting the JSON output of scripts, re-calling with the `--host` option in cases where the `_meta.hostvars` key was not provided.
### Switch to Ansible Inventory
The CLI entry point `ansible-inventory` was introduced in Ansible 2.4. In Tower 3.2, inventory imports began running this command as an intermediary between the inventory and the import's logic to save content to the database. Using `ansible-inventory` eliminates the need to maintain source-specific logic, relying on Ansible's code instead. This also enables consistent data structure output from `ansible-inventory`. There are many valid structures that a script can provide, but the output from `ansible-inventory` will always be the same, thus the AWX logic to parse the content is simplified. This is why even scripts must be ran through the `ansible-inventory` CLI.
Along with this switchover, a backported version of `ansible-inventory` was provided, which supports Ansible versions 2.2 and 2.3.
### Removal of Backport
In AWX 3.0.0 (and Tower 3.5), the backport of `ansible-inventory` was removed, and support for using custom virtual environments was added. This set the minimum version of Ansible necessary to run _any_ inventory update to 2.4.
## Inventory Plugin Versioning
Beginning in Ansible 2.5, inventory sources in Ansible started migrating away from `contrib` scripts (meaning they lived in the `contrib` folder) to the inventory plugin model.
In AWX 4.0.0 (and Tower 3.5) inventory source types start to switch over to plugins, provided that sufficient compatibility is in place for the version of Ansible present in the local virtualenv.
To see in which version the plugin transition will happen, see `awx/main/models/inventory.py` and look for the source name as a subclass of `PluginFileInjector`, and there should be an `initial_version`, which is the first version that was deemed (via testing) to have sufficient parity in the content for its inventory plugin returns. For example, `openstack` will begin using the inventory plugin in Ansible version 2.8. If you run an OpenStack inventory update with Ansible 2.7.x or lower, it will use the script.
The eventual goal is for all source types to have moved to plugins. For any given source, after the `initial_version` for plugin use is higher than the lowest supported Ansible version, the script can be removed and the logic for script credential injection will also be removed.
For example, after AWX no longer supports Ansible 2.7, the script `awx/plugins/openstack_inventory.py` will be removed.
## Changes to Expect in Imports
An effort was made to keep imports working in the exact same way after the switchover. However, the inventory plugins are a fundamental rewrite and many elements of default behavior have changed. These changes also include many backward-incompatible changes. Because of this, what you get via an inventory import will be a superset of what you get from the script but will not match the default behavior you would get from the inventory plugin on the CLI.
Due to the fact that inventory plugins add additional variables, if you downgrade Ansible, you should turn on `overwrite` and `overwrite_vars` to get rid of stale variables (and potentially groups) no longer returned by the import.
### Changes for Compatibility
Programatically-generated examples of inventory file syntax used in updates (with dummy data) can be found in `awx/main/tests/data/inventory/scripts`. These demonstrate the inventory file syntax used to restore old behavior from the inventory scripts.
#### Hostvar Keys and Values
More `hostvars` will appear if the inventory plugins are used. To maintain backward compatibility, the old names are added back where they have the same meaning as a variable returned by the plugin. New names are not removed.
A small number of `hostvars` will be lost because of general deprecation needs.
#### Host Names
In many cases, the host names will change. In all cases, accurate host tracking will still be maintained via the host `instance_id`.
## Writing Your Own Inventory File
If you do not want any of this compatibility-related functionality, then you can add an SCM inventory source that points to your own file. You can also apply a credential of a `managed_by_tower` type to that inventory source that matches the credential you are using, as long as it is not `gce` or `openstack`.
All other sources provide _secrets_ via environment variables. These can be re-used without any problems for SCM-based inventory, and your inventory file can be used securely to specify non-sensitive configuration details such as the `keyed_groups` (to provide) or `hostvars` (to construct).
## Notes on Technical Implementation of Injectors
For an inventory source with a given value of the `source` field that is of the built-in sources, a credential of the corresponding credential type is required in most cases (ec2 IAM roles are an exception). This privileged credential is obtained by the method `get_cloud_credential`.
The `inputs` for this credential constitute one source of data for running inventory updates. The following fields from the `InventoryUpdate` model are also data sources, including:
- `source_vars`
- `source_regions`
- `instance_filters`
- `group_by`
The way this data is applied to the environment (including files and environment vars) is highly dependent on the specific source.
With plugins, the inventory file may reference files that contain secrets from the credential. With scripts, typically an environment variable will reference a filename that contains a ConfigParser format file with parameters for the update, and possibly including fields from the credential.
**Caution:** Please do not put secrets from the credential into the inventory file for the plugin. Right now there appears to be no need to do this, and by using environment variables to specify secrets, this keeps open the possibility of showing the inventory file contents to the user as a latter enhancement.
Logic for setup for inventory updates using both plugins and scripts live in the inventory injector class, specific to the source type.
Any credentials which are not source-specific will use the generic injection logic which is also used in playbook runs.

165
docs/inventory/inventory_refresh.md Normal file
View File

@@ -0,0 +1,165 @@
# Inventory Refresh Overview
Tower should have an inventory view that is more aligned towards systems management
rather than merely maintaining inventory for automation.
## Inventory Source Promotion
Starting with Tower 3.2, `InventorySource` will be associated directly with an `Inventory`.
### API Concerns / Deprecation
The `group` field for `InventorySource` has been renamed to `deprecated_group` and will be removed
from `InventorySource` completely in Tower 3.3. As a result the related field on `Group` ,
`inventory_source` has been renamed `deprecated_inventory_source` and will also be removed in Tower 3.3.
## Fact Searching
Facts generated by an Ansible playbook during a Job Template run are stored by Tower into the database
whenever `use_fact_cache=True` is set per-Job-Template. New facts are merged with existing
facts and are per-host. These stored facts can be used to filter hosts via the
`/api/v2/hosts` endpoint, using the GET query parameter `host_filter` *i.e.*,
`/api/v2/hosts?host_filter=ansible_facts__ansible_processor_vcpus=8`
The grammar of `host_filter` allows for:
* grouping via `()`
* the boolean `and` operator
* `__` to reference related fields in relational fields
* `__` is used on `ansible_facts` to separate keys in a JSON key path
* `[]` is used to denote a JSON array in the path specification
* `""` can be used in the value when spaces are utilized
* "classic" Django queries may be embedded in the `host_filter`
Examples:
```
/api/v2/hosts/?host_filter=name=localhost
/api/v2/hosts/?host_filter=ansible_facts__ansible_date_time__weekday_number="3"
/api/v2/hosts/?host_filter=ansible_facts__ansible_processor[]="GenuineIntel"
/api/v2/hosts/?host_filter=ansible_facts__ansible_lo__ipv6[]__scope="host"
/api/v2/hosts/?host_filter=ansible_facts__ansible_processor_vcpus=8
/api/v2/hosts/?host_filter=ansible_facts__ansible_env__PYTHONUNBUFFERED="true"
/api/v2/hosts/?host_filter=(name=localhost or name=database) and (groups__name=east or groups__name="west coast") and ansible_facts__ansible_processor_vcpus=8
```
## Smart Inventory
Starting in Tower 3.2, Tower will support the ability to define a _Smart Inventory_.
Users will define the inventories using the same language that is currently supported
in _Smart Search_.
### Inventory Changes
* The `Inventory` model has a new field called `kind`. The default of this field will be blank
for normal inventories and set to `smart` for smart inventories.
* `Inventory` model has a new field called `host_filter`. The default of this field will be blank
for normal inventories. When `host_filter` is set AND the inventory `kind` is set to `smart`, this combination makes a _Smart Inventory_.
* `Host` model has a new field called `smart_inventories`. This field uses the `SmartInventoryMemberships`
lookup table to provide a set of all of the _Smart Inventory_ a host is a part of. The memberships
are generated by the `update_host_smart_inventory_memberships` task. The task is launched when:
* New Host is added.
* Existing Host is changed (update/delete).
* New Smart Inventory is added.
* Existing Smart Inventory is changed (update/delete).
* **NOTE:** This task is only run if the `AWX_REBUILD_SMART_MEMBERSHIP` is set to `True`. It defaults to `False`.
### Smart Filter (`host_filter`)
The `SmartFilter` class handles our translation of the smart search string. We store the
filter value in the `host_filter` field for an inventory. This value should be expressed
the same way for existing smart searches.
host_filter="search=foo"
host_filter="group__search=bar"
host_filter="search=baz and group__search=bang"
host_filter="name=localhost or group__name=local"
Creating a new _Smart Inventory_ for all of our GCE and EC2 groups might look like this:
HTTP POST /api/v2/inventories/
{
"name": "GCE and EC2 Smart Inventory",
"kind": "smart",
"host_filter": "group__search=ec2 and group__search=gce"
...
}
### More On Searching
The `host_filter` that is set will search over the entirety of the hosts the user has
access to in Tower. If the user wants to restrict their search in anyway, they will
want to declare that in their host filter.
For example, if wanting to restrict the search to only hosts in an inventory
named "US-East", create a `host_filter` that looks something like this:
{
"name": "NYC Hosts",
"kind": "smart",
"host_filter": "inventory__name='US-East' and search='nyc'",
...
}
In the above example, the search is limited to the "US-East" inventory and
hosts with a name containing "nyc".
### Acceptance Critera
When verifying acceptance, ensure the following statements are true:
``
* `Inventory` has a new field named `kind` that defaults to empty and
can only be set to `smart`.
* `Inventory` has a new field named `host_filter` to empty and can only be
set to a valid _SmartFilter_ string.
* `Inventory` with a `host_filter` set and a `kind` of `smart`:
* `hosts` list reflecting the results of searching `/api/v2/hosts` with the same
search that is set in the `host_filter`.
* Not allow creation of Hosts
* Not allow creation of Groups
* Not allow creation of Inventory Sources
### API Concerns
There are no breaking or backwards-incompatible changes for this feature.
## Other Changes
### Inventory Updates All `inventory_sources`
A new endpoint `/api/v2/inventories/:id/update_inventory_sources` has been added. This endpoint
functions in the same way that `/api/v2/inventory_source/:id/update` functions for a single
`InventorySource` with the exception that it updates all of the inventory sources for the
`Inventory`.
`HTTP GET /api/v2/inventories/:id/update_inventory_sources` will list all of the inventory
sources and whether or not they will be updated when a POST to the same endpoint is made. The result of
this request will look like this:
{
results: [
"inventory_source": 1, "can_update": True,
"inventory_source": 2, "can_update": False,
]
}
> *Note:* All manual inventory sources (`source=''`) will be ignored by the `update_inventory_sources` endpoint.
When making a POST to the same endpoint, the response will contain a status as well as the job ID for the update:
POST /api/v2/inventories/:id/update_inventory_sources
{
results: [
"inventory_update": 20, "inventory_source": 1, "status": "started",
"inventory_update": 21, "inventory_source": 2, "status": "Could not start because `can_update` returned False"
]
}
The response code from this action will be:
- 200 if all inventory source updates were successful
- 202 if some inventory source updates were successful, but some failed
- 400 if all of the inventory source updates failed
- 400 if there are no inventory sources in the inventory
### Background Deletion of Inventory
If a DELETE request is submitted to an inventory, the field `pending_delete` will be True until a separate task fully completes the task of deleting the inventory and all its contents.

173
docs/inventory/scm_inventory.md Normal file
View File

@@ -0,0 +1,173 @@
# SCM Inventory
Users can create inventory sources that use content in the source tree of
a project as an Ansible inventory source.
## Usage Details
Fields that should be specified on creation of SCM inventory source:
- `source_project` - project to use
- `source_path` - relative path inside of the project indicating a
directory or a file; if left blank, `""` is still a relative path
indicating the root directory of the project
- the `source` field should be set to `"scm"`
Additionally:
- `source_vars` - if these are set on a "file" type inventory source
then they will be passed to the environment vars when running
- `update_on_project_update` - if set, a project update of the source
project will automatically update this inventory source as a side effect
If `update_on_project_update` is not set, then they can manually update
just the inventory source with a POST to its update endpoint,
`/inventory_sources/N/update/`.
If `update_on_project_update` is set, the POST to the inventory source's
update endpoint will trigger an update of the source project, which may,
in turn, trigger an update of the inventory source.
Also, with this flag set, an update _of the project_ is
scheduled immediately after creation of the inventory source.
Also, if this flag is set, no inventory updates will be triggered
_unless the SCM revision of the project changes_.
### RBAC
The user needs the `use` role for the project in order to utilize it as a source
project for the inventory (this entails permission to run arbitrary scripts).
To update the project, they need `update` permission to the project,
even if the update is done indirectly.
### Inventory File Suggestions
The project should show a listing of suggested inventory locations at the `/projects/N/inventories/` endpoint, but this is not a comprehensive list of
all paths that could be used as an Ansible inventory because of the wide
range of inclusion criteria. The list will also max out at 50 entries.
The user should be allowed to specify a location manually in the UI.
This listing should be refreshed to the latest SCM info on a project update.
If no inventory sources use a project as an SCM inventory source, then
the inventory listing may not be refreshed on update.
### Still-to-come 3.2 Changes
As a part of a different feature, it is planned to have all inventory sources
inside of an inventory all update with a single button click. When this
happens for an inventory containing an SCM inventory source, it should
update the project.
### Inventory Source Restriction
Since automatic inventory updates (triggered by a project update) do not
go through the task system, typical protection against conflicting updates
is not available. To avoid problems, only one inventory source is allowed for
inventories that use this feature. That means that if an inventory source
has `source=scm` and `update_on_project_update=true`, it can be the only
inventory source for its inventory.
## Supported File Syntax
> Any Inventory Ansible supports should be supported by this feature.
This is accomplished by making use of the `ansible-inventory` command.
The inventory import `tower-manage` command will check for the existence
of `ansible-inventory` and if it is not present, it will call a backported
version of it. The backport is maintained as its own GPL3 licensed
repository.
https://github.com/ansible/ansible-inventory-backport
Because the internal mechanism is different, there needs to be some coverage
testing with Ansible versions pre-2.4 and after.
### Vars Restrictions
When creating any `scm` type inventory source, the `overwrite_vars` field
must be set to `true`. This should be enforced by API validation and
the UI should also force this setting.
Why? This is because `ansible-inventory` is planned to
return group vars at the group-level in its JSON output, but the backported
script returns them on the host-level. In Ansible 2.4, inventory modules are
being refactored into plugins, and showing vars on the group-level depends on
this. Since it is not possible to _also_ backport the inventory module
refactor to prior Ansible versions, this discrepancy can not be resolved.
While "flattening" the group vars down to the host-level is functionally
equivalent, upgrading Ansible would cause an anomaly in variable precedence.
This future variable precedence problem is avoided by forcing overwriting
of variables until Ansible 2.3 is deprecated, after which all updates
will consistently utilize group-level variables.
# Acceptance Criteria Use Cases
Some test scenarios to look at:
- Test projects that use scripts
- Test projects that have multiple inventory files in a directory,
`group_vars`, `host_vars`, etc.
- Test scripts in the project repo
- Test scripts that use environment variables provided by a credential
in Tower
- Test multiple inventories that use the same project, pointing to different
files / directories inside of the project
- Feature works correctly even if project doesn't have any playbook files
- File-related errors should surface as inventory import failures
+ missing file
+ invalid syntax in file
- If the project SCM update encounters errors, it should not run the
inventory updates
# Notes for Official Documentation
The API guide should summarize what is in the use details.
Once the UI implementation is done, the product docs should cover its
standard use.
## Update-On-Launch
If the SCM inventory source is configured to follow the project updates,
the `update_on_launch` field can not to be set to True. This is because
of concerns related to the task manager job dependency tree.
Below are some alternatives methods which allow a user to accomplish the same thing in a different way:
### Alternative 1: Use same project for playbook
A user can make a job template that uses a project as well as an inventory
that updates from that same project. In this case, they can set the project
to `update_on_launch`, in which case it will trigger an inventory update
if needed.
### Alternative 2: Use the project in a workflow
If a user must utilize a different project for the playbook than for the inventory
source, then they can still place the project in a workflow and then have
a job template run on success of the project update.
This is guaranteed to have the inventory update "on time" (this means
that the inventory changes are complete before the job template is launched),
because the project does not transition to the completed state
until the inventory update is finished.
Note that a failed inventory update does not mark the project as failed.
## Restricting Instance Group to Run Script On
Since SCM inventory sources are running scripts written by people with
access to the source-control, a user may want to restrict which instance
groups the inventory update runs on.
If the inventory source is set to update on project update, it will run
on the same instance (inside of the Tower cluster) as the project update.
This can be restricted by limiting the instance groups of the organization
that contains the `source_project` of the inventory source.
If the inventory source is not configured to update on project update,
then it will inherit the allowed instance groups from its inventory,
like all other inventory syncs.