External-Mirrors/awx
2
0
Fork 0
mirror of https://github.com/ansible/awx.git synced 2026-01-09 23:12:08 -03:30
Code Issues Packages Projects Releases Wiki Activity

remove a bunch of RabbitMQ references

Browse Source
This commit is contained in:
Ryan Petrello 2020-03-24 18:34:20 -04:00
parent bd7c048113
commit 8f1db173c1
No known key found for this signature in database
GPG Key ID: F2AA5F2122351777
10 changed files with 11 additions and 83 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

11
CONTRIBUTING.md
View File

@ -155,12 +155,11 @@ If you start a second terminal session, you can take a look at the running conta
(host)$ docker ps
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
aa4a75d6d77b gcr.io/ansible-tower-engineering/awx_devel:devel "/tini -- /bin/sh ..." 23 seconds ago Up 15 seconds 0.0.0.0:5555->5555/tcp, 0.0.0.0:7899-7999->7899-7999/tcp, 0.0.0.0:8013->8013/tcp, 0.0.0.0:8043->8043/tcp, 22/tcp, 0.0.0.0:8080->8080/tcp tools_awx_1
e4c0afeb548c postgres:10 "docker-entrypoint..." 26 seconds ago Up 23 seconds 5432/tcp tools_postgres_1
0089699d5afd tools_logstash "/docker-entrypoin..." 26 seconds ago Up 25 seconds tools_logstash_1
4d4ff0ced266 memcached:alpine "docker-entrypoint..." 26 seconds ago Up 25 seconds 0.0.0.0:11211->11211/tcp tools_memcached_1
92842acd64cd rabbitmq:3-management "docker-entrypoint..." 26 seconds ago Up 24 seconds 4369/tcp, 5671-5672/tcp, 15671/tcp, 25672/tcp, 0.0.0.0:15672->15672/tcp tools_rabbitmq_1
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
44251b476f98 gcr.io/ansible-tower-engineering/awx_devel:devel "/entrypoint.sh /bin…" 27 seconds ago Up 23 seconds 0.0.0.0:6899->6899/tcp, 0.0.0.0:7899-7999->7899-7999/tcp, 0.0.0.0:8013->8013/tcp, 0.0.0.0:8043->8043/tcp, 0.0.0.0:8080->8080/tcp, 22/tcp, 0.0.0.0:8888->8888/tcp tools_awx_run_9e820694d57e
b049a43817b4 memcached:alpine "docker-entrypoint.s…" 28 seconds ago Up 26 seconds 0.0.0.0:11211->11211/tcp tools_memcached_1
40de380e3c2e redis:latest "docker-entrypoint.s…" 28 seconds ago Up 26 seconds 0.0.0.0:6379->6379/tcp tools_redis_1
b66a506d3007 postgres:10 "docker-entrypoint.s…" 28 seconds ago Up 26 seconds 0.0.0.0:5432->5432/tcp tools_postgres_1
```
**NOTE**

12
INSTALL.md
View File

@ -128,7 +128,6 @@ For convenience, you can create a file called `vars.yml`:
```
admin_password: 'adminpass'
pg_password: 'pgpass'
rabbitmq_password: 'rabbitpass'
secret_key: 'mysupersecret'
```
@ -555,16 +554,7 @@ $ ansible-playbook -i inventory -e docker_registry_password=password install.yml
### Post-install
After the playbook run completes, Docker will report up to 5 running containers. If you chose to use an existing PostgresSQL database, then it will report 4. You can view the running containers using the `docker ps` command, as follows:
```bash
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
e240ed8209cd awx_task:1.0.0.8 "/tini -- /bin/sh ..." 2 minutes ago Up About a minute 8052/tcp awx_task
1cfd02601690 awx_web:1.0.0.8 "/tini -- /bin/sh ..." 2 minutes ago Up About a minute 0.0.0.0:443->8052/tcp awx_web
55a552142bcd memcached:alpine "docker-entrypoint..." 2 minutes ago Up 2 minutes 11211/tcp memcached
84011c072aad rabbitmq:3 "docker-entrypoint..." 2 minutes ago Up 2 minutes 4369/tcp, 5671-5672/tcp, 25672/tcp rabbitmq
97e196120ab3 postgres:9.6 "docker-entrypoint..." 2 minutes ago Up 2 minutes 5432/tcp postgres
```
After the playbook run completes, Docker starts a series of containers that provide the services that make up AWX. You can view the running containers using the `docker ps` command.
If you're deploying using Docker Compose, container names will be prefixed by the name of the folder where the docker-compose.yml file is created (by default, `awx`).

13
awx/main/management/commands/deprovision_instance.py
View File

@ -1,8 +1,6 @@
# Copyright (c) 2016 Ansible, Inc.
# All Rights Reserved
import subprocess
from django.db import transaction
from django.core.management.base import BaseCommand, CommandError
@ -33,18 +31,9 @@ class Command(BaseCommand):
with advisory_lock('instance_registration_%s' % hostname):
instance = Instance.objects.filter(hostname=hostname)
if instance.exists():
isolated = instance.first().is_isolated()
instance.delete()
print("Instance Removed")
if isolated:
print('Successfully deprovisioned {}'.format(hostname))
else:
result = subprocess.Popen("rabbitmqctl forget_cluster_node rabbitmq@{}".format(hostname), shell=True).wait()
if result != 0:
print("Node deprovisioning may have failed when attempting to "
"remove the RabbitMQ instance {} from the cluster".format(hostname))
else:
print('Successfully deprovisioned {}'.format(hostname))
print('Successfully deprovisioned {}'.format(hostname))
print('(changed: True)')
else:
print('No instance found matching name {}'.format(hostname))

2
awx/main/tests/unit/test_tasks.py
View File

@ -126,10 +126,8 @@ def test_send_notifications_list(mock_notifications_filter, mock_job_get, mocker
@pytest.mark.parametrize("key,value", [
('REST_API_TOKEN', 'SECRET'),
('SECRET_KEY', 'SECRET'),
('RABBITMQ_PASS', 'SECRET'),
('VMWARE_PASSWORD', 'SECRET'),
('API_SECRET', 'SECRET'),
('CALLBACK_CONNECTION', 'amqp://tower:password@localhost:5672/tower'),
('ANSIBLE_GALAXY_SERVER_PRIMARY_GALAXY_PASSWORD', 'SECRET'),
('ANSIBLE_GALAXY_SERVER_PRIMARY_GALAXY_TOKEN', 'SECRET'),
])

4
awx/ui_next/src/screens/Host/data.hostFacts.json
View File

@ -102,10 +102,6 @@
"INVENTORY_ID": "1",
"MAX_EVENT_RES": "700000",
"PROOT_TMP_DIR": "/tmp",
"RABBITMQ_HOST": "rabbitmq",
"RABBITMQ_PASS": "guest",
"RABBITMQ_USER": "guest",
"RABBITMQ_VHOST": "/",
"ANSIBLE_LIBRARY": "/awx_devel/awx/plugins/library",
"SDB_NOTIFY_HOST": "docker.for.mac.host.internal",
"AWX_GROUP_QUEUES": "tower",

4
awx/ui_next/src/screens/Inventory/shared/data.hostFacts.json
View File

@ -102,10 +102,6 @@
"INVENTORY_ID": "1",
"MAX_EVENT_RES": "700000",
"PROOT_TMP_DIR": "/tmp",
"RABBITMQ_HOST": "rabbitmq",
"RABBITMQ_PASS": "guest",
"RABBITMQ_USER": "guest",
"RABBITMQ_VHOST": "/",
"ANSIBLE_LIBRARY": "/awx_devel/awx/plugins/library",
"SDB_NOTIFY_HOST": "docker.for.mac.host.internal",
"AWX_GROUP_QUEUES": "tower",

6
awx/ui_next/src/screens/Job/shared/data.job.json
View File

@ -121,16 +121,13 @@
"job_env": {
"HOSTNAME": "awx",
"MAKEFLAGS": "w",
"RABBITMQ_USER": "guest",
"OS": "Operating System: Docker for Mac",
"LC_ALL": "en_US.UTF-8",
"RABBITMQ_VHOST": "/",
"SDB_HOST": "0.0.0.0",
"MAKELEVEL": "2",
"VIRTUAL_ENV": "/venv/ansible",
"MFLAGS": "-w",
"PATH": "/venv/ansible/bin:/venv/awx/bin:/venv/awx/bin:/usr/local/n/versions/node/10.15.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
"RABBITMQ_PASS": "**********",
"SUPERVISOR_GROUP_NAME": "tower-processes",
"PWD": "/awx_devel",
"LANG": "\"en-us\"",
@ -142,7 +139,6 @@
"AWX_GROUP_QUEUES": "tower",
"SUPERVISOR_SERVER_URL": "unix:///tmp/supervisor.sock",
"SUPERVISOR_PROCESS_NAME": "awx-dispatcher",
"RABBITMQ_HOST": "rabbitmq",
"CURRENT_UID": "501",
"_": "/venv/awx/bin/python3",
"DJANGO_SETTINGS_MODULE": "awx.settings.development",
@ -191,4 +187,4 @@
"task_count": 1
},
"custom_virtualenv": "/venv/ansible"
}
}

2
docs/ansible_runner_integration.md
View File

@ -7,7 +7,7 @@ Much of the code in AWX around ansible and `ansible-playbook` invocation has bee
In AWX, a task of a certain job type is kicked off (_i.e._, RunJob, RunProjectUpdate, RunInventoryUpdate, etc.) in `tasks.py`. A temp directory is built to house `ansible-runner` parameters (_i.e._, `envvars`, `cmdline`, `extravars`, etc.). The `temp` directory is filled with the various concepts in AWX (_i.e._, `ssh` keys, `extra vars`, etc.). The code then builds a set of parameters to be passed to the `ansible-runner` Python module interface, `ansible-runner.interface.run()`. This is where AWX passes control to `ansible-runner`. Feedback is gathered by AWX via callbacks and handlers passed in.
The callbacks and handlers are:
* `event_handler`: Called each time a new event is created in `ansible-runner`. AWX will dispatch the event to `rabbitmq` to be processed on the other end by the callback receiver.
* `event_handler`: Called each time a new event is created in `ansible-runner`. AWX will dispatch the event to `redis` to be processed on the other end by the callback receiver.
* `cancel_callback`: Called periodically by `ansible-runner`; this is so that AWX can inform `ansible-runner` if the job should be canceled or not.
* `finished_callback`: Called once by `ansible-runner` to denote that the process that was asked to run is finished. AWX will construct the special control event, `EOF`, with the associated total number of events that it observed.
* `status_handler`: Called by `ansible-runner` as the process transitions state internally. AWX uses the `starting` status to know that `ansible-runner` has made all of its decisions around the process that it will launch. AWX gathers and associates these decisions with the Job for historical observation.

39
docs/clustering.md
View File

@ -19,7 +19,6 @@ It's important to point out a few existing things:
* PostgreSQL is still a standalone instance and is not clustered. Replica configuration will not be managed. If the user configures standby replicas, database failover will also not be managed.
* All instances should be reachable from all other instances and they should be able to reach the database. It's also important for the hosts to have a stable address and/or hostname (depending on how you configure the Tower host).
* RabbitMQ is the cornerstone of Tower's Clustering system. A lot of AWX's configuration requirements and behavior are dictated by its needs. For this reason, it is generally inflexible to customize beyond what the setup playbook allows. Each AWX/Tower instance has a deployment of RabbitMQ which will cluster with the other instances' RabbitMQ instances.
* Existing old-style HA deployments will be transitioned automatically to the new HA system during the upgrade process to 3.1.
* Manual projects will need to be synced to all instances by the customer.
@ -29,7 +28,6 @@ Ansible Tower 3.3 adds support for container-based clusters using Openshift or K
## Important Changes
* There is no concept of primary/secondary in the new Tower system. *All* systems are primary.
* Set up playbook changes to configure RabbitMQ and give hints to the type of network the hosts are on.
* The `inventory` file for Tower deployments should be saved/persisted. If new instances are to be provisioned, the passwords and configuration options as well as host names will need to be available to the installer.
@ -67,38 +65,6 @@ hostC
hostDB
```
* It's common for customers to provision Tower instances externally but prefer to reference them by internal addressing. This is most significant for RabbitMQ clustering, where the service isn't available at all on an external interface. Because of this, it is necessary to assign the internal address for RabbitMQ links as such:
```
[tower]
hostA rabbitmq_host=10.1.0.2
hostB rabbitmq_host=10.1.0.3
hostC rabbitmq_host=10.1.0.3
```
* The `redis_password` field is removed from `[all:vars]`.
* There are various new fields for RabbitMQ:
- `rabbitmq_port=5672` - RabbitMQ is installed on each instance and is not optional, it's also not possible to externalize. It is possible to configure what port it listens on and this setting controls that.
- `rabbitmq_vhost=tower` - Tower configures a rabbitmq virtualhost to isolate itself. This controls that setting.
- `rabbitmq_username=tower` and `rabbitmq_password=tower` - Each instance will be configured with these values and each instance's Tower instance will be configured with it also. This is similar to our other uses of usernames/passwords.
- `rabbitmq_cookie=<somevalue>` - This value is unused in a standalone deployment but is critical for clustered deployments. This acts as the secret key that allows RabbitMQ cluster members to identify each other.
- `rabbitmq_use_long_names` - RabbitMQ is pretty sensitive to what each instance is named. We are flexible enough to allow FQDNs (_host01.example.com_), short names (`host01`), or IP addresses (192.168.5.73). Depending on what is used to identify each host in the `inventory` file, this value may need to be changed. For FQDNs and IP addresses, this value needs to be `true`. For short names it should be `false`
- `rabbitmq_enable_manager` - Setting this to `true` will expose the RabbitMQ management web console on each instance.
The most important field to point out for variability is `rabbitmq_use_long_name`. This cannot be detected and no reasonable default is provided for it, so it's important to point out when it needs to be changed. If instances are provisioned to where they reference other instances internally and not on external addresses, then `rabbitmq_use_long_name` semantics should follow the internal addressing (*i.e.*, `rabbitmq_host`).
Other than `rabbitmq_use_long_name`, the defaults are pretty reasonable:
```
rabbitmq_port=5672
rabbitmq_vhost=tower
rabbitmq_username=tower
rabbitmq_password=''
rabbitmq_cookie=cookiemonster
# Needs to be true for fqdns and ip addresses
rabbitmq_use_long_name=false
rabbitmq_enable_manager=false
```
Recommendations and constraints:
- Do not create a group named `instance_group_tower`.
@ -238,7 +204,6 @@ Tower itself reports as much status as it can via the API at `/api/v2/ping` in o
* The instance servicing the HTTP request.
* The last heartbeat time of all other instances in the cluster.
* The RabbitMQ cluster status.
* Instance Groups and Instance membership in those groups.
A more detailed view of Instances and Instance Groups, including running jobs and membership
@ -252,7 +217,7 @@ Each Tower instance is made up of several different services working collaborati
* **HTTP Services** - This includes the Tower application itself as well as external web services.
* **Callback Receiver** - Receives job events that result from running Ansible jobs.
* **Celery** - The worker queue that processes and runs all jobs.
* **RabbitMQ** - A Message Broker, this is used as a signaling mechanism for Celery as well as any event data propagated to the application.
* **Redis** - this is used as a queue for AWX to process ansible playbook callback events.
* **Memcached** - A local caching service for the instance it lives on.
Tower is configured in such a way that if any of these services or their components fail, then all services are restarted. If these fail sufficiently (often in a short span of time), then the entire instance will be placed offline in an automated fashion in order to allow remediation without causing unexpected behavior.
@ -262,7 +227,7 @@ Tower is configured in such a way that if any of these services or their compone
Ideally a regular user of Tower should not notice any semantic difference to the way jobs are run and reported. Behind the scenes it is worth pointing out the differences in how the system behaves.
When a job is submitted from the API interface, it gets pushed into the Dispatcher queue on RabbitMQ. A single RabbitMQ instance is the responsible master for individual queues, but each Tower instance will connect to and receive jobs from that queue using a fair-share scheduling algorithm. Any instance on the cluster is just as likely to receive the work and execute the task. If an instance fails while executing jobs, then the work is marked as permanently failed.
When a job is submitted from the API interface, it gets pushed into the dispatcher queue via postgres notify/listen (https://www.postgresql.org/docs/10/sql-notify.html), and the task is handled by the dispatcher process running on that specific Tower node. If an instance fails while executing jobs, then the work is marked as permanently failed.
If a cluster is divided into separate Instance Groups, then the behavior is similar to the cluster as a whole. If two instances are assigned to a group then either one is just as likely to receive a job as any other in the same group.

1
tools/docker-compose-cluster.yml
View File

@ -14,7 +14,6 @@ services:
- "8013:8013"
- "8043:8043"
- "1936:1936"
- "15672:15672"
awx-1:
user: ${CURRENT_UID}
container_name: tools_awx_1_1