under the hood, Host.ansible_facts is a postgres jsonb field which
performs match operations using the JSON containment operator (@>)
this operator _only_ works on exact matches on containment (i.e.,
"does the `ansible_distribution` jsonb value contain _this exact_ JSON
structure"):
SELECT ...
FROM main_host
WHERE ansible_facts @> '{"ansible_distribution": "centos"}'
SELECT ...
FROM main_host
WHERE ansible_facts @> '{"packages": {"dnsmasq": [{"version": 2}]}}'
postgres does _not_ expose any operator for fuzzy or lookup-based
matches with this operator, so host filter values like these don't
really make sense (postgres can't _filter_ in the way intended in these
examples):
ansible_distribution__startswith=\"Cent\"
ansible_distribution__icontains=\"CentOS\"
ansible_facts__packages__dnsmasq[]__version__startswith=\"2\"
a recent change (65641c7) made it so that we call
ansible-playbook using its _absolute path_ (i.e.,
/usr/bin/ansible-playbook, /var/lib/venv/xyz/bin/ansible-playbook), so
this logic is no longer correct
we shouldn't call signal.disconnect in __del__ because it can lead to
deadlocks in Django signal dispatch code
The Signal.connect, Signal.disconnect, and Signal._live_receivers
methods all share a threading.Lock():
https://github.com/django/django/blob/22a60f8d0b331bf06c066ccba4eea5bb5e4ac9f2/django/dispatch/dispatcher.py#L49
It's possible for this to lead to a deadlock:
1. Have code that calls Signal._live_receivers and enter the critical
path inside the shared threading.Lock()
2. Python garbage collection occurs and finds one or more LDAPBackend
objects with no more references
3. This __del__ is called, which calls Signal.disconnect
4. Code in Signal._disconnect attempts to obtain the (already held)
threading.Lock
5. Python hangs forever while attempting to garbage collect
this code was added to detect celerybeat shelve .db corruption, but it caused a different issue; opening the shelve in this way puts a write lock on it, which means that when we attempt to open it _again_ moments later, we can't.
when we're done checking the validity of the file, we need to close it
once we merge in runner support for isolated environments, we can
revert this commit (because we'll always run isolated code using python3
executables)
this _technically_ prevents a remote code exploit where a user who has
access to publish AMQP messages to the dispatch queue could craft
a special message that would import and run arbitrary Python functions;
that said, the types of user with this privilege level are generally
_already_ the awx user (so they can already do this by hand if they
want)
if a user has an active session that just sits on the dashboard or job
list, websocket messages that come in (for e.g., job status changes)
will trigger AJAX requests for more data; this process causes a user
with an idle login to continue to generate API requests, which in turn
ticks their expiry timer. As a result, users with active sessions
sitting on these two (popular) pages will never be automatically logged
out via SESSION_MAX_AGE.
this change introduces a special header that the UI can use to signify
that a request shouldn't bump the expiry timer
this is a simple sanity check, but it should help us avoid shooting
ourselves in the foot in complicated scenarios, such as:
1. A dispatcher worker is running a job, and it's killed with `kill -9`
2. The dispatcher attempts to reap jobs with a matching celery_task_id
3. The associated sync project update has the *same* celery_task_id
(an implementation detail of how we implemented that), and it ends
up getting reaped _even though_ it's already finished and has
status=successful
reading capacity values using the jsonfile cache plugin is more robust
in scenarios where ansible-playbook may print non-JSON output (such as
-vvv or when a custom callback plugin like timer is enabled)
1. Install awx w/ a single node.
2. Start a long-running job.
3. Forcibly kill the `awx-manage run_dispatcher` process (e.g.,
SIGKILL) and do not start it again.
4. The job remains in running - without a second cluster to discover
the job, it is never reaped.
5. This PR allows you to cancel the job from the UI+API.
it looks like choices can also be a list and _maybe_ comma delimited;
clearly there's a lot of history here; let's verify and test what's _really_ supported and _then_ add any necessary validation
when `--reload` is sent to the dispatcher, it sends a special QUIT
message to each worker in the pool so that it will exit gracefully at
the next opportunity
when a worker process exits unexpectedly, the dispatcher attempts to
recover its queued messages and sends them to another worker in the
pool; in this scenario, we should _never_ re-enqueue these special
QUIT messages (because the process doesn't need to quit, it's already
gone)
To reproduce this race condition:
1. Launch an adhoc that does `sleep 60`
2. Run `awx-manage run_dispatcher --reload` to enqueue a `QUIT` message
into the worker's queue
3. Find the pid of the worker running the `sleep 60` and `SIGKILL` it.
4. Observe that dispatcher attempts to requeue the `QUIT` message and
logs a confusing error.
this commit implements the bulk of `awx-manage run_dispatcher`, a new
command that binds to RabbitMQ via kombu and balances messages across
a pool of workers that are similar to celeryd workers in spirit.
Specifically, this includes:
- a new decorator, `awx.main.dispatch.task`, which can be used to
decorate functions or classes so that they can be designated as
"Tasks"
- support for fanout/broadcast tasks (at this point in time, only
`conf.Setting` memcached flushes use this functionality)
- support for job reaping
- support for success/failure hooks for job runs (i.e.,
`handle_work_success` and `handle_work_error`)
- support for auto scaling worker pool that scale processes up and down
on demand
- minimal support for RPC, such as status checks and pool recycle/reload
if you have a license that doesn't allow use of workflows, invalid
credentials yielded an HTTP 402; this commit changes the precedence
see: https://github.com/ansible/tower/issues/2950
If database connectivity is lost/interrupted in this block of celery
internals, beat is *not* smart enough to recover, and it gets stuck in
an endless fail loop. We don't _need_ to talk to the database here
anyways; just use settings.CLUSTER_HOST_ID to get what we need.
see: https://github.com/ansible/tower/issues/2957
Ryan Petrello