this fixes several scenarios where certain POST endpoints don't show up
in our generated Swagger doc; /api/swagger is only
discoverable/accessible in the development environment where we generate
the schema
This has a few benefits:
1. It makes adding new fields to built-in CredentialTypes _much_
simpler. In the past, we've had to write a migration every time we
want to modify an existing type (changing a label/help text,
changing options like the recent become_method changes) or
when adding a new field entirely
2. It paves the way for third party credential plugins support, where
importable libraries will define their own source code-based schema
run this command on _any_ node in an awx cluster:
$ awx-manage profile_sql --threshold=2.0 --minutes=1
...and for 1 minute, the timing for _every_ SQL query in _every_ awx
Python process that uses the Django ORM will be measured
queries that run longer than (in this example) 2 seconds will be
written to a per-process sqlite database in /var/lib/awx/profile, and
the file will contain an EXPLAIN VERBOSE for the query and the full
Python stack that led to that SQL query's execution (this includes not
just WSGI requests, but background processes like the runworker and
dispatcher)
$ awx-manage profile_sql --threshold=0
...can be used to disable profiling again (if you don't want to wait for
the minute to expire)
if we don't preload this column, Django needs it, and so it generates
one query per-host (!!!) to get it. For large (10k+ host) inventories,
this is incredibly slow.
see: https://github.com/ansible/awx/issues/3214
under the hood, Host.ansible_facts is a postgres jsonb field which
performs match operations using the JSON containment operator (@>)
this operator _only_ works on exact matches on containment (i.e.,
"does the `ansible_distribution` jsonb value contain _this exact_ JSON
structure"):
SELECT ...
FROM main_host
WHERE ansible_facts @> '{"ansible_distribution": "centos"}'
SELECT ...
FROM main_host
WHERE ansible_facts @> '{"packages": {"dnsmasq": [{"version": 2}]}}'
postgres does _not_ expose any operator for fuzzy or lookup-based
matches with this operator, so host filter values like these don't
really make sense (postgres can't _filter_ in the way intended in these
examples):
ansible_distribution__startswith=\"Cent\"
ansible_distribution__icontains=\"CentOS\"
ansible_facts__packages__dnsmasq[]__version__startswith=\"2\"
a recent change (65641c7) made it so that we call
ansible-playbook using its _absolute path_ (i.e.,
/usr/bin/ansible-playbook, /var/lib/venv/xyz/bin/ansible-playbook), so
this logic is no longer correct
we shouldn't call signal.disconnect in __del__ because it can lead to
deadlocks in Django signal dispatch code
The Signal.connect, Signal.disconnect, and Signal._live_receivers
methods all share a threading.Lock():
https://github.com/django/django/blob/22a60f8d0b331bf06c066ccba4eea5bb5e4ac9f2/django/dispatch/dispatcher.py#L49
It's possible for this to lead to a deadlock:
1. Have code that calls Signal._live_receivers and enter the critical
path inside the shared threading.Lock()
2. Python garbage collection occurs and finds one or more LDAPBackend
objects with no more references
3. This __del__ is called, which calls Signal.disconnect
4. Code in Signal._disconnect attempts to obtain the (already held)
threading.Lock
5. Python hangs forever while attempting to garbage collect
this code was added to detect celerybeat shelve .db corruption, but it caused a different issue; opening the shelve in this way puts a write lock on it, which means that when we attempt to open it _again_ moments later, we can't.
when we're done checking the validity of the file, we need to close it
once we merge in runner support for isolated environments, we can
revert this commit (because we'll always run isolated code using python3
executables)
this _technically_ prevents a remote code exploit where a user who has
access to publish AMQP messages to the dispatch queue could craft
a special message that would import and run arbitrary Python functions;
that said, the types of user with this privilege level are generally
_already_ the awx user (so they can already do this by hand if they
want)
if a user has an active session that just sits on the dashboard or job
list, websocket messages that come in (for e.g., job status changes)
will trigger AJAX requests for more data; this process causes a user
with an idle login to continue to generate API requests, which in turn
ticks their expiry timer. As a result, users with active sessions
sitting on these two (popular) pages will never be automatically logged
out via SESSION_MAX_AGE.
this change introduces a special header that the UI can use to signify
that a request shouldn't bump the expiry timer
this is a simple sanity check, but it should help us avoid shooting
ourselves in the foot in complicated scenarios, such as:
1. A dispatcher worker is running a job, and it's killed with `kill -9`
2. The dispatcher attempts to reap jobs with a matching celery_task_id
3. The associated sync project update has the *same* celery_task_id
(an implementation detail of how we implemented that), and it ends
up getting reaped _even though_ it's already finished and has
status=successful
reading capacity values using the jsonfile cache plugin is more robust
in scenarios where ansible-playbook may print non-JSON output (such as
-vvv or when a custom callback plugin like timer is enabled)
1. Install awx w/ a single node.
2. Start a long-running job.
3. Forcibly kill the `awx-manage run_dispatcher` process (e.g.,
SIGKILL) and do not start it again.
4. The job remains in running - without a second cluster to discover
the job, it is never reaped.
5. This PR allows you to cancel the job from the UI+API.
it looks like choices can also be a list and _maybe_ comma delimited;
clearly there's a lot of history here; let's verify and test what's _really_ supported and _then_ add any necessary validation
when `--reload` is sent to the dispatcher, it sends a special QUIT
message to each worker in the pool so that it will exit gracefully at
the next opportunity
when a worker process exits unexpectedly, the dispatcher attempts to
recover its queued messages and sends them to another worker in the
pool; in this scenario, we should _never_ re-enqueue these special
QUIT messages (because the process doesn't need to quit, it's already
gone)
To reproduce this race condition:
1. Launch an adhoc that does `sleep 60`
2. Run `awx-manage run_dispatcher --reload` to enqueue a `QUIT` message
into the worker's queue
3. Find the pid of the worker running the `sleep 60` and `SIGKILL` it.
4. Observe that dispatcher attempts to requeue the `QUIT` message and
logs a confusing error.
Ryan Petrello