* Unit tests do not create CredentialType records for Credential
plugins. Instead, they explicitly instantiate CredentialType(s) for
Credential plugins. They rely on CredentialType.defaults[key] to do
so. This change makes sure custom_injectors get bolted onto the
created CredentialType.
* There isn't a great reason to allow the UI to edit these meta-data
fields that denote the last time an analytics job ran.
* The only reason I hesitate to mark them uneditable in the API is that
they are useful to change in order to influence when the jobs run.
Mostly for debug purposes or 1-off.
* Register all discovered CredentialType(s) after Django finishes
loading
* Protect parallel registrations using shared postgres advisory lock
* The down-side of this is that this will run when it does not need to,
adding overhead to the init process.
* Only register discovered credential types in the database IF
migrations have ran and are up-to-date.
* #egg _could_ be awx-plugins.some.other.provided.package
* Also point at ansible devel instead of a forked branch since the
entrypoints PR has now merged to devel
```
/var/lib/awx/venv/awx/lib64/python3.11/site-packages/_pytest/python.py:163:
PytestReturnNotNoneWarning: Expected None, but
awx/main/tests/unit/test_tasks.py::TestJobCredentials::test_custom_environment_injectors_with_boolean_extra_vars
returned ['successful', 0], which will be an error in a future version
of pytest. Did you mean to use `assert` instead of `return`?
```
* Dug into the git blame for this one
060585434a is the commit for any
historians. It was wrongfully carried over from a mock pexpect
implementation. Our new tests are nice. They don't go as far as trying
to run the task so they do not need to mock pexpect. That is why it is
safe to remove this code without finding it a new home.
* Check upstream django-ansible-base releases. If the version upstream
does not match the version we are pinned to then submit a PR with the
upstream version.
* Include a bit of context into the name of the delete function. The
HTTP_ added prepended string may be unexpected if Django's header
transformation isn't top of mind.
* Always output awx logs to a file via otel
* That log file can always be later replayed into a product that
supports otlp at a later date.
* Useful when you find a problem that you need a time series DB to help
find and solve.
* Useful if a community member or customer has a problem where a time
series db would be helpful. You can take a "remote" users log and
replay it locally for analysis.
* Otherwise, settings value changes bleeds over into other tests.
* Remove django.conf settings import so that we do not accidentally
forget to use the settings fixture.
* Before, the optional url prefix feature required calling our
versioning version of reverse(). This worked _ok_ until we added more
and more urls from 3rd party apps. Those 3rd party apps do not call
our reverse(), writefully so.
* This implementation looks at the incoming request path. If it includes
the special optional prefix url, then we register ALL the urls WITH
the optional url prefix.
If the incoming request path does NOT contain the options url prefix
then we register ALL the urls WITHOUT the optional url prefix.
* Before this, we were registering BOTH sets of urls and then reverse()
+ the request as context to decide which url.
* Middleware classes can be instantiated multiple times in testing. To
make this a non-issue, move the init code for named urls out of the
middleware init and into the app init.
* This makes it easier to use other testing facilities, like
LiveServerTestCase, without having to mock the named url middleware
init.
* We didn't really make use of json formatting across the app. Remove
the special case json formatter. Instead, output all of the meta-data
associated with a job lifecycle event every time. Before, we tried to
only output this extra meta data when in DEBUG mode. It turns out this
information is smaller than we thought and more useful than we thought
so always output it.
* Previously, the params were passed without quotes and each directory
was being interpreted as a seperate command line flag.
* Added some structure around the error messages returned from
receptorctl so we can more easily decide how to handle each case. For
example, releasing the cleanup job from receptor doesn't absolutely
need to succeed because we have a periodic job that does that. In
fact, that is the thing that is making it fail .. but I digress.
* No harm in adding it to the list. If a JWT auth header is provided,
then process it (valid or not). If a JWT is not provided, move on to
the next auth.
* authorized/not authorized tests for wsrelay endpoint
* not authorized test for web browser websockets
* skeleton of a test for authorized web browser websockets
* Channels doesn't really give you an interface to support per-endpoint
auth ... so this adds one.
* The web browser and node <--> node communication have different auth
needs.
* Previously, the nginx location would match on /foo/websocket... or
/foo/api/websocket... Now, we require these two paths to start at the
root i.e. <host>/websocket/... /api/websocket/...
* Note: We now also require an ending / and do NOT support
<host>/websocket_foobar but DO support <host>/websocket/foobar. This
was always the intended behavior. We want to keep
<host>/api/websocket/... "open" and routing to daphne in case we want
to add more websocket urls in the future.
* Always support cookies, session, and also allow rest_framework
configured auth methods over the browser websocket.
* The node -> node websocket auth remains locked down and unchanged
* Organize metrics into their respective service
* Server per-service metrics on a per-service http server
* Increase prometheus client usage over our custom metrics fields
* Not many, if any, folks use the notebook feature. It kind of goes in
and out of popularity. We've used it in the past when we work on
features that require visualization (i.e. network graphs, workflows).
Might as well keep it around in case we use it again.
* We introduced multi networks to our docker env. The code this replaces
would return both networks' ip addresses concatinated i.e.
'192.168.2.1192.168.2.3'.
* Put the awx node(s) on a service-mesh docker network so they can be
proxied to. Also put all the other containers on an explicit awx
network otherwise they can not talk to each other. We might could be
more surgical about what containers we put on awx but I just added all
of them.
* Move awxkit import code into a pytest fixture to better control when
the import happens
* Ensure /awx_devel/awxkit is added to sys path before awxkit import
runs
* Basic export tests
* Added test that highlights a problem with running Schedule exports as
non-root user. We rely on the POST key in the OPTIONS response to
determine the fields to export for a resource. The POST key is not
present if a user does NOT have create privileges.
* Fixed up forwarding all headers from the API server back to the test
code. This was causing a problem in awxkit code that checks for
allowed HTTP Verbs in the headers.
* We observed daphne giving tracebacks when accessing logging settings.
Originally, configure tower in tower settings was no a suspect because
daphne is not multi-process. We've had issues with configure tower in
tower settings and multi-process before. We later learned that Daphne
is multi-threaded. Configure tower in tower was back to being a
suspect. We constructed a minimal reproducer to show that multiple
threads accessing settings can cause the same traceback that we saw in
daphne. See
https://gist.github.com/chrismeyersfsu/7aa4bdcf76e435efd617cb078c64d413
for that recreator. These fixes stop the recreation.
* We trigger notifications when the callback receiver processes the
playbook_on_stats event. This is the last event in ansible-playbook and
the process should exist very shortly after this event is emitted. The
trouble comes in with the isolated node feature. There is a management
playbook that runs periodically that pulls the events from the remote
node. It's possible that the management playbooks runs, gets the
playbook_on_stats event, but does not see that the playbook is finished
running. Therefore the job status is still seen as 'running' BUT we have
kicked of the notification for the job. The notification worker will
enter a loop waiting on the job to enter the finished state. In this
case the time it takes for the job to enter the finished state can be
long, roughly 2 * the management playbook run time.
* This new setting allows the user to increase the time that the
notification spends waiting for the job to enter the finished state.
* Note in the job lifecycle when the controller_node and execution_node
are chosen. This event occurs most commonly in the task manager with a
couple of exceptions that happen when we dynamically create dependenct
jobs on the fly in tasks.py
* Unsure exactly why this happens but there seems to be a race condition
related to the time window between Receptor work_results and work
release. This sleep extends that window and hopefully avoids the race
condition.
* Our tests could consistently get awx jobs into a deadlocked state
whenever the parallelism was high. Even podman ps would hang when the
system was in this state. We don't know exactly where in runc the bug is
but the deadlocks stopped happening when we changed the OCI runtime
environment to crun.
* job_created is a fake field as far as Django is concerned. Under the
hood, in postgres, this is the partition key so it is real. sqlite
doesn't support partitioning so we need to fake some things.
Specifically, we need to remove job_created from being auto-added to
get_event_queryset()
* Add pagination tests for <unified_job_name>/<id>/<job_events>?limit=x
endpoint to make sure the paginator is wired up.
* trigger via jobs/<id>/job_events/?limit=10
* Can and should be used in conjunction with an indexed set of fields to
generate efficient pagination queries. i.e.
jobs/<id>/job_events?limit=10&start_line__gte=10
* If limit is not specified in the query params then the default
pagination will be used.
* Before, we would get the min and max pk of the set we are to gather.
This changeset removes that.
* Before, we would, basically, know the size of the set we are to gather
and would query 100,000 of those job event records at a time. That logic
is now gone.
* Now, for unpartitioned job events we gather 4 hours at a time by
created time.
* Now, for partitioned job events we gather 4 hours at a time by
modified time.
* The order by results in an in-memory sort that COULD blow out the
worker mem buffer and result in sorting having to take place on disk.
* This WILL happen with a default postgres 4MB mem buffer. We saw as
much as 20MB used. Note that AWX defaults postgres mem worker buffer to
3% of the DB memory on external installs and 1% on same-node installs.
So for a 16GB remote DB this would not be a problem.
* We are going to avoid this problem all together by NOT doing a sort
when gathering. Instead, we will sort remotely, in analytics.
* Old, _unpartitioned_main_jobevent table does not have the job_created
column
* New, main_jobevent does.
* Always in clude the job_created column. NULL if old, job_created if
new
* Bump events_table schema version from 1.2 to 1.3 because of the
job_created field
* We found that having multiple `::json` casts in a query slows down
queries more and more by =~> 33%.
* This change coerces postgres into only casting once. Micro
benchmarking shows =~ 2-3x performance boost
Changes in old unpartitioned cleanup logic:
* Manually cascade delete events related to job(s)
(new partitions cleanup logic) For each event type:
* Get the event partitions that are within the cleanup date range
* Get a list of jobs to delete that are in the cutoff range.
* Jobs that are running, pending, or waiting in the job list are special.
* Use the special list to further filter the partition drop list.
* Drop partitions
* delete jobs
* AWX_ISOLATION_SHOW_PATHS will be shared between containers. Strange
file not found error can crop up when concurrently accessing shared
directories between multiple containers that are bind mounted with big
Z. So make sure we use little z.
Chris Meyers