* Gather brroadcast websocket metrics and push them into redis every
configurable seconds.
* Pop metrics from redis in web view layer to display via the api on
demand
* 100 is the default capacity for a channel. If the client doesn't read
the socket fast enough, websocket messages can and will be lost. This
increases the default to 10,000
* Do not return from blocking unsubscribe until _after_ putting the
gotten unsubscribe message on the queue so that it can be read by the
thread of execution that was unblocked.
* New tower nodes that are (de)registered in the Instance table are seen
by the websocket layer and connected to or disconnected from by the
websocket broadcast backplane using a polling mechanism.
* This is especially useful for openshift and kubernetes. This will be
useful for standalone Tower in the future when the restarting of Tower
services is not required.
* Sending health about websockets over websockets is not a great idea.
* I tried sending health data via prometheus and encountered problems
that will need PR's to prometheus_client library to solve. Circle back
to this later.
* We can not query the dispatcher running on isolated nodes to see if
the playbook is still running because that is the nature of isolated
nodes, they don't run the dispatcher nor do they run the message broker.
Therefore, we should query the control node that is arbitrating the
isolated work. If the control node process in the dispatcher is dead,
consider the iso job dead.
* Instead of waiting an arbitrary number of seconds. We can now wait the
exact amount of time needed to KNOW that we are unsubscribed. This
changeset takes advantage of the new subscribe reply semantic.
* This change adds more than just an unsubscribe reply.
* Websockets canrequest to join/leave groups. They do so using a single
idempotent request. This change replies to group requests over the
websockets with the diff of the group subscription. i.e. what groups the
user currenntly is in, what groups were left, and what groups were
joined.
* User in channels session is a lazy user class. This does not conform
to what the generic Role ancestry code expects. The Role ancestry code
expects a User objects. This change converts the lazy object into a
proper User object before calling the permission code path.
* asgiref async_to_sync was causing a Redis connection _for each_ call
to emit_channel_notification i.e. every event that the callback receiver
processes. This is a "known" issue
https://github.com/django/channels_redis/pull/130#issuecomment-424274470
and the advise is to slow downn the rate at which you call
async_to_sync. That is not an option for us. Instead, we put the async
group_send call onto the event loop for the current thread and wait for
it to be processed immediately.
The known issue has to do with event loop + socket relationship. Each
connection to redis is achieved via a socket. That conection can only be
waiting on by the event loop that corresponds to the calling thread.
async_to_sync creates a _new thread_ for each invocation. Thus, a new
connection to redis is required. Thus, the excess redis connections that
can be observed via netstat | grep redis | wc -l.
* Under the new postgres backed notify/listen message queue, this never
actually worked. Without using the database to store state, we can not
provide a at-most-once delivery mechanism w/ multi-readers.
* With this change, work is done ONLY on the node that requested for the
work to be done. Under rabbitmq, the node that was first to get the
message off the queue would do the work; presumably the least busy node.
* requirements/updater.sh does pip magic. In doing this magic, devel
system packages are required to download/install/build. This change
ensures those dev packages are available.
* The user awx is passed to the launch of our dev docker container. The
docker system automagically creates that user for us and sets the home
dir to /tmp in /etc/passwd. Many methods of detecting the user home dir
don't use that. Instead, they use the HOME env var. This is a half-way
solution that solves the problem of python expanding the ~ dir.
* If other things break because they determine the users home dir via
/etc/passwd entry then a more in-depth fix will be needed.
* Two job templates that use the same smart inventory running at the
same time can easily race to recompute the smart inventory <-> host
mapping. In this case, bulk_create() can throw an error when racing.
* The per-smart-inventory advisory lock ensures that the state of the
system is consistent & that bulk_create() runs in isolation.
* Disabling an instance is used to stop and instance from being the
target of new jobs to run.
* The instance should still perform it's heartbeat so that it isn't
considered offline.
* If the instance was allowed to go offline on an openshift cluster it
would be deleted from the database.
* Instantiating the callback dispatch queue on each job event callback
is expensive. Instead, instantiate it only once. Note, we do not need to
instantiate the callback queue in the iso case so we do not.
* Continue to pick up facts from scan_insights.py
* This PR adds the ability to pickup facts from
/etc/ansible/facts.d/insights.facts
* Log what transport the insights system_id was found via
* We include a special header value in the user agent when tower proxies
requests to get per-host rules.
* This extends that header logic to when we fetch plans (playbooks)
* all inventory updates continue to occur in parallel up to the point
that they update the database with their results.
* the "funnel" is achieved by using a global per-inventory postgres
named lock
* instead, set the ident passed to ansible runner to be the job id. That
way, on we know what directory to look in for results when the directory
structure is created.
* Previously, parent_uuid was expected only on events generated for a
Job run. Now, there maybe a parent_uuid for any job type. AWX does not
support parenting events for any job type other than Job.
* fixup task TestGenericRun
* make runner callback functions accessable to testing
* reduce isintance() usage in run() by using build_ pattern
* move process_isolation param building to build_ function so it can be
tested
* v2_playbook_on_play_start is called multiple times for the same UUID.
Specifically, once for each host in the play. This changes makes the
uuid unique before going to the dispatcher.
* While parsing host_filter in the smart inventory code it was
triggering sql queries. This changset avoids executing the query that is
being constructed.
* The periodic scheduler that runs and spawns jobs from Schedule()'s can
end up spawning more jobs than intended, for a single Schedule.
Specifically, when tower clustering is involed. This change adds a
"global" database lock around this critical code. If another process is
already doing the scheduling, short circuit.
* Removed the emailing of admins on request error. When turned on, the
handler will include all django settings in the email. This is not
desirable from a security standpoint.
* Tried to re-use the topological sort order to crawl the graph to find
the next node(s) to run. This is incorrect, we need to take into account
the fail/success of jobs and directionally crawl the graph.
* Leave workflow nodes with no related unified job template nodes
do_not_run = False. If we mark it True, we can't differentiate between
the actual want to not take that path vs. do not run this because I do
not have a valid related unified job template.
* Perform topological sort on graph nodes before looping over them to
mark do not run. This guarantees that parent nodes will be processed
before calling dependent child nodes. The complexity of the sorting is
N. The complexity of marking the the nodes is N*V
* Compute largest depth of each node and traverse graph by depth. This
allows us to check a node once, and only once, to determine if it needs
to be marked for do not run.
* Workflow Node without unified_job_template is treated as a job marked
as failure; when deciding what path to execute.
* Remove optimization of marking dnr nodes due to it making the
algorithm incorrect.
* When workflow job fails because a workflow job node doesn't have a
related unified_job_template note that with an error on the workflow
job's job_description
* When a workflow job fails because a failure path isn't defined, note
that on the workflow job job_description
* Instead of traversing the workflow graph to determine if a workflow is
done or has failed; instead, loop through all the nodes in the graph and
grab only the relevant nodes.
chris meyers