* Simple patches to make jobs robust to database restarts
* Add some wait time before retrying loop due to DB error
* Apply dispatcher downtime setting to job updates, fix dispatcher bug
This resolves a bug where the pg_is_down property
never had the right value
the loop is normally stuck in the conn.events() iterator
so it never recognized successful database interactions
this lead to serial database outages terminating jobs
New setting for allowable PG downtime is shared with task code
any calls to update_model will use _max_attempts parameter
to make it align with the patience time that the dispatcher
respects when consuming new events
* To avoid restart loops, handle DB errors on startup with prejudice
* If reconnect consistently fails, exit with non-zero code
* Process unresponsive and newly responsive hop nodes
* Use more natural way to zero hop node capacity, add test
* Use warning as opposed to warn for log messages
This is to avoid references to settings in threads,
this is known to create problems when caches expire
this leads to KeyError in environments with heavy load
This requires corresponding ansible-runner changes
which are only available in devel branch
to do this, requirements are changed
to install ansible-runner devel as it did before
Revert "Use ansible-runner 2.1.1 build"
This reverts commit f0ede01017.
Add back in change from updater.sh that we want to keep
* Allow for customizing the receptor image
* Hook in receptor image to docker-compose template
* Fix missing -e to pass into Dockerfile playbook
* Add some docs
Add new logic to cleanup orphaned work units
from administrative tasks
Remove noisy log which is often irrelevant
about running-cleanup-on-execution-nodes
we already have other logs for this
* Primary development of integrating runner cleanup command
* Fixup image cleanup signals and their tests
* Use alphabetical sort to solve the cluster coordination problem
* Update test to new pattern
* Clarity edits to interface with ansible-runner cleanup method
* Another change corresponding to ansible-runner CLI updates
* Fix incomplete implementation of receptor remote cleanup
* Share receptor utils code between worker_info and cleanup
* Complete task logging from calling runner cleanup command
* Wrap up unit tests and some contract changes that fall out of those
* Fix bug in CLI construction
* Fix queryset filter bug
* Write tests to assure air-tightness of overwrite
* Candidate fix for group overwrite air-tightness
* Another proposed fix for the id mapping
* Further double down on tracking old instance_id
* Separate unchanging data case and fix some test issues
* parametrize final confusion test
* cut down some more on test cases and fix bug with prior fix
* Rewrite of _delete_host code sharing with update method
This is a start-to-finish rewrite of the host overwrite bug fix
this method is much more conservative,
it does this by keeping the overall code structure where hosts
are deleted before host updates are made
To fix the bug, we share code between the method that deletes hosts
and the method that updates the hosts
A data structure is created and passed to both methods
By having both methods use the same data structure which maps
the in-memory hosts to DB hosts, we assure that the deletions
will ONLY delete hosts that will not be updated
* Full finalize the planned work for health checks of execution nodes
* Implementation of instance health_check endpoint
* Also do version conditional to node_type
* Do not use receptor mesh to check main cluster nodes health
* Fix bugs from testing health check of cluster nodes, add doc
* Add a few fields to health check serializer missed before
* Light refactoring of error field processing
* Fix errors clearing error, write more unit tests
* Update health check info in docs
* Bump migration of health check after rebase
* Mark string for translation
* Add related health_check link for system auditors too
* Handle health_check cluster node timeout, add errors for peer judgement
* Add wording about expections for enabled status and default group
* fix pluralization
Co-authored-by: Alex Corey <acorey@redhat.com>
* Correct grammar mistake
Co-authored-by: Alex Corey <acorey@redhat.com>
* Change the location of the receptor socket
to /var/run/awx-receptor, to match what the installer is currently doing.
* Sync awx and receptor nodes for control socket
Co-authored-by: Jeff Bradberry <jeff.bradberry@gmail.com>
* Exclude control-only nodes from IG policy calculations
Also, as a reverse to this, exclude execution-only nodes from
the calculations if the group in question is the controlplane
* Incorporate review comments
Alan Rominger
