Commit Graph
100 Commits
Author SHA1 Message Date
Alan Rominger 799bac4066 Merge pull request #11860 from AlanCoding/hybrid_artifacts
Do not remove artifacts for local work
2022-03-18 10:37:06 -04:00
Alan Rominger f99820a391 Merge pull request #11898 from AlanCoding/hop_node_lost_found
Only clear hop node errors if they were lost before, not lost now
2022-03-15 13:03:33 -04:00
Alan Rominger 2321f06c8a Only clear hop node errors if they were lost before, not lost now 2022-03-14 16:04:55 -04:00
Alan Rominger d4a4ba7fdb Move location of AWX_ISOLATION_SHOW_PATHS so it is editable 2022-03-11 11:08:04 -05:00
Alan Rominger 99bbc347ec Fill in errors for hop nodes when Last Seen is out of date, and clear them when not (#11714)
* Process unresponsive and newly responsive hop nodes

* Use more natural way to zero hop node capacity, add test

* Use warning as opposed to warn for log messages
2022-03-09 13:21:32 -05:00
Alan Rominger cd86310562 Do not remove artifacts for local work 2022-03-08 09:07:09 -05:00
Alan Rominger e02e91adaa Merge pull request #11837 from AlanCoding/thread_key_error
Move model and settings operations out of threaded code
2022-03-05 14:55:13 -05:00
Alan Rominger 264c508c80 Move model and settings operations out of threaded code
This is to avoid references to settings in threads,
  this is known to create problems when caches expire
  this leads to KeyError in environments with heavy load
2022-03-04 15:31:12 -05:00
Alan Rominger 17756f0e72 Add job execution environment image to analytics data (#11835)
* Add job execution environment image to analytics data

* Add EE image to UJT analytics data

* Bump the unified job templates table
2022-03-03 11:13:11 -05:00
Alan Rominger 128400bfb5 Add resolved_action to analytics event data (#11816)
* Add resolved_action to analytics event data

* Bump collector version
2022-03-03 10:11:54 -05:00
Alan Rominger eb52095670 Fix bug where translated strings will cause log error to error (#11813)
* Fix bug where translated strings will cause log error to error

* Use force_str for ensuring string
2022-02-28 08:38:01 -05:00
Alan Rominger fe7a2fe229 Add loop label with docker-compose playbook 2022-02-15 13:05:59 -05:00
Alan Rominger 21bf698c81 Merge pull request #11617 from AlanCoding/task_job_id
Fix error on timeout with non-job types
2022-02-04 09:41:25 -05:00
Alan Rominger 54b5884943 Merge pull request #11642 from AlanCoding/new_black_rule
Fix newly-added black rules
2022-01-31 10:01:50 -05:00
Alan Rominger 6b25fcaa80 Fix newly-added black rules 2022-01-29 23:17:58 -05:00
Alan Rominger 2de5ffc8d9 Merge pull request #11627 from AlanCoding/fast_heartbeat
Prevent duplicate query in local health check
2022-01-28 13:19:56 -05:00
Alan Rominger 285ff080d0 Prevent duplicate query in local health check 2022-01-27 15:27:07 -05:00
Alan Rominger cd8d382038 Fix error on timeout with non-job types 2022-01-26 17:00:59 -05:00
Alan Rominger 8d4d1d594b Merge pull request #11608 from AlanCoding/mount_awx_devel
Mount awx_devel in execution nodes for developer utility
2022-01-25 16:42:56 -05:00
Alan Rominger c86fafbd7e Mount awx_devel in execution nodes for developer utility 2022-01-25 12:28:26 -05:00
Alan Rominger 4f1c662691 Merge pull request #11570 from AlanCoding/keycloak_docs
Minor docs tweaks for keycloak setup
2022-01-20 11:52:21 -05:00
Alan Rominger 9abd4e05d0 Minor docs tweaks for keycloak setup 2022-01-20 11:01:32 -05:00
Alan Rominger add54bfd0b Merge pull request #11472 from AlanCoding/process_ident
Pass new ansible-runner parameters to reduce number of artifacts we don't need on file system
2022-01-20 09:48:44 -05:00
Alan Rominger 2609ee5ed0 Delete artifact dir after transmit phase is finished 2022-01-18 14:51:40 -05:00
Alan Rominger 8fac1c18c8 Make task logic use consistent artifact dir location 2022-01-18 13:00:39 -05:00
Alan Rominger eb64fde885 Pass ident to "process" cmd and disable stdout file
This requires corresponding ansible-runner changes
  which are only available in devel branch
  to do this, requirements are changed
  to install ansible-runner devel as it did before

Revert "Use ansible-runner 2.1.1 build"

This reverts commit f0ede01017.

Add back in change from updater.sh that we want to keep
2022-01-18 13:00:39 -05:00
Alan Rominger 696c0b0055 Merge pull request #11503 from AlanCoding/no_version
Remove unused ansible version method
2022-01-14 22:15:15 -05:00
Alan Rominger 9664aed1f2 Remove unused ansible version method 2022-01-14 14:55:35 -05:00
Alan Rominger 72cd73ca71 Update to cover stuff from tasks.py changes 2022-01-14 13:42:24 -05:00
Alan Rominger 02e18cf919 Fix more F811 linter violations 2022-01-14 13:23:05 -05:00
Alan Rominger 82671680e3 Respect linter rule F811 for trivial re-definition 2022-01-14 13:23:04 -05:00
Alan Rominger b0c59ee330 Merge pull request #11375 from AlanCoding/missing_image_error_devel
Fail with specific error message if protected image is not available
2022-01-12 11:05:17 -05:00
Alan Rominger 8126f734e3 Remove unnecessary DEBUG logger level settings (#11441)
* Remove unnecessary DEBUG logger level settings
2022-01-05 14:44:57 -05:00
Alan Rominger 7219f8fed8 Merge pull request #11462 from AlanCoding/forgot_this
Add the cancel_callback to system job interface
2021-12-14 14:17:47 -05:00
Alan Rominger b6a5f834d6 Merge pull request #11408 from amolgautam25/receptor_tech_debt
Removing time.sleep(3)
2021-12-14 11:54:49 -05:00
Alan Rominger 99b9d53bbb Add the cancel_callback to system job interface 2021-12-14 10:50:39 -05:00
Alan Rominger 21895bd09b Merge pull request #11448 from AlanCoding/revert_again
Revert "cancel job if receptor no longer knows about the work item"
2021-12-10 16:35:12 -05:00
Alan Rominger 411ef5f9e8 Revert "cancel job if receptor no longer knows about the work item"
This reverts commit 2a11bb4f3b.
2021-12-10 16:18:44 -05:00
Alan Rominger 1b0fca8026 Merge pull request #11386 from AlanCoding/logs_on_the_fire
Remove dev-only log filters and downgrade periodic logs
2021-12-07 16:13:45 -05:00
Alan Rominger d6679a1e9b Respect dynamic log setting for console, downgrade exit log 2021-12-07 14:35:03 -05:00
Alan Rominger b721a4b361 Remove dev-only log filters and downgrade periodic logs 2021-12-07 14:35:02 -05:00
Alan Rominger 19b4849345 Merge pull request #11394 from notok/cfg_from_template_branch
Load ansible.cfg from the branch specified on job template
2021-12-06 11:09:36 -05:00
Alan Rominger 5512b71e16 Merge pull request #11412 from AlanCoding/cookie_revert
Revert "Set SESSION_COOKIE_NAME by default"
2021-12-02 11:00:56 -05:00
Alan Rominger 97b60c43b7 Merge pull request #11385 from AlanCoding/my_cluster_host
Do not overwrite file-based CLUSTER_HOST_ID written by installer
2021-12-02 10:53:59 -05:00
Alan Rominger 35b62f8526 Revert "Set SESSION_COOKIE_NAME by default"
This reverts commit 59c6f35b0b.
2021-12-01 17:51:47 -05:00
Alan Rominger 776c4a988a Do not overwrite file-based CLUSTER_HOST_ID written by installer 2021-11-30 20:15:10 -05:00
Alan Rominger 099efb883d Allow customizing the receptor image in the development environment (#11374)
* Allow for customizing the receptor image

* Hook in receptor image to docker-compose template

* Fix missing -e to pass into Dockerfile playbook

* Add some docs
2021-11-19 14:00:23 -05:00
Alan Rominger eeefd19ad3 Fail with specific error message if protected image is not available locally 2021-11-19 11:52:54 -05:00
Alan Rominger d67007f777 Move only_transmit_kwargs calculation out of thread 2021-11-19 01:11:18 +00:00
Alan Rominger 3a3fffb2dd Fixed error dropped on floor - save receptor detail when it applies 2021-11-10 08:50:16 +08:00
Alan Rominger a2acbe9fe6 Fix incorrect (changed: True) frequent in OCP task logs 2021-11-10 08:50:16 +08:00
Alan Rominger 0d1f8a06ce Revert default EE authfile support for inventory_import 2021-11-10 08:50:15 +08:00
Alan Rominger d42fe921db Re-order authfile option to make inventory import command work 2021-11-10 08:50:15 +08:00
Alan Rominger 329caad681 In admin reaper skip work units w/o params 2021-11-10 08:50:14 +08:00
Alan Rominger ecb84e090c Revert "Merge pull request #5354 from ansible/jobs_killed_via_receptor_should_get_reaped"
This reverts commit 8736858d80, reversing
changes made to 84e77c9db9.
2021-11-10 08:50:14 +08:00
Alan Rominger 2839091b22 Avoid extra check if we have job_explanation 2021-11-10 08:50:13 +08:00
Alan Rominger 47e67481b3 Avoid reaping tentative jobs 2021-11-10 08:50:13 +08:00
Alan Rominger 55059b015f Avoid resultsock shutdown before reading from it 2021-11-10 08:50:13 +08:00
Alan Rominger eb6c58682d Alternative for reaping lost jobs, in work unit reaper 2021-11-10 08:50:13 +08:00
Alan Rominger e54db3ce50 Gracefully handle receptorctl RuntimeError in health check 2021-11-10 08:50:11 +08:00
Alan Rominger 77076dbd67 Reduce the number of triggers for execution node health checks 2021-11-10 08:50:11 +08:00
Alan Rominger 6f20a798ab Allow testing a single hybrid instance like the good old days 2021-11-10 08:50:11 +08:00
Alan Rominger f34c96ecf5 Error handling when node is missing from mesh for jobs and checks 2021-11-10 08:50:11 +08:00
Alan Rominger 7b35902d33 Respect settings to keep files and work units
Add new logic to cleanup orphaned work units
  from administrative tasks

Remove noisy log which is often irrelevant
  about running-cleanup-on-execution-nodes
  we already have other logs for this
2021-11-10 08:50:10 +08:00
Alan Rominger 12f417d0a3 Merge pull request #11286 from StarryInternet/enpaul-multiuse-mesh
Skip additional instance checks on unrecognized hosts
2021-10-29 15:09:33 -04:00
Alan Rominger 74af187568 Fix Makefile conditional used for docker-refresh (#11238) 2021-10-12 13:52:52 -04:00
Alan Rominger ed8498f43f Change search location for job private data (#11217) 2021-10-07 20:33:57 -04:00
Alan Rominger e52416fd47 Report single node clusters as non-ha (#11212)
* Report single node clusters as non-ha

* Move test file so we can make it use the database

* Update unit test to accomidate different node types
2021-10-06 10:50:18 -04:00
Alan Rominger b70793db5c Consolidate cleanup actions under new ansible-runner worker cleanup command (#11160)
* Primary development of integrating runner cleanup command

* Fixup image cleanup signals and their tests

* Use alphabetical sort to solve the cluster coordination problem

* Update test to new pattern

* Clarity edits to interface with ansible-runner cleanup method

* Another change corresponding to ansible-runner CLI updates

* Fix incomplete implementation of receptor remote cleanup

* Share receptor utils code between worker_info and cleanup

* Complete task logging from calling runner cleanup command

* Wrap up unit tests and some contract changes that fall out of those

* Fix bug in CLI construction

* Fix queryset filter bug
2021-10-05 16:32:03 -04:00
Alan Rominger af5f8e8a4a Always set project sync execution_node to current host (#11204) 2021-10-05 13:08:40 -04:00
Alan Rominger 7c9626b0e7 Fix bug that would run --worker-info health checks on control or hybrid nodes (#11161)
* Fix bug that would run health check on control nodes

* Prevent running execution node health check against main cluster nodes
2021-09-29 09:34:22 -04:00
Alan Rominger 3664cc3369 Disable autodiscovery except for docker-compose (#11142) 2021-09-27 11:36:11 -04:00
Alan Rominger 3fc63489f1 Filter controller_node selection to online nodes (#11120) 2021-09-24 23:01:32 -04:00
Alan Rominger b5dee61e57 Delete wording that we have reversed position on (#11129) 2021-09-24 09:38:48 -04:00
Alan Rominger 62e9e7ea80 Avoid setting controller_node to an execution node for container jobs (#11117) 2021-09-23 09:16:10 -04:00
Alan Rominger 0ac3a377fd Make some needed updates to docker-refresh target (#11089) 2021-09-17 09:11:52 -04:00
Alan Rominger 1319fadc60 Fix overwrite bug where hosts with no instance ID var are re-created (#10910)
* Write tests to assure air-tightness of overwrite

* Candidate fix for group overwrite air-tightness

* Another proposed fix for the id mapping

* Further double down on tracking old instance_id

* Separate unchanging data case and fix some test issues

* parametrize final confusion test

* cut down some more on test cases and fix bug with prior fix

* Rewrite of _delete_host code sharing with update method

This is a start-to-finish rewrite of the host overwrite bug fix
this method is much more conservative,
it does this by keeping the overall code structure where hosts
are deleted before host updates are made

To fix the bug, we share code between the method that deletes hosts
and the method that updates the hosts
A data structure is created and passed to both methods

By having both methods use the same data structure which maps
the in-memory hosts to DB hosts, we assure that the deletions
will ONLY delete hosts that will not be updated
2021-09-16 15:29:57 -04:00
Alan Rominger e914c23c42 Pass --delete flag to worker for execution node cleanup (#11078)
* Pass --delete flag to worker for execution node cleanup

* Remove the pdd_wrapper_ directory
2021-09-16 11:21:41 -04:00
Alan Rominger 46ac9506e6 Assure consistent ordering with default IG first (#11034)
* Assure consistent ordering with default IG first

* Write conditional a little more defensively to pass tests
2021-09-08 11:11:46 -04:00
Alan Rominger 6a17e5b65b Allow manually running a health check, and make other adjustments to the health check trigger (#11002)
* Full finalize the planned work for health checks of execution nodes

* Implementation of instance health_check endpoint

* Also do version conditional to node_type

* Do not use receptor mesh to check main cluster nodes health

* Fix bugs from testing health check of cluster nodes, add doc

* Add a few fields to health check serializer missed before

* Light refactoring of error field processing

* Fix errors clearing error, write more unit tests

* Update health check info in docs

* Bump migration of health check after rebase

* Mark string for translation

* Add related health_check link for system auditors too

* Handle health_check cluster node timeout, add errors for peer judgement
2021-09-03 16:37:37 -04:00
Alan RomingerandAlex Corey 22782f8c5f Add wording about expectations for enabled status and default group (#10993)
* Add wording about expections for enabled status and default group

* fix pluralization

Co-authored-by: Alex Corey <acorey@redhat.com>

* Correct grammar mistake

Co-authored-by: Alex Corey <acorey@redhat.com>
2021-09-03 10:35:29 -04:00
Alan RomingerandJeff Bradberry eb0528c157 dev environment - change location of receptor socket and sync awx and receptor nodes function (#11005)
* Change the location of the receptor socket

to /var/run/awx-receptor, to match what the installer is currently doing.

* Sync awx and receptor nodes for control socket

Co-authored-by: Jeff Bradberry <jeff.bradberry@gmail.com>
2021-09-02 09:18:25 -04:00
Alan Rominger 1d3a36d821 Fix the hostname of execution nodes in dev environment (#10992) 2021-09-01 13:35:39 -04:00
Alan Rominger 5a6e9a06e2 Exclude control-only nodes from IG policy calculations (#10985)
* Exclude control-only nodes from IG policy calculations

Also, as a reverse to this, exclude execution-only nodes from
  the calculations if the group in question is the controlplane

* Incorporate review comments
2021-09-01 08:09:46 -04:00
Alan Rominger dc4b014d12 Make status command in error handling cleaner (#10823) 2021-08-31 12:02:39 -04:00
Alan Rominger 573b2bc44f Redefine execution plane (#10979) 2021-08-31 10:33:14 -04:00
Alan Rominger 68f79a1f3a Always use controlplane as project update backup IG (#10949)
* Always use controlplane as project update backup IG

Before, this was done conditionally to container_group jobs
this logic changes it so that controlgroup will always be a
firm backstop for project updates

* Code a little more defensively to make unit tests pass

* Fix unit tests
2021-08-30 14:23:09 -04:00
Alan Rominger 424dbe8208 Use ansible-runner imports for cpu and memory calculation (#10954)
* Use ansible-runner imports for cpu and memory calculation

* Fix bug with capacity and memory adjustment
2021-08-27 21:46:53 -04:00
Alan Rominger 711e5e09ba Delete images by id instead of tag in docker-refresh (#10957) 2021-08-27 11:51:58 -04:00
Alan Rominger ee4792dbf8 Add an option to create a cluster with control-only nodes (#10946) 2021-08-26 13:37:13 -04:00
Alan Rominger daf4310176 Clean up work_type processing and fix execution vs control capacity (#10930)
* Clean up added work_type processing for mesh_code branch

* track both execution and control capacity

* Remove unused execution_capacity property

* Count all forms of capacity to make test pass

* Force jobs to be on execution nodes, updates on control nodes

* Introduce capacity_type property to abstract some details out

* Update test to cover all job types at same time

* Register OpenShift nodes as control types

* Remove unqualified consumed_capacity from task manager and make unit tests work

* Remove unqualified consumed_capacity from task manager and make unit tests work

* Update unit test to execution vs control TM logic changes

* Fix bug, else handling for work_type method
2021-08-26 07:24:14 -04:00
Alan Rominger e7dbe90cb5 Merge pull request #10727 from ansible/mesh_code
Code changes to support execution-only nodes in receptor mesh
2021-08-24 13:39:19 -04:00
Alan Rominger 42484cf98d Obtain receptor sockfile from the receptor.conf file (#10932) 2021-08-24 11:20:21 -04:00
Alan Rominger 940c189c12 Corresponding AWX changes for runner --worker-info schema update (#10926) 2021-08-24 08:41:36 -04:00
Alan Rominger c3ad479fc6 Minor tweaks for the mesh_code branch from review (#10902) 2021-08-24 08:41:35 -04:00
Alan Rominger 928c35ede5 Model changes for instance last_seen field to replace modified (#10870)
* Model changes for instance last_seen field to replace modified

* Break up refresh_capacity into smaller units

* Rename execution node methods, fix last_seen clustering

* Use update_fields to make it clear save only affects capacity

* Restructing to pass unit tests

* Fix bug where a PATCH did not update capacity value
2021-08-24 08:41:35 -04:00
Alan Rominger 3b1e40d227 Use the ansible-runner worker --worker-info to perform execution node capacity checks (#10825)
* Introduce utilities for --worker-info health check integration

* Handle case where ansible-runner is not installed

* Add ttl parameter for health check

* Reformulate return data structure and add lots of error cases

* Move up the cleanup tasks, close sockets

* Integrate new --worker-info into the execution node capacity check

* Undo the raw value override from the PoC

* Additional refinement to execution node check frequency

* Put in more complete network diagram

* Followup on comment to remove modified from from health check responsibilities
2021-08-24 08:41:35 -04:00
Alan Rominger 4e84c7c4c4 Use the existing get_receptor_ctl method (#10813) 2021-08-24 08:41:35 -04:00
Alan Rominger f47eb126e2 Adopt the node_type field in receptor logic (#10802)
* Adopt the node_type field in receptor logic

* Refactor Instance.objects.register so we do not reset capacity to 0
2021-08-24 08:41:34 -04:00
Alan Rominger 5d4ab13386 Add topology of docker-compose to docs, remove old mount (#10773) 2021-08-24 08:41:34 -04:00