Add a periodic task to reap unreleased receptor work units

- Add work_unit_id field to UnifiedJob
This commit is contained in:
Seth Foster
2021-06-11 00:36:51 -04:00
committed by Shane McDonald
parent 7b9bcd0481
commit 75a27c38c2
6 changed files with 64 additions and 6 deletions

View File

@@ -156,6 +156,11 @@ One of the most important tasks in a clustered AWX installation is the periodic
If a node in an AWX cluster discovers that one of its peers has not updated its heartbeat within a certain grace period, it is assumed to be offline, and its capacity is set to zero to avoid scheduling new tasks on that node. Additionally, jobs allegedly running or scheduled to run on that node are assumed to be lost, and "reaped", or marked as failed.
## Reaping Receptor Work Units
When an AWX job is launched via receptor, files such as status, stdin, and stdout are created in a specific receptor directory. This directory on disk is a random 8 character string, e.g. qLL2JFNT
This is also called the work Unit ID in receptor, and is used in various receptor commands, e.g. "work results qLL2JFNT"
After an AWX job executes, the receptor work unit directory is cleaned up by issuing the work release command. In some cases the release process might fail, or if AWX crashes during a job's execution, the work release command is never issued to begin with.
As such, there is a periodic task that will obtain a list of all receptor work units, and find which ones belong to AWX jobs that are in a completed state (status is canceled, error, or succeeded). This task will call "work release" on each of these work units to clean up the files on disk.
## AWX Jobs