NUL characters are not allowed in text fields in the database
We used to strip them out of stdout but the exception changed
And we want to be sure to strip them out of JSONBlob fields
This is expected to free up 4 additional database connections per traditional node
compare to roughly 12 in total before this change
Out of these 3 are accomplished by using existing connection for recently added services
then 1 is obtained by closing the connection for the idle callback receiver main process
Signed-off-by: jessicamack <jmack@redhat.com>
Co-authored-by: jessicamack <jmack@redhat.com>
This adds a handful of metrics to /api/v2/metrics/ recorded from the dispatcher main process
Adds logic in the dispatcher period tasks to calculate these for the last collection interval
Reports worker count, task count, scale up events, and availability
Add data to demo grafana dashboard
* Introduce new method in settings, import in-line w NOQA mark
* Further refine the app_name to use shorter service names like dispatcher
* Clean up listener logic, change some names
get_local_queuename will return the pod name of the instance
now that web and task are in different pods when web container queue a task it will be put into a queue without as task worker to execute the task
* Fix race with heartbeat and reaper logic
* Fix tests to fail when over drift over heartbeat time
* replaced modified with started time for reap() code and added test
* fixed logic bug and cleaned up tests
* Added comments to tests to call out reasoning
* Workaround for events with NUL char, touch up error loop
This fixes an error where some events would not save
due to having the 0x00 character which errors in postgres
this adds a line to replace it with empty text
Hitting that kind of event put us in an infinite error loop
so this change makes a number of changes to prevent similar loops
the showcase example is a negative counter,
this is not realistic in the real world but works for unit tests
These error loop fixes seek to esablish the cases where we clear the buffer
Some logic is removed from the outer loop, with the idea that
ensure_connection will better distinguish flake
* From review comments, delay NUL char sanitization to later
Use pop to make list operations more clear
* Fix incorrect use of pop
If canceled attempted before, still allow attempting another cancel
in this case, attempt to send the sigterm signal again.
Keep clicking, you might help!
Replace other cancel_callbacks with sigterm watcher
adapt special inventory mechanism for this too
Get rid of the cancel_watcher method with exception in main thread
Handle academic case of sigterm race condition
Process cancelation as control signal
Fully connect cancel method and run_dispatcher to control
Never transition workflows directly to canceled, add logs
Make it so that submitting a task to the dispatcher happens as part of the transaction.
this applies to dispatcher task "publishers" which NOTIFY the pg_notify queue
if the transaction is not successful, it will not be sent, as per postgres docs
This keeps current behavior for pg_notify listeners
practically, this only applies for the awx-manage run_dispatcher service
this requires creating a separate connection and keeping it long-lived
arbitrary code will occasionally close the main connection, which would stop listening
Stop sending the waiting status websocket message
this is required because the ordering cannot be maintained with other changes here
the instance group data is moved to the running websocket message payload
Move call to create_partition from task manager to pre_run_hook
mock this in relevant unit tests
Avoid running jobs that have already been reapted
Co-authored-by: Elijah DeLee <kdelee@redhat.com>
Remove unnecessary extra actions
Fix waiting jobs in other cases of reaping
add settings to define task manager timeout and grace period
This gives us still TASK_MANAGER_TIMEOUT_GRACE_PERIOD amount of time to
get out of the task manager.
Also, apply start task limit in WorkflowManager to starting pending
workflows
* Reap jobs on dispatcher startup to increase clarity, replace existing reaping logic
* Exit jobs if receiving SIGTERM signal
* Fix unwanted reaping on shutdown, let subprocess close out
* Add some sanity tests for signal module
* Add a log for an unhandled dispatcher error
* Refine wording of error messages
Co-authored-by: Elijah DeLee <kdelee@redhat.com>
grafana via prometheus.
This metric is a good indicator of how far behind the callback receiver
is. The higher the load the further behind/the greater the number of
seconds the metric will display.
This number being high may indicate the need for horizontal scaling in
the control plane or vertically scaling the number of callback
receivers.
* Track host_status_counts and use that to process notifications
* Remove now unused setting
* Back out changes to callback class not needed after all
* Skirt the need for duck typing by leaning on the cached field
* Delete tests for deleted task
* Revert "Back out changes to callback class not needed after all"
This reverts commit 3b8ae350d218991d42bffd65ce4baac6f41926b2.
* Directly hardcode stats_event_type for callback class
* Fire notifications if stats event was never sent
* Remove test content for deleted methods
* Add placeholder for when no hosts matched
* Make field default be None, denote events processed with empty dict
* Make UI process null value for host_status_counts
* Fix tracking of EOF dispatch for system jobs
* Reorganize EVENT_MAP into class properties
* Consolidate conditional I missed from EVENT_MAP refactor
* Give up on the null condition, also applies for empty hosts
* Remove cls position argument not being used
* Move wrapup method out of class, add tests
* Simple patches to make jobs robust to database restarts
* Add some wait time before retrying loop due to DB error
* Apply dispatcher downtime setting to job updates, fix dispatcher bug
This resolves a bug where the pg_is_down property
never had the right value
the loop is normally stuck in the conn.events() iterator
so it never recognized successful database interactions
this lead to serial database outages terminating jobs
New setting for allowable PG downtime is shared with task code
any calls to update_model will use _max_attempts parameter
to make it align with the patience time that the dispatcher
respects when consuming new events
* To avoid restart loops, handle DB errors on startup with prejudice
* If reconnect consistently fails, exit with non-zero code
