Patches to make jobs robust to database restarts (#11905)

* Simple patches to make jobs robust to database restarts

* Add some wait time before retrying loop due to DB error

* Apply dispatcher downtime setting to job updates, fix dispatcher bug

This resolves a bug where the pg_is_down property
  never had the right value
  the loop is normally stuck in the conn.events() iterator
  so it never recognized successful database interactions
  this lead to serial database outages terminating jobs

New setting for allowable PG downtime is shared with task code
  any calls to update_model will use _max_attempts parameter
  to make it align with the patience time that the dispatcher
  respects when consuming new events

* To avoid restart loops, handle DB errors on startup with prejudice

* If reconnect consistently fails, exit with non-zero code

awx/main/tasks/callback.py

@@ -32,9 +32,10 @@ class RunnerCallback:
         self.safe_env = {}
         self.event_ct = 0
         self.model = model
+        self.update_attempts = int(settings.DISPATCHER_DB_DOWNTOWN_TOLLERANCE / 5)
 
     def update_model(self, pk, _attempt=0, **updates):
-        return update_model(self.model, pk, _attempt=0, **updates)
+        return update_model(self.model, pk, _attempt=0, _max_attempts=self.update_attempts, **updates)
 
     def event_handler(self, event_data):
         #
@@ -147,7 +148,7 @@ class RunnerCallback:
         Ansible runner callback to tell the job when/if it is canceled
         """
         unified_job_id = self.instance.pk
-        self.instance.refresh_from_db()
+        self.instance = self.update_model(unified_job_id)
         if not self.instance:
             logger.error('unified job {} was deleted while running, canceling'.format(unified_job_id))
             return True