Commit Graph

4 Commits

Author SHA1 Message Date
Buymov Ivan
f2676064fd Fix error with rejoining node to cluster after lost connection to postgres 2019-09-27 01:17:27 -04:00
Ryan Petrello
38bf174bda don't reap jobs that aren't running
this is a simple sanity check, but it should help us avoid shooting
ourselves in the foot in complicated scenarios, such as:

1.  A dispatcher worker is running a job, and it's killed with `kill -9`
2.  The dispatcher attempts to reap jobs with a matching celery_task_id
3.  The associated sync project update has the *same* celery_task_id
    (an implementation detail of how we implemented that), and it ends
    up getting reaped _even though_ it's already finished and has
    status=successful
2018-11-28 18:11:12 -05:00
Ryan Petrello
0d29bbfdc6 make the dispatcher more fault-tolerant to prolonged database outages 2018-10-18 20:00:07 -04:00
Ryan Petrello
ff1e8cc356 replace celery task decorators with a kombu-based publisher
this commit implements the bulk of `awx-manage run_dispatcher`, a new
command that binds to RabbitMQ via kombu and balances messages across
a pool of workers that are similar to celeryd workers in spirit.
Specifically, this includes:

- a new decorator, `awx.main.dispatch.task`, which can be used to
  decorate functions or classes so that they can be designated as
  "Tasks"
- support for fanout/broadcast tasks (at this point in time, only
  `conf.Setting` memcached flushes use this functionality)
- support for job reaping
- support for success/failure hooks for job runs (i.e.,
  `handle_work_success` and `handle_work_error`)
- support for auto scaling worker pool that scale processes up and down
  on demand
- minimal support for RPC, such as status checks and pool recycle/reload
2018-10-11 10:53:30 -04:00