Implement a more dynamic celery queue system

* Meant to be a starting point to more efficiently manage work routing
  and to balance work across all tower nodes
* Integrate flower as a dev tool that starts alongside other nodes.
  Helpful for observing and monitoring the queues/exchanges
* For the moment, force the task manager to only run on one node (not
  sure if this is needed)
* Define queues and routes for all task work
* Bump celery version to 3.1.23
* Expose flower through haproxy
This commit is contained in:
Matthew Jones
2016-09-09 15:18:18 -04:00
parent 13a0fd749f
commit 807cced571
9 changed files with 63 additions and 11 deletions

View File

@@ -17,6 +17,11 @@ frontend localnodes
mode http
default_backend nodes
frontend flower
bind *:5555
mode http
default_backend flower_nodes
backend nodes
mode http
balance roundrobin
@@ -29,6 +34,18 @@ backend nodes
server tower_2 tower_2:8013 check
server tower_3 tower_3:8013 check
backend flower_nodes
mode http
balance roundrobin
option forwardfor
option http-pretend-keepalive
http-request set-header X-Forwarded-Port %[dst_port]
http-request add-header X-Forwarded-Proto https if { ssl_fc }
#option httpchk HEAD / HTTP/1.1\r\nHost:localhost
server tower_1 tower_1:5555
server tower_2 tower_2:5555
server tower_3 tower_3:5555
listen stats
bind *:1936
stats enable