broadcast queues get a per-node stable queue name

* Using Kombu's default Broadcast() constructor requires only 1
parameter. That parameter defines the exchange name and the queue name
is randomly generated per-node.
* This caused problems if/when celery enters an infinite restart loop
because too many rabbit queues get created and rabbit OOM's
(gracefully).
* To remedy this we tell Broadcast the queue name to use, which is
derived from some constant + the node name so that the per-node queue
name is stable.
This commit is contained in:
chris meyers
2018-04-30 15:18:05 -04:00
parent a009d21edc
commit 648d9165ff
4 changed files with 40 additions and 27 deletions

View File

@@ -189,7 +189,7 @@ def apply_cluster_membership_policies(self):
handle_ha_toplogy_changes.apply([])
@shared_task(queue='tower_broadcast_all', bind=True)
@shared_task(exchange='tower_broadcast_all', bind=True)
def handle_setting_changes(self, setting_keys):
orig_len = len(setting_keys)
for i in range(orig_len):
@@ -208,7 +208,7 @@ def handle_setting_changes(self, setting_keys):
restart_local_services(['uwsgi'])
@shared_task(bind=True, queue='tower_broadcast_all')
@shared_task(bind=True, exchange='tower_broadcast_all')
def handle_ha_toplogy_changes(self):
(changed, instance) = Instance.objects.get_or_register()
if changed: