switch the periodic scheduler to a child process (instead of a thread)

I have a hunch that our usage of a daemon thread is causing import lock
contention related to https://github.com/ansible/awx/issues/5617
We've encountered similar issues before with threads across dispatcher
processes at fork time, and cpython has had bugs like this in recent
history:

https://bugs.python.org/issue38884

My gut tells me this might be related.

The prior implementation - based on celerybeat - ran its code in
a process (not a thread), and the timing of that merge matches the
period of time we started noticing issues.

Currently testing it to see if it resolves some of the issues we're
seeing.
This commit is contained in:
Ryan Petrello
2020-02-26 16:06:49 -05:00
parent 154b9c36ac
commit 5364e78397
2 changed files with 28 additions and 25 deletions

View File

@@ -53,7 +53,7 @@ class Command(BaseCommand):
# spawn a daemon thread to periodically enqueues scheduled tasks
# (like the node heartbeat)
cease_continuous_run = periodic.run_continuously()
periodic.run_continuously()
reaper.reap()
consumer = None
@@ -87,7 +87,6 @@ class Command(BaseCommand):
)
consumer.run()
except KeyboardInterrupt:
cease_continuous_run.set()
logger.debug('Terminating Task Dispatcher')
if consumer:
consumer.stop()