after some prolonged RFC reading and tinkering w/ rsyslogd...
cpython's SysLogHandler doesn't emit RFC3164 formatted messages
in the format you'd expect; it's missing the ISO date, hostname, etc...
along with other header values; the handler implementation relies on you
to specify a syslog-like formatter (we've replaced all of this with our
own *custom* logstash-esque formatter that effectively outputs valid JSON
- without dates and other syslog header values prepended)
because of this unanticipated format, rsyslogd chokes when trying to
parse the message's parts; AWX is emitting:
<priority>RAWJSON
...so the usage of `%msg%` isn't going to work for us, because rsyslog
tries to parse *all* of the possible headers (and yells, because it
can't find a date to parse):
see: https://www.rsyslog.com/files/temp/doc-indent/configuration/properties.html#message-properties
this is fine, because we don't *need* any of that message parsing
anyways; in the end, we're *just* interested in forwarding the raw
JSON/text content to the third party log handler
when rsyslogd restarts due to config changes, there's a brief moment
where the socket will refuse connections on teardown; exception handling
is needed here to deal with that
after some prolonged RFC reading and tinkering w/ rsyslogd...
cpython's SysLogHandler doesn't emit RFC3164 formatted messages
in the format you'd expect; it's missing the ISO date, hostname, etc...
along with other header values; the handler implementation relies on you
to specify a syslog-like formatter (we've replaced all of this with our
own *custom* logstash-esque formatter that effectively outputs valid JSON
- without dates and other syslog header values prepended)
because of this unanticipated format, rsyslogd chokes when trying to
parse the message's parts; AWX is emitting:
<priority>RAWJSON
...so the usage of `%msg%` isn't going to work for us, because rsyslog
tries to parse *all* of the possible headers (and yells, because it
can't find a date to parse):
see: https://www.rsyslog.com/files/temp/doc-indent/configuration/properties.html#message-properties
this is fine, because we don't *need* any of that message parsing
anyways; in the end, we're *just* interested in forwarding the raw
JSON/text content to the third party log handler
there's a race condition if we do this pre-commit where the correct
value isn't actually *persisted* to the database yet, and we end up
saving the *prior* setting values
- this change adds rsyslog (https://github.com/rsyslog/rsyslog) as
a new service that runs on every AWX node (managed by supervisord)
in particular, this feature requires a recent version (v8.38+) of
rsyslog that supports the omhttp module
(https://github.com/rsyslog/rsyslog-doc/pull/750)
- the "external_logger" handler in AWX is now a SysLogHandler that ships
logs to the local UDP port where rsyslog is configured to listen (by
default, 51414)
- every time a LOG_AGGREGATOR_* setting is changed, every AWX node
reconfigures and restarts its local instance of rsyslog so that its
fowarding settings match what has been configured in AWX
- unlike the prior implementation, if the external logging aggregator
(splunk/logstash) goes temporarily offline, rsyslog will retain the
messages and ship them when the log aggregator is back online
- 4xx or 5xx level errors are recorded at /var/log/tower/external.err
for some reason (unsure why), django-extensions has begun noticing
ipython importability and treating "shell_plus" as "start an IPython
notebook by default
it could be that this is a bug in django-extensions that will be fixed
soon, but for now, this fixes the issue
success/failure notifications for *playbooks* include summary data about
the hosts in based on the contents of the playbook_on_stats event
the current implementation suffers from a number of race conditions that
sometimes can cause that data to be missing or incomplete; this change
makes it so that for *playbooks* we build (and send) the notification in
response to the playbook_on_stats event, not the EOF event
postgres has a limitation on its notify message size (8k), and the
messages we generate for deep copying functionality easily go over this
limit; instead of passing a giant nested data structure across the
message bus, this change makes it so that we temporarily store the JSON
structure in memcached, and look it up from *within* the task
see: https://github.com/ansible/tower/issues/4162
a side effect of this bug is that `awx schedules create` doesn't work
properly for non-admin users (i.e., users who have execute access for
a JT)
see: https://github.com/ansible/awx/issues/5717
these aren't top-level serializer fields; they're summary fields
if we want to support these properly, we should treat them as
enhancements, and write support, tests, and documentation
since the custom notification template refactor, grafana notification
support has been broken; this is largely because grafana functions more
like the webhooks, and needs to send JSON in its notification body
see: https://github.com/ansible/awx/issues/6137
I have a hunch that our usage of a daemon thread is causing import lock
contention related to https://github.com/ansible/awx/issues/5617
We've encountered similar issues before with threads across dispatcher
processes at fork time, and cpython has had bugs like this in recent
history:
https://bugs.python.org/issue38884
My gut tells me this might be related.
The prior implementation - based on celerybeat - ran its code in
a process (not a thread), and the timing of that merge matches the
period of time we started noticing issues.
Currently testing it to see if it resolves some of the issues we're
seeing.
if the awx_capacity module runs on an isolated node with missing
libraries (i.e., psutil) or bad permissions, then the runner status will
be "failed"
in this scenario, we *still* want to react by recording a capacity=0
Ryan Petrello