* We log stats using a safe hostname because of prometheus requirements.
However, when we display users the hostname we should use the Instance
hostname. This change outputs the Instance.hostname instead of the safe
prometheus name.
* We strip out the json select fields in our tests since it is an sql
lite database underneath. Ideally, we would do something fancier, but we
aren't. In doing this stipping, we could strip the last element in the
projection list. This would result in an extra dangling comma. This
commit removes the danging comma in the projection list after the
removal of JSON projections.
when rsyslogd restarts due to config changes, there's a brief moment
where the socket will refuse connections on teardown; exception handling
is needed here to deal with that
after some prolonged RFC reading and tinkering w/ rsyslogd...
cpython's SysLogHandler doesn't emit RFC3164 formatted messages
in the format you'd expect; it's missing the ISO date, hostname, etc...
along with other header values; the handler implementation relies on you
to specify a syslog-like formatter (we've replaced all of this with our
own *custom* logstash-esque formatter that effectively outputs valid JSON
- without dates and other syslog header values prepended)
because of this unanticipated format, rsyslogd chokes when trying to
parse the message's parts; AWX is emitting:
<priority>RAWJSON
...so the usage of `%msg%` isn't going to work for us, because rsyslog
tries to parse *all* of the possible headers (and yells, because it
can't find a date to parse):
see: https://www.rsyslog.com/files/temp/doc-indent/configuration/properties.html#message-properties
this is fine, because we don't *need* any of that message parsing
anyways; in the end, we're *just* interested in forwarding the raw
JSON/text content to the third party log handler
there's a race condition if we do this pre-commit where the correct
value isn't actually *persisted* to the database yet, and we end up
saving the *prior* setting values
- Fix messages getting contatenated at 8k
- Fix rsyslog cutting off the opening brace of log messages
- Make valid default conf and emit logs based on prescence of .sock and
settings
- Add a placeholder rsyslog.conf so it doesn't fail on start
- Create access restricted directory for unix socket to be created in
- Create RSyslogHandler to exit early when logging socket doesn't exist
- Write updated logging settings when dispatcher comes up and restart rsyslog so they take effect
- Move rsyslogd to the web container and create rpc supervisor.sock
- Add env var for supervisor.conf path
- Add proper paths for rsyslog's supervisor logs
- Do not enable debug mode for rsyslogd
- Include system rsyslog.conf, and specify tower logging conf when
starting rsyslog.
- log aggregator url paths were not being passed to rsyslog
- http log services like loggly will now truly use http and port 80
- add rsyslog.pid to .gitignore
- this change adds rsyslog (https://github.com/rsyslog/rsyslog) as
a new service that runs on every AWX node (managed by supervisord)
in particular, this feature requires a recent version (v8.38+) of
rsyslog that supports the omhttp module
(https://github.com/rsyslog/rsyslog-doc/pull/750)
- the "external_logger" handler in AWX is now a SysLogHandler that ships
logs to the local UDP port where rsyslog is configured to listen (by
default, 51414)
- every time a LOG_AGGREGATOR_* setting is changed, every AWX node
reconfigures and restarts its local instance of rsyslog so that its
fowarding settings match what has been configured in AWX
- unlike the prior implementation, if the external logging aggregator
(splunk/logstash) goes temporarily offline, rsyslog will retain the
messages and ship them when the log aggregator is back online
- 4xx or 5xx level errors are recorded at /var/log/tower/external.err
Christian Adams
chris meyers
softwarefactory-project-zuul[bot]
Bill Nottingham
Ryan Petrello
Shane McDonald