Commit Graph

74 Commits

Author SHA1 Message Date
Alan Rominger
dce5ac73c5 Apply new rules from black update (#16232) 2026-01-19 12:58:07 -05:00
jessicamack
de86b93690 AAP-59874: Update to Python 3.12 (#16208)
* update to Python 3.12

* remove use of utcnow

* switch to timezone.utc

datetime.UTC is an alias of datetime.timezone.utc. if we're doing the double import for datetime it's more straightforward to just import timezone as well and get it directly

* debug python env version issue

* change python version

* pin to SHA and remove debug portion
2026-01-07 11:57:24 -05:00
Alan Rominger
49240ca8e8 Fix environment-specific rough edges of logging setup (#15193) 2025-04-14 12:12:06 -04:00
Chris Meyers
0eb465531c Centralized logging via otel 2024-05-31 13:55:17 -04:00
Seth Foster
131dd088cd fix linting (#14302) 2023-07-31 20:37:37 -04:00
Alan Rominger
f5785976be Update to comply with new black rules 2023-02-01 14:59:38 -05:00
Seth Foster
6492c03965 Fix console color logs 2023-01-05 12:55:20 -05:00
Seth Foster
de41601f27 make job lifecycle Cyan again 2022-10-24 13:50:42 -04:00
Alan Rominger
c59bbdecdb Refactor canceling to work through messaging and signals, not database
If canceled attempted before, still allow attempting another cancel
in this case, attempt to send the sigterm signal again.
Keep clicking, you might help!

Replace other cancel_callbacks with sigterm watcher
  adapt special inventory mechanism for this too

Get rid of the cancel_watcher method with exception in main thread

Handle academic case of sigterm race condition

Process cancelation as control signal

Fully connect cancel method and run_dispatcher to control

Never transition workflows directly to canceled, add logs
2022-09-01 15:20:31 -04:00
Alan Rominger
eb52095670 Fix bug where translated strings will cause log error to error (#11813)
* Fix bug where translated strings will cause log error to error

* Use force_str for ensuring string
2022-02-28 08:38:01 -05:00
Ryan Petrello
4eb85ad23e fix up a bug in rsyslogd error handling
see: https://github.com/ansible/tower/issues/4915
2021-03-26 16:27:53 -04:00
Ryan Petrello
c2ef0a6500 move code linting to a stricter pep8-esque auto-formatting tool, black 2021-03-23 09:39:58 -04:00
Ryan Petrello
4456ae2d71 if rsyslogd cannot be reached, note the failure in sys.stderr
see: https://github.com/ansible/awx/issues/8505
2021-03-03 09:46:33 -05:00
Seth Foster
41d0a2f7b9 Add job lifecycle logging
Various	points (e.g. created, running, processing events), are
structured into	json format and	output to /var/log/tower/job_lifecycle.log

As part	of this	work, the DependencyGraph is reworked to return
which job object is doing the blocking, rather than a boolean.
2021-02-04 12:25:51 -05:00
Alan Rominger
7734def856 Fix inventory log timestamp, organize logging code
The fixes and issue where the timestaps in the stdout for
inventory updates gave the time since the start of the dispatcher
instead of the time since the start of the update.

This commit also moves the handler into the utils module where
other custom AWX handlers live, instead of tasks.py
this is to keep tasks.py relatively clean, as best as possible
2020-11-30 11:36:16 -05:00
Ryan Petrello
7164049062 don't block on log aggregator socket.send() calls
see: https://github.com/ansible/tower/issues/4391
2020-06-08 18:07:33 -04:00
Ryan Petrello
b0db2b7bec add some exception handling for dealing with logging connection resets
when rsyslogd restarts due to config changes, there's a brief moment
where the socket will refuse connections on teardown; exception handling
is needed here to deal with that
2020-04-13 11:44:00 -04:00
Christian Adams
b942fde59a Ensure log messages have valid json
- Fix messages getting contatenated at 8k
 - Fix rsyslog cutting off the opening brace of log messages
 - Make valid default conf and emit logs based on prescence of .sock and
 settings
2020-04-13 11:44:00 -04:00
Ryan Petrello
ce82b87d9f rsyslog hardening (fixing a few weird things we noticed) 2020-04-13 11:44:00 -04:00
Christian Adams
d350551547 Tweaks to Test Button logic and cleans up flake8 and test failures 2020-04-13 11:43:59 -04:00
Christian Adams
7fd79b8e54 Remove unneeded logging sock variable 2020-04-13 11:43:59 -04:00
Shane McDonald
c0af3c537b Configure rsyslog to listen over a unix domain socket instead of a port
- Add a placeholder rsyslog.conf so it doesn't fail on start
 - Create access restricted directory for unix socket to be created in
 - Create RSyslogHandler to exit early when logging socket doesn't exist
 - Write updated logging settings when dispatcher comes up and restart rsyslog so they  take effect
 - Move rsyslogd to the web container and create rpc supervisor.sock
 - Add env var for supervisor.conf path
2020-04-13 11:43:59 -04:00
Ryan Petrello
589d27c88c POC: replace our external log aggregation feature with rsyslog
- this change adds rsyslog (https://github.com/rsyslog/rsyslog) as
  a new service that runs on every AWX node (managed by supervisord)
  in particular, this feature requires a recent version (v8.38+) of
  rsyslog that supports the omhttp module
  (https://github.com/rsyslog/rsyslog-doc/pull/750)
- the "external_logger" handler in AWX is now a SysLogHandler that ships
  logs to the local UDP port where rsyslog is configured to listen (by
  default, 51414)
- every time a LOG_AGGREGATOR_* setting is changed, every AWX node
  reconfigures and restarts its local instance of rsyslog so that its
  fowarding settings match what has been configured in AWX
- unlike the prior implementation, if the external logging aggregator
  (splunk/logstash) goes temporarily offline, rsyslog will retain the
  messages and ship them when the log aggregator is back online
- 4xx or 5xx level errors are recorded at /var/log/tower/external.err
2020-04-13 11:43:59 -04:00
Ryan Petrello
e18639b26b further optimize conf.settings access when logging is enabled
the callback receiver is still fairly slow when logging is enabled due
to constant setting lookups; this speeds things up considerably

related: https://github.com/ansible/awx/pull/5618
2020-01-22 16:17:33 -05:00
Ryan Petrello
8bc6367e1e fix a bug introduced upstream with settings.LOG_AGGREGATOR_AUDIT 2019-10-29 11:24:12 -04:00
Martin Juhl
321aa3b01d Update handlers.py
The setFormatter tries to create the external.log file.. So we should check if LOG_AGGREGATOR_AUDIT is active here as well
2019-10-29 11:24:11 -04:00
Ryan Petrello
a076e84a33 add a settings flag for writing all external logs to disk 2019-10-09 13:54:54 -04:00
Ryan Petrello
dfa8d44eb8 don't ship external logs from the main thread of the dispatcher
this is a fairly esoteric change that attempts to work around a bug
we've discovered in cpython itself

context: https://github.com/ansible/awx/issues/4181
2019-06-27 16:24:36 -04:00
AlanCoding
d3f047d731 Move dynamic log level logic to filter 2019-05-22 14:00:39 -04:00
Ryan Petrello
b1d75327e3 add the ability to toggle DEBUG logging on dynamically 2019-05-16 07:58:31 -04:00
Ryan Petrello
daeeaf413a clean up unnecessary usage of the six library (awx only supports py3) 2019-01-25 00:19:48 -05:00
Ryan Petrello
f223df303f convert py2 -> py3 2019-01-15 14:09:01 -05:00
Ryan Petrello
6bd9792518 default HTTP-based log emits to HTTPS
see: https://github.com/ansible/awx/issues/2048
2018-07-09 11:52:04 -04:00
AlanCoding
015b19d8c3 interpret null protocol as logging.NullHandler 2018-05-30 15:21:51 -04:00
AlanCoding
ac20aa954a Replace logging-related restart with dynamic handler
refactor existing handlers to be the related
  "real" handler classes, which are swapped
  out dynamically by external logger "proxy" handler class

real handler swapout only done on setting change

remove restart_local_services method
get rid of uWSGI fifo file

change TCP/UDP return type contract so that it mirrors
  the request futures object
add details to socket error messages
2018-05-02 09:47:22 -04:00
Ryan Petrello
982539f444 fix a bug when testing UDP-based logging configuration
see: https://github.com/ansible/ansible-tower/issues/7868
2018-01-29 12:05:51 -05:00
AlanCoding
67867cf0c8 flake8: comply with new E722 rule 2017-12-01 12:16:44 -05:00
Matthew Jones
c7a85d9738 Mass rename from ansible_(awx|tower) -> (awx|tower) 2017-07-26 13:33:26 -04:00
Alan Rominger
047624a857 Merge pull request #6944 from AlanCoding/splunk_host
handle Splunk hostname case from support
2017-07-10 21:07:12 -04:00
AlanCoding
459c33d272 More consistently provide fields in job_events logger
* Change scheme from using event dict to JobEvent object
* Add processing to grok object fields
* Allow override of provided formatter in case of future issues
2017-07-10 10:34:04 -04:00
AlanCoding
d23e11c43c handle Splunk hostname case from support 2017-07-07 21:03:16 -04:00
Aaron Tan
9874e123be Allow specifying log handler level. 2017-07-06 09:39:54 -04:00
Chris Meyers
fee16fe391 Merge pull request #6588 from chrismeyersfsu/feature-fact_cache
initial tower fact cache implementation
2017-06-22 09:58:28 -04:00
Ryan Petrello
45bdd9f747 improve readability of the honcho console logs in the dev environment
* colorize uwsgi and celery logs; DEBUG lines are green, WARN lines
  are yellow, ERROR lines (and tracebacks) are red
* pretty-print fact callback receiver JSON
* simplify the uwsgi log format so it's more legible
2017-06-22 09:19:32 -04:00
Chris Meyers
965159a1fb rename store_facts to use_fact_cache 2017-06-21 13:45:27 -04:00
Chris Meyers
4c118159ed system_tracking logging for fact cache 2017-06-21 13:45:27 -04:00
Matthew Jones
e1e83598e9 Merge branch 'release_3.1.3' into devel
* release_3.1.3: (52 commits)
  ack fact scan messages
  making ldap user/group search fields into codemirror instances
  removing UI parsing for LDAP User and Group Search fields
  Allow exception view to accept all valid HTTP methods.
  Restore ability of parsing extra_vars string for provisioning callback.
  Fix up backup/restore role broken in f7a8e45809758322d9ee41c5305850dd70ed5faf
  Stop / start ansible-tower-service during restores
  value_to_python should encode lookup fields as ascii
  fix brace interpolation on standard out pane
  Adjust some hardcoded usages of 'awx' to use 'aw_user' and 'aw_group'.
  Pull Spanish updates from Zanata
  Temporarily grant awx user createdb role
  Stop giving ownership of backups to postgres
  don't display chunked lines'
  Add dropdown li truncation with ellipsis
  CTiT -> adhoc modules should allow the user to add new modules
  Remove task that was replacing the supervisor systemd tmp file
  Fix failing supervisorctl commands on RH-based distros
  Give ownership of the supervisor socket to awx
  Setting for external log emissions cert verification
  ...
2017-04-28 13:57:04 -04:00
Aaron Tan
8d2ee8c30f Refactor log handler and support TCP/UDP communications 2017-04-25 11:07:57 -04:00
AlanCoding
3af0f505d8 Setting for external log emissions cert verification
Stopgap solution for issue for 3.1.3 where Splunk servers with a
self-signed certificate could not receive logs from Tower.
Users should set `LOG_AGGREGATOR_VERIFY_CERT = False` in settings
to use this.
2017-04-17 14:50:01 -04:00
Alan Rominger
f3dd7f7987 Remove async from logging test kwargs 2017-04-12 09:22:50 -04:00