Commit Graph

97 Commits

Author SHA1 Message Date
Chris Meyers
8a902debd5 Per-service metrics http server
* Organize metrics into their respective service
* Server per-service metrics on a per-service http server
* Increase prometheus client usage over our custom metrics fields
2024-02-05 15:17:24 -05:00
Ryan Petrello
c2ef0a6500 move code linting to a stricter pep8-esque auto-formatting tool, black 2021-03-23 09:39:58 -04:00
Ryan Petrello
57f8e48894 make --status more robust for dispatcher, and add support for receiver
make the --status flag work by fetching a periodically recorded snapshot
of internal process state; additionally, update the callback receiver to
*also* record these statistics so we can gain more insight into any
performance issues
2020-09-17 15:33:37 -04:00
chris meyers
be58906aed remove kombu 2020-03-18 16:10:17 -04:00
chris meyers
2a2c34f567 combine all the broker replacement pieces
* local redis for event processing
* postgres for message broker
* redis for websockets
2020-03-18 16:10:15 -04:00
chris meyers
355fb125cb redis events 2020-03-18 16:10:15 -04:00
chris meyers
c8eeacacca POC channels 2 2020-03-18 16:10:12 -04:00
Ryan Petrello
40b1e89b67 add the ability to disable RabbitMQ queue durability 2019-05-28 15:49:32 -04:00
Ryan Petrello
17a803f49c remove the old callback plugin import paths and callback-specific tests 2019-04-12 16:11:23 -04:00
Ryan Petrello
da74f1d01f refactor and test the callback receiver as a base for a task dispatcher 2018-10-11 10:53:26 -04:00
Ryan Petrello
87adfe5889 close DB and cache sockets _immediately_ before we fork callback workers 2018-08-15 15:10:08 -04:00
Jake McDermott
60a38a196a add final_counter to EOF websocket 2018-06-11 18:36:28 -04:00
AlanCoding
f54ac776cd add final_line_count to EOF websocket 2018-05-21 15:47:43 -04:00
Ryan Petrello
f59f47435b send job notification templates _after_ all events have been processed
see: https://github.com/ansible/awx/issues/500
2018-03-22 09:30:41 -04:00
Wayne Witzel III
91c6d406c5 Rollback celery 2018-02-22 09:37:14 -05:00
Ryan Petrello
51f7907a01 optimize OutputEventFilter for large stdout streams
update our event data search algorithm to be a bit lazier in event data
discovery; this drastically improves processing speeds for stdout >5MB

see: https://github.com/ansible/awx/issues/417
2018-01-16 14:41:35 -05:00
Ryan Petrello
c6b6a3ad89 send and subcribe to summary events for all new events 2018-01-03 09:09:45 -05:00
Ryan Petrello
fc94b3a943 generalize stdout event processing to emit events for *all* job types
* introduces three new models: `ProjectUpdateEvent`,
  `InventoryUpdateEvent`, and `SystemJobEvent`
* simplifies the stdout callback management in `tasks.py` - now _all_
  job run types capture and emit events to the callback receiver
* supports stdout reconstruction from events for stdout downloads for
  _all_ job types
* configures `ProjectUpdate` runs to configure the awx display callback
  (so we can capture real playbook events for `project_update.yml`)
* ProjectUpdate, InventoryUpdate, and SystemJob runs no longer write
  text blobs to the deprecated `main_unifiedjob.result_stdout_text` column

see: https://github.com/ansible/awx/issues/200
2018-01-03 09:09:42 -05:00
Matthew Jones
9dbcc5934e Merge remote-tracking branch 'tower/release_3.2.2' into devel 2017-12-13 12:25:47 -05:00
Wayne Witzel III
de376292ba Update management commands 2017-11-09 17:18:18 -05:00
Ryan Petrello
8d2ab3de42 improve the callback worker's ability to deal with idle/disconnected DB
if database connectivity is lost, callback workers currently raise an
uncaught exception and hang; this can cause the entire process to stop
handling callback events

see: https://github.com/ansible/ansible-tower/issues/7660
2017-10-31 09:51:13 -04:00
AlanCoding
9ad34edc38 graceful killing of receiver worker processes 2017-08-15 12:13:47 -04:00
AlanCoding
8be1bea33a import os, fixing bug that forced SIGKILL 2017-08-15 06:32:21 -04:00
Ryan Petrello
45bdd9f747 improve readability of the honcho console logs in the dev environment
* colorize uwsgi and celery logs; DEBUG lines are green, WARN lines
  are yellow, ERROR lines (and tracebacks) are red
* pretty-print fact callback receiver JSON
* simplify the uwsgi log format so it's more legible
2017-06-22 09:19:32 -04:00
Matthew Jones
8c4d4a0543 Protect Tower from Ansible core making changes to the event uuids 2017-02-13 11:55:08 -05:00
Matthew Jones
1cbbb6a87c Make full queue writes silent unless we can't write to any of them
This will still keep general queue errors other than Queue.Full
2017-02-06 13:53:41 -05:00
Matthew Jones
abdd91bd1f Fixes for data corruption/exception in cache usage
Specifically as it relates to serializers and job event writing at
high speeds
2017-01-23 14:39:03 -05:00
Matthew Jones
73f92bba5d Integrate multiprocessing writer worker in callback receiver 2017-01-18 12:05:29 -05:00
Matthew Jones
1cdeb4d2af Refactor event emitter to not use event partials
event emitter now caches to memcache where it is eventually picked up
by the stdout event emitter. This obviates event reassembly in the
callback receiver.
2017-01-17 15:59:43 -05:00
Matthew Jones
e594a3ee3e Implement a worker-based callback receiver system 2017-01-10 14:25:41 -05:00
Chris Church
b85c98afd2 Split job event data between callback queue and stdout. Send most of event data directly over queue and capture only stdout/counter/start_line/end_line in celery task; recombine into single event in callback receiver. 2016-12-12 16:34:36 -05:00
Aaron Tan
9e4655419e Fix flake8 E302 errors. 2016-11-15 20:59:39 -05:00
Matthew Jones
c6cf02a602 Merge branch 'release_3.0.3' into devel
* release_3.0.3: (55 commits)
  Revert "Revert "Add needed types for selinux change""
  Revert "Add needed types for selinux change"
  interpret backslash escapes when displaying url in welcome message
  Bump the SELinux policy version
  Add needed types for selinux change
  Update SELinux policy to allow httpd_t to execute files in lib_t and var_lib_t
  Bumping changelog for 3.0.3
  Update rax.py inventory
  Revert "filter internal User.admin_roles from the /roles API list view"
  fix spelling of disassociated
  Resolves 404 when assigning resources/users to organizations in card view. Sidesteps a bug in the Refresh() utility, where pagination calculations are not made against filtered results.
  Sync azure changes to Tower virtual environment
  Add regions here as well.
  Also bump boto for new regions, per ryansb.
  More regions!
  Revert "bump shade version"
  bump shade version
  Hack copying of job_template.related.survey_spec into ui job copy flow, resolves #3737
  Revert "bump shade version"
  bump shade version
  ...
2016-11-01 11:49:28 -04:00
Chris Church
440f0539b0 Flake8 fixes. 2016-10-28 21:58:24 -04:00
Chris Church
c18b6c1352 Add support for capturing stdout associated with job events and ad hoc command events.
* New event types for stdout lines not associated with a callback event.
* New stdout, start_line, end_line and verbosity fields for job/ahc events.
* Callback plugins to wrap Ansible default/minimal stdout callbacks and embed callback event data using ANSI escape sequences.
* Callback plugin library to wrap ansible.display.Display class methods.
* Output filter to extract event data from stdout and create job/ahc events.
* Update stdout formats to strip new ANSI escape sequences.
2016-10-28 21:58:24 -04:00
AlanCoding
5d4cf9d4fc Add job artifacts and workflow artifact passing
artifacts redact from job when no_log is set
parent no_log artifacts treated as survey passwords
2016-10-12 17:45:50 -04:00
Aaron Tan
38f5ae21f8 Set up post_process flag for JobEvent save in callback receiver. 2016-09-28 14:14:29 -04:00
Matthew Jones
3de4aae548 Fixing up HA induced flake8 issues 2016-09-15 13:51:17 -04:00
Matthew Jones
56d4210047 Implement Performance improvements shown in #3492
* Make callback workers tunable
* Disable callback worker recycling - doesn't appear to be needed
  anymore
* Tweak pexpect behavior by limiting its read buffer size and search
  window
* Use copy instead of deepcopy for job event callback emitter censor
2016-09-14 16:08:47 -04:00
Matthew Jones
ab395b0009 Integrate callback receiver refactoring
* Drop ZMQ as the communication mechanism between job_event_callback and
  callback_receiver
* Setup queue and exchange for callback broker communication
* Refactor event plugin and callback receiver to efficiently handle
  message submission and processing
* Integrate django caching for parent processing
2016-09-14 11:42:13 -04:00
Matthew Jones
0c1e1fa2fb Refactor Tower HA Instance logic and models
* Gut the HA middleware
* Purge concept of primary and secondary.
* UUID is not the primary host identifier, now it's based mostly on the
  username.  Some work probably still left to do to make sure this is
  legit.  Also removed unique constraint from the uuid field.  This
  might become the cluster ident now... or it may just deprecate
* No more secondary -> primary redirection
* Initial revision of /api/v1/ping
* Revise and gut tower-manage register_instance
* Rename awx/main/socket.py to awx/main/socket_queue.py to prevent
  conflict with the "socket" module from python base
* Revist/gut the Instance manager... not sure if this manager is really
  needed anymore
2016-09-08 13:37:53 -04:00
Akita Noek
b57739a800 Converted except T,e expressions to except T as e 2016-06-27 15:12:37 -04:00
Chris Meyers
ae73be12f1 remove statsd 2016-05-10 11:56:04 -04:00
Matthew Jones
0e2184902e Handle runner items from ansible v2
Also denote whether the trailing runner_on_ was a loop event
2016-03-22 12:31:05 -04:00
Chris Church
60224cdbe4 Update Django to 1.8 and DRF to 3.3, add new Django migrations, update serializers/pagination/metadata, update browsable API styling. 2016-02-02 17:48:04 -05:00
Joshua "jag" Ginsberg
f25b76aee9 Integrate statsd metrics into ansible playbook execution.
* Add dependencies for pystatsd and django-statsd-mozilla
* Default turned off except for development environment
* Modify docker-compose to install statsd/graphite host
2016-01-26 16:54:24 -05:00
Matthew Jones
b579b76099 Handle playbook_on_include events from v2 ansible 2015-10-20 11:18:07 -04:00
John Mitchell
32d1c0e4db fixed copyright date 2015-06-11 16:10:23 -04:00
Matthew Jones
b3da3b34a3 Changing some legal headers for python source files 2015-05-29 12:10:39 -04:00
Chris Church
f7b8d510dc Implement support for ad hoc commands. 2015-03-30 13:04:19 -04:00