136 Commits

Author SHA1 Message Date
Matthew Jones
8505783350
Merge remote-tracking branch 'tower/release_3.2.3' into devel
* tower/release_3.2.3:
  fix unicode bugs with log statements
  use --export option for ansible-inventory
  add support for new "BECOME" prompt in Ansible 2.5+ for adhoc commands
  enforce strings for secret password inputs on Credentials
  fix a bug for "users should be able to change type of unused credential"
  fix xss vulnerabilities - on host recent jobs popover - on schedule name tooltip
  fix a bug when testing UDP-based logging configuration
  bump templates form credential_types page limit
  Wait for Slack RTM API websocket connection to be established
  don't process artifacts from custom `set_stat` calls asynchronously
  don't overwrite env['ANSIBLE_LIBRARY'] when fact caching is enabled
  only allow facts to cache in the proper file system location
  replace our memcached-based fact cache implementation with local files
  add support for new "BECOME" prompt in Ansible 2.5+
  fix a bug in inventory generation for isolated nodes
  properly handle unicode for isolated job buffers
2018-02-20 12:22:25 -05:00
cclauss
260aec543e Misc Python 3 changes: execfile(), file(), reduce(), StandardError 2018-02-09 17:17:05 +01:00
cclauss
e1a8b69736 from six.moves import xrange for Python 3 2018-02-08 22:41:33 +01:00
Ryan Petrello
d57470ce49
don't process artifacts from custom set_stat calls asynchronously
previously, we persisted custom artifacts to the database on
`Job.artifacts` via the callback receiver.  when the callback receiver
is backed up processing events, this can result in race conditions for
workflows where a playbook calls `set_stat()`, but the artifact data is
not persisted in the database before the next job in the workflow starts

see: https://github.com/ansible/ansible-tower/issues/7831
2018-01-23 17:09:23 -05:00
Ryan Petrello
fc94b3a943
generalize stdout event processing to emit events for *all* job types
* introduces three new models: `ProjectUpdateEvent`,
  `InventoryUpdateEvent`, and `SystemJobEvent`
* simplifies the stdout callback management in `tasks.py` - now _all_
  job run types capture and emit events to the callback receiver
* supports stdout reconstruction from events for stdout downloads for
  _all_ job types
* configures `ProjectUpdate` runs to configure the awx display callback
  (so we can capture real playbook events for `project_update.yml`)
* ProjectUpdate, InventoryUpdate, and SystemJob runs no longer write
  text blobs to the deprecated `main_unifiedjob.result_stdout_text` column

see: https://github.com/ansible/awx/issues/200
2018-01-03 09:09:42 -05:00
AlanCoding
6269b43456
update tests to new Ansible core code 2017-12-01 12:31:56 -05:00
AlanCoding
67867cf0c8
flake8: comply with new E722 rule 2017-12-01 12:16:44 -05:00
AlanCoding
e5d2eb9f3d
update tests to new Ansible core code 2017-10-26 08:34:00 -04:00
AlanCoding
90f63774f4
flake8: comply with new E722 rule 2017-10-23 14:36:48 -04:00
Chris Meyers
4e2cf62e89 fixes ansible callback import json warning
[WARNING]: Failure using method (v2_runner_on_ok) in callback plugin
(<awx_display_callback.module.AWXDefaultCallbackModule object at
0x47f6090>):
'module' object has no attribute 'dumps'

The above error is thrown by ansible if callback plugins don't respect
the same import precedence configuration as Ansible. ansible callback/*
dir includes a json.py file. This is imported by ansible
callback/__init__.py when a callback plugin implementation imports from
Ansible callback base without setting the correct import precedence.
2017-10-16 10:29:41 -04:00
Matthew Jones
d4b1a07495 Rename tower display plugins to awx display 2017-07-26 13:33:30 -04:00
AlanCoding
b393a60ece prevent overwriting of playbook data when censoring 2017-07-11 08:45:55 -04:00
Ryan Petrello
405c01a847 more isolated production tinkering
see: #5903
see: #6507
2017-06-29 09:35:26 -04:00
AlanCoding
40287d8e78 multi-host isolated heartbeat w tower-isolated check
* use tower-expect command to determine job status when running
  the isolated heartbeat playbook
* grok JSON output of playbook to obtain result information
* run playbook against multiple isolated hosts at the same time
  (addresses scalability concerns)
2017-06-20 14:36:18 -04:00
AlanCoding
7eb8fdd0f1 Change use of subprocess lib for calculating system capacity
Suggested by Ryan Petrello as a part of the PR feedback
for the isolated instance setup work.
2017-06-19 12:13:36 -04:00
AlanCoding
1d9f2be0e4 add task in isolated heartbeat to delete rogue files 2017-06-19 12:13:36 -04:00
AlanCoding
dd1a261bc3 setup playbook and heartbeat for isolated deployments
* Allow isolated_group_ use in setup playbook
* Tweaks to host/queue registration commands complementing setup
* Create isolated heartbeat task and check capacity
* Add content about isolated instances to acceptance docs
2017-06-19 12:13:36 -04:00
Ryan Petrello
44e0c8621a isolated ramparts: replace systemd unit with a tower-expect binary
instead of launching isolated tasks via `systemctl`, treat
`awx.main.isolated.run` as an executable that knows how to daemonize

additionally, add `setup.py isolated_build` for isolated Tower source
distribution
2017-06-16 09:59:21 -04:00
Ryan Petrello
422950f45d Support for executing job and adhoc commands on isolated Tower nodes (#6524) 2017-06-14 11:47:30 -04:00
Ryan Petrello
f6046d47c9 work around a change in ansible 2.3 to Task.get_name()
Properly detect setup tasks so that we can filter out sensitive
variables from the `ansible_facts` portion of the event payload.
2017-05-01 10:38:05 -04:00
Aaron Tan
6884e44bb3 Unit test added. 2017-04-11 11:40:47 -04:00
Aaron Tan
9b1107c054 Prevent unexpected callback module attribute warning. 2017-04-10 14:40:57 -04:00
Ryan Petrello
f9b5b9aa91 property filter no_log for item event loops
see: #5691
2017-03-22 11:15:33 -04:00
Ryan Petrello
6a7743b274 fix a callback bug that causes a task_args leak between job events
see: #5802
2017-03-20 12:05:41 -04:00
Ryan Petrello
999a304c3b add no_log filtering for runner_item_on_xyz events
see: #5691
2017-03-17 09:54:46 -04:00
Ryan Petrello
66c786b971 simplify memcache patching in our callback plugin tests 2017-03-14 14:07:53 -04:00
Ryan Petrello
a5ea468bfe add a test to ensure that job events properly strip out environment vars
see: #5601
2017-03-14 12:56:44 -04:00
Ryan Petrello
5bbb824865 add tests for no_log filtering
see: #5691
2017-03-14 12:56:44 -04:00
Ryan Petrello
1005782ee9 add tests for our custom ansible callback plugin 2017-03-14 12:56:40 -04:00
Ryan Petrello
6af9638d81 job event data marked with _ansible_no_log should be censored
see: #5691
2017-03-10 15:47:39 -05:00
Ryan Petrello
ed5a01c0cf strip ansible_env from job event callback data
this often contains sensitive details, and the data is already filtered
and stored on the job itself

see: #5601
2017-03-07 16:48:48 -05:00
Matthew Jones
2621f48189 We specifically need to convert the ansible included file type
for include events
2017-02-01 15:30:22 -05:00
Matthew Jones
d9a7f2fadb Silently ignore psutil errors when the process is already gone
During ssh CP cleanuip
2017-01-30 13:39:05 -05:00
Matthew Jones
7ca496b6ef Merge branch 'release_3.1.0' into multi_worker_callbacks 2017-01-18 20:39:15 -05:00
Matthew Jones
1cdeb4d2af Refactor event emitter to not use event partials
event emitter now caches to memcache where it is eventually picked up
by the stdout event emitter. This obviates event reassembly in the
callback receiver.
2017-01-17 15:59:43 -05:00
Wayne Witzel III
9592c67c31 fixing flake8 errors 2017-01-16 14:16:24 -05:00
Wayne Witzel III
cd5eed9828 remove extra censoring code from tower, not needed since Ansible 2.1+ 2017-01-16 13:54:07 -05:00
Matthew Jones
974fe1d244 Fix an issue with set_stats for unsupported Ansible versions 2017-01-11 10:24:17 -05:00
Chris Meyers
86a9a921eb remove deepcopy, not sure how that slipped in 2017-01-06 11:47:45 -05:00
Chris Meyers
a1267a3dee update artifacts to work with ansible set_stats 2017-01-06 10:04:57 -05:00
Chris Meyers
03734015c5 set_stats support 2017-01-05 13:42:29 -05:00
Matthew Jones
a61e729ebb Purge event res dict if it is over a certain size
Also purge/update some old settings values
2016-12-14 15:05:28 -05:00
Chris Church
b85c98afd2 Split job event data between callback queue and stdout. Send most of event data directly over queue and capture only stdout/counter/start_line/end_line in celery task; recombine into single event in callback receiver. 2016-12-12 16:34:36 -05:00
Chris Church
ae16cadcc2 Ignore exception in sitecustomize. 2016-12-07 23:50:58 -05:00
Matthew Jones
24b858c6fb Switch away from deepcopy to improve performance 2016-12-06 13:33:14 -05:00
Chris Church
d6d5d4263a Only run ssh control persist cleanup from main process, flush after writing end marker for job event data. 2016-12-01 14:47:24 -05:00
Chris Church
1db697e4eb Add multiprocessing RLock around event data output to ensure only one process is writing to stdout at a time. 2016-11-17 09:09:19 -05:00
Aaron Tan
9e4655419e Fix flake8 E302 errors. 2016-11-15 20:59:39 -05:00
Aaron Tan
473ed016de Fix flake8 E305 errors. 2016-11-15 16:32:27 -05:00
Chris Church
5760654671 Fix warning from v2_playbook_on_task_start with ad hoc commands. 2016-11-07 21:42:39 -05:00