Commit Graph

480 Commits

Author SHA1 Message Date
Matthew Jones
ea8b78ca49 Protect cluster nodes after an upgrade
* Modify instance model to container a version number for the node
* Update that version number during the heartbeat
* If during a heartbeat any of the nodes are of a newer version then
  shutdown the current node.

The idea behind this is that if all nodes were upgraded at the same
time then at the moment of the healthcheck they should all be at the
newer version. Otherwise we put the system in a state where it can
receive the upgrade but stay down until that happens. During setup
playbook run the services will be fully restarted.
2017-04-10 15:37:33 -04:00
Ryan Petrello
b3733d233e always use async http logging - even in celery workers
additionally, add a timeout to requests to avoid thread starvation

see: #5718
see: #5729
2017-03-15 13:17:01 -04:00
Matthew Jones
bc4174adf8 Ignore .retry files for project updates 2017-03-14 13:23:14 -04:00
Chris Meyers
4005cf927e copy existing env before making a safe env
* We should not modify the original env because it's what is going to be
passed to the Ansible process. This will contain all of the passwords,
unobscured. The safe env is what will get saved into the model.
2017-03-07 16:48:48 -05:00
Ryan Petrello
ae321536cd fix a regex that doesn't properly strip certain environment variables
see: #5601
2017-03-07 16:48:48 -05:00
Matthew Jones
95a2ed6336 Updates to insights playbook project syncing based on feedback
* Support "name" if given, else use a more reasonable file name for
  the playbook file
* Fix an issue where we wouldn't populate the playbooks correctly in
  the database
* Force downloading the playbooks every time since we can't tell when
  the file might change
2017-03-07 16:48:48 -05:00
Matthew Jones
2656e3f3fa Insights project api side integration
* Extend scm credential type to support username and password for
  redhat portal accounts
* Update project update playbook to work with remote insights server
2017-03-07 16:48:44 -05:00
Matthew Jones
90a26de180 Fix an issue where notifications might not always be triggered
In the case where a notification template were assigned to "failed"
events but not "success" or "any" events.
2017-03-06 15:41:43 -05:00
Matthew Jones
98daaece73 Switch job_type to check from sync when detecting delete_on_update 2017-02-27 16:22:05 -05:00
Chris Meyers
dd513621f0 Revert "Merge pull request #5553 from chrismeyersfsu/fix-waiting_blocked"
This reverts commit 9ba2122f4f85eecaeb6fa53ac92ea2811b05e83f, reversing
changes made to c3a5f2c96fd85dd1405a8f5c875ffc988dee16a4.
2017-02-27 09:38:45 -05:00
Chris Meyers
d020c81047 Merge pull request #5553 from chrismeyersfsu/fix-waiting_blocked
just like we fail running tasks fail waiting tasks
2017-02-24 16:18:02 -05:00
Chris Meyers
903d0472f0 just like we fail running tasks fail waiting tasks
* Associate the celery_id with the job at the earliest point possible.
This ensures that a waiting job has a celery id. Thus, we are free to
fail waiting jobs that don't have a celery id.
2017-02-24 12:07:04 -05:00
Matthew Jones
1a946d51c1 Fix an issue with sync vs run project updates
We need to not remove the source tree on project update run which
happens in the context of the job run to prevent the directory from
being deleted out from under the job runners. Since those will
inherently not block each other as the project update would
2017-02-24 11:03:48 -05:00
Matthew Jones
fd7c2550ee Fix a potential security issue in revision storage path name
It's too stable and could lead to a race condition
2017-02-23 16:21:13 -05:00
Matthew Jones
6f84849f1a Disable retry files on ansible runs 2017-02-17 09:55:52 -05:00
AlanCoding
7e3a5fd2c2 move reload functionality to its own file 2017-02-15 13:42:49 -05:00
AlanCoding
186b672e4f move service definition into settings 2017-02-15 12:07:30 -05:00
AlanCoding
d8f133d175 Add supervisor command to restart select services on reload event 2017-02-15 09:13:59 -05:00
AlanCoding
cdf28f1bca remove old logger reconfig method 2017-02-10 11:54:19 -05:00
AlanCoding
91031cbbc8 modularize logging config in proper Django fashion 2017-02-10 11:38:32 -05:00
AlanCoding
1d6a36876b have tasks.py deduce location of uwsgi fifo 2017-02-09 15:27:58 -05:00
AlanCoding
93034ef1d6 surface dry_run option for system jobs through the API 2017-02-07 10:37:41 -05:00
Matthew Jones
1fc69038a6 Propogate stdout change to mgmt jobs 2017-02-03 15:46:16 -05:00
Ryan Petrello
fde4d5c1f6 Merge pull request #5083 from ryanpetrello/fix-5030
don't stringify source_vars['private'] for Openstack inventory updates
2017-02-01 08:53:17 -05:00
Alan Rominger
242f769f7e Merge pull request #5059 from AlanCoding/yaml_extra_data
consume YAML extra_data on schedule spawn
2017-01-31 16:00:48 -05:00
Matthew Jones
e389084298 Add a periodic celery task to clean up local stdout files
This works in tandem with the on-demand stdout file generator and
decouples stdout locally from the persistent database representation
ob the job itself
2017-01-31 15:52:45 -05:00
Ryan Petrello
f4a21a2650 don't stringify source_vars['private'] for Openstack inventory updates
shade's OS_CLIENT_CONFIG_FILE expects the generated YAML value to be
a boolean, not a stringified boolean

Addresses #5030
2017-01-31 14:57:12 -05:00
Matthew Jones
53256f4c2d Integrate proper cleanup for tmp paths for inventory updates 2017-01-31 11:31:44 -05:00
AlanCoding
025bc5de8d consume YAML extra_data on schedule spawn 2017-01-31 08:37:33 -05:00
Matthew Jones
587a843acd Shorten the default control persist path
Customer needed a few more characters, this should give a bit more breathingroom
2017-01-30 13:49:20 -05:00
Matthew Jones
66426aa7dd Store project update and inventory update stdout in the database
So things work across the cluster
2017-01-27 16:51:03 -05:00
Alan Rominger
1b0c2cbb32 Merge pull request #4986 from AlanCoding/destroyer_of_notifications
Add notifications to cleanup_jobs
2017-01-27 10:54:24 -05:00
Matthew Jones
cf084850c2 Merge pull request #4960 from ansible/priv_esc_hangs
Fixes some issues where Tower would not emit the password on prompts
2017-01-27 09:51:57 -05:00
AlanCoding
33e97d5c5a add notifications to cleanup_jobs 2017-01-27 09:31:11 -05:00
Matthew Jones
e15273c993 Fixes some issues where Tower would not emit the password on prompts
Carat seems to only want to match newlines or the beginning of the
string, since we seem to get CR+LF this seems to keep the carat from
matching so here we're expanding the regex matches.

This also extends support for DZDO where it was only originally added
on ad-hoc commands inadvertantly.

Remove PMRUN support since that was never merged into core
2017-01-26 11:19:23 -05:00
Ryan Petrello
f061505895 move the uwsgi master FIFO from /tmp/awxfifo to /var/lib/awx/awxfifo
Resolves #4900
2017-01-25 16:37:11 -05:00
Matthew Jones
986ddae33e Merge pull request #4882 from matburt/cache_behavior_fixes
Fixes for data corruption/exception in cache usage
2017-01-24 11:12:28 -05:00
Matthew Jones
b3f4f3bea5 Fix issue reassembling ad-hoc event partials 2017-01-23 16:30:44 -05:00
Matthew Jones
abdd91bd1f Fixes for data corruption/exception in cache usage
Specifically as it relates to serializers and job event writing at
high speeds
2017-01-23 14:39:03 -05:00
Matthew Jones
94ffce07a4 Fix an issue processing job events when uuid isn't present 2017-01-19 09:47:10 -05:00
Matthew Jones
7ca496b6ef Merge branch 'release_3.1.0' into multi_worker_callbacks 2017-01-18 20:39:15 -05:00
AlanCoding
81eb16523c add workflow jobs to cleanup_jobs management job 2017-01-18 11:50:28 -05:00
Matthew Jones
1cdeb4d2af Refactor event emitter to not use event partials
event emitter now caches to memcache where it is eventually picked up
by the stdout event emitter. This obviates event reassembly in the
callback receiver.
2017-01-17 15:59:43 -05:00
Alan Rominger
6921cbfdd0 Merge pull request #4538 from AlanCoding/celery_log_reset
Celery log reset on CTiT change / task start
2017-01-06 15:36:45 -05:00
AlanCoding
df90328531 remove duplicate conversion of keys into set 2017-01-05 15:08:52 -05:00
Matthew Jones
89304b2ee0 Don't ignore ANSIBLE_NET env items when hiding sensitive information 2017-01-05 12:35:50 -05:00
AlanCoding
f76680b625 add back in the Tower logger, fix bug 2017-01-05 10:28:23 -05:00
AlanCoding
cc4aa49556 pool restart with thread-based external log config
for CTiT enablement of celery task Tower logs
2017-01-04 14:53:49 -05:00
Chris Meyers
55047a5643 letters, they matter 2017-01-04 11:25:39 -05:00
Chris Meyers
721de9f10a pass network ssh key via an env var, not ssh-agent 2017-01-03 16:17:52 -05:00