Commit Graph
100 Commits
Author SHA1 Message Date
chris meyers 6606a29f57 celery 4.x -> 3.x change route config name 2018-02-27 14:13:05 -05:00
chris meyers 148baf7674 add explicit awx_celery container version 2018-02-27 11:37:10 -05:00
chris meyers 5918fa5573 remove () from postgres port value
* awx task container uses postgres port to wait for postgres to become
available before the container init continues. The () are problematic
and are removed.
* () was originally added to fix an openshift issues. That error does
NOT occur with this fix.
2018-02-27 11:36:55 -05:00
chris meyers e4470aa4cf remove uneeded celery configs
* Celery routes and queues are set and defined at runtime. Thus, a
static definition of routes and queues is not needed.
2018-02-27 11:36:55 -05:00
chris meyers fe05b4c0d5 use celery 3.x BROKER_URL
* Celery 4.x specifies the broker via CELERY_BROKER_URL. Since we are
now on 3.x, use 3.x way of specifying the broker via BROKER_URL
2018-02-27 11:36:55 -05:00
Chris Meyers aaf87c0c04 xss test for per-host recent jobs popup 2018-02-06 10:37:00 -05:00
Chris Meyers 0e97dc4b84 Beat and celery clustering fixes
* use embedded beat rather than standalone
* dynamically set celeryd hostname at runtime
* add embeded beat flag to celery startup
* Embedded beat mode routes will piggyback off of celery worker setup
signal
2018-02-01 16:47:33 -05:00
Chris Meyers c9ff3e99b8 celeryd attach to queues dynamically
* Based on the tower topology (Instance and InstanceGroup
relationships), have celery dyamically listen to queues on boot
* Add celery task capable of "refreshing" what queues each celeryd
worker listens to. This will be used to support changes in the topology.
* Cleaned up some celery task definitions.
* Converged wrongly targeted job launch/finish messages to 'tower'
queue, rather than a 1-off queue.
* Dynamically route celery tasks destined for the local node
* separate beat process

add support for separate beat process
2018-02-01 16:37:33 -05:00
Chris Meyers 290a296f9f add xss test for jobs schedules
* Test for tooltip regression on job schedules list entries
2018-02-01 10:55:13 -05:00
Chris Meyers 0402064c0f expose ansible_facts_modified 2018-01-17 10:28:34 -05:00
Chris Meyers e33265e12c add job_id to fact cache log output 2018-01-17 10:19:27 -05:00
Chris Meyers e49dfd6ee2 only run saml pipeline if saml social auth
* Do not trigger saml social auth pipeline methods if the user logging
in was not created by the saml social auth backend.
2018-01-11 16:20:49 -05:00
Chris Meyers 2ed97aeb0c implement multiple ldap servers 2018-01-11 09:03:14 -05:00
Chris Meyers de02138dfd spelling is hard 2018-01-10 09:26:11 -05:00
Chris Meyers 0a9d3d47b9 more efficiently determine saml team mapping 2018-01-09 12:16:07 -05:00
Chris Meyers 664bdec57f add documentation 2018-01-05 14:43:33 -05:00
Chris Meyers 9d58b15135 allow for saml attributes to define team and org
related to https://github.com/ansible/awx/issues/217

* Adds a configure tower in tower setting for users to configure a saml
attribute that tower will use to put users into teams and orgs.
2018-01-04 15:35:11 -05:00
Chris Meyers a0038276a4 do not use a custom task exception
* Celery + json pickling do not handle custom Exceptions (and may never
do so). Mentioning of, if handling custom Exceptions then the code would
be susceptible to same arbitrary code execution that python pickle is
vulnerable to.
* So don't use custom Exceptions.
2018-01-04 15:30:52 -05:00
Chris Meyers f0ff6ecb0a handle_work_error signature to work
* celery error callback signature isn't well defined. Thus, our error
callback signature is made to handle just about any call signature and
depend on only 1 attribute, id, existing.

See https://github.com/celery/celery/issues/3709
2018-01-04 15:23:13 -05:00
Chris Meyers 032318494b added tests for new settings field type 2017-11-29 11:52:00 -05:00
Chris Meyers 383c3cfe3e add more saml fields 2017-11-28 13:49:35 -05:00
Chris Meyers 6c89935521 correctly cascade job cancel
* Check the reason for a dependent project update failure. If it's
because of a cancel, then let the normal cancel mechanisms update the
jobs status and explanation. Do not update the dependent job's status
for a project update that was canceled, in the run code.
2017-11-27 12:34:55 -05:00
Chris Meyers 98f2d936d9 allow support for saml + 2-factor
* python-social-auth has SOCIAL_AUTH_SAML_SECURITY_CONFIG, which is
forwarded to python-saml settings configuration. This commit exposes
SOCIAL_AUTH_SAML_SECURITY_CONFIG to configure tower in tower to allow
users to set requestedAuthnContext, which will disable the requesting of
password type auth from the idp. Thus, it's up to the idp to choose
which auth to use (i.e. 2-factor).
2017-11-17 09:25:50 -05:00
Chris Meyers c4e6fc23fc add docs for job events
* Focus on the ordering of Job Event creation. Important to know when
thinking through different Ansible execution strategies.
2017-11-16 12:48:48 -05:00
Chris Meyers 9288b53015 cascade cancel proj update when job canceled
* Implicit project update, launch_type='sync', get "associated" with a
job via project_update. When a job is canceled, so should this implicit
project update. This change enforces that logic.
2017-11-15 11:17:52 -05:00
Chris Meyers 4e2cf62e89 fixes ansible callback import json warning
[WARNING]: Failure using method (v2_runner_on_ok) in callback plugin
(<awx_display_callback.module.AWXDefaultCallbackModule object at
0x47f6090>):
'module' object has no attribute 'dumps'

The above error is thrown by ansible if callback plugins don't respect
the same import precedence configuration as Ansible. ansible callback/*
dir includes a json.py file. This is imported by ansible
callback/__init__.py when a callback plugin implementation imports from
Ansible callback base without setting the correct import precedence.
2017-10-16 10:29:41 -04:00
Chris Meyers 62987196cb add speedup support to event replay and stats
* add tests
* add verbosity support
2017-10-13 09:25:18 -04:00
Chris Meyers e19a57c50a add job event replay awx-manage command
* awx-manage replay_job_event --job_id <id>
2017-10-12 09:40:30 -04:00
Chris Meyers cad8710ac7 fix scan job migration unicode issue 2017-10-03 11:31:43 -04:00
Chris Meyers a7ec5876ce add awx install type to issue template
* Recreation of user-issues increasing depends on the chosen install
path used. Thus, we need this information when recreating the issue
locally. Ask the user to include this information in the issue template.
2017-09-28 14:37:26 -04:00
Chris Meyers 26d393e5c2 2-level memoize
* Allows for invalidating an entire function from the memoizer
2017-09-21 15:34:51 -04:00
Chris Meyers 062ff7153d resurrect cchurch's license feature caching 2017-09-20 14:57:24 -04:00
Chris Meyers c265ed2722 Revert "use request response cache for license checks"
This reverts commit 93acae9367023fec153aa7980f2edd9f96822ba2.
2017-09-20 13:47:44 -04:00
Chris Meyers 0b40331107 azure migrations
* delete azure credentials
* delete azure inventory sources
2017-09-18 10:35:42 -04:00
Chris Meyers a08a158672 remove azure 2017-09-18 10:35:32 -04:00
Chris Meyers 1ce3c7937b use request response cache for license checks 2017-09-13 13:34:39 -04:00
Chris Meyers 45813bea16 do not re-create django session on every request
* The django middleware call stack behavior is changed by DRF. As a
result, during the process_request in sso/middlware.py request.user
is not set as you would expect it to be set from the middleware
django.contrib.auth.middleware.AuthenticationMiddleware
2017-08-21 11:19:52 -04:00
Chris Meyers 1df47a2ddd account for waiting tasks not having execution_nodes yet
* Reap running tasks on non-netsplit nodes
* Reap running tasks on known to be offline nodes
* Reap waiting tasks with no celery id anywhere if waiting >= 60 seconds
2017-08-16 13:18:25 -04:00
Chris Meyers d615e2e9ff do not include workflow jobs for reaping
* Workflow jobs are virtual jobs that don't actually run. Thus they
won't have a celery id and aren't candidates for the generic reaping.
* Better error logging when Instance not found in reaping code.
2017-08-16 08:51:27 -04:00
Chris Meyers de82707581 fail all jobs on an offline node 2017-08-15 14:46:28 -04:00
Chris Meyers 90ea9b8bde presume 401 from insights means invalid credential
* Tell the user about the invalid credentials
2017-08-15 08:31:52 -04:00
Chris Meyers 9314db646b only reap non-netsplit nodes 2017-08-15 08:30:14 -04:00
Chris Meyers 5ad29c9081 pass through insights rule.ansible 2017-08-14 15:21:13 -04:00
Chris Meyers 80a90beae8 remove dead code 2017-08-14 10:37:57 -04:00
Chris Meyers 7e86ef9a46 remove scan job choice from launch endpoint 2017-08-10 08:52:54 -04:00
Chris Meyers c0b9c76a41 stringify uuid so it can be serialized 2017-08-10 07:32:03 -04:00
Chris Meyers c3f24d878d reap waiting processes if crash 2017-08-09 14:01:33 -04:00
Chris Meyers 4e2f075fa6 better scm inv restriction error message 2017-08-08 10:42:38 -04:00
Chris Meyers 95c12ac1ab enforce muutally exclusivity on update_on_project_update and update_on_launch for scm inventory 2017-08-07 10:50:36 -04:00
Chris Meyers 4b830e1176 fix fact cache timeout env var key name 2017-08-04 09:39:18 -04:00
Chris Meyers 3c89aeaba4 fix single update_on_project_update per-inventory 2017-08-03 18:54:05 -04:00
Chris Meyers 729d86c485 standardize on 502 for insights errors 2017-08-03 16:12:09 -04:00
Chris Meyers 9b12d88e8c lay the foundation for checking license on upgrade 2017-08-02 12:24:58 -04:00
Chris Meyers 6cc5f14e16 remove conditional inventory sources POST
* Move logic from validator down to model
* Allow only 1 inventory source of type scm with
update_on_project_update set to True; for each inventory
2017-08-02 11:49:10 -04:00
Chris Meyers 3dea02ceac use insights v3 api 2017-07-31 11:04:15 -04:00
Chris Meyers ed8db2419e make specific saml fields required 2017-07-31 09:28:52 -04:00
Chris Meyers f20f4f40a0 trim insights content to only what the UI needs 2017-07-27 13:58:54 -04:00
Chris Meyers 1d0e125aad make password survey field behave like text field
related to https://github.com/ansible/ansible-tower/issues/6083
2017-07-27 12:57:31 -04:00
Chris Meyers ef055110f5 handle unicode host names in fact cache 2017-07-26 12:41:13 -04:00
Chris Meyers 89a39f4bed add scm to cloud sources so computed fields works 2017-07-25 14:33:07 -04:00
Chris Meyers ec212db83b fix api docs sample host_filter queries 2017-07-24 13:48:20 -04:00
Chris Meyers eaddafc920 allow smart inventory to be created
related to https://github.com/ansible/ansible-tower/issues/7261
2017-07-24 13:43:15 -04:00
Chris Meyers 525490a9a0 all dependent jobs must finish before starting job
related to https://github.com/ansible/ansible-tower/issues/6570 https://github.com/ansible/ansible-tower/issues/6489

* Ensure that all jobs dependent on a job have finished (i.e. error,
success, failed) before starting the dependent job.
* Fixes a bug where a smaller set of dependent jobs to fail upon a
update_on_launch job failing is chosen.
* This fixes the bug of jobs starting before dependent Project Updates
created via update on launch. This also fixes similar bugs associated
with inventory updates.
2017-07-24 12:31:35 -04:00
Chris Meyers 41ae7b5120 validate schedule extra_data
related to #4779

* Ensure days is a non-negative integer
* Ensure that extra_data is a json payload
2017-07-21 10:46:44 -04:00
Chris Meyers c11b870084 remove job_type scan from options
* Need to keep the scan job in job_type for migrations so just remove it
from the constructed OPTIONS
2017-07-21 10:45:17 -04:00
Chris Meyers 22e4957698 remove duplicate hosts from host_filter 2017-07-19 17:09:39 -04:00
Chris Meyers c2204e5f27 disallow insights creds on smart inventory
related to #7083
2017-07-19 17:09:00 -04:00
Chris Meyers 802f722e95 do not accept array as extra_vars
related to #5645

* Since we pass extra_vars to ansible as -e; we require that the top
most data-structure be a key-value. NOT an array.
2017-07-19 17:07:50 -04:00
Chris Meyers 815732bcba add API docs for host_filter 2017-07-19 10:18:13 -04:00
Chris Meyers 352e83aa84 fix diff_mode migration error 2017-07-13 07:54:58 -04:00
Chris Meyers 6511d2968e use newest aws env var names for creds 2017-07-12 15:27:18 -04:00
Chris Meyers d3e4be66dd disallow new survey spec with password default $encrypted$ 2017-07-12 15:21:55 -04:00
Chris Meyers 376170124e require that insights projects have insights cred 2017-07-12 12:52:26 -04:00
Chris Meyers 6652a960b7 make insights playbook fetching an Ansible module
* Gone are the days of playbook abuse.
* Our project update can now call a single Insight Ansible module to
loop over the set of maintenance plans to download and save playbooks
for use by job templates.
2017-07-11 16:28:01 -04:00
Chris Meyers c34a89714c rework hosts without insights creds error message 2017-07-11 14:25:12 -04:00
Chris Meyers e1a24d9849 structure scan modules as dicts not arrays 2017-07-11 14:14:48 -04:00
Chris Meyers f119b94134 correctly create jobs in migrations
* Do not import Project. Instead, use get_model() and ensure the
polymorphic ctype is set correctly.
* Point migrated job templates at the new fact scan playbook
2017-07-11 13:44:08 -04:00
Chris Meyers 7fda3c0658 back out thought to cause deadlock scenario 2017-07-10 10:43:29 -04:00
Chris Meyers aeb7119796 fix 2 data source inconcistency with failing tasks
* Do not "trust" the list of celery ids for database entries that were
modified after the list of celery ids was gotten.
* err on the side of caution and just let the next heartbeat celery
killer try killing the task if it needs to be reaped.
2017-07-10 10:43:01 -04:00
Chris Meyers 5b9a0b504a celery task fail check now uses pglock
* Align locking used by celery task cleaner upper with regular task manager.
* Uses pglock/advisory lock instead of abusing Instance table lock.
2017-07-10 10:41:54 -04:00
Chris Meyers 7a795b8681 fix job launch deadlock
* This both fixes the deadlock problem and a logic problem. We shouldn't
set the job's job_template current_job to pending jobs.
2017-07-10 10:34:36 -04:00
Chris Meyers df572d1477 migrations do not call overwitten save() methods
related to #6941

* We don't want our special logic to trigger anyway.
2017-07-07 16:47:07 -04:00
Chris Meyers e04c4e30d8 update tests to fix the removal of the word Tower 2017-07-06 11:22:16 -04:00
Chris Meyers 0a25cc2d19 back out parent jt status update short circuit 2017-07-06 10:48:28 -04:00
Chris Meyers 1331865749 take into account memcached key restrictions
* Keys can't contain spaces or control characters
2017-07-05 15:39:04 -04:00
Chris Meyers 0d3fc3389d flake8 fix 2017-07-05 13:37:32 -04:00
Chris Meyers e226b0ab37 noop pglock for unit tests 2017-07-05 13:34:07 -04:00
Chris Meyers f3f9782c0b fix 2 data source inconcistency with failing tasks
* Do not "trust" the list of celery ids for database entries that were
modified after the list of celery ids was gotten.
* err on the side of caution and just let the next heartbeat celery
killer try killing the task if it needs to be reaped.
2017-07-05 13:33:29 -04:00
Chris Meyers 15aee1f8ac celery task fail check now uses pglock
* Align locking used by celery task cleaner upper with regular task manager.
* Uses pglock/advisory lock instead of abusing Instance table lock.
2017-07-05 13:27:08 -04:00
Chris Meyers 3d4f8b0074 support clear_facts 2017-07-05 10:33:36 -04:00
Chris Meyers 4b84cec1e8 do not launch proj update on migrate 2017-07-05 10:32:45 -04:00
Chris Meyers 88b250b2f1 migrate scan jobs to use fact caching instead 2017-07-05 10:32:45 -04:00
Chris Meyers 318b1aebd1 remove special scan job logic 2017-07-05 10:32:45 -04:00
Chris Meyers eda0679eb3 Revert "support clear_facts"
This reverts commit e5432c231d1ab38807c8be10133d35ede237a519.
2017-07-05 10:32:13 -04:00
Chris Meyers 06edb6774a support clear_facts 2017-07-05 10:26:49 -04:00
Chris Meyers f1b1c4ee97 avoid instance record deadlock by using pglock 2017-07-05 09:38:19 -04:00
Chris Meyers 29adb39cae add insights_cred_id to host->inventory summary fields 2017-06-30 13:52:41 -04:00
Chris Meyers 668bce8212 fix job launch deadlock
* This both fixes the deadlock problem and a logic problem. We shouldn't
set the job's job_template current_job to pending jobs.
2017-06-30 09:54:08 -04:00
Chris Meyers d6082a976b retain support for insights system_id discovery 2017-06-22 09:49:45 -04:00
Chris Meyers d679dc7407 do not inject facts for isolated node runs 2017-06-21 13:45:28 -04:00