Commit Graph

384 Commits

Author SHA1 Message Date
Matthew Jones
cd48427189 Merge pull request #3757 from ansible/AlanCoding-patch-1
Only fill scm_revision env var if project for job exists
2016-10-24 13:27:19 -04:00
Matthew Jones
7c7d2e37ed Fix some issues syncing playbooks
* Build a list of playbooks and store it in the database at sync time
* Fix an issue running playbook sync on jobs for scan jobs
* Remove a TODO that was unneeded
2016-10-24 11:32:21 -04:00
Alan Rominger
3121839704 only fill scm_revision var if project for job exists 2016-10-24 09:43:17 -04:00
Matthew Jones
6e22460f1e Add scm revision to the job model
This also cleans up flake8 issues
2016-10-20 15:18:05 -04:00
Matthew Jones
ca32c5fd21 scm_revision should be used for job_type run 2016-10-20 14:56:15 -04:00
Matthew Jones
4562faa858 Pass scm revision in env and extra var for job run 2016-10-20 14:42:28 -04:00
Matthew Jones
ba469a8c13 Remove spurious workflow job class 2016-10-20 13:34:10 -04:00
Matthew Jones
e6bcc039f2 Rearchitect project update strategy
* Instead of using a fanout for project updates initial project
  updates will sync the latest commit hash
* Prior to a node running a job it will ensure that the latest project
  is synced
2016-10-20 13:30:44 -04:00
Aaron Tan
5fdd44c3f4 Fix mutable default argument issue. 2016-10-18 09:48:19 -04:00
Aaron Tan
21feae7c11 Rename default timeout and remove runtime_flag. 2016-10-17 16:56:39 -04:00
Aaron Tan
2362f34d27 Implement pluggable job explanation. 2016-10-17 16:56:39 -04:00
Aaron Tan
701c23171f Remove timeout fields from sys job & adhoc cmd. 2016-10-17 16:56:39 -04:00
Aaron Tan
96e3228c9d Change global settings structure and fix potential celery issue. 2016-10-17 16:56:39 -04:00
Aaron Tan
651a09f5fb Remove stdout indicator. 2016-10-17 16:56:39 -04:00
Aaron Tan
33952808c4 Readability improvement and job explaination added. 2016-10-17 16:56:39 -04:00
Aaron Tan
69be94feb9 Implement global job timeout mechanism. 2016-10-17 16:56:39 -04:00
Aaron Tan
37139301b4 Add job timeout mechanism to tasks. 2016-10-17 16:53:54 -04:00
Alan Rominger
1979d63c25 Merge pull request #3708 from AlanCoding/ancestors_PR
Store vars from set_artifact tasks, pipe data through workflows
2016-10-13 15:59:58 -04:00
AlanCoding
ef861e5256 move set_artifact module to its own repository 2016-10-12 17:54:35 -04:00
AlanCoding
5d4cf9d4fc Add job artifacts and workflow artifact passing
artifacts redact from job when no_log is set
parent no_log artifacts treated as survey passwords
2016-10-12 17:45:50 -04:00
AlanCoding
3cfe544ada update workflows to work with new channels implementation 2016-10-12 13:54:13 -04:00
Wayne Witzel III
5d8e7c17c2 Merge branch 'devel' into jtabor-sockets 2016-10-07 09:50:03 -04:00
Matthew Jones
babe29ebfa Implement cluster health checks
* Add a local node queue to execute targeted jobs
* Add a setting for active cluster node id (per-node)
* Base the heartbeat time on the `modified` time on the Instance table
* Add periodic task that calls save() on the instance to update the
  heartbeat time if services are up
* Purge/update any ha/instance management commands
* Fix up CELERY_ROUTES settings data structure
2016-10-06 16:05:39 -04:00
Wayne Witzel III
530a5c0c88 merging devel 2016-10-03 10:35:50 -04:00
Chris Meyers
ec2b0ac90d add back in removed method 2016-09-29 16:01:15 -04:00
Chris Meyers
dd3c3c9f29 Merge branch 'feature-ha_task_manager' into devel 2016-09-29 10:27:17 -04:00
Chris Meyers
fbc1dff4de flake8 fixes 2016-09-29 10:23:00 -04:00
Wayne Witzel III
5faa21b72d add group_name to channel data 2016-09-27 18:39:14 -04:00
Wayne Witzel III
4c8aaf1aed converting from socketio to channels websocket 2016-09-27 18:39:07 -04:00
Chris Meyers
04f69727f2 fully message driven job execution
TODO:
* Need a distributed lock (leverage postgres)
* Less memory-intensive graph representation
* Maybe serializer/deserializer graph to database
  * Iterative graph building instead of full rebuild.
2016-09-27 14:16:18 -04:00
Chris Meyers
cdb65ccac9 replace task manager with event driven scheduler 2016-09-27 14:16:18 -04:00
Chris Meyers
cc90204b0f task manager using messages
* First pass, adapt singleton task manager to process messages and run
jobs based on events instead of a busy loop.
* Still need to make message handing run in celery, not in a consumption
loop
2016-09-27 14:05:29 -04:00
Chris Church
6ebe45b1bd Configure Tower in Tower:
* Add separate Django app for configuration: awx.conf.
* Migrate from existing main.TowerSettings model to conf.Setting.
* Add settings wrapper to allow get/set/del via django.conf.settings.
* Update existing references to tower_settings to use django.conf.settings.
* Add a settings registry to allow for each Django app to register configurable settings.
* Support setting validation and conversion using Django REST Framework fields.
* Add /api/v1/settings/ to display a list of setting categories.
* Add /api/v1/settings/<slug>/ to display all settings in a category as a single object.
* Allow PUT/PATCH to update setting singleton, DELETE to reset to defaults.
* Add "all" category to display all settings across categories.
* Add "changed" category to display only settings configured in the database.
* Support per-user settings via "user" category (/api/v1/settings/user/).
* Support defaults for user settings via "user-defaults" category (/api/v1/settings/user-defaults/).
* Update serializer metadata to support category, category_slug and placeholder on OPTIONS responses.
* Update serializer metadata to handle child fields of a list/dict.
* Hide raw data form in browsable API for OPTIONS and DELETE.
* Combine existing licensing code into single "TaskEnhancer" class.
* Move license helper functions from awx.api.license into awx.conf.license.
* Update /api/v1/config/ to read/verify/update license using TaskEnhancer and settings wrapper.
* Add support for caching settings accessed via settings wrapper.
* Invalidate cached settings when Setting model changes or is deleted.
* Preload all database settings into cache on first access via settings wrapper.
* Add support for read-only settings than can update their value depending on other settings.
* Use setting_changed signal whenever a setting changes.
* Register configurable authentication, jobs, system and ui settings.
* Register configurable LDAP, RADIUS and social auth settings.
* Add custom fields and validators for URL, LDAP, RADIUS and social auth settings.
* Rewrite existing validator for Credential ssh_private_key to support validating private keys, certs or combinations of both.
* Get all unit/functional tests working with above changes.
* Add "migrate_to_database_settings" command to determine settings to be migrated into the database and comment them out when set in Python settings files.
* Add support for migrating license key from file to database.
* Remove database-configuable settings from local_settings.py example files.
* Update setup role to no longer install files for database-configurable settings.

f 94ff6ee More settings work.
f af4c4e0 Even more db settings stuff.
f 96ea9c0 More settings, attempt at singleton serializer for settings.
f 937c760 More work on singleton/category views in API, add code to comment out settings in Python files, work on command to migrate settings to database.
f 425b0d3 Minor fixes for sprint demo.
f ea402a4 Add support for read-only settings, cleanup license engine, get license support working with DB settings.
f ec289e4 Rename migration, minor fixmes, update setup role.
f 603640b Rewrite key/cert validator, finish adding social auth fields, hook up signals for setting_changed, use None to imply a setting is not set.
f 67d1b5a Get functional/unit tests passing.
f 2919b62 Flake8 fixes.
f e62f421 Add redbaron to requirements, get file to database migration working (except for license).
f c564508 Add support for migrating license file.
f 982f767 Add support for regex in social map fields.
2016-09-26 22:14:47 -04:00
Aaron Tan
609a3e6f2f Merge pull request #3390 from jangsutsr/2719_update_task_failure_traceback_text
update task failure traceback text
2016-09-26 11:47:22 -04:00
Chris Church
b7a6aa01a3 Fixes to get flake8 and unit/functional tests passing. 2016-09-18 19:11:29 -04:00
Matthew Jones
f5d2c5c18a Merge branch 'ramparts_and_ha' into devel
* ramparts_and_ha: (21 commits)
  Rename database migrations for devel integration
  Integrate callback receiver refactoring
  Fix an issue running jobs in the cluster
  Implement a more dynamic celery queue system
  Purge old munin monitors and tools
  Refactor Tower HA Instance logic and models
  Docker compose improvements
  Initial Docker Compose workflow for Tower cluster
  Add memcached role for setup playbook
  Removing qpid from deb packaging
  Refactor rabbitmq role
  Integrate memcached into setup playbook
  Remove mongodb shutdown task
  Remove dependency on erlang_sd_notify
  Add initial rabbitmq role
  Initial rabbitmq setup playbook integration
  Update development environment for rabbit
  Replace qpid with rabbitmq
  Remove redis role from setup playbook
  Update qpid packaging, remove migrations
  ...
2016-09-15 10:19:57 -04:00
Chris Meyers
c3bb7521de Merge pull request #3474 from chrismeyersfsu/feature-workflows
workflows
2016-09-15 10:12:58 -04:00
Matthew Jones
ab395b0009 Integrate callback receiver refactoring
* Drop ZMQ as the communication mechanism between job_event_callback and
  callback_receiver
* Setup queue and exchange for callback broker communication
* Refactor event plugin and callback receiver to efficiently handle
  message submission and processing
* Integrate django caching for parent processing
2016-09-14 11:42:13 -04:00
Matthew Jones
807cced571 Implement a more dynamic celery queue system
* Meant to be a starting point to more efficiently manage work routing
  and to balance work across all tower nodes
* Integrate flower as a dev tool that starts alongside other nodes.
  Helpful for observing and monitoring the queues/exchanges
* For the moment, force the task manager to only run on one node (not
  sure if this is needed)
* Define queues and routes for all task work
* Bump celery version to 3.1.23
* Expose flower through haproxy
2016-09-09 15:18:18 -04:00
Chris Meyers
0e68481a85 consolidate workflow migration 2016-09-08 12:57:54 -04:00
Chris Meyers
39ac2c047b launch workflows 2016-09-01 14:11:20 -04:00
jangsutsr
f5c095eb8a update task failure traceback text 2016-09-01 10:07:02 -04:00
Matthew Jones
f988877bae Merge branch 'stable' into devel
* stable: (24 commits)
  Updating changelogs for 3.0.2 release
  fixing deprecated_team.organization credential migration
  Fix issue when installing bundled ansible on el6
  fixed localed date stuff
  update test to check org_auditor access
  ensure team organizations are assigned to credentials
  temporarily pin the pytest version until the ldap error can be fixed
  fixed locale
  fix date locale angular scheduler
  Make sure org admins can see credential after migration, comment updates on related tests add clause in test to verify automatic setting of org of new team credential
  Update team admin credential migration test to current state-of-knowledge
  fix ng-toast rel, resolves #3197 (#3316)
  allow users to edit their first and last name
  Revert "Prevent ignored task from being displayed as failing."
  Revert "Modify job event save behavior"
  fixing old tests for new user creation permissions
  Clean venv on 'make clean'
  Resolve KeyError by coercing instance_id to a str
  Update cloudforms dynamic inventory
  Update foreman inventory script
  ...
2016-08-31 14:34:52 -04:00
Chris Meyers
2cfdee3b21 turn job templates in jobs via launch 2016-08-31 09:00:25 -04:00
Chris Meyers
4c876b40e4 initial models and endpoints added for workflows 2016-08-30 15:05:28 -04:00
Chris Meyers
27b9fb8dab refactor notifications 2016-08-30 14:36:18 -04:00
Bill Nottingham
c2f85d5bfb Remove compatiblity code for pre-1.9 Ansible versions. 2016-08-25 15:41:47 -04:00
James Laska
188f81ef3d Update cloudforms dynamic inventory
Pulling the latest from
https://github.com/ansible/ansible/blob/devel/contrib/inventory/cloudforms.py

Related #3168
2016-08-19 15:37:28 -04:00
Matthew Jones
8fcc194c82 Attempt to wait for job host summaries
Sometimes the job host summaries can land a little later after the job
has finished so sometimes events are still filtering in when the
notifications are triggered
2016-08-10 11:30:02 -04:00
James Laska
11d68d1ebe Complete Sat6 integration
Various fixes to get sat-6 integration working.

* inventory_import.py - the inventory script is called foreman.py
* tasks.py - be sure to call `add_section()` before using it
* defaults.py - Add various SATELLITE6_* enablement variables

Relates #3119
2016-07-25 15:13:39 -04:00