Commit Graph
100 Commits
Author SHA1 Message Date
Matthew Jones 062ace975c Add the super-handy pprofile to dev reqs 2016-05-03 12:01:58 -04:00
Matthew Jones 9cc9ef6b33 Use shallow copy in fact cache plugin
This deep copy would cause ansible-playbook to get hung up and fail to
spawn new subprocesses thus forcing extremely large scan jobs to
serialize their connections through one process.    This had the effect
of forcing `forks=1` behavior.

This change was for citco, a customer who is running scan jobs against
600 hosts with a forks value of 200.
2016-05-02 13:34:47 -04:00
Matthew Jones 256487a963 Strip hash marks from slack channels at send time 2016-04-25 09:49:36 -04:00
Matthew Jones 9ac8ee3ef0 Updating api side vendored licenses 2016-04-22 15:23:20 -04:00
Matthew Jones cf7b71ae2c Merge branch 'release_2.4.5' into devel
* release_2.4.5:
  Remove distribute from the setup virtualenv if installed (#1631)
  Update changelogs for 2.4.5 release
  Only export changed targets in reprepro
  Improve the efficiency of the stdout dump database migration
  Added logic to not show the loop summary events in the UI by looking at event_data.event_loop.
  Handle runner items from ansible v2
  bump boto
  fix case of Ansible v2 _result.cmd is list for release 2.4.5
  Handle both string and list hosts in our hosts->name conversion
  Default play names to the hosts the play was run against
  Use better isinstance(x) type checking
  Backporting test fixes from PR #1020: Fix error with ad hoc command events when running in check mode.
  Fix for tasks breaking when using 'yum' with ansible 1.9.4
  Bump 2.4.5 version, changelogs, and reprepo
2016-04-22 10:54:23 -04:00
Matthew Jones fe360ca8e3 Updates to network credentials
Separating out the network credential used fields
2016-04-21 16:43:55 -04:00
Matthew Jones 42247fa3c1 Add job events top level api object 2016-04-21 15:30:35 -04:00
Matthew Jones fcc9c1e6e4 Switch org projects list back to allowing create 2016-04-21 15:09:47 -04:00
Matthew Jones e1808a6ed7 Update indentation for flake8 2016-04-21 14:13:03 -04:00
Matthew Jones 0cd4b52582 Remove azure dependency from tower venv 2016-04-21 14:03:27 -04:00
Matthew Jones 296f70ce17 Implement Azure RM creds and inventory
* Vendor ansible's azure_rm inventory script
* Add new inventory type
* Add new credential type
* Expand host instance_id column from varchar 100 to 1024 to accept the
  long instance ids returned by Azure
* Make the inventory_import azure match rename more explicit.
2016-04-21 13:51:30 -04:00
Matthew Jones 16fbb7d14c Add default playbook to job template preload 2016-04-20 11:29:21 -04:00
Matthew Jones d87213705b Adjust job launch blocking logic
Previously a job template would always block another job template launch
regardless of the details of the job template.   We now restrict that
blocking logic to only block in the case that the job template was
launched with the same inventory.   We keep the exclusion where if the
launch type is 'callback' and the limits differ then they won't be blocked
2016-04-18 15:57:31 -04:00
Matthew Jones 226cb9acdf Protect job template and project template notifiers in the case where
the project or org may not be available.
2016-04-18 14:24:53 -04:00
Matthew Jones ba7f48abf1 Updating tests around data preload 2016-04-18 11:27:39 -04:00
Matthew Jones d1801bdf9f Demo data flake8 cleanup 2016-04-15 15:14:10 -04:00
Matthew Jones 8c4e4f4c0d Creating pre-loaded demo data
* Bootstrap some initial data if there is no default org
* Renamed default org script appropriately
2016-04-15 14:42:51 -04:00
Matthew Jones 6f79ec8830 Revert "Merge pull request #1534 from chrismeyersfsu/fix-config_performance"
This reverts commit 7cc144fa59f157a5e1789a913e0d9a31720141ec, reversing
changes made to 86af9950e754b06f40cb293a36ab365f589273db.
2016-04-14 11:40:07 -04:00
Matthew Jones 6150addc2e Updates to makefile to coerce pip install and usage for dependencies 2016-04-13 11:18:52 -04:00
Matthew Jones 28ffa57258 Implement rbac and related unit tests for notifications 2016-04-12 17:00:38 -04:00
Matthew Jones b4ef0304e3 Add funcsigs as a direct dependency for the venvs 2016-04-12 10:48:49 -04:00
Matthew Jones d6c2b931f9 Remove stray comma from network credential in JobOptions 2016-04-11 14:49:23 -04:00
Matthew Jones f437d39d06 Add psutil to ansible venv dependencies 2016-04-11 12:57:56 -04:00
Matthew Jones 673c5929f9 Improve the efficiency of the stdout dump database migration 2016-04-11 11:40:56 -04:00
Matthew Jones 62218cfd7d Fix virtualenv path association in job runs 2016-04-08 11:56:51 -04:00
Matthew Jones a0fd546bdf Force jenkins pip dependencies to override the base 2016-04-07 10:33:20 -04:00
Matthew Jones 86cca77b43 Make sure requirements_jenkins invokes the other requirements sections 2016-04-06 20:33:44 -04:00
Matthew Jones 547506a3bf Remove contextual statsd reference 2016-04-06 16:47:36 -04:00
Matthew Jones c0afa3ca0f More commenting out of statsd 2016-04-06 16:42:30 -04:00
Matthew Jones 3ba591cea2 Remove statsd from callback
This doesn't work with our current virtualenv setup and ansible
dependencies.  Will revisit later
2016-04-06 16:29:50 -04:00
Matthew Jones 0a6df40066 Improvements to docker compose workflow with new packaging changes 2016-04-06 12:24:19 -04:00
Matthew Jones ae9222f716 Fix flake8 after packaging changes merge 2016-04-06 11:03:34 -04:00
Matthew Jones b89d183433 Import pytest for skipping settings tests 2016-04-04 14:19:08 -04:00
Matthew Jones 5460bac498 Add historical notifications to notifier summary 2016-04-04 13:38:34 -04:00
Matthew Jones 469bff1d0b Fix required fields in notifier serializer
Handle the situation where the password field was left as-is when
posting the existing encrypted field
2016-04-04 12:04:31 -04:00
Matthew Jones b01fe663af Fix up tests for vacated settings 2016-04-04 11:57:40 -04:00
Matthew Jones b6706662a4 Disable database settings until after 3.0 2016-04-04 09:59:24 -04:00
Matthew Jones f2284a0098 Handle runner items from ansible v2
Also denote whether the trailing runner_on_ was a loop event
2016-03-23 11:38:11 -04:00
Matthew Jones 7f8fae566e Further decouple survey spec from enablement
We now show the survey summary field only if the contents of the survey
spec or valid (not the default {})
2016-03-22 14:59:56 -04:00
Matthew Jones 0e2184902e Handle runner items from ansible v2
Also denote whether the trailing runner_on_ was a loop event
2016-03-22 12:31:05 -04:00
Matthew Jones f286dc7486 Fix an issue with the email notifier
Incorrect body format assumptions in the email notifier
2016-03-21 15:40:41 -04:00
Matthew Jones 46bbc83d0d Changes for Tower virtualenv support
* Break requirements down into ansible and tower reqs
* Generate separate Ansible and Tower virtual environments
* Install appropriate requirements files into each one
* Modify development tools to use these venvs instead of our old
  site-packages
* Modify settings to indicate venv enablement and location of venvs
* Modify tasks to use the proper virtual environment for its purpose
2016-03-21 12:11:19 -04:00
Matthew Jones c7e33d0471 Changes for Tower virtualenv support
* Break requirements down into ansible and tower reqs
* Generate separate Ansible and Tower virtual environments
* Install appropriate requirements files into each one
* Modify development tools to use these venvs instead of our old
  site-packages
* Modify settings to indicate venv enablement and location of venvs
* Modify tasks to use the proper virtual environment for its purpose
2016-03-21 11:47:19 -04:00
Matthew Jones 526a6ec7dd Remove unneeded fetch for r/o fields 2016-03-16 13:12:13 -04:00
Matthew Jones 45f95bf2b2 Disallow related elements to be treated as choices
DRF will try to resolve potential candidates into the OPTIONS endpoint.
This is mainly to support their POST field in the browseable API.   We
don't need this and it can yield some expensive queries so we bypass
generating choices for any RelatedField fields
2016-03-16 13:11:05 -04:00
Matthew Jones 63b01bb04b Fix up the docker-refresh Makefile target
This allows you to cleanup the images without requiring a rebuild necessarily
2016-03-16 13:11:05 -04:00
Matthew Jones 7e818319b5 Pass, don't return pass when checking mongo connectivity 2016-03-14 13:53:34 -04:00
Matthew Jones 001eb1f69f Handle a mongo OperationFailure
This seems to happenw hen the database is up, allows us to connect but
we don't have permission to access the objects (or they don't exist
yet).
2016-03-14 11:57:34 -04:00
Matthew Jones eab223d229 Make sure we are covering system jobs and template on notifications 2016-03-11 15:11:08 -05:00
Matthew Jones cdc342a6ee Fix fact database migration
For new installs the database may not be online and thus the migration
can't happen.  This will allow it to continue but we may want more
messaging to the user running potential upgrades
2016-03-11 12:59:21 -05:00
Matthew Jones c2ea9c3c00 Make survey always visible if license enables
Previously this would only be visible if survey is enabled.  It should
be possible to define and view a survey even if this is disabled.   The
enablement flag controls whether the survey prompt is shown and required.
2016-03-10 16:09:34 -05:00
Matthew Jones 4f8763ff90 Changes for Tower virtualenv support
* Break requirements down into ansible and tower reqs
* Generate separate Ansible and Tower virtual environments
* Install appropriate requirements files into each one
* Modify development tools to use these venvs instead of our old
  site-packages
* Modify settings to indicate venv enablement and location of venvs
* Modify tasks to use the proper virtual environment for its purpose
2016-03-09 14:33:42 -05:00
Matthew Jones 823e54a0b9 Fix an error sending success notifications 2016-03-08 14:14:57 -05:00
Matthew Jones 714d13eefd Update docker development workflow to not require make develop 2016-03-02 12:09:42 -05:00
Matthew Jones 1f19c13ac9 Fix up a Notification unit test issue after the merge 2016-03-01 12:16:10 -05:00
Matthew Jones 533c629040 Add irc requirements dependency 2016-02-29 23:50:16 -05:00
Matthew Jones 99ec07b8a5 Merge branch 'notifications_work' into devel
* notifications_work: (23 commits)
  Updates to notification unit tests after @wwitzel3's feedback
  Fix some notifications issues and write some tests
  Add notification system documentation
  Clean up flake8 related issues
  Fixing up some unicode issues
  Implement tower ui view url on models
  Sanity check and force proper types in admin check
  Proper type for in check
  Adding migration and base notification type
  Add a periodic administrative notification
  Refactor message generator
  Support notification password field encryption
  Notification configuration type checking
  Refactor NotificationTemplate to Notifier
  Implement irc notification backend
  Add webhook notification backend
  Pagerduty and Hipchat backends plus some cleanup
  Notification serializers, views, and tasks
  Implement notification serializer and validations
  Notification endpoints and url expositions
  ...
2016-02-29 16:50:33 -05:00
Matthew Jones b892ee5f96 Updates to notification unit tests after @wwitzel3's feedback 2016-02-29 13:25:49 -05:00
Matthew Jones 0ee12901fe Fix some notifications issues and write some tests
* Fixes some notifier merging issues
* Fixes some more unicode problems
* Implements unit tests
2016-02-29 12:30:00 -05:00
Matthew Jones a9bf29d2a7 Add the id field to the default summary fields
This is to support the UI.

ref: #1063
2016-02-26 11:30:34 -05:00
Matthew Jones b35d7a3c6b Add notification system documentation 2016-02-24 13:26:43 -05:00
Matthew Jones 7eb74cd011 Bump 2.4.5 version, changelogs, and reprepo 2016-02-24 09:51:12 -05:00
Matthew Jones 41d5393af0 Clean up flake8 related issues 2016-02-23 15:30:07 -05:00
Matthew Jones 4b1493f456 Fixing up some unicode issues 2016-02-23 15:26:29 -05:00
Matthew Jones 75ef0dd395 Implement tower ui view url on models 2016-02-23 14:19:42 -05:00
Matthew Jones b88892be49 Sanity check and force proper types in admin check 2016-02-23 12:33:30 -05:00
Matthew Jones 050ed8a200 Proper type for in check
Still an __in for when we need to add another trigger obj
2016-02-23 12:18:31 -05:00
Matthew Jones 39c942e98b Adding migration and base notification type 2016-02-23 11:20:08 -05:00
Matthew Jones 4928badd3f Add a periodic administrative notification 2016-02-23 10:59:08 -05:00
Matthew Jones ab3669efa9 Refactor message generator
* Job object can now control the output and generate K:V output for
  notification types that can support it
* Notifications store the body as json/dict now to encode more
  information
* Notification Type can further compose the message based on what is
  sensible for the notification type
* This will also allow customizing the message template in the future
* All notification types use sane defaults for the level of detail now
2016-02-22 17:09:36 -05:00
Matthew Jones 718a9c2a97 Fix up some merge requirements conflicts 2016-02-22 10:21:34 -05:00
Matthew Jones c3121211c3 Merge branch 'release_2.4.4' into devel
* release_2.4.4: (35 commits)
  Update changelog for 2.4.4 release
  Make pycompile non-fatal during deb build
  Revert "Add virtualenv site-pagkages to Python path before system dist-packages, to get new setuptools"
  Roll back mock version due to packaging issues
  Add virtualenv site-pagkages to Python path before system dist-packages, to get new setuptools
  change to warning behavior
  Resolve bug when building with /bin/sh on Ubuntu
  Attempt to workaround pip install issue
  point at packages with source on pypi
  Mock requires a newer setuptools when building requirements
  requests needs openssl
  Properly set the shell during directory migration
  pyrax bumpb new python license
  Typo's are bad and should be vanquished
  Conditionally install 2.6 python requirements
  separate pip requirements file for python2.6
  Added missing 'skipped' field for no_log
  Obey no_log even more when using ansible 2.0
  bump shade from 0.5.0 to 1.4
  RHEL5 compatibility and handling of error scenarios
  ...
2016-02-22 10:09:47 -05:00
Matthew Jones 60c64d9fe8 Merge branch 'release_2.4.4' into stable
* release_2.4.4: (35 commits)
  Update changelog for 2.4.4 release
  Make pycompile non-fatal during deb build
  Revert "Add virtualenv site-pagkages to Python path before system dist-packages, to get new setuptools"
  Roll back mock version due to packaging issues
  Add virtualenv site-pagkages to Python path before system dist-packages, to get new setuptools
  change to warning behavior
  Resolve bug when building with /bin/sh on Ubuntu
  Attempt to workaround pip install issue
  point at packages with source on pypi
  Mock requires a newer setuptools when building requirements
  requests needs openssl
  Properly set the shell during directory migration
  pyrax bumpb new python license
  Typo's are bad and should be vanquished
  Conditionally install 2.6 python requirements
  separate pip requirements file for python2.6
  Added missing 'skipped' field for no_log
  Obey no_log even more when using ansible 2.0
  bump shade from 0.5.0 to 1.4
  RHEL5 compatibility and handling of error scenarios
  ...
2016-02-22 09:49:04 -05:00
Matthew Jones eb3d663d18 Support notification password field encryption
Modify encrypt_field and decrypt_field to support sub-fields under a
dictionary object.  It still uses the parent key when encrypting.
2016-02-21 23:11:17 -05:00
Matthew Jones 52974648df Notification configuration type checking 2016-02-17 16:01:54 +00:00
Matthew Jones dde70dafec Refactor NotificationTemplate to Notifier 2016-02-17 15:18:18 +00:00
Matthew Jones 9d6739045a Implement irc notification backend 2016-02-11 15:34:27 -05:00
Matthew Jones 7abcb6e306 Add webhook notification backend 2016-02-10 17:03:57 -05:00
Matthew Jones 24b33a52c8 Roll back mock version due to packaging issues 2016-02-10 17:01:18 -05:00
Matthew Jones 34ebe0a848 Pagerduty and Hipchat backends plus some cleanup 2016-02-10 15:17:11 -05:00
Matthew Jones 8db2f60405 Notification serializers, views, and tasks
* Implement concrete Notification model for notification runs
* Implement NotificationTemplate and Notification serializers and views
* Implement ancillary views
* Implement NotificationTemplate trigger m2m fields on all job templates
  via a fields mixin
* Link NotificationTemplates with an org
* Link notifications with the activity stream
* Implement Notification celery tasks
* Extend Backend field parameters to identify sender and receiver as
  parameters needed by the message and not the backend itself
* Updates to backends to better fit the django email backend model as it
  relates to Messages
* Implement success job chain task + notifications
* Implement notifications in error job chain task
2016-02-09 23:12:55 -05:00
Matthew Jones 319deffc18 Implement notification serializer and validations 2016-02-04 15:52:23 -05:00
Matthew Jones 172207cd4b Notification endpoints and url expositions
Also some changes to the footprint of the notification handler classes
2016-02-04 15:52:23 -05:00
Matthew Jones 96b0fb168f Updating makefile migration generator 2016-02-04 15:52:23 -05:00
Matthew Jones 805514990b Changes to celery tasks to support success signals
Linking in a success callback that will be invoked by our UnifiedJobs in
the case they terminate normally.  This is where we'll hook in the
success notification type.
2016-02-04 15:52:23 -05:00
Matthew Jones 7385efef35 Adding some early Notifications stubs
* A basic NotificationTemplate model class with early notification type
  definitions
* Initial implementations of the Email, Slack, and Twilio Notification
  backends using the Django email backend system
* Some dependencies thereof
2016-02-04 15:52:23 -05:00
Matthew Jones 7ab839c17f Fix up another statsd formatting error 2016-02-02 14:26:17 -05:00
Matthew Jones 5d21ad5ae6 Fix up statsd work to support python 2.6
Format specifiers must include field specifier
2016-02-02 09:58:19 -05:00
Matthew Jones f9f10a4ed7 Add some more statsd dummy methods 2016-01-30 12:05:10 -05:00
Matthew Jones 96b754c5ec Switch to StatsClient from StatsClientBase
So as not to use a purely abstract class
2016-01-29 16:27:32 -05:00
Matthew Jones c2875c65ca Fix stats client import 2016-01-29 15:49:51 -05:00
Matthew Jones fd2339dbb8 Fix up flake8 line seperation issue 2016-01-29 14:52:02 -05:00
Matthew Jones c7fd21ae52 Fix up flake8 and unit test failures
* Make sure we default to localhost to not fail at the GAI check
* Cleanup some flake8 issues in the metrics module
2016-01-29 13:59:37 -05:00
Matthew Jones 56d0cbfeff Cover DELETEing the license from database config
Fix the standard license delete behavior on the /config endpoint.  It
was covered elsewhere but this is the common path for license posting/removing
2016-01-26 16:31:38 -05:00
Matthew Jones 769fe18724 Wrap up other places where the old license access method is used 2016-01-25 20:59:18 -05:00
Matthew Jones 483fc507d6 tasks env vars should be strings
This was causing rax unit test failures which are now alleviated
2016-01-25 20:48:20 -05:00
Matthew Jones 32366b5544 Fix for rax cache timeout issue
2.4.0 level of rax.py introduces a default cache timeout of 600s, this
fixes that and includes a new rax.py inventory module that has been
submitted upstream.
2016-01-25 20:48:08 -05:00
Matthew Jones f775fa9680 tasks env vars should be strings
This was causing rax unit test failures which are now alleviated
2016-01-25 16:51:33 -05:00
Matthew Jones b35f84b401 Bypass license database check for cleanup schedule
We pre-create some cleanup jobs if a license is already present, this
doesn't work with the database config as-is.  These can be removed at
some point in the future once the 2.4 migration path is not needed
2016-01-25 15:53:17 -05:00
Matthew Jones 3132d4efee Fix license enablement utility helpers
We know call "from_database" which can implicitly call "from_file" if
the database config isn't available.
2016-01-25 14:09:04 -05:00