Commit Graph
100 Commits
Author SHA1 Message Date
Matthew Jones 93ce2a5dfd Set proot/bubblewrap enabled by default 2016-12-12 15:26:31 -05:00
Matthew Jones 8b8c29b9dc Purge has_schedules
Also... migration for big field docs updates
2016-12-12 12:33:09 -05:00
Matthew Jones bb7a201c2f Add missing field to ad-hoc command partial 2016-12-12 12:25:19 -05:00
Matthew Jones 349d497bb4 Make max page size tunable 2016-12-12 12:10:33 -05:00
Matthew Jones 08edbb1b72 Set an upper limit of 200 on the max page size 2016-12-12 12:01:48 -05:00
Matthew Jones 2d6dff1a3b Better error handling on deprovision_node management command 2016-12-12 10:55:03 -05:00
Matthew Jones 4e10d55501 Updating clustering acceptance docs 2016-12-07 11:08:39 -05:00
Matthew Jones 24b858c6fb Switch away from deepcopy to improve performance 2016-12-06 13:33:14 -05:00
Matthew Jones cd11a9c1c2 peg compose tag to active branch 2016-12-06 13:32:33 -05:00
Matthew Jones 71c600cad0 more missing punctuation 2016-12-06 13:18:34 -05:00
Matthew Jones 6efeeeb083 Adding missing period for timeout 2016-12-06 12:29:56 -05:00
Matthew Jones 51461a27da Adding help text to various job fields 2016-12-06 12:18:17 -05:00
Matthew Jones ebb0f06b17 Updating host field help text 2016-12-06 12:12:08 -05:00
Matthew Jones d7ffdf7020 Updating project help text 2016-12-06 12:08:23 -05:00
Matthew Jones eebdd13aa1 Updating schedules help text 2016-12-06 11:39:08 -05:00
Matthew Jones e5195032f6 Add back in some older dependencies that we had un-tracked
We'll need to figure out why dm.xmlsec.binding is segfaulting without these
2016-12-03 15:51:56 -05:00
Matthew Jones b17ca58fdd Trigger local cache flush before broadcasting on settings change 2016-12-02 16:46:48 -05:00
Matthew Jones faf4c84eca Purging old workload generator 2016-12-02 14:19:28 -05:00
Matthew Jones 765b7d9cc2 Add TOWER_HOST in the form of TOWER_URL_BASE to job env 2016-12-01 12:01:17 -05:00
Matthew Jones 9cecb7d870 Cleaning up some parts of the logging branch
* Fixing some flake8 warnings
* Removing some superflous print statements
2016-11-29 15:02:20 -05:00
Matthew Jones 1921f8cf04 Re-sync tower and ansible dependencies 2016-11-29 15:02:20 -05:00
Matthew Jones 7df11c59c0 Adding requests-futures dependency 2016-11-29 13:40:35 -05:00
Matthew Jones cad112e8da Add python-logstash dependency 2016-11-29 13:29:13 -05:00
Matthew Jones 34add726b7 Hide active tasks if anonymous user 2016-11-28 13:35:43 -05:00
Matthew Jones ddaf4ff2d0 Merge branch 'fix_fact_caching' into devel
* fix_fact_caching:
  Convert the fact data structure to the right type
  Fixing up fact cachine related unit tests
  Port fact caching system to rabbitmq
2016-11-21 23:15:13 -05:00
Matthew Jones ccd284526d Fix serializer related unit tests 2016-11-21 23:11:12 -05:00
Matthew Jones fd56a7acbb Remove unneeded summary fields element 2016-11-21 14:15:18 -05:00
Matthew Jones 9d044c1f41 Store the job's project refresh if it happened
* Create a field on the job to track this
* Add a summarizeable model to the summarizable fk fields so we can
  pull that in where needed
2016-11-21 14:11:20 -05:00
Matthew Jones c6ebdb9b26 Switch to using callback queue by default
* Fix an issue where we grab the wrong setting for the callback
  connection
* Remove old ansible.cfg that we shipped with the setup playbook for
  centos 6
2016-11-21 13:26:32 -05:00
Matthew Jones f77caf8b8b Convert the fact data structure to the right type 2016-11-18 12:13:09 -05:00
Matthew Jones 35b9786e06 Fixing up fact cachine related unit tests 2016-11-18 10:23:07 -05:00
Matthew Jones 3b07d9745d Port fact caching system to rabbitmq
* Purge all references to zmq also
* New setting to control the queue it works on
2016-11-18 09:44:17 -05:00
Matthew Jones 30984a3a79 Fix up flake8 errors 2016-11-10 13:07:48 -05:00
Matthew Jones 1a3d452a2d Fix an auth issue in the browsable api
Trying to get the object could raise an unhandleable permission error if
the user wasn't logged in for certain views
2016-11-10 11:46:25 -05:00
Matthew Jones 5186e7d8b4 Remove some unneeded func tests for instances 2016-11-10 11:46:25 -05:00
Matthew Jones 78b8876ed9 Support expiring of capacity if a node is down
For a certain amount of time
2016-11-10 09:52:04 -05:00
Matthew Jones 343966f744 Implement gathering overall task capacity
For use when running/planning jobs
2016-11-07 13:45:01 -05:00
Matthew Jones fa0780b9d0 Optimize uwsgi launching in dev container and prod
* Set --master flag so we have a preforking master process to manage
  subprocs
* Set max-requests to 1000 so uwsgi will recycle processes after 1000
  requests
* Set --no-orphans so uwsgi will track child processes and clean them up
  in order to respawn children
* Turn stats on in the container and install uwsgitop in dev
  requirements so we can track uwsgi stats
2016-11-04 13:38:23 -04:00
Matthew Jones 06dd5bb5d2 Fix an invalid database backend in unit test settings 2016-11-03 16:09:17 -04:00
Matthew Jones b0e40b6c29 Remove proot from dev base image in favor of bubblewrap
Remove python utils from base system, they are included with the venvs
2016-11-03 16:06:03 -04:00
Matthew Jones 8e77deea27 Push celery queue stats to memcached periodically
We'll piggyback off the task that checks for inconsistent jobs between
celery and Tower itself. These are read off via /api/v1/ping
2016-11-03 12:10:38 -04:00
Matthew Jones e01c86e88f Fix missing import for failure signal handler 2016-11-03 09:43:18 -04:00
Matthew Jones 1908b56ef7 Add event handler to manifest 2016-11-02 14:16:27 -04:00
Matthew Jones 4cb2e5aa02 Add failure event handler for supervisor process 2016-11-02 13:44:01 -04:00
Matthew Jones 23bf9cd2be Cleanup tower services and exit handling
* Replace apache with nginx in various service scripts
* Add a failure handler that will shut down all services if any of them
  fail
* Use INT signal for uwsgi shutdown procedure
2016-11-02 13:00:09 -04:00
Matthew Jones 96355854a8 Bump django version for CVE
https://www.djangoproject.com/weblog/2016/nov/01/security-releases/
2016-11-01 14:40:32 -04:00
Matthew Jones a1de353543 Shifting migrations after 3.0.3 merge 2016-11-01 14:40:32 -04:00
Matthew Jones c6cf02a602 Merge branch 'release_3.0.3' into devel
* release_3.0.3: (55 commits)
  Revert "Revert "Add needed types for selinux change""
  Revert "Add needed types for selinux change"
  interpret backslash escapes when displaying url in welcome message
  Bump the SELinux policy version
  Add needed types for selinux change
  Update SELinux policy to allow httpd_t to execute files in lib_t and var_lib_t
  Bumping changelog for 3.0.3
  Update rax.py inventory
  Revert "filter internal User.admin_roles from the /roles API list view"
  fix spelling of disassociated
  Resolves 404 when assigning resources/users to organizations in card view. Sidesteps a bug in the Refresh() utility, where pagination calculations are not made against filtered results.
  Sync azure changes to Tower virtual environment
  Add regions here as well.
  Also bump boto for new regions, per ryansb.
  More regions!
  Revert "bump shade version"
  bump shade version
  Hack copying of job_template.related.survey_spec into ui job copy flow, resolves #3737
  Revert "bump shade version"
  bump shade version
  ...
2016-11-01 11:49:28 -04:00
Matthew Jones defd271c90 Make sure we bootstrap the static dir prior to starting 2016-10-28 14:21:19 -04:00
Matthew Jones 7c7d2e37ed Fix some issues syncing playbooks
* Build a list of playbooks and store it in the database at sync time
* Fix an issue running playbook sync on jobs for scan jobs
* Remove a TODO that was unneeded
2016-10-24 11:32:21 -04:00
Matthew Jones 3d8eb48986 Test for HA license before allowing cluster job start 2016-10-20 16:50:49 -04:00
Matthew Jones 73b5b47e94 flake8 cleanup 2016-10-20 15:27:59 -04:00
Matthew Jones 6e22460f1e Add scm revision to the job model
This also cleans up flake8 issues
2016-10-20 15:18:05 -04:00
Matthew Jones ca32c5fd21 scm_revision should be used for job_type run 2016-10-20 14:56:15 -04:00
Matthew Jones 4562faa858 Pass scm revision in env and extra var for job run 2016-10-20 14:42:28 -04:00
Matthew Jones 2a504ff040 Shift migration files around for conflict 2016-10-20 14:42:12 -04:00
Matthew Jones 03ca8a571a Update clustering acceptance docs 2016-10-20 13:37:22 -04:00
Matthew Jones ba469a8c13 Remove spurious workflow job class 2016-10-20 13:34:10 -04:00
Matthew Jones 0f93b875ca Add migrations for execution node and project upds 2016-10-20 13:30:44 -04:00
Matthew Jones e6bcc039f2 Rearchitect project update strategy
* Instead of using a fanout for project updates initial project
  updates will sync the latest commit hash
* Prior to a node running a job it will ensure that the latest project
  is synced
2016-10-20 13:30:44 -04:00
Matthew Jones b906469b40 Add execution node information 2016-10-20 13:27:36 -04:00
Matthew Jones e8669b0cbf Support celeryd autoreloading for dev environment 2016-10-20 13:27:36 -04:00
Matthew Jones 2a844b9c42 Add api node request servicer to response headers 2016-10-07 11:40:44 -04:00
Matthew Jones 20b5aa7424 Fix some issues with management commands for clustering 2016-10-07 09:58:47 -04:00
Matthew Jones babe29ebfa Implement cluster health checks
* Add a local node queue to execute targeted jobs
* Add a setting for active cluster node id (per-node)
* Base the heartbeat time on the `modified` time on the Instance table
* Add periodic task that calls save() on the instance to update the
  heartbeat time if services are up
* Purge/update any ha/instance management commands
* Fix up CELERY_ROUTES settings data structure
2016-10-06 16:05:39 -04:00
Matthew Jones 4b57263adf Updates to clustering doc
* Notes about routing/dns
* Mention upgrade testing
2016-10-04 12:22:43 -04:00
Matthew Jones ff4d071be9 Adding/updating documentation for Cluster/HA Tower 2016-10-03 14:50:54 -04:00
Matthew Jones 959342a5fd Fixing up unit tests from HA/Clustering work
* Also purge old fact migration tests
* Upgrade some services on various container images
2016-10-03 10:40:30 -04:00
Matthew Jones 4d22af0d3c Fix up ubuntu apt repo paths 2016-09-29 13:16:58 -04:00
Matthew Jones be18737802 Merge branch 'ha_installer' into devel
* ha_installer:
  Fixing up some minor nits with ha setup playbook
  Update postgres yum/apt repo locations
  Add note about when to use short/long name
  Setup playbook modifications for HA/Ramparts
2016-09-29 12:29:25 -04:00
Matthew Jones a648beba90 Rev django security version 2016-09-29 12:03:06 -04:00
Matthew Jones 6b0e337886 Update postgres yum/apt repo locations
*Thanks postgres team
2016-09-29 11:25:08 -04:00
Matthew Jones f30c8c0bb8 Update postgres yum/apt repo locations
*Thanks postgres team

(cherry picked from commit b398fcc5b97ed478273e3d124aaaaa3af2879965)
2016-09-29 10:42:08 -04:00
Matthew Jones 162f0b81d1 Remove unneeded maxread
searchwindowsize was the important bit here
2016-09-28 13:15:53 -04:00
Matthew Jones a4ec0739ea Temporarily disable instance id gathering
During requests
2016-09-16 15:21:42 -04:00
Matthew Jones d8ea0920c5 Update configuration and playbook for rabbitmq
* Purge remaining zeromq settings
* Make sure we have CALLBACK_QUEUE defined where needed
* Update setup playbook to not be tied to old ha primary/secondary concept
2016-09-16 11:35:46 -04:00
Matthew Jones 3de4aae548 Fixing up HA induced flake8 issues 2016-09-15 13:51:17 -04:00
Matthew Jones 2872409271 Integrate rabbitmqctl interface to cluster and standalone containers 2016-09-15 12:07:16 -04:00
Matthew Jones c6f87607f2 Shifting migrations after integration with devel for ha 2016-09-15 10:55:00 -04:00
Matthew Jones 3255a17856 Exposing rabbitmq management console interface on compose 2016-09-15 10:46:45 -04:00
Matthew Jones f5d2c5c18a Merge branch 'ramparts_and_ha' into devel
* ramparts_and_ha: (21 commits)
  Rename database migrations for devel integration
  Integrate callback receiver refactoring
  Fix an issue running jobs in the cluster
  Implement a more dynamic celery queue system
  Purge old munin monitors and tools
  Refactor Tower HA Instance logic and models
  Docker compose improvements
  Initial Docker Compose workflow for Tower cluster
  Add memcached role for setup playbook
  Removing qpid from deb packaging
  Refactor rabbitmq role
  Integrate memcached into setup playbook
  Remove mongodb shutdown task
  Remove dependency on erlang_sd_notify
  Add initial rabbitmq role
  Initial rabbitmq setup playbook integration
  Update development environment for rabbit
  Replace qpid with rabbitmq
  Remove redis role from setup playbook
  Update qpid packaging, remove migrations
  ...
2016-09-15 10:19:57 -04:00
Matthew Jones 574a0fde05 Rename database migrations for devel integration 2016-09-15 10:15:14 -04:00
Matthew Jones 56d4210047 Implement Performance improvements shown in #3492
* Make callback workers tunable
* Disable callback worker recycling - doesn't appear to be needed
  anymore
* Tweak pexpect behavior by limiting its read buffer size and search
  window
* Use copy instead of deepcopy for job event callback emitter censor
2016-09-14 16:08:47 -04:00
Matthew Jones ab395b0009 Integrate callback receiver refactoring
* Drop ZMQ as the communication mechanism between job_event_callback and
  callback_receiver
* Setup queue and exchange for callback broker communication
* Refactor event plugin and callback receiver to efficiently handle
  message submission and processing
* Integrate django caching for parent processing
2016-09-14 11:42:13 -04:00
Matthew Jones 799f321760 Fix an issue running jobs in the cluster
The old VENV_PATH settings were still in place
2016-09-12 10:43:40 -04:00
Matthew Jones 988bbc4f4f Purge more qpid configuration 2016-09-12 09:50:42 -04:00
Matthew Jones 807cced571 Implement a more dynamic celery queue system
* Meant to be a starting point to more efficiently manage work routing
  and to balance work across all tower nodes
* Integrate flower as a dev tool that starts alongside other nodes.
  Helpful for observing and monitoring the queues/exchanges
* For the moment, force the task manager to only run on one node (not
  sure if this is needed)
* Define queues and routes for all task work
* Bump celery version to 3.1.23
* Expose flower through haproxy
2016-09-09 15:18:18 -04:00
Matthew Jones 13a0fd749f Purge old munin monitors and tools 2016-09-09 15:17:16 -04:00
Matthew Jones 0c1e1fa2fb Refactor Tower HA Instance logic and models
* Gut the HA middleware
* Purge concept of primary and secondary.
* UUID is not the primary host identifier, now it's based mostly on the
  username.  Some work probably still left to do to make sure this is
  legit.  Also removed unique constraint from the uuid field.  This
  might become the cluster ident now... or it may just deprecate
* No more secondary -> primary redirection
* Initial revision of /api/v1/ping
* Revise and gut tower-manage register_instance
* Rename awx/main/socket.py to awx/main/socket_queue.py to prevent
  conflict with the "socket" module from python base
* Revist/gut the Instance manager... not sure if this manager is really
  needed anymore
2016-09-08 13:37:53 -04:00
Matthew Jones eafb6c92b5 Docker compose improvements
* Make sure we explicitly set a hostname for tower nodes
* Switch rabbit vhost to use the root
2016-09-08 11:02:51 -04:00
Matthew Jones ebf103f345 Initial Docker Compose workflow for Tower cluster
The goal is to share a common pattern with the existing development work
2016-09-08 10:18:14 -04:00
Matthew Jones 27ab6705e7 Merge branch 'ramparts_and_ha' into devel
* ramparts_and_ha:
  Removing qpid from deb packaging
  Refactor rabbitmq role
  Integrate memcached into setup playbook
  Remove mongodb shutdown task
  Remove dependency on erlang_sd_notify
  Add initial rabbitmq role
  Initial rabbitmq setup playbook integration
  Update development environment for rabbit
  Replace qpid with rabbitmq
  Remove redis role from setup playbook
  Update qpid packaging, remove migrations
  Integrate packaging for qpid/memcached
2016-09-06 09:47:11 -04:00
Matthew Jones f3b7fe18ce Integrate memcached into setup playbook 2016-09-01 11:41:20 -04:00
Matthew Jones 95563c7637 Add Inventory and Project Update list endpoints 2016-08-31 16:17:22 -04:00
Matthew Jones f0b7622672 Update development environment for rabbit 2016-08-31 14:47:20 -04:00
Matthew Jones f988877bae Merge branch 'stable' into devel
* stable: (24 commits)
  Updating changelogs for 3.0.2 release
  fixing deprecated_team.organization credential migration
  Fix issue when installing bundled ansible on el6
  fixed localed date stuff
  update test to check org_auditor access
  ensure team organizations are assigned to credentials
  temporarily pin the pytest version until the ldap error can be fixed
  fixed locale
  fix date locale angular scheduler
  Make sure org admins can see credential after migration, comment updates on related tests add clause in test to verify automatic setting of org of new team credential
  Update team admin credential migration test to current state-of-knowledge
  fix ng-toast rel, resolves #3197 (#3316)
  allow users to edit their first and last name
  Revert "Prevent ignored task from being displayed as failing."
  Revert "Modify job event save behavior"
  fixing old tests for new user creation permissions
  Clean venv on 'make clean'
  Resolve KeyError by coercing instance_id to a str
  Update cloudforms dynamic inventory
  Update foreman inventory script
  ...
2016-08-31 14:34:52 -04:00
Matthew Jones 48e15d98b5 Merge branch 'release_3.0.2' into stable
* release_3.0.2: (150 commits)
  Updating changelogs for 3.0.2 release
  fixing deprecated_team.organization credential migration
  Fix issue when installing bundled ansible on el6
  fixed localed date stuff
  update test to check org_auditor access
  ensure team organizations are assigned to credentials
  temporarily pin the pytest version until the ldap error can be fixed
  fixed locale
  fix date locale angular scheduler
  Make sure org admins can see credential after migration, comment updates on related tests add clause in test to verify automatic setting of org of new team credential
  Update team admin credential migration test to current state-of-knowledge
  fix ng-toast rel, resolves #3197 (#3316)
  allow users to edit their first and last name
  Revert "Prevent ignored task from being displayed as failing."
  Revert "Modify job event save behavior"
  fixing old tests for new user creation permissions
  Clean venv on 'make clean'
  Resolve KeyError by coercing instance_id to a str
  Update cloudforms dynamic inventory
  Update foreman inventory script
  ...
2016-08-31 14:30:58 -04:00
Matthew Jones 116e40dbe7 Fix some venv issues related to the new dev env 2016-08-30 14:21:11 -04:00
Matthew Jones c112fc3cf4 Update qpid packaging, remove migrations
* Remove old system migrations
* Update qpid install deps for RH
2016-08-30 10:46:13 -04:00
Matthew Jones 3f0311a969 Integrate packaging for qpid/memcached
Also

* Remove redis packaging
* Fix typo in contributing
2016-08-26 14:59:43 -04:00