269 Commits

Author SHA1 Message Date
Chris Meyers
ca1eb28d1f don't assume insights id is a uuid 2017-05-25 17:52:46 -04:00
Chris Meyers
eb27233e5c clear insights system id if not found
* Host previously found to have an insights system id that then later
are found to not have an insights system id, have their system id
cleared.
2017-05-12 12:00:30 -04:00
Chris Meyers
3cdc332497 rename insight tracking field; handle duplicate
* rename from insights_machine_id to insights_system_id
* gracefully handle duplicate insights machine id's
2017-05-12 10:47:49 -04:00
Chris Meyers
b9b0b29d97 associate insights machine id w/ host on fact scan
* Add inisghts fact scan module
* Update fact scan playbook to call new insight fact scan module
* JT run w/ store_facts=True will save scanned facts. We "skim" the
machine_id fact from Insights fact scans and associate it with the host
that the fact scan came from.
2017-05-11 11:20:52 -04:00
Aaron Tan
cfb633e8a6 Dependency Updates
* Dynamic Inventory Source
Template against ansible 2.3 dynamic inventory sources.
The major change is removal of `rax.py`. Most upstream scripts except
`foreman.py` has quite trivial coding style changes, or minor functional
extensions  that does not affect Tower inventory update runs.
`foreman.py`, on the other hand, went through quite a major refactoring,
but functionalities stay the same.

Major python dependency updates include apache-libcloud (1.3.0 -->
2.0.0), boto (2.45.0 --> 2.46.1) and shade (1.19.0 --> 1.20.0). Minor
python dependency updates include indirect updates via `pip-compile`,
which are determined by base dependencies.

Some minor `task.py` extensions:
 - `.ini` file for ec2 has one more field `stack_filter=False`, which
   reveals changes in `ec2.py`.
 - `.ini` file for cloudforms will catch these four options from
   `source_vars_dict` of inventory update: `'version', 'purge_actions',
   'clean_group_keys', 'nest_tags'`. These four options have always been
   available in `cloudforms.py` but `cloudforms.ini.example` has not
   mentioned them until the latest version. For consistency with upstream
   docs, we should make these fields available for tower user to customize.
 - YAML file of openstack will catch ansible options `use_hostnames`,
   `expand_hostvars` and `fail_on_errors` from `source_vars_dict` of
   inventory update as a response to issue #6075.

* Remove Rackspace support
Supports of Rackspace as both a dynamic inventory source and a cloud
credential are fully removed. Data migrations have been added to support
arbitrary credential types feature and delete rackspace inventory
sources.

Note also requirement `jsonschema` has been moved from
`requirements.txt` to `requirements.in` as a primary dependency to
reflect it's usage in `/main/fields.py`.

Connected issue: #6080.

* `pexpect` major update
`pexpect` stands at the very core of our task system and underwent a
major update from 3.1 to 4.2.1. Although verified during devel, please
still be mindful of any suspicious issues on celery side even after this
PR gets merged.

* Miscellaneous
 - requests now explicitly declared in `requirements.in` at version 2.11.1
   in response to upstream issue
 - celery: 3.1.17 -> 3.1.25
 - django-extensions: 1.7.4 -> 1.7.8
 - django-polymorphic: 0.7.2 -> 1.2
 - django-split-settings: 0.2.2 -> 0.2.5
 - django-taggit: 0.21.3 -> 0.22.1
 - irc: 15.0.4 -> 15.1.1
 - pygerduty: 0.35.1 -> 0.35.2
 - pyOpenSSL: 16.2.0 -> 17.0.0
 - python-saml: 2.2.0 -> 2.2.1
 - redbaron: 0.6.2 -> 0.6.3
 - slackclient: 1.0.2 -> 1.0.5
 - tacacs_plus: 0.1 -> 0.2
 - xmltodict: 0.10.2 -> 0.11.0
 - pip: 8.1.2 -> 9.0.1
 - setuptools: 23.0.0 -> 35.0.2
 - (requirements_ansible.in only)kombu: 3.0.35 -> 3.0.37
2017-05-08 12:03:02 -04:00
Alan Rominger
84b8dcece0 Merge pull request #6120 from AlanCoding/ansible-inventory
Inventory updates through `ansible-inventory`
2017-05-02 16:19:58 -04:00
Chris Meyers
4795fd28bf always gather facts
* bolt on the fact scan gather to all jobs
2017-05-01 16:03:16 -04:00
AlanCoding
a92c7cec7d remove code to parse inventory files 2017-05-01 14:37:42 -04:00
AlanCoding
8e6020436c modularization of inventory_import command
This separates file parsing logic that was mixed
in with other important code inside of the
inventory import command.

The logic around MemObject data structures was
moved to utils, and the file parsing was moved
to a legacy module. As of this commit, that
module can operate within the Tower environment
but it will be removed.

Also refactor the loggers to fix old bug and
work inside of the different contexts - the
Loader classes, mem objects, and hopefully
the inventory modules eventually.
2017-05-01 14:37:42 -04:00
AlanCoding
2488dabca6 add ansible-inventory backport from upstream
This is a separately maintained repo
https://github.com/ansible/ansible-inventory-backport
tests and compatibility is managed in that location
2017-05-01 14:37:42 -04:00
Graham Mainwaring
5098d0e99c Fix case where Get-ChildItems does not return any results 2017-03-10 14:53:05 -05:00
Chris Meyers
608975ff2f bump vmware inventory script 2017-01-26 16:01:57 -05:00
Bill Nottingham
e12ae193db Merge foreman inventory script performance fixes.
Sent & accepted upstream.
2017-01-24 16:40:59 -05:00
Chris Meyers
b72a148cbc vmware_inventory dyn inv script support 2016-12-19 16:39:10 -05:00
Chris Meyers
2f98716c0b Merge pull request #4493 from chrismeyersfsu/fix-3954
fix license not found err msg
2016-12-19 13:01:09 -05:00
Chris Meyers
1c4e6c55f5 bump foreman and expose new grouping features
related to #3467
related to #5226
related to #4373
2016-12-19 11:26:43 -05:00
Chris Meyers
a95880b60a fix license not found err msg 2016-12-19 10:39:28 -05:00
Chris Meyers
a566cec5db rectifiy arch name in fact scans 2016-12-16 10:06:34 -05:00
Chris Meyers
248f030744 bump dyn inv sync scripts 2016-12-13 17:14:43 -05:00
Matthew Jones
35b9786e06 Fixing up fact cachine related unit tests 2016-11-18 10:23:07 -05:00
Matthew Jones
3b07d9745d Port fact caching system to rabbitmq
* Purge all references to zmq also
* New setting to control the queue it works on
2016-11-18 09:44:17 -05:00
Aaron Tan
9e4655419e Fix flake8 E302 errors. 2016-11-15 20:59:39 -05:00
Aaron Tan
473ed016de Fix flake8 E305 errors. 2016-11-15 16:32:27 -05:00
Matthew Jones
c6cf02a602 Merge branch 'release_3.0.3' into devel
* release_3.0.3: (55 commits)
  Revert "Revert "Add needed types for selinux change""
  Revert "Add needed types for selinux change"
  interpret backslash escapes when displaying url in welcome message
  Bump the SELinux policy version
  Add needed types for selinux change
  Update SELinux policy to allow httpd_t to execute files in lib_t and var_lib_t
  Bumping changelog for 3.0.3
  Update rax.py inventory
  Revert "filter internal User.admin_roles from the /roles API list view"
  fix spelling of disassociated
  Resolves 404 when assigning resources/users to organizations in card view. Sidesteps a bug in the Refresh() utility, where pagination calculations are not made against filtered results.
  Sync azure changes to Tower virtual environment
  Add regions here as well.
  Also bump boto for new regions, per ryansb.
  More regions!
  Revert "bump shade version"
  bump shade version
  Hack copying of job_template.related.survey_spec into ui job copy flow, resolves #3737
  Revert "bump shade version"
  bump shade version
  ...
2016-11-01 11:49:28 -04:00
Chris Church
440f0539b0 Flake8 fixes. 2016-10-28 21:58:24 -04:00
Chris Church
c18b6c1352 Add support for capturing stdout associated with job events and ad hoc command events.
* New event types for stdout lines not associated with a callback event.
* New stdout, start_line, end_line and verbosity fields for job/ahc events.
* Callback plugins to wrap Ansible default/minimal stdout callbacks and embed callback event data using ANSI escape sequences.
* Callback plugin library to wrap ansible.display.Display class methods.
* Output filter to extract event data from stdout and create job/ahc events.
* Update stdout formats to strip new ANSI escape sequences.
2016-10-28 21:58:24 -04:00
James Laska
f129c42929 Update rax.py inventory
Resolves https://github.com/ansible/ansible/pull/18204 and continues
support for disabling SSL verification.

Fixes #3709
2016-10-26 15:59:41 -04:00
AlanCoding
ef861e5256 move set_artifact module to its own repository 2016-10-12 17:54:35 -04:00
AlanCoding
5d4cf9d4fc Add job artifacts and workflow artifact passing
artifacts redact from job when no_log is set
parent no_log artifacts treated as survey passwords
2016-10-12 17:45:50 -04:00
Graham Mainwaring
180209236e Change names of Windows scan modules to avoid ambiguity 2016-10-05 16:43:19 -04:00
nitzmahone
898cbbf26e update azure_rm inventory script to 2.0.0rc5-target 2016-09-23 10:28:24 -04:00
Matthew Jones
3de4aae548 Fixing up HA induced flake8 issues 2016-09-15 13:51:17 -04:00
Matthew Jones
f5d2c5c18a Merge branch 'ramparts_and_ha' into devel
* ramparts_and_ha: (21 commits)
  Rename database migrations for devel integration
  Integrate callback receiver refactoring
  Fix an issue running jobs in the cluster
  Implement a more dynamic celery queue system
  Purge old munin monitors and tools
  Refactor Tower HA Instance logic and models
  Docker compose improvements
  Initial Docker Compose workflow for Tower cluster
  Add memcached role for setup playbook
  Removing qpid from deb packaging
  Refactor rabbitmq role
  Integrate memcached into setup playbook
  Remove mongodb shutdown task
  Remove dependency on erlang_sd_notify
  Add initial rabbitmq role
  Initial rabbitmq setup playbook integration
  Update development environment for rabbit
  Replace qpid with rabbitmq
  Remove redis role from setup playbook
  Update qpid packaging, remove migrations
  ...
2016-09-15 10:19:57 -04:00
Matthew Jones
56d4210047 Implement Performance improvements shown in #3492
* Make callback workers tunable
* Disable callback worker recycling - doesn't appear to be needed
  anymore
* Tweak pexpect behavior by limiting its read buffer size and search
  window
* Use copy instead of deepcopy for job event callback emitter censor
2016-09-14 16:08:47 -04:00
Matthew Jones
ab395b0009 Integrate callback receiver refactoring
* Drop ZMQ as the communication mechanism between job_event_callback and
  callback_receiver
* Setup queue and exchange for callback broker communication
* Refactor event plugin and callback receiver to efficiently handle
  message submission and processing
* Integrate django caching for parent processing
2016-09-14 11:42:13 -04:00
Matthew Jones
f988877bae Merge branch 'stable' into devel
* stable: (24 commits)
  Updating changelogs for 3.0.2 release
  fixing deprecated_team.organization credential migration
  Fix issue when installing bundled ansible on el6
  fixed localed date stuff
  update test to check org_auditor access
  ensure team organizations are assigned to credentials
  temporarily pin the pytest version until the ldap error can be fixed
  fixed locale
  fix date locale angular scheduler
  Make sure org admins can see credential after migration, comment updates on related tests add clause in test to verify automatic setting of org of new team credential
  Update team admin credential migration test to current state-of-knowledge
  fix ng-toast rel, resolves #3197 (#3316)
  allow users to edit their first and last name
  Revert "Prevent ignored task from being displayed as failing."
  Revert "Modify job event save behavior"
  fixing old tests for new user creation permissions
  Clean venv on 'make clean'
  Resolve KeyError by coercing instance_id to a str
  Update cloudforms dynamic inventory
  Update foreman inventory script
  ...
2016-08-31 14:34:52 -04:00
James Laska
188f81ef3d Update cloudforms dynamic inventory
Pulling the latest from
https://github.com/ansible/ansible/blob/devel/contrib/inventory/cloudforms.py

Related #3168
2016-08-19 15:37:28 -04:00
James Laska
a847495bea Update foreman inventory script
Per request from Red Hat Satellite team, update to the latest foreman.py

https://github.com/theforeman/foreman_ansible_inventory/blob/master/foreman_ansible_inventory.py
2016-08-19 15:37:28 -04:00
Matthew Jones
8893f8278a Merge branch 'release_3.0.2' into devel
* release_3.0.2: (126 commits)
  Disable permissions tab in Credential > Edit form if Credential is private (#3288)
  Tweaked the popover text for job and skip tags on JT add/edit
  Workaround a cascade setnull polymorphic issue
  flake8
  Fixed old test expectations
  Made it so the credential organization field can't be changed
  Skip some unit tests
  Fixed org auditor visibility of team credentials
  Fix sosreport
  fix credential kind options for list
  interpret any code below 300 as success
  bail when status code is over 300
  Make CloudForms inventory_script work
  Use no_log when handling passwords
  Prevent ignored task from being displayed as failing.
  making ec2 cred optional on group->edit
  making ec2 credential optional for ec2 inventory
  Revert "Fix to ensure org auditors can see team credentials"
  Fixed team credential list to work with corrected permissions
  Making the username and password fields optional
  ...
2016-08-18 22:52:55 -04:00
James Laska
ee66fd4aa5 Make CloudForms inventory_script work
Fixes a few flake8 issues while at it.
2016-08-11 18:50:42 -04:00
Graham Mainwaring
7ef92a609f Windows scan jobs (#3034)
Implements simple package, service and file scanning for Windows.
2016-08-10 10:19:26 -04:00
James Laska
13bd9211ae Add copyright back with utf-8 encoding
Fixes encoding exception
2016-07-25 15:18:10 -04:00
James Laska
11d68d1ebe Complete Sat6 integration
Various fixes to get sat-6 integration working.

* inventory_import.py - the inventory script is called foreman.py
* tasks.py - be sure to call `add_section()` before using it
* defaults.py - Add various SATELLITE6_* enablement variables

Relates #3119
2016-07-25 15:13:39 -04:00
James Laska
d182280ff7 Correctly set verify_ssl to a boolean value 2016-06-17 20:53:46 -04:00
James Laska
af8011cd91 Honor CLOUD_VERIFY_SSL variable in rax import 2016-06-16 15:49:08 -04:00
Matthew Jones
4d52d0748f Include core ec2 fix for tag sync limit 2016-06-13 11:24:47 -04:00
Matthew Jones
bdf5ee1fea Import latest ec2.py and fix an issue
Upgrading ec2.py to fix an issue where the module wasn't obeying
"replace_dash_in_groups" property.
2016-06-07 14:01:00 -04:00
Matthew Jones
e5ec5c9726 Change prefix for imported sattelite groups 2016-05-18 10:22:12 -04:00
Matthew Jones
ae4cfc98ce Update to house's latest legacy azure inv script 2016-05-13 10:47:18 -04:00
Matthew Jones
b0f4dc06c4 Fix fail/skip scenario for scan service jobs
In the case where no services could be found we'll set the status as
skipped instead of failed
2016-05-09 15:30:56 -04:00