Commit Graph
100 Commits
Author SHA1 Message Date
Chris Meyers e1a24d9849 structure scan modules as dicts not arrays 2017-07-11 14:14:48 -04:00
Chris Meyers 7fda3c0658 back out thought to cause deadlock scenario 2017-07-10 10:43:29 -04:00
Chris Meyers aeb7119796 fix 2 data source inconcistency with failing tasks
* Do not "trust" the list of celery ids for database entries that were
modified after the list of celery ids was gotten.
* err on the side of caution and just let the next heartbeat celery
killer try killing the task if it needs to be reaped.
2017-07-10 10:43:01 -04:00
Chris Meyers 5b9a0b504a celery task fail check now uses pglock
* Align locking used by celery task cleaner upper with regular task manager.
* Uses pglock/advisory lock instead of abusing Instance table lock.
2017-07-10 10:41:54 -04:00
Chris Meyers 7a795b8681 fix job launch deadlock
* This both fixes the deadlock problem and a logic problem. We shouldn't
set the job's job_template current_job to pending jobs.
2017-07-10 10:34:36 -04:00
Chris Meyers df572d1477 migrations do not call overwitten save() methods
related to #6941

* We don't want our special logic to trigger anyway.
2017-07-07 16:47:07 -04:00
Chris Meyers e04c4e30d8 update tests to fix the removal of the word Tower 2017-07-06 11:22:16 -04:00
Chris Meyers 0a25cc2d19 back out parent jt status update short circuit 2017-07-06 10:48:28 -04:00
Chris Meyers 1331865749 take into account memcached key restrictions
* Keys can't contain spaces or control characters
2017-07-05 15:39:04 -04:00
Chris Meyers 0d3fc3389d flake8 fix 2017-07-05 13:37:32 -04:00
Chris Meyers e226b0ab37 noop pglock for unit tests 2017-07-05 13:34:07 -04:00
Chris Meyers f3f9782c0b fix 2 data source inconcistency with failing tasks
* Do not "trust" the list of celery ids for database entries that were
modified after the list of celery ids was gotten.
* err on the side of caution and just let the next heartbeat celery
killer try killing the task if it needs to be reaped.
2017-07-05 13:33:29 -04:00
Chris Meyers 15aee1f8ac celery task fail check now uses pglock
* Align locking used by celery task cleaner upper with regular task manager.
* Uses pglock/advisory lock instead of abusing Instance table lock.
2017-07-05 13:27:08 -04:00
Chris Meyers 3d4f8b0074 support clear_facts 2017-07-05 10:33:36 -04:00
Chris Meyers 4b84cec1e8 do not launch proj update on migrate 2017-07-05 10:32:45 -04:00
Chris Meyers 88b250b2f1 migrate scan jobs to use fact caching instead 2017-07-05 10:32:45 -04:00
Chris Meyers 318b1aebd1 remove special scan job logic 2017-07-05 10:32:45 -04:00
Chris Meyers eda0679eb3 Revert "support clear_facts"
This reverts commit e5432c231d1ab38807c8be10133d35ede237a519.
2017-07-05 10:32:13 -04:00
Chris Meyers 06edb6774a support clear_facts 2017-07-05 10:26:49 -04:00
Chris Meyers f1b1c4ee97 avoid instance record deadlock by using pglock 2017-07-05 09:38:19 -04:00
Chris Meyers 29adb39cae add insights_cred_id to host->inventory summary fields 2017-06-30 13:52:41 -04:00
Chris Meyers 668bce8212 fix job launch deadlock
* This both fixes the deadlock problem and a logic problem. We shouldn't
set the job's job_template current_job to pending jobs.
2017-06-30 09:54:08 -04:00
Chris Meyers d6082a976b retain support for insights system_id discovery 2017-06-22 09:49:45 -04:00
Chris Meyers d679dc7407 do not inject facts for isolated node runs 2017-06-21 13:45:28 -04:00
Chris Meyers 6c67834694 beautify code 2017-06-21 13:45:28 -04:00
Chris Meyers 45421c84a6 fact cache docs 2017-06-21 13:45:27 -04:00
Chris Meyers 965159a1fb rename store_facts to use_fact_cache 2017-06-21 13:45:27 -04:00
Chris Meyers ec2e537f63 remove fache cache receiver 2017-06-21 13:45:27 -04:00
Chris Meyers 4c118159ed system_tracking logging for fact cache 2017-06-21 13:45:27 -04:00
Chris Meyers 12cdbcf8b5 adds per-host timeout 2017-06-21 13:45:27 -04:00
Chris Meyers 817dbe8d33 add fact modified time 2017-06-21 13:45:27 -04:00
Chris Meyers 626e2d1c9b tower fact cache implementation
* Tower now injects facts into jobs via memcached for use by Ansible
playbooks. On the Ansible side, this is accomplished by the existing
mechanism, an Ansible Fact Cache Plugin + memcached. On the Tower side,
memcached is leveraged heavily.
2017-06-21 13:44:08 -04:00
Chris Meyers e437ddb3a5 insights integration docs 2017-06-12 12:01:30 -04:00
Chris Meyers c8ae48f1ac add host filter docs 2017-06-12 10:12:40 -04:00
Chris Meyers d0dc41daac turn single element array into string 2017-06-08 08:24:42 -04:00
Chris Meyers 2317687468 allow changing source_project
* scm project backed inventory now allows source_project to be mutated.
We reset the tracked revision when this happens.
2017-06-07 15:37:51 -04:00
Chris Meyers 3dc329bb0f copy paste fix 2017-06-07 15:04:44 -04:00
Chris Meyers aeb68f02f6 change settings comments 2017-06-07 15:03:53 -04:00
Chris Meyers ba5f553f0c fix up scm inventory updates
* implicit project updates inherit inv update instance group
* added needed settings for SCM group filter and others
2017-06-07 14:51:56 -04:00
Chris Meyers 0d56f42c83 better insights_credential reverse relationship
* From an insights cred, access all related inventories via
cred.insights_inventories rather than cred.insights_credential
* This fixes auto-attaching insights_credential to credential summary
fields. Fixes by NOT attaching them. That is the desired behavior.
2017-06-05 13:48:41 -04:00
Chris Meyers eb6e598f6c add insights_credential to summary fields on inventory 2017-06-01 08:47:21 -04:00
Chris Meyers 0c034f29af use insights etag as a cache
* During insights project updates, if the etag from insights matches the
last etag (scm_revision) then don't get all the maintenance playbooks.
Otherwise, get all the maintenance playbooks and update scm_revision
with the etag.
2017-05-30 18:30:13 -04:00
Chris Meyers 8b97db06b0 abuse group_by and instance_filters 2017-05-30 18:29:02 -04:00
Chris Meyers 9ede92a426 Revert "support vmware via source_vars"
This reverts commit cc64104f4bad6c360dcbd2f0291006956180a416.
2017-05-30 16:16:12 -04:00
Chris Meyers f7f4560de1 support vmware via source_vars
* support vmware host_filters and groupby_patterns by just doing nothing
2017-05-30 15:58:30 -04:00
Chris Meyers 441946d332 add v1 vs. v2 inclusion/exclusion 2017-05-26 14:04:15 -04:00
Chris Meyers b25b397e14 remove insights logic from backwards compat logic
* spelling fixes
2017-05-26 09:58:20 -04:00
Chris Meyers 2f43f0fe2b flake8 2017-05-26 08:52:13 -04:00
Chris Meyers 7c0513d5ee migrate insights credentials
* A credential is considered an insights credential if a project of
scm_type 'insights' is associated.
* all non-insights Projects that point at an insights credential are
nulled
2017-05-26 08:50:53 -04:00
Chris Meyers 87eea59845 add per-inventory insights credential 2017-05-26 07:25:36 -04:00
Chris Meyers ca1eb28d1f don't assume insights id is a uuid 2017-05-25 17:52:46 -04:00
Chris Meyers 4461c8fe91 more robust insights error handling 2017-05-22 17:32:17 -04:00
Chris Meyers 3793f1ec09 insights proxy robust error condition handling 2017-05-22 15:02:16 -04:00
Chris Meyers 9ba0ba07ec nest insights api response data 2017-05-22 10:11:31 -04:00
Chris Meyers fb53ca8455 insights proxy
* Issue request to Red Hat Insights API from Tower
/hosts/<id>/insights endpoint. User the first found Satellite 6
Credential as Basic Auth requests to Insights API.
2017-05-22 10:11:31 -04:00
Chris Meyers cf26678cea remove not operator code instead of commenting 2017-05-19 16:38:41 -04:00
Chris Meyers fceb09dc8c remove not operator
* We've switched from Q() to Host.objects.filter() and you can't do
~(Host.objects.filter()). We'd need to create an AST to track the
operations in order to perform the boolean logic ~ and we don't have
time for that now.
2017-05-19 14:40:02 -04:00
Chris Meyers 136bdc06e9 remove insights_system_id uniqueness constraint 2017-05-18 12:51:25 -04:00
Chris Meyers e81a7e1c25 allow insights_system_id to be set to null via api 2017-05-18 11:31:58 -04:00
Chris Meyers 067fdac8b1 prefetch optimizations for task manager
* Prefetch all Jobs Types related instance group
* Prefetch inventory updates inventory source. The attribute
inventory_source.inventory_id is accessed when building the hash
tables of the running tasks.
2017-05-16 15:05:20 -04:00
Chris Meyers 9b771ae907 only consider update_on_launch inventory sources
* When generating dependencies (i.e. dynamically launching Project
Update and Inventory Update) only create the dynamic dependencies if
update_on_launch is True.
2017-05-16 13:18:15 -04:00
Chris Meyers cc4692932a update Host manager to handle Query Sets
* filter_host_string now returns a query set instead of Q(). This change
updates the Host manager to remove the .filter() call since the results
returned from filter_host_string() has already turned Q() -> Query Set
2017-05-12 12:14:32 -04:00
Chris Meyers ac534abe24 expose insights_system_id via api
* insights_system_id is read-only. Only modified via the fact gathering
code path.
2017-05-12 12:09:20 -04:00
Chris Meyers eb27233e5c clear insights system id if not found
* Host previously found to have an insights system id that then later
are found to not have an insights system id, have their system id
cleared.
2017-05-12 12:00:30 -04:00
Chris Meyers 3cdc332497 rename insight tracking field; handle duplicate
* rename from insights_machine_id to insights_system_id
* gracefully handle duplicate insights machine id's
2017-05-12 10:47:49 -04:00
Chris Meyers b55af095ef boolean logic performed on filter results
* Before, the boolean logic operators were performed against the Q()
objects, iteratively. Now, boolean logic is done after
Host.objects.filter(Q()). This results in the wanted and expected
results.
2017-05-12 09:26:11 -04:00
Chris Meyers b9b0b29d97 associate insights machine id w/ host on fact scan
* Add inisghts fact scan module
* Update fact scan playbook to call new insight fact scan module
* JT run w/ store_facts=True will save scanned facts. We "skim" the
machine_id fact from Insights fact scans and associate it with the host
that the fact scan came from.
2017-05-11 11:20:52 -04:00
Chris Meyers 0be384af18 rename field to store_facts and fix up tests 2017-05-01 16:39:48 -04:00
Chris Meyers 6dec9d1014 per-jt most recent fact scan gathering
* Remove system tracking piggybacked most-recent-fact-gathering
* Add explicit per-jobtemplate configurable fact gathering
* Continue to support system tracking
* Fixed a bug where we weren't ack()ing system tracking rabbit msgs
2017-05-01 16:03:16 -04:00
Chris Meyers 4795fd28bf always gather facts
* bolt on the fact scan gather to all jobs
2017-05-01 16:03:16 -04:00
Chris Meyers 633da940a8 create .lock file without needing the proj dir to exist
* We create <project_name>.lock file in the tower project root
directory. This should not require that the project directory exists,
only that the project directory name is generatable.
2017-05-01 11:15:18 -04:00
Chris Meyers d14a55911b ack fact scan messages
* Fixes a bug where fact rabbit messages would build up because we never
explicitly ack them.
2017-04-28 13:46:54 -04:00
Chris Meyers d69ae2cc92 coarse json queries to use gin index 2017-04-25 10:21:22 -04:00
Chris Meyers 5a7b78220d pin twisted and asgiref for asgi-amqp
* asgi-amqp translation layer needs a specific version of channels which
requires a specific version of daphne and all of the fall out.
2017-04-19 13:56:36 -04:00
Chris Meyers 8c8ccba5a5 add mock to ansible dev env requirements 2017-04-18 12:31:05 -04:00
Chris Meyers 70a8ccef0f return count, results, next, prev
* This returns /api/vx/hosts/ to having all the nice response items like
count, results, next, prev. I had "chopped" that off by explicitly only
returning result.
2017-04-17 16:43:29 -04:00
Chris Meyers 55268a8ff0 remove facts_latest in favor of ansible_facts
* Closer align our facts data structure with ansible facts data
structure for purposes of ... wait for it ... 2-way fact caching
2017-04-17 14:27:33 -04:00
Chris Meyers 775093294e run host listing through v1 vs. v2 logic
* api/v1/hosts related fields were getting v2 in the related urls. The
context is now included in the serializer instantiation.
2017-04-17 11:43:44 -04:00
Chris Meyers 5ff4966526 add json a=null support and still support a="null"
related to #6016
2017-04-13 16:26:35 -04:00
Chris Meyers 040f9dd545 support > 2 sequential and, also or
* Fixed a bug in the fact search language parser where it would not loop
over all the <and, or> operations. The result, we can support (a=b or
a=c or a=d) also support (a=b and b=b and c=b)
2017-04-13 13:36:38 -04:00
Chris Meyers 23ce5ab9de fix up test 2017-04-13 10:44:28 -04:00
Chris Meyers bd1f8b088f 400 on invalid host filter query instead of 500 2017-04-13 09:40:11 -04:00
Chris Meyers 1a2cbaf5e1 flake8 fix 2017-04-12 15:35:42 -04:00
Chris Meyers a2b37aa7e6 add not logic to json search
* Also fix up single contains [] logic. Values with a single contains
should be wrapped in [] and not, like I thought before, \"\"
2017-04-12 15:11:36 -04:00
Chris Meyers 70fafe75d2 clean up commented code 2017-04-12 11:42:31 -04:00
Chris Meyers d1eba4b607 log lock errors and test it 2017-04-12 11:41:11 -04:00
Chris Meyers 7919c47370 put lock file in project root dir 2017-04-12 09:36:15 -04:00
Chris Meyers 648aa470d7 lock projects on project sync
* Use filesystem, blocking, locks to prevent two project syncs for the
same project from running the project update playbook at the same time.
2017-04-11 15:47:46 -04:00
Chris Meyers a4fe04cbfa handle unicode and types 2017-04-10 14:57:37 -04:00
Chris Meyers 5ce4c80821 support json/python types 2017-04-07 12:24:20 -04:00
Chris Meyers ee09bca558 change facts_recent to facts_latest 2017-04-05 16:50:38 -04:00
Chris Meyers 02795e526c inherit from jsonbfield package not jsonfield
* jsonbfield supports json querying. jsonfield package is still a jsonb
postgres data type, but doesn't support jsonb style querying.
* add undo migration support to GIN index
2017-04-05 13:49:51 -04:00
Chris Meyers f7fb541fe2 add facts recent endpoints
* Hang most recent listing of facts per-host off of /hosts/n/
2017-04-04 17:54:02 -04:00
Chris Meyers f5d7d0bce5 add model fact recent
* Copy of the most recent system tracking fact for each module type.
Utimately, this allows us to GIN index the jsonb object to support
fact searching.
2017-04-04 16:32:30 -04:00
Chris Meyers a94c8ef1e3 add functional tests for host_filter 2017-04-04 09:13:56 -04:00
Chris Meyers 67633b2387 add pyparsing a top level dep and fix other deps
* bump shade
* remove pyrax as a dep
* pin daphne so channels doens't pick up the wrong, too new, version
2017-04-03 16:17:05 -04:00
Chris Meyers 443f14ed85 flake8 is hard 2017-04-03 16:14:43 -04:00
Chris Meyers 80e89b1116 spelling is hard 2017-04-03 15:53:54 -04:00
Chris Meyers 6cbdb8d0e6 faster first time parse generator
* Generating the set of valid unicode characters is expensive in terms
of memory storage. Instead, we define the grammer by the negation of
allowed unicode characters. Much faster.
2017-04-03 15:32:21 -04:00
Chris Meyers a8213661fd basic fact search grammar
* Establish a base grammar for handling json path specification and
value matching. With boolean logic support and parenthesis grouping
i.e. (a.b.c="value") and ((a.b="foo") or (a="bar"))
* generate Q() results for passing to Host.objects.filter()
* Hooked up via /api/v1/hosts?host_filter=...
* DynamicFilterField added to store host filter as string with grammar
parser attached as static methods for later use by DynamicInventory &
continued use by host_filter=...
2017-04-03 15:32:21 -04:00