Commit Graph

62 Commits

Author SHA1 Message Date
AlanCoding
9e07b3f777 Tag jobs started via special cases with node & group 2017-06-22 16:24:12 -04:00
AlanCoding
d69b4e00ff select isolated node inside of queue based on capacity 2017-06-21 15:48:23 -04:00
Ryan Petrello
422950f45d Support for executing job and adhoc commands on isolated Tower nodes (#6524) 2017-06-14 11:47:30 -04:00
Matthew Jones
adf4be29b7 Fix taskmanager failing to launch workflow job
This was caused by taskmanager instance group refactoring. This
re-enables that functionality
2017-05-23 11:15:57 -04:00
Chris Meyers
067fdac8b1 prefetch optimizations for task manager
* Prefetch all Jobs Types related instance group
* Prefetch inventory updates inventory source. The attribute
inventory_source.inventory_id is accessed when building the hash
tables of the running tasks.
2017-05-16 15:05:20 -04:00
Chris Meyers
9b771ae907 only consider update_on_launch inventory sources
* When generating dependencies (i.e. dynamically launching Project
Update and Inventory Update) only create the dynamic dependencies if
update_on_launch is True.
2017-05-16 13:18:15 -04:00
AlanCoding
901c77bcfb dependent IU launch_type reduced to 'dependency' 2017-05-15 14:24:41 -04:00
Matthew Jones
fc6630fd25 Consider that the project could be none when launching dependencies 2017-05-15 12:33:36 -04:00
Matthew Jones
8bc1490368 Increase test coverage for task scheduler inventory updates
Also fixes some bugs found in that process
2017-05-11 14:36:13 -04:00
Matthew Jones
5508bad97c Updating unit tests for task manager refactoring
* Purging old task manager unit tests
* Migrating those tests to functional tests
* Updating fixtures and factories to support a change in the way the
  task manager is tested
* Fix an issue with the mk_credential fixture when used in functional
  tests. Previously it had trouble with multiplel invocations when
  persistence was used
2017-05-10 12:45:37 -04:00
Matthew Jones
4ced911c00 Implementing models for instance groups, updating task manager
* New InstanceGroup model and associative relationship with Instances
* Associative instances between Organizations, Inventory, and Job
  Templates and InstanceGroups
* Migrations for adding fields and tables for Instance Groups
* Adding activity stream reference for instance groups
* Task Manager Refactoring:
** Simplifying task manager relationships and move away from the
   interstitial hash tables
** Simplify dependency determination logic
** Reduce task manager runtime complexity by removing the partial
   references and moving the logic into the task manager directly or
   relying on Job model logic for determinism
2017-05-10 12:32:54 -04:00
Chris Meyers
c4fb88c0d9 remove uneeded post commit wrapper
* Since we changed the lower level method to always use post commit
message emit
2017-02-28 13:10:00 -05:00
Chris Meyers
dd513621f0 Revert "Merge pull request #5553 from chrismeyersfsu/fix-waiting_blocked"
This reverts commit 9ba2122f4f85eecaeb6fa53ac92ea2811b05e83f, reversing
changes made to c3a5f2c96fd85dd1405a8f5c875ffc988dee16a4.
2017-02-27 09:38:45 -05:00
Chris Meyers
903d0472f0 just like we fail running tasks fail waiting tasks
* Associate the celery_id with the job at the earliest point possible.
This ensures that a waiting job has a celery id. Thus, we are free to
fail waiting jobs that don't have a celery id.
2017-02-24 12:07:04 -05:00
Chris Meyers
08825a1f49 fix check running status 2017-02-23 15:09:50 -05:00
Chris Meyers
6e9488a59b ensure job deps are created only once 2017-02-15 15:54:30 -05:00
Chris Meyers
dfb7b08380 check job status before marking as failed
* When rectifying celery <-> db job status, re-check the job status to
ensure it did not finish between the time that we pulled the list of
celery tasks from celery and the time in which we rectify.
2017-02-09 15:27:08 -05:00
AlanCoding
b54de150db corner-case verbose message for users who do not provide an inventory for their job 2017-01-31 12:49:26 -05:00
AlanCoding
a8e46021f8 conservative method to validating that a job can be launched from a node 2017-01-18 12:16:45 -05:00
AlanCoding
5bfb0f229c graceful handling of null project and inventory in workflow 2017-01-17 14:19:46 -05:00
AlanCoding
8f259f2bb7 handle case of node with null UJT 2017-01-10 18:50:52 -05:00
Chris Meyers
4509f402a8 send notification when recifiying celery jobs 2016-12-15 16:31:39 -05:00
Chris Meyers
812b7c5f5f fix two inv updates created from 1 jt run 2016-12-09 14:09:58 -05:00
Chris Meyers
cc7c2957cf always chain failures
* When inv and proj updates trigger from a JT run, if either update
fails then the job template should get marked failed. Before this
commit, the job template would get marked failed ONLY if there was
enough capacity to run all the associated updates within the same
schedule() call. If, instead, the associated updates were ran in another
schedule() call, the failure chain was lost. This changeset fixes that
by saving the necessary data in the dependent_jobs relationship so that
the failure is always chained.
2016-12-09 14:09:54 -05:00
Aaron Tan
a458bb3c89 Remove source_workflow_job and use workflow_job_id instead. 2016-11-29 11:55:04 -05:00
Aaron Tan
54f99749eb Provide linkage from spawned job back to wfj. 2016-11-25 21:23:29 -05:00
Aaron Tan
9e4655419e Fix flake8 E302 errors. 2016-11-15 20:59:39 -05:00
Chris Meyers
9aaf694acf always allow running at least 1 job
* Regardless of capacity
* Add unit tests around this logic
2016-11-10 11:07:05 -05:00
Aaron Tan
cdbc5127a6 Workflow notification backend. 2016-11-08 15:55:15 -05:00
Chris Meyers
930b3651a0 Merge pull request #3917 from chrismeyersfsu/task-manager_prov_callback_inv_sync_dedup
on host callback, upate dynamic inv only once
2016-11-08 15:45:38 -05:00
Chris Meyers
adcf0af2a0 on host callback, upate dynamic inv only once
* During a job template launch via the host callback launch feature,
when the host launching the job template is not found; we try to launch
associated dyn inv updates to pick up the host. Then the inv is searched
again for the host. If the host is found, then the jt is launched. This
fix carries forward the feature that prevents the dyn inv update with
cache timeout of 0 from being re-launched with the new task manager.
2016-11-08 15:06:47 -05:00
Aaron Tan
05790c5b5c Merge pull request #3848 from jangsutsr/3528_workflow_job_cancel_endpoint
Implement workflow job cancel endpoint
2016-11-08 12:16:39 -05:00
Aaron Tan
9e244b640b Simplify internal node job cancel mechanism. 2016-11-08 11:32:33 -05:00
Aaron Tan
3c0d60075c Fix bug by using cancel_flag. 2016-11-08 11:32:33 -05:00
Aaron Tan
3778914aa9 Implement recursive spawned job cancel. 2016-11-08 11:31:06 -05:00
Chris Meyers
dcb8789fd9 fix job blocking job with allow simultaneous
related to #3911
2016-11-08 11:00:52 -05:00
Matthew Jones
343966f744 Implement gathering overall task capacity
For use when running/planning jobs
2016-11-07 13:45:01 -05:00
Matthew Jones
8e77deea27 Push celery queue stats to memcached periodically
We'll piggyback off the task that checks for inconsistent jobs between
celery and Tower itself. These are read off via /api/v1/ping
2016-11-03 12:10:38 -04:00
Chris Meyers
87dd91e849 rename Scheduler to TaskManager 2016-11-01 13:50:42 -05:00
Chris Meyers
c4a3b604f8 add back in alan's workflow fail detection 2016-11-01 13:50:42 -05:00
Chris Meyers
ed37e68c53 run dependencies when capacity is available 2016-11-01 13:50:42 -05:00
Chris Meyers
0f98e1edec remove todo's 2016-11-01 13:50:42 -05:00
Chris Meyers
454b3edb7c rectify celery<->db inconsistent running job 2016-11-01 13:50:42 -05:00
Chris Meyers
4ef4b4709b workflow execution added 2016-11-01 13:50:42 -05:00
Chris Meyers
9802b1f379 AdHocCommand support added to task manager 2016-11-01 13:50:42 -05:00
Chris Meyers
46faeffbb3 added task manager system job support 2016-11-01 13:50:42 -05:00
Chris Meyers
306562cd67 inventory updates running correctly 2016-11-01 13:50:42 -05:00
Chris Meyers
555f0bb90f project and jobs running correctly 2016-11-01 13:50:42 -05:00
AlanCoding
a2c972e513 Workflow status original commit 2016-10-27 12:52:07 -04:00
Alan Rominger
76fae6c052 Revert "Introduce workflow failure condition" 2016-10-27 12:21:31 -04:00