From bb2e84f61190563a8efa4ee399f02a9b77343e77 Mon Sep 17 00:00:00 2001 From: Aaron Tan Date: Wed, 9 Nov 2016 10:57:46 -0500 Subject: [PATCH 01/11] Initial commit. --- docs/workflow.md | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 docs/workflow.md diff --git a/docs/workflow.md b/docs/workflow.md new file mode 100644 index 0000000000..d704ce4449 --- /dev/null +++ b/docs/workflow.md @@ -0,0 +1,7 @@ +## Tower Workflow Overview + +## Acceptance Criteria + +## Testing Considerations + +## Performance Testing From 643e294245883a7fe5687963e90297b7ddb4d03e Mon Sep 17 00:00:00 2001 From: Aaron Tan Date: Wed, 9 Nov 2016 18:25:18 -0500 Subject: [PATCH 02/11] Overview and usage manual --- docs/workflow.md | 42 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/docs/workflow.md b/docs/workflow.md index d704ce4449..ec715fe659 100644 --- a/docs/workflow.md +++ b/docs/workflow.md @@ -1,4 +1,46 @@ ## Tower Workflow Overview +Workflows are structured pipelines of multiple tower job runs. Think of workflows as jobs that have no native content, but triggers other job in specific orders to accomplish the purpose of tracking the full set of jobs that were part of a release process as a single unit. + +A workflow job is composed of one or more decision trees, each node of which wraps around zero to one tower job. A workflow job run starts with creating and running jobs of the root node of each decision tree. When a parent node has its job finished running, it would pick up a subset of its child nodes according to its job status, and create new jobs for them. The selected child nodes would then run their own jobs and so forth till all nodes that needs to run finish running, which marks the end of the underlying workflow run. + +## Usage Manual + +### Workflow CUID +Like other job resources, workflow jobs are created based on workflow job templates. Workflow job templates are equipped with common funtional fields including labels, schedules, notification templates, extra variables and survey specifications. It also comes with a custom `workflow_nodes` field which refers to a list of related workflow job template nodes. + +The CUID operations against a workflow job template and its corresponding workflow jobs are almost identical to those of normal job templates and related jobs. However, from RBAC perspective, CUID on workflow job templates/jobs are limited to administrators. That is, an organization administrator takes full control over all workflow job templates/jobs under the same organization, while an organization auditor is able to see workflow job templates/jobs under the same organization. On the other hand, ordinary organization members have no, and are not able to gain, permission over any workflow-related resources. + +### Workflow Nodes +Workflow Nodes are containers of workflow spawned jobs and function as nodes of workflow decision trees. Like that of workflow itself, the two types of workflow nodes are workflow job template nodes and workflow job nodes. + +Workflow job template nodes are listed and created under endpoint `/workflow_job_templates/\d/workflow_nodes/` to be associated with underlying workflow job template, or under endpoint `/workflow_job_template_nodes/` directly. The most important fields of a workflow job template node are `success_nodes`, `failure_nodes`, `always_nodes`, `unified_job_template` and `workflow_job_template`. The former three are lists of workflow job template nodes that in union forms the set of all it s child nodes, in specific, `success_nodes` are triggered when parnent node job succeeds, `failure_nodes` are triggered when parent node job fails, and `always_nodes` are triggered regardless of whether parent job fails. The later two reference the job template resource it contains and workflow job template it belongs to. + +Workflow job nodes are created by workflow job template nodes at the beginning of workflow job run. Other than the fields inherited from its template, workflow job nodes also come with a `job` field which references the to-be-spawned job resource. + +### Decision Tree Formation and Restrictions +The decision tree structure of a workflow is enforced by associating workflow job template nodes via endpoints `/workflow_job_template_nodes/\d/*_nodes/`, where `*` has options `success`, `failure` and `always`. However there are restrictions that must be enforced when setting up new connections. Here are the three restrictions that will raise validation error when break: +* Cycle restriction: According to decision tree definition, no cycle is allowed. +* Convergent restriction: Different paths should not come into the same node, in other words, a node cannot have multiple parents. +* Mutex restriction: A node cannot have all three types of child nodes. It contains either always nodes only, or any type other than always nodes. + +### Workflow Run Details +A typical workflow run starts by either POSTing to endpoint `/workflow_job_templates/7/launch/`, or being triggered automatically by related schedule. The first step is the workflow job template creating workflow job, and all related workflow job template nodes creating workflow job node. Right after that, all root nodes are populated with corresponding job resources and start running. If nothing goes wrong, each decision tree will follow its own route to completion. The entire workflow finishes running when all its decision trees completes. + +Job resources spawned by workflow jobs are needed by workflow to run correctly. Therefore deletion of spawned job resources is blocked while the underlying workflow job is executing. + +Other than success and failure, a workflow spawned job resource can also end with status 'error' and 'canceled'. When a workflow spawned job resource errors, all branches starting from that job will stop executing while the rest keep their own paces. Canceling a workflow spawned job resource follows the same path. + +A workflow job itself can also be canceled altogether. In this case all its spawned job resources will be canceled and following paths stop executing. + +Like job templates, workflow job templates can associated with notification templates and notifications works exactly the same as that of job templates. One distinction is the notification message body. Workflow jobs sends notification that contains not only the status of itself, but also status of all its spawned jobs. Typical notification body looks like this: +``` +Workflow job summary: + +- node #141 spawns no job. +- node #139 spawns job #212, "foo", which finished with status successful. +- node #140 spawns job #213, "bar", which finished with status failed. +... +``` ## Acceptance Criteria From 41e281d233c94d6aa04cf9d02b58fd55a1ab79f9 Mon Sep 17 00:00:00 2001 From: Aaron Tan Date: Thu, 10 Nov 2016 13:12:50 -0500 Subject: [PATCH 03/11] Test Coverage & Test Notes added --- docs/workflow.md | 28 ++++++++++++++++++++++++---- 1 file changed, 24 insertions(+), 4 deletions(-) diff --git a/docs/workflow.md b/docs/workflow.md index ec715fe659..ea60ca6f0a 100644 --- a/docs/workflow.md +++ b/docs/workflow.md @@ -42,8 +42,28 @@ Workflow job summary: ... ``` -## Acceptance Criteria +## Test Coverage +* Verify that CUID operations on all workflow resources are working properly. Note workflow job nodes cannot be created or deleted independently, but verifications are needed to make sure when a workflow job is deleted, all its related workflow job nodes are deleted. +* Verify the RBAC property of workflow resources. In specific: + * Workflow job templates can only be accessible by super users ---- system admin, admin of the same organization and system auditor and auditor of the same organization with read permission only. + * Workflow jobs follows the permission rules of its associated workflow job template. + * Workflow job template nodes rely their permission rules on the permission rules of both their associated workflow job template and unified job template. + * Workflow job nodes follows the permission rules of both its associated workflow job and unified job. +* Verify that workflow job template nodes can be created under, or (dis)associated with workflow job templates. +* Verify that only the permitted types of unified job templates can be associated with a workflow job template node. Currently the permitted types are **job templates, inventory sources and projects**. +* Verify that workflow job template nodes under the same workflow job template can be associated to form parent-child relationship of parent trees. In specific, one node takes another as its child node by POSTing another node's id to one of the three endpoints: `/success_nodes/`, `/failure_nodes/` and `/always_nodes/`. +* Verify that workflow job template nodes are not allowed to have invalid association. Any attempt that causes invalidity will trigger validation error. The three types of invalid associations are cycle, convergence(multiple parent) and mutex('always' XOR the rest). +* Verify that workflow jobs can be launched by POSTing to endpoint `/workflow_job_templates/\d/launch/`. +* Verify that schedules can be successfully (dis)associated with a workflow job template, and workflow jobs can be triggered by the schedule of associated workflow job template at specified time point. +* Verify that during a workflow job run, all its decision trees follow their correct paths of execution. Unwarranted behaviors include child node executing before its parent and wrong path being selected (failure nodes are executed when parent node succeeds and so on). +* Verify that a subtree of execution will never start if its root node runs into internal error (*not ends with failure*). +* Verify that a subtree of execution will never start if its root node is successfully canceled. +* Verify that cancelling a workflow job that is cancellable will consequently cancel any of its cancellable spawned jobs and thus end the whole workflow execution. +* Verify that during a workflow job run, its spawned jobs are blocked from deletion. +* Verify that notification templates can be successfully (dis)associated with a workflow job template. Later when its spawned workflow jobs finish running, verify that the correct type of notifications will be sent according to the job status. -## Testing Considerations - -## Performance Testing +## Test Notes +* Please apply non-trivial topology when testing workflow run. A non-trivial topology for a workflow job template should include: + * Multiple decision trees. + * Relatively large hight in each decision tree. + * All three types of relationships (`success`, `failure` and `always`). From ede6bef4293e01514b640c2a18a0825f31b4ea3b Mon Sep 17 00:00:00 2001 From: Aaron Tan Date: Thu, 10 Nov 2016 15:01:01 -0500 Subject: [PATCH 04/11] Typo correction --- docs/workflow.md | 37 ++++++++++++++++++++----------------- 1 file changed, 20 insertions(+), 17 deletions(-) diff --git a/docs/workflow.md b/docs/workflow.md index ea60ca6f0a..08a2d01277 100644 --- a/docs/workflow.md +++ b/docs/workflow.md @@ -1,21 +1,21 @@ ## Tower Workflow Overview Workflows are structured pipelines of multiple tower job runs. Think of workflows as jobs that have no native content, but triggers other job in specific orders to accomplish the purpose of tracking the full set of jobs that were part of a release process as a single unit. -A workflow job is composed of one or more decision trees, each node of which wraps around zero to one tower job. A workflow job run starts with creating and running jobs of the root node of each decision tree. When a parent node has its job finished running, it would pick up a subset of its child nodes according to its job status, and create new jobs for them. The selected child nodes would then run their own jobs and so forth till all nodes that needs to run finish running, which marks the end of the underlying workflow run. +A workflow job is composed of one or more decision trees, each node of which contains zero to one tower job resource (the permitted types are jobs, inventory updates and project updates). A workflow job run starts from creating and running jobs of the root node of each decision tree. When a parent node has its job resource finished running, it would pick up a subset of its child nodes according to its job status, and create new job resources for them. The selected child nodes would then run their own jobs and so forth till all nodes that needs to run finish running, which marks the end of the underlying workflow job run. ## Usage Manual ### Workflow CUID -Like other job resources, workflow jobs are created based on workflow job templates. Workflow job templates are equipped with common funtional fields including labels, schedules, notification templates, extra variables and survey specifications. It also comes with a custom `workflow_nodes` field which refers to a list of related workflow job template nodes. +Like other job resources, workflow jobs are created based on workflow job templates. Workflow job templates are equipped with common relational fields including labels, schedules, notification templates, extra variables and survey specifications. It also comes with a custom `workflow_nodes` field which refers to a list of related workflow job template nodes. -The CUID operations against a workflow job template and its corresponding workflow jobs are almost identical to those of normal job templates and related jobs. However, from RBAC perspective, CUID on workflow job templates/jobs are limited to administrators. That is, an organization administrator takes full control over all workflow job templates/jobs under the same organization, while an organization auditor is able to see workflow job templates/jobs under the same organization. On the other hand, ordinary organization members have no, and are not able to gain, permission over any workflow-related resources. +The CUID operations against a workflow job template and its corresponding workflow jobs are almost identical to those of normal job templates and related jobs. However, from RBAC perspective, CUID on workflow job templates/jobs are limited to super users. That is, an organization administrator takes full control over all workflow job templates/jobs under the same organization, while an organization auditor is able to see workflow job templates/jobs under the same organization. On the other hand, ordinary organization members have no, and are not able to gain, permission over any workflow-related resources. ### Workflow Nodes -Workflow Nodes are containers of workflow spawned jobs and function as nodes of workflow decision trees. Like that of workflow itself, the two types of workflow nodes are workflow job template nodes and workflow job nodes. +Workflow Nodes are containers of workflow spawned job resources and function as nodes of workflow decision trees. Like that of workflow itself, the two types of workflow nodes are workflow job template nodes and workflow job nodes. -Workflow job template nodes are listed and created under endpoint `/workflow_job_templates/\d/workflow_nodes/` to be associated with underlying workflow job template, or under endpoint `/workflow_job_template_nodes/` directly. The most important fields of a workflow job template node are `success_nodes`, `failure_nodes`, `always_nodes`, `unified_job_template` and `workflow_job_template`. The former three are lists of workflow job template nodes that in union forms the set of all it s child nodes, in specific, `success_nodes` are triggered when parnent node job succeeds, `failure_nodes` are triggered when parent node job fails, and `always_nodes` are triggered regardless of whether parent job fails. The later two reference the job template resource it contains and workflow job template it belongs to. +Workflow job template nodes are listed and created under endpoint `/workflow_job_templates/\d/workflow_nodes/` to be associated with underlying workflow job template, or under endpoint `/workflow_job_template_nodes/` directly. The most important fields of a workflow job template node are `success_nodes`, `failure_nodes`, `always_nodes`, `unified_job_template` and `workflow_job_template`. The former three are lists of workflow job template nodes that, in union, forms the set of all its child nodes, in specific, `success_nodes` are triggered when parnent node job succeeds, `failure_nodes` are triggered when parent node job fails, and `always_nodes` are triggered regardless of whether parent job succeeds or fails; The later two reference the job template resource it contains and workflow job template it belongs to. -Workflow job nodes are created by workflow job template nodes at the beginning of workflow job run. Other than the fields inherited from its template, workflow job nodes also come with a `job` field which references the to-be-spawned job resource. +Workflow job nodes are created by workflow job template nodes at the beginning of workflow job runs. Other than the fields inherited from its template, workflow job nodes also come with a `job` field which references the to-be-spawned job resource. ### Decision Tree Formation and Restrictions The decision tree structure of a workflow is enforced by associating workflow job template nodes via endpoints `/workflow_job_template_nodes/\d/*_nodes/`, where `*` has options `success`, `failure` and `always`. However there are restrictions that must be enforced when setting up new connections. Here are the three restrictions that will raise validation error when break: @@ -24,15 +24,15 @@ The decision tree structure of a workflow is enforced by associating workflow jo * Mutex restriction: A node cannot have all three types of child nodes. It contains either always nodes only, or any type other than always nodes. ### Workflow Run Details -A typical workflow run starts by either POSTing to endpoint `/workflow_job_templates/7/launch/`, or being triggered automatically by related schedule. The first step is the workflow job template creating workflow job, and all related workflow job template nodes creating workflow job node. Right after that, all root nodes are populated with corresponding job resources and start running. If nothing goes wrong, each decision tree will follow its own route to completion. The entire workflow finishes running when all its decision trees completes. +A typical workflow run starts by either POSTing to endpoint `/workflow_job_templates/7/launch/`, or being triggered automatically by related schedule. At the very first, the workflow job template creats workflow job, and all related workflow job template nodes create workflow job nodes. Right after that, all root nodes are populated with corresponding job resources and start running. If nothing goes wrong, each decision tree will follow its own route to completion. The entire workflow finishes running when all its decision trees complete. Job resources spawned by workflow jobs are needed by workflow to run correctly. Therefore deletion of spawned job resources is blocked while the underlying workflow job is executing. -Other than success and failure, a workflow spawned job resource can also end with status 'error' and 'canceled'. When a workflow spawned job resource errors, all branches starting from that job will stop executing while the rest keep their own paces. Canceling a workflow spawned job resource follows the same path. +Other than success and failure, a workflow spawned job resource can also end with status 'error' and 'canceled'. When a workflow spawned job resource errors, all branches starting from that job will stop executing while the rest keep their own paces. Canceling a workflow spawned job resource follows the same rules. -A workflow job itself can also be canceled altogether. In this case all its spawned job resources will be canceled and following paths stop executing. +A workflow job itself can also be canceled. In this case all its spawned job resources will be canceled if cancelable and following paths stop executing. -Like job templates, workflow job templates can associated with notification templates and notifications works exactly the same as that of job templates. One distinction is the notification message body. Workflow jobs sends notification that contains not only the status of itself, but also status of all its spawned jobs. Typical notification body looks like this: +Like job templates, workflow job templates can be associated with notification templates and notifications work exactly the same as that of job templates. One distinction is the notification message body. Workflow jobs sends notification body that contains not only the status of itself, but also status of all its spawned jobs. A typical notification body looks like this: ``` Workflow job summary: @@ -43,23 +43,26 @@ Workflow job summary: ``` ## Test Coverage +### CUID-related * Verify that CUID operations on all workflow resources are working properly. Note workflow job nodes cannot be created or deleted independently, but verifications are needed to make sure when a workflow job is deleted, all its related workflow job nodes are deleted. * Verify the RBAC property of workflow resources. In specific: - * Workflow job templates can only be accessible by super users ---- system admin, admin of the same organization and system auditor and auditor of the same organization with read permission only. + * Workflow job templates can only be accessible by superusers ---- system admin, admin of the same organization and system auditor and auditor of the same organization with read permission only. * Workflow jobs follows the permission rules of its associated workflow job template. * Workflow job template nodes rely their permission rules on the permission rules of both their associated workflow job template and unified job template. * Workflow job nodes follows the permission rules of both its associated workflow job and unified job. * Verify that workflow job template nodes can be created under, or (dis)associated with workflow job templates. -* Verify that only the permitted types of unified job templates can be associated with a workflow job template node. Currently the permitted types are **job templates, inventory sources and projects**. -* Verify that workflow job template nodes under the same workflow job template can be associated to form parent-child relationship of parent trees. In specific, one node takes another as its child node by POSTing another node's id to one of the three endpoints: `/success_nodes/`, `/failure_nodes/` and `/always_nodes/`. -* Verify that workflow job template nodes are not allowed to have invalid association. Any attempt that causes invalidity will trigger validation error. The three types of invalid associations are cycle, convergence(multiple parent) and mutex('always' XOR the rest). +* Verify that only the permitted types of job template types can be associated with a workflow job template node. Currently the permitted types are *job templates, inventory sources and projects*. +* Verify that workflow job template nodes under the same workflow job template can be associated to form parent-child relationship of decision trees. In specific, one node takes another as its child node by POSTing another node's id to one of the three endpoints: `/success_nodes/`, `/failure_nodes/` and `/always_nodes/`. +* Verify that workflow job template nodes are not allowed to have invalid association. Any attempt that causes invalidity will trigger 400-level response. The three types of invalid associations are cycle, convergence(multiple parent) and mutex('always' XOR the rest). + +### Task-related * Verify that workflow jobs can be launched by POSTing to endpoint `/workflow_job_templates/\d/launch/`. * Verify that schedules can be successfully (dis)associated with a workflow job template, and workflow jobs can be triggered by the schedule of associated workflow job template at specified time point. -* Verify that during a workflow job run, all its decision trees follow their correct paths of execution. Unwarranted behaviors include child node executing before its parent and wrong path being selected (failure nodes are executed when parent node succeeds and so on). +* Verify that during a workflow job run, all its decision trees follow their correct paths of execution. Unwarranted behaviors include child node executing before its parent and wrong path being selected (*failure nodes* are executed when parent node *succeeds* and so on). * Verify that a subtree of execution will never start if its root node runs into internal error (*not ends with failure*). * Verify that a subtree of execution will never start if its root node is successfully canceled. -* Verify that cancelling a workflow job that is cancellable will consequently cancel any of its cancellable spawned jobs and thus end the whole workflow execution. -* Verify that during a workflow job run, its spawned jobs are blocked from deletion. +* Verify that cancelling a workflow job that is cancellable will consequently cancel any of its cancellable spawned jobs and thus interrupts the whole workflow execution. +* Verify that during a workflow job run, deleting its spawned jobs are prohibited. * Verify that notification templates can be successfully (dis)associated with a workflow job template. Later when its spawned workflow jobs finish running, verify that the correct type of notifications will be sent according to the job status. ## Test Notes From cc7f2c4884ebd521e90349a1e0af75526d839a7e Mon Sep 17 00:00:00 2001 From: Aaron Tan Date: Mon, 14 Nov 2016 17:52:43 -0500 Subject: [PATCH 05/11] prompt field permutation added --- docs/workflow.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/workflow.md b/docs/workflow.md index 08a2d01277..33c92106e7 100644 --- a/docs/workflow.md +++ b/docs/workflow.md @@ -15,6 +15,8 @@ Workflow Nodes are containers of workflow spawned job resources and function as Workflow job template nodes are listed and created under endpoint `/workflow_job_templates/\d/workflow_nodes/` to be associated with underlying workflow job template, or under endpoint `/workflow_job_template_nodes/` directly. The most important fields of a workflow job template node are `success_nodes`, `failure_nodes`, `always_nodes`, `unified_job_template` and `workflow_job_template`. The former three are lists of workflow job template nodes that, in union, forms the set of all its child nodes, in specific, `success_nodes` are triggered when parnent node job succeeds, `failure_nodes` are triggered when parent node job fails, and `always_nodes` are triggered regardless of whether parent job succeeds or fails; The later two reference the job template resource it contains and workflow job template it belongs to. +Apart from the core fields, workflow job template nodes have optional fields `credential`, `inventory` and `char_prompts`. These fields will be passed on to corresponding fields of underlying jobs if those fields are set prompted at runtime. + Workflow job nodes are created by workflow job template nodes at the beginning of workflow job runs. Other than the fields inherited from its template, workflow job nodes also come with a `job` field which references the to-be-spawned job resource. ### Decision Tree Formation and Restrictions @@ -58,11 +60,13 @@ Workflow job summary: ### Task-related * Verify that workflow jobs can be launched by POSTing to endpoint `/workflow_job_templates/\d/launch/`. * Verify that schedules can be successfully (dis)associated with a workflow job template, and workflow jobs can be triggered by the schedule of associated workflow job template at specified time point. +* Verify that extra variables and surveys work for workflow job templates the same way as they work for normal job templates. * Verify that during a workflow job run, all its decision trees follow their correct paths of execution. Unwarranted behaviors include child node executing before its parent and wrong path being selected (*failure nodes* are executed when parent node *succeeds* and so on). * Verify that a subtree of execution will never start if its root node runs into internal error (*not ends with failure*). * Verify that a subtree of execution will never start if its root node is successfully canceled. * Verify that cancelling a workflow job that is cancellable will consequently cancel any of its cancellable spawned jobs and thus interrupts the whole workflow execution. * Verify that during a workflow job run, deleting its spawned jobs are prohibited. +* Verify that at the beginning of each spawned job run, its prompted fields will be populated by the wrapping workflow job node with corrected values. For example, `credential` field of workflow job node goes to `credential` field of spawned job. * Verify that notification templates can be successfully (dis)associated with a workflow job template. Later when its spawned workflow jobs finish running, verify that the correct type of notifications will be sent according to the job status. ## Test Notes From 2c9a475d4692e2bea36cb8e71893b2d21df85983 Mon Sep 17 00:00:00 2001 From: Aaron Tan Date: Tue, 15 Nov 2016 22:29:24 -0500 Subject: [PATCH 06/11] Update according to review feedback --- docs/workflow.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/workflow.md b/docs/workflow.md index 33c92106e7..787750a2ac 100644 --- a/docs/workflow.md +++ b/docs/workflow.md @@ -1,23 +1,23 @@ ## Tower Workflow Overview -Workflows are structured pipelines of multiple tower job runs. Think of workflows as jobs that have no native content, but triggers other job in specific orders to accomplish the purpose of tracking the full set of jobs that were part of a release process as a single unit. +Workflows are structured compositions of Tower job resources. The only job of a workflow is to trigger other jobs in specific orders to achieve certain goals, such as tracking the full set of jobs that were part of a release process as a single unit. -A workflow job is composed of one or more decision trees, each node of which contains zero to one tower job resource (the permitted types are jobs, inventory updates and project updates). A workflow job run starts from creating and running jobs of the root node of each decision tree. When a parent node has its job resource finished running, it would pick up a subset of its child nodes according to its job status, and create new job resources for them. The selected child nodes would then run their own jobs and so forth till all nodes that needs to run finish running, which marks the end of the underlying workflow job run. +A workflow has an associated tree-graph that is composed of multiple nodes. Each node in the tree has one associated job template (job template, inventory update, or project update) along with related resources that, if defined, will override the associated job template resources (i.e. credential, inventory, etc.) if the job template associated with the node is chosen to run. ## Usage Manual -### Workflow CUID -Like other job resources, workflow jobs are created based on workflow job templates. Workflow job templates are equipped with common relational fields including labels, schedules, notification templates, extra variables and survey specifications. It also comes with a custom `workflow_nodes` field which refers to a list of related workflow job template nodes. +### Workflow Create-Read-Update-Delete (CRUD) +Like other job resources, workflow jobs are created from workflow job templates. The API exposes common fields similar to job templates, including labels, schedules, notification templates, extra variables and survey specifications. Other than that, in the API, the related workflow graph nodes can be gotten to via the related workflow_nodes field. -The CUID operations against a workflow job template and its corresponding workflow jobs are almost identical to those of normal job templates and related jobs. However, from RBAC perspective, CUID on workflow job templates/jobs are limited to super users. That is, an organization administrator takes full control over all workflow job templates/jobs under the same organization, while an organization auditor is able to see workflow job templates/jobs under the same organization. On the other hand, ordinary organization members have no, and are not able to gain, permission over any workflow-related resources. +The CRUD operations against a workflow job template and its corresponding workflow jobs are almost identical to those of normal job templates and related jobs. However, from an RBAC perspective, CRUD on workflow job templates/jobs are limited to super users. That is, an organization administrator takes full control over all workflow job templates/jobs under the same organization, while an organization auditor is able to see workflow job templates/jobs under the same organization. On the other hand, ordinary organization members have no, and are not able to gain, permission over any workflow-related resources. ### Workflow Nodes Workflow Nodes are containers of workflow spawned job resources and function as nodes of workflow decision trees. Like that of workflow itself, the two types of workflow nodes are workflow job template nodes and workflow job nodes. -Workflow job template nodes are listed and created under endpoint `/workflow_job_templates/\d/workflow_nodes/` to be associated with underlying workflow job template, or under endpoint `/workflow_job_template_nodes/` directly. The most important fields of a workflow job template node are `success_nodes`, `failure_nodes`, `always_nodes`, `unified_job_template` and `workflow_job_template`. The former three are lists of workflow job template nodes that, in union, forms the set of all its child nodes, in specific, `success_nodes` are triggered when parnent node job succeeds, `failure_nodes` are triggered when parent node job fails, and `always_nodes` are triggered regardless of whether parent job succeeds or fails; The later two reference the job template resource it contains and workflow job template it belongs to. +Workflow job template nodes are listed and created under endpoint `/workflow_job_templates/\d/workflow_nodes/` to be associated with underlying workflow job template, or directly under endpoint `/workflow_job_template_nodes/`. The most important fields of a workflow job template node are `success_nodes`, `failure_nodes`, `always_nodes`, `unified_job_template` and `workflow_job_template`. The former three are lists of workflow job template nodes that, in union, forms the set of all its child nodes, in specific, `success_nodes` are triggered when parnent node job succeeds, `failure_nodes` are triggered when parent node job fails, and `always_nodes` are triggered regardless of whether parent job succeeds or fails; The later two reference the job template resource it contains and workflow job template it belongs to. Apart from the core fields, workflow job template nodes have optional fields `credential`, `inventory` and `char_prompts`. These fields will be passed on to corresponding fields of underlying jobs if those fields are set prompted at runtime. -Workflow job nodes are created by workflow job template nodes at the beginning of workflow job runs. Other than the fields inherited from its template, workflow job nodes also come with a `job` field which references the to-be-spawned job resource. +When a workflow job template is launched a workflow job is created. A workflow job node is create for each WFJT node and all fields from the WFJT node are copied. Note that workflow job nodes contain all fields that a workflow job template node contains plus an additional `job` field, which is a reference to the to-be-spawned job resource. ### Decision Tree Formation and Restrictions The decision tree structure of a workflow is enforced by associating workflow job template nodes via endpoints `/workflow_job_template_nodes/\d/*_nodes/`, where `*` has options `success`, `failure` and `always`. However there are restrictions that must be enforced when setting up new connections. Here are the three restrictions that will raise validation error when break: @@ -45,8 +45,8 @@ Workflow job summary: ``` ## Test Coverage -### CUID-related -* Verify that CUID operations on all workflow resources are working properly. Note workflow job nodes cannot be created or deleted independently, but verifications are needed to make sure when a workflow job is deleted, all its related workflow job nodes are deleted. +### CRUD-related +* Verify that CRUD operations on all workflow resources are working properly. Note workflow job nodes cannot be created or deleted independently, but verifications are needed to make sure when a workflow job is deleted, all its related workflow job nodes are deleted. * Verify the RBAC property of workflow resources. In specific: * Workflow job templates can only be accessible by superusers ---- system admin, admin of the same organization and system auditor and auditor of the same organization with read permission only. * Workflow jobs follows the permission rules of its associated workflow job template. From 16f1828a531da2ba09807131543f88e974b3497d Mon Sep 17 00:00:00 2001 From: Aaron Tan Date: Wed, 16 Nov 2016 16:02:57 -0500 Subject: [PATCH 07/11] Workflow copy/relaunch added --- docs/workflow.md | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/docs/workflow.md b/docs/workflow.md index 787750a2ac..bd61e02361 100644 --- a/docs/workflow.md +++ b/docs/workflow.md @@ -13,20 +13,20 @@ The CRUD operations against a workflow job template and its corresponding workfl ### Workflow Nodes Workflow Nodes are containers of workflow spawned job resources and function as nodes of workflow decision trees. Like that of workflow itself, the two types of workflow nodes are workflow job template nodes and workflow job nodes. -Workflow job template nodes are listed and created under endpoint `/workflow_job_templates/\d/workflow_nodes/` to be associated with underlying workflow job template, or directly under endpoint `/workflow_job_template_nodes/`. The most important fields of a workflow job template node are `success_nodes`, `failure_nodes`, `always_nodes`, `unified_job_template` and `workflow_job_template`. The former three are lists of workflow job template nodes that, in union, forms the set of all its child nodes, in specific, `success_nodes` are triggered when parnent node job succeeds, `failure_nodes` are triggered when parent node job fails, and `always_nodes` are triggered regardless of whether parent job succeeds or fails; The later two reference the job template resource it contains and workflow job template it belongs to. +Workflow job template nodes are listed and created under endpoint `/workflow_job_templates/\d+/workflow_nodes/` to be associated with underlying workflow job template, or directly under endpoint `/workflow_job_template_nodes/`. The most important fields of a workflow job template node are `success_nodes`, `failure_nodes`, `always_nodes`, `unified_job_template` and `workflow_job_template`. The former three are lists of workflow job template nodes that, in union, forms the set of all its child nodes, in specific, `success_nodes` are triggered when parnent node job succeeds, `failure_nodes` are triggered when parent node job fails, and `always_nodes` are triggered regardless of whether parent job succeeds or fails; The later two reference the job template resource it contains and workflow job template it belongs to. Apart from the core fields, workflow job template nodes have optional fields `credential`, `inventory` and `char_prompts`. These fields will be passed on to corresponding fields of underlying jobs if those fields are set prompted at runtime. When a workflow job template is launched a workflow job is created. A workflow job node is create for each WFJT node and all fields from the WFJT node are copied. Note that workflow job nodes contain all fields that a workflow job template node contains plus an additional `job` field, which is a reference to the to-be-spawned job resource. -### Decision Tree Formation and Restrictions -The decision tree structure of a workflow is enforced by associating workflow job template nodes via endpoints `/workflow_job_template_nodes/\d/*_nodes/`, where `*` has options `success`, `failure` and `always`. However there are restrictions that must be enforced when setting up new connections. Here are the three restrictions that will raise validation error when break: -* Cycle restriction: According to decision tree definition, no cycle is allowed. +### Tree-Graph Formation and Restrictions +The tree-graph structure of a workflow is enforced by associating workflow job template nodes via endpoints `/workflow_job_template_nodes/\d+/*_nodes/`, where `*` has options `success`, `failure` and `always`. However there are restrictions that must be enforced when setting up new connections. Here are the three restrictions that will raise validation error when break: +* Cycle restriction: According to tree definition, no cycle is allowed. * Convergent restriction: Different paths should not come into the same node, in other words, a node cannot have multiple parents. * Mutex restriction: A node cannot have all three types of child nodes. It contains either always nodes only, or any type other than always nodes. ### Workflow Run Details -A typical workflow run starts by either POSTing to endpoint `/workflow_job_templates/7/launch/`, or being triggered automatically by related schedule. At the very first, the workflow job template creats workflow job, and all related workflow job template nodes create workflow job nodes. Right after that, all root nodes are populated with corresponding job resources and start running. If nothing goes wrong, each decision tree will follow its own route to completion. The entire workflow finishes running when all its decision trees complete. +A typical workflow run starts by either POSTing to endpoint `/workflow_job_templates/\d+/launch/`, or being triggered automatically by related schedule. At the very first, the workflow job template creats workflow job, and all related workflow job template nodes create workflow job nodes. Right after that, all root nodes are populated with corresponding job resources and start running. If nothing goes wrong, each decision tree will follow its own route to completion. The entire workflow finishes running when all its decision trees complete. Job resources spawned by workflow jobs are needed by workflow to run correctly. Therefore deletion of spawned job resources is blocked while the underlying workflow job is executing. @@ -44,6 +44,13 @@ Workflow job summary: ... ``` +### Workflow Copy and Relaunch +Other than the normal way of creating workflow job templates/jobs, it is also possible to copy existing workflow job templates/jobs. The resulting new workflow job resource will be almost identical to which it copies from, with several unique fields to identify itself as well as to indicate its copy nature. + +Workflow job templates can be copied by POSTing to endpoint `/workflow_job_templates/\d+/copy/`. After copy finished, the resulting new workflow job template will have identical fields including description, extra_vars, labels, 'launch_type' and survey-related fields (survey_passwords, survey_spec and survey_enabled). More importantly, all workflow job template nodes of the original workflow job template, as well as the topology they bear, will be copied. On the other hand, schedules and notification templates of the original workflow job template will not be copied nor shared, and the name of the created workflow job template is the original name plus a special-formatted suffix to indicate its copy origin, such as **TODO**. + +Worflow jobs cannot be copied directly, instead a workflow job is implicitly copied when it needs to relaunch. Relaunching an existing workflow job is done by POSTing to endpoint `/workflow_jobs/\d+/relaunch/`. What happens next is the original workflow job is copied to create a new workflow job. The new workflow job then gets a copy of all nodes of the original as well as the topology they bear. Finally the full-fledged new workflow job is triggered to run, thus fulfilling the purpose of relaunch. + ## Test Coverage ### CRUD-related * Verify that CRUD operations on all workflow resources are working properly. Note workflow job nodes cannot be created or deleted independently, but verifications are needed to make sure when a workflow job is deleted, all its related workflow job nodes are deleted. @@ -56,6 +63,7 @@ Workflow job summary: * Verify that only the permitted types of job template types can be associated with a workflow job template node. Currently the permitted types are *job templates, inventory sources and projects*. * Verify that workflow job template nodes under the same workflow job template can be associated to form parent-child relationship of decision trees. In specific, one node takes another as its child node by POSTing another node's id to one of the three endpoints: `/success_nodes/`, `/failure_nodes/` and `/always_nodes/`. * Verify that workflow job template nodes are not allowed to have invalid association. Any attempt that causes invalidity will trigger 400-level response. The three types of invalid associations are cycle, convergence(multiple parent) and mutex('always' XOR the rest). +* Verify that a workflow job template can be successfully copied and the created workflow job template does not miss any field that should be copied or intentionally modified. ### Task-related * Verify that workflow jobs can be launched by POSTing to endpoint `/workflow_job_templates/\d/launch/`. @@ -68,6 +76,7 @@ Workflow job summary: * Verify that during a workflow job run, deleting its spawned jobs are prohibited. * Verify that at the beginning of each spawned job run, its prompted fields will be populated by the wrapping workflow job node with corrected values. For example, `credential` field of workflow job node goes to `credential` field of spawned job. * Verify that notification templates can be successfully (dis)associated with a workflow job template. Later when its spawned workflow jobs finish running, verify that the correct type of notifications will be sent according to the job status. +* Verify that a workflow job can be successfully relaunched. ## Test Notes * Please apply non-trivial topology when testing workflow run. A non-trivial topology for a workflow job template should include: From 3d9a13abfc8f15cd8ac66e0dbba442242a7c4331 Mon Sep 17 00:00:00 2001 From: Aaron Tan Date: Thu, 17 Nov 2016 16:46:15 -0500 Subject: [PATCH 08/11] Fill in the missing pieces --- docs/workflow.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/workflow.md b/docs/workflow.md index bd61e02361..fd175e55f2 100644 --- a/docs/workflow.md +++ b/docs/workflow.md @@ -15,9 +15,9 @@ Workflow Nodes are containers of workflow spawned job resources and function as Workflow job template nodes are listed and created under endpoint `/workflow_job_templates/\d+/workflow_nodes/` to be associated with underlying workflow job template, or directly under endpoint `/workflow_job_template_nodes/`. The most important fields of a workflow job template node are `success_nodes`, `failure_nodes`, `always_nodes`, `unified_job_template` and `workflow_job_template`. The former three are lists of workflow job template nodes that, in union, forms the set of all its child nodes, in specific, `success_nodes` are triggered when parnent node job succeeds, `failure_nodes` are triggered when parent node job fails, and `always_nodes` are triggered regardless of whether parent job succeeds or fails; The later two reference the job template resource it contains and workflow job template it belongs to. -Apart from the core fields, workflow job template nodes have optional fields `credential`, `inventory` and `char_prompts`. These fields will be passed on to corresponding fields of underlying jobs if those fields are set prompted at runtime. +Apart from the core fields, workflow job template nodes have optional fields `credential`, `inventory`, `job_type`, `job_tags`, `skip_tags` and `limit`. These fields will be passed on to corresponding fields of underlying jobs if those fields are set prompted at runtime. -When a workflow job template is launched a workflow job is created. A workflow job node is create for each WFJT node and all fields from the WFJT node are copied. Note that workflow job nodes contain all fields that a workflow job template node contains plus an additional `job` field, which is a reference to the to-be-spawned job resource. +When a workflow job template is launched a workflow job is created. A workflow job node is create for each WFJT node and all fields from the WFJT node are copied. Note that workflow job nodes contain all fields that a workflow job template node contains plus two additional fields: `job` and `ancestor_artifacts`. `job` is a reference to the to-be-spawned job resource, whereas `ancestor_artifacts` is a read-only JSON field that capture `artifacts` field of containing spawned job and combine it with parent node's `ancestor_artifacts`. ### Tree-Graph Formation and Restrictions The tree-graph structure of a workflow is enforced by associating workflow job template nodes via endpoints `/workflow_job_template_nodes/\d+/*_nodes/`, where `*` has options `success`, `failure` and `always`. However there are restrictions that must be enforced when setting up new connections. Here are the three restrictions that will raise validation error when break: @@ -28,6 +28,8 @@ The tree-graph structure of a workflow is enforced by associating workflow job t ### Workflow Run Details A typical workflow run starts by either POSTing to endpoint `/workflow_job_templates/\d+/launch/`, or being triggered automatically by related schedule. At the very first, the workflow job template creats workflow job, and all related workflow job template nodes create workflow job nodes. Right after that, all root nodes are populated with corresponding job resources and start running. If nothing goes wrong, each decision tree will follow its own route to completion. The entire workflow finishes running when all its decision trees complete. +As stated, workflow job templates can be created with populated `extra_vars`. These extra_vars are the runtime extra variables of *any* spawned job, meaning that they will overwrite the existing extra variables of spawned jobs if present, and add new extra variables to spawned jobs if not. Note before the extra_vars set is applied as runtime job extra variables, it might be expaneded and over-written twice: first by corresponding survey spec, then by `ancestor_artifacts` field of parent workflow job node. + Job resources spawned by workflow jobs are needed by workflow to run correctly. Therefore deletion of spawned job resources is blocked while the underlying workflow job is executing. Other than success and failure, a workflow spawned job resource can also end with status 'error' and 'canceled'. When a workflow spawned job resource errors, all branches starting from that job will stop executing while the rest keep their own paces. Canceling a workflow spawned job resource follows the same rules. @@ -45,7 +47,7 @@ Workflow job summary: ``` ### Workflow Copy and Relaunch -Other than the normal way of creating workflow job templates/jobs, it is also possible to copy existing workflow job templates/jobs. The resulting new workflow job resource will be almost identical to which it copies from, with several unique fields to identify itself as well as to indicate its copy nature. +Other than the normal way of creating workflow job templates, it is also possible to copy existing workflow job templates. The resulting new workflow job template will be identical to which it copies from, with several unique fields to identify itself as well as to indicate its copy nature. Workflow job templates can be copied by POSTing to endpoint `/workflow_job_templates/\d+/copy/`. After copy finished, the resulting new workflow job template will have identical fields including description, extra_vars, labels, 'launch_type' and survey-related fields (survey_passwords, survey_spec and survey_enabled). More importantly, all workflow job template nodes of the original workflow job template, as well as the topology they bear, will be copied. On the other hand, schedules and notification templates of the original workflow job template will not be copied nor shared, and the name of the created workflow job template is the original name plus a special-formatted suffix to indicate its copy origin, such as **TODO**. @@ -68,7 +70,9 @@ Worflow jobs cannot be copied directly, instead a workflow job is implicitly cop ### Task-related * Verify that workflow jobs can be launched by POSTing to endpoint `/workflow_job_templates/\d/launch/`. * Verify that schedules can be successfully (dis)associated with a workflow job template, and workflow jobs can be triggered by the schedule of associated workflow job template at specified time point. -* Verify that extra variables and surveys work for workflow job templates the same way as they work for normal job templates. +* Verify that extra variables work for workflow job templates as described. In specific, verify the role of workflow job extra variables as a set of global runtime variables over all its spawned jobs. +* Verify that extra variables of a workflow job node are correctly overwritten in order by its own `survey_spec` if survey is enabled and `ancestor_artifacts` field of parent node. +* Verify that when a contained job finished running with its `artifacts` field populated, the underlying workfow job node will populate its `ancestor_artifacts` field with the union of the job's `artifacts` field and the `ancestor_artifacts` field of its parent node. * Verify that during a workflow job run, all its decision trees follow their correct paths of execution. Unwarranted behaviors include child node executing before its parent and wrong path being selected (*failure nodes* are executed when parent node *succeeds* and so on). * Verify that a subtree of execution will never start if its root node runs into internal error (*not ends with failure*). * Verify that a subtree of execution will never start if its root node is successfully canceled. From 35da7215ca92fc129512a4b91a204861b64a5c9b Mon Sep 17 00:00:00 2001 From: Aaron Tan Date: Mon, 21 Nov 2016 10:24:48 -0500 Subject: [PATCH 09/11] Remove 'ancestor_artifacts' --- docs/workflow.md | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/workflow.md b/docs/workflow.md index fd175e55f2..5a7d9d250b 100644 --- a/docs/workflow.md +++ b/docs/workflow.md @@ -17,7 +17,7 @@ Workflow job template nodes are listed and created under endpoint `/workflow_job Apart from the core fields, workflow job template nodes have optional fields `credential`, `inventory`, `job_type`, `job_tags`, `skip_tags` and `limit`. These fields will be passed on to corresponding fields of underlying jobs if those fields are set prompted at runtime. -When a workflow job template is launched a workflow job is created. A workflow job node is create for each WFJT node and all fields from the WFJT node are copied. Note that workflow job nodes contain all fields that a workflow job template node contains plus two additional fields: `job` and `ancestor_artifacts`. `job` is a reference to the to-be-spawned job resource, whereas `ancestor_artifacts` is a read-only JSON field that capture `artifacts` field of containing spawned job and combine it with parent node's `ancestor_artifacts`. +When a workflow job template is launched a workflow job is created. A workflow job node is created for each WFJT node and all fields from the WFJT node are copied. Note that workflow job nodes contain all fields that a workflow job template node contains plus an additional field, `job`, which is a reference to the to-be-spawned job resource. ### Tree-Graph Formation and Restrictions The tree-graph structure of a workflow is enforced by associating workflow job template nodes via endpoints `/workflow_job_template_nodes/\d+/*_nodes/`, where `*` has options `success`, `failure` and `always`. However there are restrictions that must be enforced when setting up new connections. Here are the three restrictions that will raise validation error when break: @@ -28,7 +28,7 @@ The tree-graph structure of a workflow is enforced by associating workflow job t ### Workflow Run Details A typical workflow run starts by either POSTing to endpoint `/workflow_job_templates/\d+/launch/`, or being triggered automatically by related schedule. At the very first, the workflow job template creats workflow job, and all related workflow job template nodes create workflow job nodes. Right after that, all root nodes are populated with corresponding job resources and start running. If nothing goes wrong, each decision tree will follow its own route to completion. The entire workflow finishes running when all its decision trees complete. -As stated, workflow job templates can be created with populated `extra_vars`. These extra_vars are the runtime extra variables of *any* spawned job, meaning that they will overwrite the existing extra variables of spawned jobs if present, and add new extra variables to spawned jobs if not. Note before the extra_vars set is applied as runtime job extra variables, it might be expaneded and over-written twice: first by corresponding survey spec, then by `ancestor_artifacts` field of parent workflow job node. +As stated, workflow job templates can be created with populated `extra_vars`. These `extra_vars` are combined with the `extra_vars` of any job template launched by the workflow with higher variable precedence, meaning they will overwrite job template variables with the same name. Note before the extra_vars set is applied as runtime job extra variables, it might be expaneded and over-written by the cumulative job artifacts of ancestor nodes. The meaning of 'cumulative' here is children overwritting parent. For example, if a node has a parent node and a grandparent node, and both ancestors generate job artifacts, then the job artifacts of grandparent node is overwritten by that of parent node to form the set of cumulative job artifacts of the current node. Job resources spawned by workflow jobs are needed by workflow to run correctly. Therefore deletion of spawned job resources is blocked while the underlying workflow job is executing. @@ -47,7 +47,7 @@ Workflow job summary: ``` ### Workflow Copy and Relaunch -Other than the normal way of creating workflow job templates, it is also possible to copy existing workflow job templates. The resulting new workflow job template will be identical to which it copies from, with several unique fields to identify itself as well as to indicate its copy nature. +Other than the normal way of creating workflow job templates, it is also possible to copy existing workflow job templates. The resulting new workflow job template will be identical to which it copies from, except for `name` field which will be appended a text to indicate it's a copy. Workflow job templates can be copied by POSTing to endpoint `/workflow_job_templates/\d+/copy/`. After copy finished, the resulting new workflow job template will have identical fields including description, extra_vars, labels, 'launch_type' and survey-related fields (survey_passwords, survey_spec and survey_enabled). More importantly, all workflow job template nodes of the original workflow job template, as well as the topology they bear, will be copied. On the other hand, schedules and notification templates of the original workflow job template will not be copied nor shared, and the name of the created workflow job template is the original name plus a special-formatted suffix to indicate its copy origin, such as **TODO**. @@ -71,8 +71,7 @@ Worflow jobs cannot be copied directly, instead a workflow job is implicitly cop * Verify that workflow jobs can be launched by POSTing to endpoint `/workflow_job_templates/\d/launch/`. * Verify that schedules can be successfully (dis)associated with a workflow job template, and workflow jobs can be triggered by the schedule of associated workflow job template at specified time point. * Verify that extra variables work for workflow job templates as described. In specific, verify the role of workflow job extra variables as a set of global runtime variables over all its spawned jobs. -* Verify that extra variables of a workflow job node are correctly overwritten in order by its own `survey_spec` if survey is enabled and `ancestor_artifacts` field of parent node. -* Verify that when a contained job finished running with its `artifacts` field populated, the underlying workfow job node will populate its `ancestor_artifacts` field with the union of the job's `artifacts` field and the `ancestor_artifacts` field of its parent node. +* Verify that extra variables of a workflow job node are correctly overwritten in order by the cumulative job artifacts of ancestors, and the overwrite policy of cumulative job artifacts is correct (artifacts of parent overwrite artifacts of grandparent). * Verify that during a workflow job run, all its decision trees follow their correct paths of execution. Unwarranted behaviors include child node executing before its parent and wrong path being selected (*failure nodes* are executed when parent node *succeeds* and so on). * Verify that a subtree of execution will never start if its root node runs into internal error (*not ends with failure*). * Verify that a subtree of execution will never start if its root node is successfully canceled. From 220f44bdbce077ef581ca6e49a5fc1495154968e Mon Sep 17 00:00:00 2001 From: Aaron Tan Date: Mon, 21 Nov 2016 10:31:52 -0500 Subject: [PATCH 10/11] Give updated copy name format --- docs/workflow.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/workflow.md b/docs/workflow.md index 5a7d9d250b..1f9ee7a3f6 100644 --- a/docs/workflow.md +++ b/docs/workflow.md @@ -49,7 +49,7 @@ Workflow job summary: ### Workflow Copy and Relaunch Other than the normal way of creating workflow job templates, it is also possible to copy existing workflow job templates. The resulting new workflow job template will be identical to which it copies from, except for `name` field which will be appended a text to indicate it's a copy. -Workflow job templates can be copied by POSTing to endpoint `/workflow_job_templates/\d+/copy/`. After copy finished, the resulting new workflow job template will have identical fields including description, extra_vars, labels, 'launch_type' and survey-related fields (survey_passwords, survey_spec and survey_enabled). More importantly, all workflow job template nodes of the original workflow job template, as well as the topology they bear, will be copied. On the other hand, schedules and notification templates of the original workflow job template will not be copied nor shared, and the name of the created workflow job template is the original name plus a special-formatted suffix to indicate its copy origin, such as **TODO**. +Workflow job templates can be copied by POSTing to endpoint `/workflow_job_templates/\d+/copy/`. After copy finished, the resulting new workflow job template will have identical fields including description, extra_vars, labels, 'launch_type' and survey-related fields (survey_passwords, survey_spec and survey_enabled). More importantly, all workflow job template nodes of the original workflow job template, as well as the topology they bear, will be copied. On the other hand, schedules and notification templates of the original workflow job template will not be copied nor shared, and the name of the created workflow job template is the original name plus a special-formatted suffix to indicate its copy origin as well as the copy time, such as 'copy_from_name@10:30:00 am'. Worflow jobs cannot be copied directly, instead a workflow job is implicitly copied when it needs to relaunch. Relaunching an existing workflow job is done by POSTing to endpoint `/workflow_jobs/\d+/relaunch/`. What happens next is the original workflow job is copied to create a new workflow job. The new workflow job then gets a copy of all nodes of the original as well as the topology they bear. Finally the full-fledged new workflow job is triggered to run, thus fulfilling the purpose of relaunch. From 2995ea310fffcedafb9eaf5b7cb524f064f1882b Mon Sep 17 00:00:00 2001 From: Aaron Tan Date: Mon, 21 Nov 2016 15:03:21 -0500 Subject: [PATCH 11/11] Modify wfjt copy behavior --- docs/workflow.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/workflow.md b/docs/workflow.md index 1f9ee7a3f6..fcabb9891f 100644 --- a/docs/workflow.md +++ b/docs/workflow.md @@ -49,7 +49,7 @@ Workflow job summary: ### Workflow Copy and Relaunch Other than the normal way of creating workflow job templates, it is also possible to copy existing workflow job templates. The resulting new workflow job template will be identical to which it copies from, except for `name` field which will be appended a text to indicate it's a copy. -Workflow job templates can be copied by POSTing to endpoint `/workflow_job_templates/\d+/copy/`. After copy finished, the resulting new workflow job template will have identical fields including description, extra_vars, labels, 'launch_type' and survey-related fields (survey_passwords, survey_spec and survey_enabled). More importantly, all workflow job template nodes of the original workflow job template, as well as the topology they bear, will be copied. On the other hand, schedules and notification templates of the original workflow job template will not be copied nor shared, and the name of the created workflow job template is the original name plus a special-formatted suffix to indicate its copy origin as well as the copy time, such as 'copy_from_name@10:30:00 am'. +Workflow job templates can be copied by POSTing to endpoint `/workflow_job_templates/\d+/copy/`. After copy finished, the resulting new workflow job template will have identical fields including description, extra_vars, labels, 'launch_type' and survey-related fields (survey_passwords, survey_spec and survey_enabled). More importantly, workflow job template node of the original workflow job template, as well as the topology they bear, will be copied. Note there are RBAC restrictions on determining which workflow job template node is copied. In specific, a workflow job template is allowed to be copied if the user has at least read permission on all related resources like credential and job template. On the other hand, schedules and notification templates of the original workflow job template will not be copied nor shared, and the name of the created workflow job template is the original name plus a special-formatted suffix to indicate its copy origin as well as the copy time, such as 'copy_from_name@10:30:00 am'. Worflow jobs cannot be copied directly, instead a workflow job is implicitly copied when it needs to relaunch. Relaunching an existing workflow job is done by POSTing to endpoint `/workflow_jobs/\d+/relaunch/`. What happens next is the original workflow job is copied to create a new workflow job. The new workflow job then gets a copy of all nodes of the original as well as the topology they bear. Finally the full-fledged new workflow job is triggered to run, thus fulfilling the purpose of relaunch. @@ -66,6 +66,7 @@ Worflow jobs cannot be copied directly, instead a workflow job is implicitly cop * Verify that workflow job template nodes under the same workflow job template can be associated to form parent-child relationship of decision trees. In specific, one node takes another as its child node by POSTing another node's id to one of the three endpoints: `/success_nodes/`, `/failure_nodes/` and `/always_nodes/`. * Verify that workflow job template nodes are not allowed to have invalid association. Any attempt that causes invalidity will trigger 400-level response. The three types of invalid associations are cycle, convergence(multiple parent) and mutex('always' XOR the rest). * Verify that a workflow job template can be successfully copied and the created workflow job template does not miss any field that should be copied or intentionally modified. +* Verify that if a user has no access to any of the related resources of a workflow job template node, that node will not be copied and will have `null` as placeholder. ### Task-related * Verify that workflow jobs can be launched by POSTing to endpoint `/workflow_job_templates/\d/launch/`.