Commit Graph
100 Commits
Author SHA1 Message Date
Seth Foster ce5270434c added awx collection support for bulk api
return more context for bulk host create

now return list of minimal info about host objects

[
    {
        "name": "lakjdsafoiaweirnladlk",
        "enabled": true,
        "instance_id": "",
        "description": "",
        "variables": "",
        "id": 4593,
        "url": "/api/v2/hosts/4593/",
        "inventory": "/api/v2/inventories/1/"
    }
]

Updated tests, but needed to work around some weird behavior with
sqlite. Apparently it behaves differently around assigning ID's to the
result of bulk_create and that is messed up my use of `reverse` to look
up the url of the hosts
2023-03-08 12:58:12 -05:00
Seth Foster 34834252ff awxkit cli support
fixes for awx cli
2023-03-08 12:58:12 -05:00
Seth Foster b87dd6dc56 tag awx-ee latest with awx release 2023-01-11 17:21:51 -05:00
Seth Foster 6492c03965 Fix console color logs 2023-01-05 12:55:20 -05:00
Seth Foster 3543644e0e bump receptorctl version to 1.3.0 2022-12-21 13:36:11 -05:00
Seth Foster 36c0d07b30 Result_traceback should not include job stdout
If a job fails, we do receptor work results and put that output
into result_traceback.

We should only do this if
1. Receptor unit has failed
2. Runner callback processed 0 events

Otherwise we risk putting too much data into this field.
2022-12-21 13:05:44 -05:00
Seth Foster 1d87e6e04c Update clustering.md to be more current 2022-12-14 22:36:29 -05:00
916b5642d2 Update task manager docs
- DependencyManager and WorkflowManager
- bulk reschedule
- global task manager timeout
- blocking logic

Co-authored-by: Elijah DeLee <kdelee@redhat.com>
Co-authored-by: John R Barker <john@johnrbarker.com>
2022-12-12 11:56:40 -05:00
Seth Foster 529a936d0a Make installing minikube optional in dev setup 2022-10-31 13:52:48 -04:00
Seth Foster de41601f27 make job lifecycle Cyan again 2022-10-24 13:50:42 -04:00
Seth Foster 6d192927ae Make nginx conf consistent with settings.STATIC_ROOT 2022-10-21 23:10:06 -04:00
Seth Foster c53228daf5 Set initial value node_type and node_state 2022-09-23 09:46:16 -04:00
Seth Foster 5b7a359c91 Add doc for adding execution node 2022-09-23 09:46:16 -04:00
Seth Foster 301807466d Only get receptor.conf lock in k8s environment
- Writing to receptor.conf only takes place in K8S, so only get a
lock if IS_K8S is true
2022-09-23 09:46:15 -04:00
Seth Foster e0c9013d9c Prevent altering certain fields on Instance
- Prevents changing hostname, listener_port, or node_type for instances
that already exist
- API default node_type is execution
- API default node_state is installed
2022-09-23 09:46:15 -04:00
Seth Foster 1fde9c4f0c add firewall rules to control node 2022-09-23 09:46:14 -04:00
Seth Foster dfe6ce1ba8 remove tests that assume health check runs in view 2022-09-23 09:46:14 -04:00
Seth Foster eaa4f2483f Run instance health check in task container
awx-web container does not have access to receptor socket, and the
execution node health check requires receptorctl.

This change runs the health check asynchronously in the task container.
2022-09-23 09:46:14 -04:00
Seth Foster 55d295c2a6 Add metric to measure task manager transaction, including on_commit calls 2022-08-15 12:44:29 -04:00
Seth Foster e6f8852b05 Cache task_impact
task_impact is now a field on the database
It is calculated and set during create_unified_job

set task_impact on .save for adhoc commands
2022-08-05 14:33:47 -04:00
Seth Foster 957b2b7188 Cache preferred instance groups
When creating unified job, stash the list of pk values from the
instance groups returned from preferred_instance_groups so that the
task management system does not need to call out to this method
repeatedly.

.preferred_instance_groups_cache is the new field
2022-08-05 14:33:28 -04:00
Seth Foster 0a47d05d26 split schedule_task_manager into 3
each call to schedule_task_manager becomes one of

ScheduleTaskManager
ScheduleDependencyManager
ScheduleWorkflowManager
2022-08-05 14:33:25 -04:00
Seth Foster b3eb9e0193 pid kill each of the 3 task managers on timeout 2022-08-05 14:33:25 -04:00
Seth Foster ff118f2177 Manage pending workflow jobs in Workflow Manager
get_tasks uses UnifiedJob
Additionally, make local overrides run after development settings
2022-08-05 14:31:48 -04:00
Seth Foster 431b9370df Split TaskManager into
- DependencyManager spawns dependencies if necessary
- WorkflowManager processes running workflows to see if a new job is
  ready to spawn
- TaskManager starts tasks if unblocked and has execution capacity
2022-08-05 14:29:02 -04:00
Seth Foster 0522233892 remove update_on_project_update from InventorySource 2022-06-24 15:27:08 -04:00
Seth Foster c92619a2dc Subsystem metrics reset_values should remove all redis keys 2022-06-16 16:54:37 -04:00
Seth Foster 987e5a084d use haproxy 2.3 with maxconn set to avoid startup failures 2022-06-14 13:09:40 -04:00
Seth Foster 2f82b75748 Add subsystem metrics for task manager 2022-06-14 11:00:11 -04:00
Seth Foster b7057fdc3e Add docs for subsystem metrics 2022-06-13 11:49:56 -04:00
Seth Foster 550d9d5e42 detect if job events are tree-like and collapsable in the UI 2022-06-09 14:25:39 -04:00
Seth Foster 9364c8e562 typo 2022-05-31 17:18:45 -04:00
Seth Foster 5831949ebf maxconn 2022-05-31 17:16:27 -04:00
Seth Foster 7fe98a670f haproxy 2022-05-31 17:12:19 -04:00
Seth Foster 6f68f3cba6 Add make prometheus and make grafana commands to dev environment 2022-05-31 17:07:15 -04:00
Seth Foster eba4a3f1c2 in case we fail a job in task manager, we need to add the project update to the inventoryupdate.source_project field 2022-05-12 15:21:17 -04:00
Seth Foster 0ae9fe3624 if dependency fails, fail job in task manager 2022-05-12 14:00:13 -04:00
Seth Foster 1b662fcca5 SCM inv source trigger project update
- scm based inventory sources should launch project updates prior to
running inventory updates for that source.
- fixes scenario where a job is based on projectA, but the inventory
source is based on projectB. Running the job will likely trigger a
sync for projectA, but not projectB.

comments
2022-05-12 14:00:12 -04:00
Seth Foster 65dfc424bc Improve help text for import and export 2022-05-10 13:18:40 -04:00
Seth Foster 1462af61b0 awxkit log which resource failed to import 2022-04-11 17:03:13 -04:00
Seth Foster cfd6df7a3b Add JobJobEventsChildrenSummary endpoint
- returns a special view to output the total number of children (and
grandchildren) events for all parents events for a job
value is the number of total children of that event
- intended to be consumed by the UI, as an efficient way to get the
number of children for a particular event
- see api/templates/api/job_job_events_children_summary.md for more info
2022-04-04 14:25:18 -04:00
Seth Foster 24152555c5 Handle error for create_partition
Occasionally the create_partition will error with,
relation "main_projectupdateevent_20220323_19" already exists

This change wraps the db command into a try except block with its
own transaction
2022-03-28 16:37:50 -04:00
Seth Foster acebff7be1 Fix sync-only operation in async context 2022-03-21 14:37:10 -04:00
Seth Foster de1df8bf28 load job meta vars after JT extra vars 2022-03-02 14:42:47 -05:00
Seth Foster 6db7cea148 variable name changes 2022-02-10 10:57:00 -05:00
Seth Foster 3993aa9524 Add metric for number of events emitted over websocket broadcast 2022-02-09 21:57:01 -05:00
Seth Foster 1cd30ceb31 remove any named pipes before unzipping artifacts 2022-02-02 15:54:31 -05:00
Seth Foster b3c20ee0ae Install ps in dev image 2022-01-26 18:12:52 -05:00
Seth Foster 956638e564 Revert "Remove unnecessary DEBUG logger level settings (#11441)"
This reverts commit 8126f734e3.
2022-01-10 11:46:19 -05:00
Seth Foster e04efad3c0 tools_receptor_1 should use whatever awx_devel tag that tools_awx_1 is using 2021-11-10 08:50:11 +08:00
Seth Foster d346dbb8ba Fix conditional to check that minikube should start 2021-07-19 11:05:05 -04:00
Seth Foster a9013c43fa fix pylint 2.9 errors in awx collection 2021-06-30 14:28:35 -04:00
Seth Foster 5da02690c1 Fix copy_from warning message
- warn() does not take keyword 'msg'
- add testing around the copy_from parameter for project creation
2021-06-26 01:28:02 -04:00
Seth Foster 7bc7c9c4bd haproxy specify user 2021-06-24 13:54:24 -04:00
Seth Foster bb3fc3caa8 controller_token, alias tower_token 2021-06-22 10:49:39 -04:00
Seth Foster 75a27c38c2 Add a periodic task to reap unreleased receptor work units
- Add work_unit_id field to UnifiedJob
2021-06-22 10:49:31 -04:00
Seth Foster 61846e88ca Add validator for ee image field name
awxkit default ee image name is now a fixed valid (but bogus) name, rather than random unicode
2021-06-15 20:06:49 -04:00
Seth Foster 9eda6359f0 False to false to fix linter 2021-06-08 14:33:30 -04:00
Seth Foster 1deaf55ba4 syntax error in runtime.yml 2021-06-08 14:33:30 -04:00
Seth Foster 44c50bbbf7 add redirect for lookup and inventory plugins 2021-06-08 14:33:29 -04:00
Seth Foster c9e7747f2d black ignore awx_collection 2021-06-08 14:33:29 -04:00
Seth Foster 199b4b6b47 fixed up jinja2 templating and documentation 2021-06-08 14:33:29 -04:00
Seth Foster f06485feca fqcn redirects, and raise from sanity test fix 2021-06-08 14:33:29 -04:00
Seth Foster 4bd910493a sanity tests 2021-06-08 14:33:28 -04:00
Seth Foster cd100fd770 fix super and kwargs 2021-06-08 14:33:28 -04:00
Seth Foster 157adb828e various sanity fixes 2021-06-08 14:33:28 -04:00
Seth Foster b26e33ca34 remove eula_accepted 2021-06-08 14:33:28 -04:00
Seth Foster ee8c1638c5 remove deprecated modules 2021-06-08 14:33:26 -04:00
Seth Foster 4add72b9d2 actually make use of option aliases 2021-06-08 14:33:25 -04:00
Seth Foster 54dd24b96b auth_plugin env alternative 2021-06-08 14:33:25 -04:00
Seth Foster 7d06fc74dd tower_username to controller_username, etc 2021-06-08 14:33:25 -04:00
Seth Foster 1a2e56c785 restore tower_legacy.py, update examples to include CONTROLLER_HOST 2021-06-08 14:33:25 -04:00
Seth Foster a7b29f6112 controller_inventory.yml in addition to tower_inventory.yml 2021-06-08 14:33:24 -04:00
Seth Foster 39b26c8f0e Added env variable CONTROLLER_HOST, et al. Add awx.awx namespace to the runtime.yml redirects, and added templating task to change to ansible.controller 2021-06-08 14:33:24 -04:00
Seth Foster 1ade9b3a7d various references to tower 2021-06-08 14:33:24 -04:00
Seth Foster 82c5803e59 mostly includes renaming non-syntax references to tower 2021-06-08 14:33:24 -04:00
Seth Foster 9f4172ce7b upstream 2021-06-08 14:33:23 -04:00
Seth Foster ef56571772 controller_meta, templating 2021-06-08 14:33:23 -04:00
Seth Foster 6911a59f39 fix unit tests 2021-06-08 14:33:23 -04:00
Seth Foster 7a63785255 more rename, mostly in test 2021-06-08 14:33:23 -04:00
Seth Foster a695274cb6 regex replace tower_*: to '' 2021-06-08 14:33:22 -04:00
Seth Foster 44fed1d7c1 pycharm refactor rename files and class, linux rename tower_ controller_ 2021-06-08 14:33:22 -04:00
Seth Foster 8ab81216f3 unused import 2021-04-20 11:16:32 -04:00
Seth Foster 1e3cfdc986 just remove dangling images 2021-04-20 11:16:32 -04:00
Seth Foster c64ec6bbf8 check if is_k8s 2021-04-20 11:16:32 -04:00
Seth Foster 4566e7a2a6 check subprocess returncode 2021-04-20 11:16:32 -04:00
Seth Foster 3912f2b57c remove migration file 2021-04-20 11:16:32 -04:00
Seth Foster fa61ec6b3c Remove system job, replace with scheduled task 2021-04-20 11:16:31 -04:00
Seth Foster 33567f8729 delete_prefix local 2021-04-20 11:16:31 -04:00
Seth Foster 1c888ca58b cleanup stdout 2021-04-20 11:16:31 -04:00
Seth Foster f98b92073d Add cleanup_images system job template
- Removes podman images on the system that are not assigned to an
execution environment
2021-04-20 11:16:31 -04:00
Seth Foster 7583525366 Return distinct labels for non-admin Users 2021-04-20 10:59:38 -04:00
Seth Foster 1e4b44e54f Prevent ee from being assigned to a new organization
- ee organization can be changed to null (less restrictive)
- if organization is null, cannot be assigned to org (more restrictive)
- if org is assigned, it cannot be set to a different org
2021-04-12 14:15:21 -04:00
Seth Foster 89e28d6d4a fix awx_collections tests by comparing Decimal to float, instead of comparing strings 2021-04-08 13:38:35 -04:00
Seth Foster f05ffa521a the returned self.elapsed should be decimal, instead of float, to match django model field type 2021-04-08 11:36:40 -04:00
Seth Foster 09c176847d string 2021-04-07 17:30:46 -04:00
Seth Foster 8c73a51730 Fix elapsed time on job showing incorrect value
- job elapsed time showed 0.0, during and after the job run
2021-04-07 15:54:43 -04:00
Seth Foster 0c569c67fd Add subsystem metrics
- Adds a Metrics() class that can track data such as number of
events the callback receiver inserted into database
- Exposes this metric data at the api/v2/metrics/ endpoint.
This data is prometheus-friendly
- Metric data is stored in memory, then periodically saved to Redis.
- Metric data is periodically broadcast to other nodes in the cluster,
so that each node has a copy of the most recent metric data collected.
2021-03-25 15:23:52 -04:00
Seth Foster 3f342feadd Fix api/v2/metrics data displaying incorrect value
- Use a locally defined prometheus registry instead of global registry
2021-03-25 14:22:17 -04:00
Seth Foster f8641bfa5e fix port conflict in dev cluster. Output only one haproxy def 2021-03-22 10:45:50 -04:00