Commit Graph
100 Commits
Author SHA1 Message Date
Seth Foster 127a0cff23 Set ip_address to empty string
ip_address cannot be null, so set to
empty instead of None

Signed-off-by: Seth Foster <fosterbseth@gmail.com>
2023-10-05 22:53:16 -04:00
Seth Foster eb33973fa3 Use receptor collection 2.0.0 2023-08-29 13:06:54 -04:00
Seth Foster 40be2e7b6e Use receptor-collection devel 2023-08-29 13:06:54 -04:00
Seth Foster 0a87bf1b5e Apply JS formatting from npm prettier 2023-08-29 13:06:54 -04:00
Seth Foster fa0e0b2576 Removed unused variable in test_instance_peers 2023-08-29 13:06:54 -04:00
Seth Foster 1d3b2f57ce No longer assert on receptor_host_identifier
receptor_host_identifier can be left out
of group_vars and will default to the
'ansible_host' variable
2023-08-29 13:06:54 -04:00
Seth Foster 0577e1ee79 Setup receptor after podman
Might help to install receptor last,
that way when nodes are first connected to the mesh
they already have podman installed and can potentially
run jobs. Otherwise it might be possible for controller
to launch jobs against nodes that aren't fully set up.
2023-08-29 13:06:54 -04:00
Seth Foster 470ecc4a4f Use itertools product instead of nested loop
Make test case cleaner by using itertools product
instead of the triple nested loop

Replace triple single quotes with triple
double quotes
2023-08-29 13:06:54 -04:00
Seth Foster 965127637b Make ip_address read only
Setting a different value for ip_address
and hostname does not work with the current
way we create receptor certs.
2023-08-29 13:06:54 -04:00
Seth Foster eba130cf41 Change username to <username> in inventory 2023-08-29 13:06:54 -04:00
Seth Foster 441336301e Ensure ip_address is empty string 2023-08-29 13:06:54 -04:00
Seth Foster 2a0be898e6 Fix detecting if peers changed in serializer
Add a check_peers_changed() utility method
to determine if peers in attrs matches
the current instance peers.

Other changes:
- Set ip_address default to "", and do not
allow null.
2023-08-29 13:06:54 -04:00
Seth Foster c47acc5988 Change PeersSerializer to SlugRelatedField
Get rid of PeersSerializer and just use SlugRelatedField,
which should be more a straightforward approach.

Other changes:
- cleanup code related to the already-removed api/v2/peers
endpoint
- add "hybrid" node type into more instance_peers test cases
2023-08-29 13:06:54 -04:00
Seth Foster 70ba32b5b2 Do not install ansible-runner or podman on hop nodes 2023-08-29 13:06:54 -04:00
Seth Foster 81e06dace2 Add listener_port to provision_instance
API changes
- cannot change peers or enable
peers_from_control_nodes on VM deployments
- allow setting ip_address
- use ip_address over hostname in the generated
group_vars/all.yml
- Drop api/v2/peers endpoint

DB changes
- add ip_address unique constraint, but ignore "" entries

Other changes
- provision_instance should take listener_port option

Tests
- test that new controls doesn't disturb other peers
relationships
- test ip_address over hostname
2023-08-29 13:06:54 -04:00
Seth Foster 3e8202590c Remove Disconnected link state
Dynamically flipping from Established
to Disconnected is not the intended
usage of InstanceLink State.

- Link state starts in Adding and becomes
Established once any control node first sees the link
is in the status KnownConnectionCosts
2023-08-29 13:06:54 -04:00
Seth Foster ad96a72ebe Remove duplicate install bundle on InstanceDetail 2023-08-29 13:06:54 -04:00
Seth Foster eb0058268b Revert "Remove duplicate install bundle on InstanceDetail"
This reverts commit cf5ccf53f4322b49b1009ca13e4f025c30529b30.
2023-08-29 13:06:54 -04:00
Seth Foster 2bf6512a8e Do not change link state if Removing
inspect_established_receptor_connections should
not change link state is current state is Removing.

Other changes:
- rename inspect_execution_nodes to inspect_execution_and_hop_nodes
- Default link state is Adding
- Set min listener_port value to 1024
- inspect_established_receptor_connections now
runs as part of cluster_node_heartbeat task
2023-08-29 13:06:54 -04:00
Seth Foster 855f61a04e Bump migration number 186 to 187 2023-08-29 13:06:54 -04:00
Seth Foster 532e71ff45 Remove extra newlines in install bundle all.yml 2023-08-29 13:06:54 -04:00
Seth Foster b9ea114cac Remove duplicate install bundle on InstanceDetail 2023-08-29 13:06:54 -04:00
Seth Foster 3bd25c682e Allow setting ip_address for execution nodes 2023-08-29 13:06:54 -04:00
Seth Foster 7169c75b1a receptor_python_packages renamed 2023-08-29 13:06:54 -04:00
Seth Foster ed2a59c1a3 receptor python packages 2023-08-29 13:06:54 -04:00
Seth Foster d15405eafe Add peers_from for reverse peers M2M
use devel receptor-collection
2023-08-29 13:06:54 -04:00
Seth Foster 2a51f23b7d Add functional API tests
add tests for calling write_receptor_config

add write_receptor_config test

Do not set default listener_port on control node
2023-08-29 13:06:54 -04:00
Seth Foster 0c41abea0e Make peers field optional 2023-08-29 13:06:54 -04:00
Seth Foster 6834568c5d Add receptor host identifier to group_vars
Add disconnected link state topology
2023-08-29 13:06:54 -04:00
Seth Foster d8abd4912b Add support in hop nodes in API 2023-08-29 13:06:54 -04:00
Seth Foster 66c3acf777 Fix content security policy 2023-05-11 10:42:23 -04:00
Seth Foster db2253601d Allow TLS 1.2 for Receptor connections
- Required for FIPS environment where TLS 1.3 is
not supported
- TLS 1.3 can still be used if the nodes
both agree to use during handshake.
2023-03-27 11:07:30 -04:00
Seth Foster 6a3282a689 docker-compose wait for PG to be ready
- periodically ping postres on port 5432 and only start
migrations if successful.
- prevents crash loop when attempting migrations before
postgres is ready.
2023-03-10 16:13:19 -05:00
Seth Foster 160508c907 TestAutoScaling wait for process to stop 2023-03-10 15:51:28 -05:00
Seth Foster ce5270434c added awx collection support for bulk api
return more context for bulk host create

now return list of minimal info about host objects

[
    {
        "name": "lakjdsafoiaweirnladlk",
        "enabled": true,
        "instance_id": "",
        "description": "",
        "variables": "",
        "id": 4593,
        "url": "/api/v2/hosts/4593/",
        "inventory": "/api/v2/inventories/1/"
    }
]

Updated tests, but needed to work around some weird behavior with
sqlite. Apparently it behaves differently around assigning ID's to the
result of bulk_create and that is messed up my use of `reverse` to look
up the url of the hosts
2023-03-08 12:58:12 -05:00
Seth Foster 34834252ff awxkit cli support
fixes for awx cli
2023-03-08 12:58:12 -05:00
Seth Foster ab6511a833 fix ui lint 2023-03-07 17:50:09 -05:00
Seth Foster a5f9506f49 spelling
add allow_override to source_project
2023-03-07 17:50:06 -05:00
Seth Foster a952ab0a75 Add scm_branch to inventory source and inventory update
add scm_branch as optional field awxkit
2023-03-07 17:49:57 -05:00
Seth Foster b87dd6dc56 tag awx-ee latest with awx release 2023-01-11 17:21:51 -05:00
Seth Foster 6492c03965 Fix console color logs 2023-01-05 12:55:20 -05:00
Seth Foster 3543644e0e bump receptorctl version to 1.3.0 2022-12-21 13:36:11 -05:00
Seth Foster 36c0d07b30 Result_traceback should not include job stdout
If a job fails, we do receptor work results and put that output
into result_traceback.

We should only do this if
1. Receptor unit has failed
2. Runner callback processed 0 events

Otherwise we risk putting too much data into this field.
2022-12-21 13:05:44 -05:00
Seth Foster 1d87e6e04c Update clustering.md to be more current 2022-12-14 22:36:29 -05:00
916b5642d2 Update task manager docs
- DependencyManager and WorkflowManager
- bulk reschedule
- global task manager timeout
- blocking logic

Co-authored-by: Elijah DeLee <kdelee@redhat.com>
Co-authored-by: John R Barker <john@johnrbarker.com>
2022-12-12 11:56:40 -05:00
Seth Foster 529a936d0a Make installing minikube optional in dev setup 2022-10-31 13:52:48 -04:00
Seth Foster de41601f27 make job lifecycle Cyan again 2022-10-24 13:50:42 -04:00
Seth Foster 6d192927ae Make nginx conf consistent with settings.STATIC_ROOT 2022-10-21 23:10:06 -04:00
Seth Foster c53228daf5 Set initial value node_type and node_state 2022-09-23 09:46:16 -04:00
Seth Foster 5b7a359c91 Add doc for adding execution node 2022-09-23 09:46:16 -04:00
Seth Foster 301807466d Only get receptor.conf lock in k8s environment
- Writing to receptor.conf only takes place in K8S, so only get a
lock if IS_K8S is true
2022-09-23 09:46:15 -04:00
Seth Foster e0c9013d9c Prevent altering certain fields on Instance
- Prevents changing hostname, listener_port, or node_type for instances
that already exist
- API default node_type is execution
- API default node_state is installed
2022-09-23 09:46:15 -04:00
Seth Foster 1fde9c4f0c add firewall rules to control node 2022-09-23 09:46:14 -04:00
Seth Foster dfe6ce1ba8 remove tests that assume health check runs in view 2022-09-23 09:46:14 -04:00
Seth Foster eaa4f2483f Run instance health check in task container
awx-web container does not have access to receptor socket, and the
execution node health check requires receptorctl.

This change runs the health check asynchronously in the task container.
2022-09-23 09:46:14 -04:00
Seth Foster 55d295c2a6 Add metric to measure task manager transaction, including on_commit calls 2022-08-15 12:44:29 -04:00
Seth Foster e6f8852b05 Cache task_impact
task_impact is now a field on the database
It is calculated and set during create_unified_job

set task_impact on .save for adhoc commands
2022-08-05 14:33:47 -04:00
Seth Foster 957b2b7188 Cache preferred instance groups
When creating unified job, stash the list of pk values from the
instance groups returned from preferred_instance_groups so that the
task management system does not need to call out to this method
repeatedly.

.preferred_instance_groups_cache is the new field
2022-08-05 14:33:28 -04:00
Seth Foster 0a47d05d26 split schedule_task_manager into 3
each call to schedule_task_manager becomes one of

ScheduleTaskManager
ScheduleDependencyManager
ScheduleWorkflowManager
2022-08-05 14:33:25 -04:00
Seth Foster b3eb9e0193 pid kill each of the 3 task managers on timeout 2022-08-05 14:33:25 -04:00
Seth Foster ff118f2177 Manage pending workflow jobs in Workflow Manager
get_tasks uses UnifiedJob
Additionally, make local overrides run after development settings
2022-08-05 14:31:48 -04:00
Seth Foster 431b9370df Split TaskManager into
- DependencyManager spawns dependencies if necessary
- WorkflowManager processes running workflows to see if a new job is
  ready to spawn
- TaskManager starts tasks if unblocked and has execution capacity
2022-08-05 14:29:02 -04:00
Seth Foster 0522233892 remove update_on_project_update from InventorySource 2022-06-24 15:27:08 -04:00
Seth Foster c92619a2dc Subsystem metrics reset_values should remove all redis keys 2022-06-16 16:54:37 -04:00
Seth Foster 987e5a084d use haproxy 2.3 with maxconn set to avoid startup failures 2022-06-14 13:09:40 -04:00
Seth Foster 2f82b75748 Add subsystem metrics for task manager 2022-06-14 11:00:11 -04:00
Seth Foster b7057fdc3e Add docs for subsystem metrics 2022-06-13 11:49:56 -04:00
Seth Foster 550d9d5e42 detect if job events are tree-like and collapsable in the UI 2022-06-09 14:25:39 -04:00
Seth Foster 9364c8e562 typo 2022-05-31 17:18:45 -04:00
Seth Foster 5831949ebf maxconn 2022-05-31 17:16:27 -04:00
Seth Foster 7fe98a670f haproxy 2022-05-31 17:12:19 -04:00
Seth Foster 6f68f3cba6 Add make prometheus and make grafana commands to dev environment 2022-05-31 17:07:15 -04:00
Seth Foster eba4a3f1c2 in case we fail a job in task manager, we need to add the project update to the inventoryupdate.source_project field 2022-05-12 15:21:17 -04:00
Seth Foster 0ae9fe3624 if dependency fails, fail job in task manager 2022-05-12 14:00:13 -04:00
Seth Foster 1b662fcca5 SCM inv source trigger project update
- scm based inventory sources should launch project updates prior to
running inventory updates for that source.
- fixes scenario where a job is based on projectA, but the inventory
source is based on projectB. Running the job will likely trigger a
sync for projectA, but not projectB.

comments
2022-05-12 14:00:12 -04:00
Seth Foster 65dfc424bc Improve help text for import and export 2022-05-10 13:18:40 -04:00
Seth Foster 1462af61b0 awxkit log which resource failed to import 2022-04-11 17:03:13 -04:00
Seth Foster cfd6df7a3b Add JobJobEventsChildrenSummary endpoint
- returns a special view to output the total number of children (and
grandchildren) events for all parents events for a job
value is the number of total children of that event
- intended to be consumed by the UI, as an efficient way to get the
number of children for a particular event
- see api/templates/api/job_job_events_children_summary.md for more info
2022-04-04 14:25:18 -04:00
Seth Foster 24152555c5 Handle error for create_partition
Occasionally the create_partition will error with,
relation "main_projectupdateevent_20220323_19" already exists

This change wraps the db command into a try except block with its
own transaction
2022-03-28 16:37:50 -04:00
Seth Foster acebff7be1 Fix sync-only operation in async context 2022-03-21 14:37:10 -04:00
Seth Foster de1df8bf28 load job meta vars after JT extra vars 2022-03-02 14:42:47 -05:00
Seth Foster 6db7cea148 variable name changes 2022-02-10 10:57:00 -05:00
Seth Foster 3993aa9524 Add metric for number of events emitted over websocket broadcast 2022-02-09 21:57:01 -05:00
Seth Foster 1cd30ceb31 remove any named pipes before unzipping artifacts 2022-02-02 15:54:31 -05:00
Seth Foster b3c20ee0ae Install ps in dev image 2022-01-26 18:12:52 -05:00
Seth Foster 956638e564 Revert "Remove unnecessary DEBUG logger level settings (#11441)"
This reverts commit 8126f734e3.
2022-01-10 11:46:19 -05:00
Seth Foster e04efad3c0 tools_receptor_1 should use whatever awx_devel tag that tools_awx_1 is using 2021-11-10 08:50:11 +08:00
Seth Foster d346dbb8ba Fix conditional to check that minikube should start 2021-07-19 11:05:05 -04:00
Seth Foster a9013c43fa fix pylint 2.9 errors in awx collection 2021-06-30 14:28:35 -04:00
Seth Foster 5da02690c1 Fix copy_from warning message
- warn() does not take keyword 'msg'
- add testing around the copy_from parameter for project creation
2021-06-26 01:28:02 -04:00
Seth Foster 7bc7c9c4bd haproxy specify user 2021-06-24 13:54:24 -04:00
Seth Foster bb3fc3caa8 controller_token, alias tower_token 2021-06-22 10:49:39 -04:00
Seth Foster 75a27c38c2 Add a periodic task to reap unreleased receptor work units
- Add work_unit_id field to UnifiedJob
2021-06-22 10:49:31 -04:00
Seth Foster 61846e88ca Add validator for ee image field name
awxkit default ee image name is now a fixed valid (but bogus) name, rather than random unicode
2021-06-15 20:06:49 -04:00
Seth Foster 9eda6359f0 False to false to fix linter 2021-06-08 14:33:30 -04:00
Seth Foster 1deaf55ba4 syntax error in runtime.yml 2021-06-08 14:33:30 -04:00
Seth Foster 44c50bbbf7 add redirect for lookup and inventory plugins 2021-06-08 14:33:29 -04:00
Seth Foster c9e7747f2d black ignore awx_collection 2021-06-08 14:33:29 -04:00
Seth Foster 199b4b6b47 fixed up jinja2 templating and documentation 2021-06-08 14:33:29 -04:00
Seth Foster f06485feca fqcn redirects, and raise from sanity test fix 2021-06-08 14:33:29 -04:00