Commit Graph
100 Commits
Author SHA1 Message Date
Alan Rominger ff6fb32297 Merge pull request #12875 from ansible/feature-prompt-on-launch-on-templates
Feature prompt on launch on templates
2022-09-23 09:16:02 -04:00
Alan Rominger e231e08869 Fix bug with missing parent field and diff with parent
Remove corresponding views for job instance_groups

Validate job_slice_count in API

Remove defaults from some job launch view prompts
  the null default is preferable
2022-09-22 16:08:23 -04:00
Alan Rominger 61093b2532 Treat instance_groups prompt as template-less 2022-09-22 16:08:22 -04:00
Alan Rominger 816e491d17 Fix another bug applying extra_vars to incompatible job types 2022-09-22 16:08:22 -04:00
Alan Rominger b38e08174a Write logic to combing workflow labels, IGs with nodes
Additionally, move the inventory-specific hacks of yesteryear
  into the prompts_dict method of the WorkflowJob model
  try to make it clear exactly what this is hacking and why

Correctly summarize label prompts, and add missing EE

Expand unit tests to apply more fields

adding missing fields to preserve during copy to workflow.py

Fix bug where empty workflow job vars blanked node vars (#12904)

* Fix bug where empty workflow job vars blanked node vars

* Fix bug where workflow job has no extra_vars, add test

* Add empty workflow job extra vars to assure fix
2022-09-22 16:08:07 -04:00
Alan Rominger 64dad61b29 Add support for instance_groups and labels on schedule create 2022-09-22 16:08:06 -04:00
Alan Rominger ef90adb67e Complete consolidation of the label views 2022-09-22 16:08:06 -04:00
Alan Rominger 34e8087aee DRY edits to access classes for new prompts
Remove if-not-data conditional from WFJTnode.can_change
  these are cannonical for can_add, but this looks like a bug

Change JTaccess.can_unattach to call same method in super()
  previously called can_attach, which is problematic

Better consolidate launch config m2m related checks

Test and fix pre-existing WFJT node RBAC bug

recognize not-provided instance group list on launch, avoiding bug where it fell back to default

fix bug where timeout field was saved on WFJT nodes after creating approval node

remove labels from schedule serializer summary_fields

remove unnecessary prefetch of credentials from WFJT node queryset
2022-09-22 15:58:15 -04:00
Alan Rominger 68e11d2b81 Add WorkflowJob.instance_groups and distinguish from char_prompts
This removes a loop that ran on import
  the loop was giving the wrong behavior
  and it initialized too many fields as char_prompts fields

With this, we will now enumerate the char_prompts type fields manually
2022-09-22 15:39:49 -04:00
Alan Rominger 15964dc395 Merge pull request #11745 from AlanCoding/cancel_rework_no_close
Close database connections while processing job output
2022-09-06 15:45:29 -04:00
Alan Rominger 430f1986c7 Merge pull request #12830 from AlanCoding/dev_stuff
Fix LDAP volume conditional, better metrics interval
2022-09-06 11:51:51 -04:00
Alan Rominger f512971991 Add project sync to job cancel chain 2022-09-05 22:29:19 -04:00
Alan Rominger 53de245877 Fix LDAP volume conditional, better metrics interval 2022-09-04 22:33:12 -04:00
Alan Rominger 725d6fa896 Merge pull request #12820 from AlanCoding/five_seconds
Make the metrics default sampling interval 5s
2022-09-02 15:21:57 -04:00
Alan Rominger ccbc8ce7de Make the metrics default sampling interval 5s 2022-09-02 13:38:49 -04:00
Alan Rominger c59bbdecdb Refactor canceling to work through messaging and signals, not database
If canceled attempted before, still allow attempting another cancel
in this case, attempt to send the sigterm signal again.
Keep clicking, you might help!

Replace other cancel_callbacks with sigterm watcher
  adapt special inventory mechanism for this too

Get rid of the cancel_watcher method with exception in main thread

Handle academic case of sigterm race condition

Process cancelation as control signal

Fully connect cancel method and run_dispatcher to control

Never transition workflows directly to canceled, add logs
2022-09-01 15:20:31 -04:00
Alan Rominger 310e354164 Merge pull request #12769 from AlanCoding/self_conn
Fix sanity check to use the relevant active connection
2022-08-29 20:36:48 -04:00
Alan Rominger 6d207d2490 Merge pull request #12754 from kdelee/fix_metrics_consumed_capacity
calcuate consumed capacity in same way in metrics
2022-08-29 16:37:53 -04:00
Alan Rominger 01037fa561 Fix sanity check to use the relevant active connection 2022-08-29 16:33:07 -04:00
Alan Rominger 61f3e5cbed Merge pull request #12702 from AlanCoding/poll_cancel
Check exit conditions in loop waiting for project flock
2022-08-29 16:29:39 -04:00
Alan Rominger 44995e944a Merge pull request #12766 from AlanCoding/lazy_no_more
Revert "Merge pull request #12584 from AlanCoding/lazy_workers"
2022-08-29 16:06:50 -04:00
Alan Rominger 2437a84b48 Minor changes to instance loop structure 2022-08-29 14:28:50 -04:00
Alan Rominger 974f845059 Revert "Merge pull request #12584 from AlanCoding/lazy_workers"
This reverts commit 64157f7207, reversing
changes made to 9e8ba6ca09.
2022-08-28 23:04:13 -04:00
Alan Rominger edc3da85cc Revert "Fix errors in websocket code due to missing template" 2022-08-20 19:09:57 -04:00
Alan Rominger 2357e24d1d Merge pull request #12701 from AlanCoding/no_more_schedules
Make schedule teardown more reliable
2022-08-20 07:05:21 -04:00
Alan Rominger 88bf03c6bf Check exit conditions in loop waiting for project flock 2022-08-19 16:08:56 -04:00
Alan Rominger 4b8a56be39 Make schedule teardown more reliable 2022-08-19 15:42:00 -04:00
Alan Rominger 2aa99234f4 Merge pull request #12692 from AlanCoding/mop_up
Fix errors in websocket code due to missing template
2022-08-19 14:46:10 -04:00
Alan Rominger 704e4781d9 Fix errors in websocket code due to missing template 2022-08-18 14:05:06 -04:00
Alan Rominger 4a8613ce4c Avoid updating modified_by from None to None (#11838)
This should help the case of inventory updates in particular
  where imported hosts are managed by the system
2022-08-18 11:39:29 -04:00
Alan Rominger e87fabe6bb Submit job to dispatcher as part of transaction (#12573)
Make it so that submitting a task to the dispatcher happens as part of the transaction.
  this applies to dispatcher task "publishers" which NOTIFY the pg_notify queue
  if the transaction is not successful, it will not be sent, as per postgres docs

This keeps current behavior for pg_notify listeners
  practically, this only applies for the awx-manage run_dispatcher service
  this requires creating a separate connection and keeping it long-lived
  arbitrary code will occasionally close the main connection, which would stop listening

Stop sending the waiting status websocket message
  this is required because the ordering cannot be maintained with other changes here
  the instance group data is moved to the running websocket message payload

Move call to create_partition from task manager to pre_run_hook
  mock this in relevant unit tests
2022-08-18 09:43:53 -04:00
Alan Rominger 532aa83555 Merge pull request #11833 from AlanCoding/facts_update_fields
Use update_fields for Ansible facts update
2022-08-17 22:37:45 -04:00
Alan Rominger d87bb973d5 Merge pull request #12090 from AlanCoding/mind_your_own_business
Avoid parent instance update when status was unchanged
2022-08-17 22:29:31 -04:00
Alan Rominger a72da3bd1a Merge pull request #12582 from AlanCoding/clean_and_forget
Move reaper logic into worker, avoiding bottlenecks
2022-08-17 18:53:47 -04:00
Alan Rominger 56df3f0c2a Merge pull request #12671 from AlanCoding/cut_the_line
Avoid dependency manager for jobs with no deps
2022-08-17 18:50:52 -04:00
Alan Rominger e0c59d12c1 Change data structure so we can conditionally reap waiting jobs 2022-08-17 16:00:30 -04:00
Alan Rominger 7645cc2707 Remove mocks for reap method that was removed 2022-08-17 15:43:29 -04:00
Alan Rominger 6719010050 Add back in cleanup call 2022-08-17 15:42:48 -04:00
Alan Rominger ccd46a1c0f Move reaper logic into worker, avoiding bottlenecks 2022-08-17 15:42:47 -04:00
Alan Rominger 4fca27c664 Merge pull request #12289 from AlanCoding/idle_help
Correct help text for job idle timeout
2022-08-17 13:55:44 -04:00
Alan Rominger 51be22aebd Merge pull request #12668 from AlanCoding/graph_tweaks
Remove an old metrics field and add a new one to dashboard
2022-08-17 13:49:17 -04:00
Alan Rominger 54b21e5872 Avoid dependency manager for jobs with no deps 2022-08-17 13:32:59 -04:00
Alan Rominger 85beb9eb70 Merge pull request #12676 from AlanCoding/forward_picks
Stability fixes, and related logging for slowdowns in dispatcher task processing
2022-08-17 13:32:34 -04:00
Alan Rominger 56739ac246 Use delay_update to set error message, according to merge note 2022-08-17 11:45:40 -04:00
Alan Rominger 1ea3c564df Apply a failed status if cancel_flag is not set 2022-08-17 11:42:09 -04:00
Alan RomingerandElijah DeLee 621833ef0e Add extra workers if computing based on memory
Co-authored-by: Elijah DeLee <kdelee@redhat.com>
2022-08-17 11:41:59 -04:00
Alan RomingerandShane McDonald 43a53f41dd Add logs about heartbeat skew
Co-authored-by: Shane McDonald <me@shanemcd.com>
2022-08-17 11:33:59 -04:00
Alan Rominger a3fef27002 Add logs to debug waiting bottlenecking 2022-08-17 11:33:49 -04:00
Alan Rominger cfc1255812 Merge pull request #12442 from AlanCoding/waiting_reaper
Fix false reaper false-positives of waiting jobs that are waiting for worker
2022-08-17 11:20:05 -04:00
Alan RomingerandElijah DeLee 278db2cdde Split reaper for running and waiting jobs
Avoid running jobs that have already been reapted

Co-authored-by: Elijah DeLee <kdelee@redhat.com>

Remove unnecessary extra actions

Fix waiting jobs in other cases of reaping
2022-08-17 10:53:29 -04:00
Alan Rominger 64157f7207 Merge pull request #12584 from AlanCoding/lazy_workers
Wait 60 seconds before scaling down a worker
2022-08-17 10:18:19 -04:00
Alan Rominger 9e8ba6ca09 Merge pull request #12494 from AlanCoding/revival
Register system again if deleted by another pod
2022-08-17 10:12:39 -04:00
Alan Rominger 268ab128d7 Merge pull request #12527 from AlanCoding/offline_db
Further resiliency changes, specifically focused on case of database going offline
2022-08-17 10:10:50 -04:00
Alan Rominger fad5934c1e Merge pull request #12356 from AlanCoding/copytree_neo
Replace git shallow clone with shutil.copytree
2022-08-17 10:07:28 -04:00
Alan Rominger c9e3873a28 Use update_fields for Ansible facts update 2022-08-17 08:22:41 -04:00
Alan Rominger 11e63e2e89 Remove an old metrics field and add a new one to dashboard 2022-08-16 22:37:27 -04:00
Alan Rominger f1bd1f1dfc Merge pull request #12658 from AlanCoding/more_panels
Add more graphs for task manager refactor
2022-08-15 16:07:43 -04:00
Alan Rominger f6da9a5073 Add more graphs for task manager refactor 2022-08-15 15:29:34 -04:00
Alan Rominger ea07aef73e Correct help text for job idle timeout 2022-08-11 09:39:29 -04:00
Alan Rominger 46be2d9e5b Replace git shallow clone with shutil.copytree
Introduce build_project_dir method
  the base method will create an empty project dir for workdir

Share code between job and inventory tasks with new mixin
  combine rest of pre_run_hook logic
  structure to hold lock for entire sync process

force sync to run for inventory updates due to UI issues

Remove reference to removed scm_last_revision field
2022-08-10 16:18:56 -04:00
Alan Rominger 998000bfbe Surface correct error from bulk_create on unrecoverable error 2022-08-10 16:16:57 -04:00
Alan Rominger 43a50cc62c Fix event counting in error handling path 2022-08-10 16:16:57 -04:00
Alan Rominger 30f556f845 Further resiliency changes focused on offline database
Make logs from database outage more manageable

Raise exception if update_model never recovers from problem
2022-08-10 16:16:57 -04:00
Alan Rominger c5985c4c81 Change lazy worker method name and adjust log 2022-08-10 16:12:03 -04:00
Alan Rominger a9170236e1 Wait 60 seconds before scaling down a worker 2022-08-10 16:12:03 -04:00
Alan Rominger a0103acbef Inherit from our own APIView, not rest framework 2022-08-10 15:31:19 -04:00
Alan Rominger f7e6a32444 Optimize task manager with debug toolbar, adjust prefetch (#12588) 2022-08-10 10:05:13 -04:00
Alan Rominger 585d3f4e2a Register system again if deleted by another pod
Avoid cases where missing instance
  would throw error on startup
  this gives time for heartbeat to register it
2022-08-08 22:36:17 -04:00
Alan Rominger 279cebcef3 Merge pull request #12586 from AlanCoding/connections_graph
Add a graph to show database connections being used
2022-08-08 15:49:20 -04:00
Alan Rominger d06a3f060d Block sliced workflow jobs on any job type from their JT (#12551) 2022-08-05 14:33:45 -04:00
Alan Rominger b94b3a1e91 [task_manager_refactor] Move approval node expiration logic into queryset (#12502)
Instead of loading all pending Workflow Approvals in the task manager,
  run a query that will only return the expired apporovals
  directly expire all which are returned by that query

Cache expires time as a new field in order to simplify WorkflowApproval filter
2022-08-05 14:33:27 -04:00
Alan Rominger 8af4dd5988 Fix unintended slice job blocking 2022-08-05 14:33:25 -04:00
Alan Rominger 3aa8320fc7 Add a graph to show database connections being used 2022-07-28 11:52:36 -04:00
Alan Rominger 2d310dc4e5 Optimize object creation by getting fewer empty relationships (#12508)
This optimizes the ActivityStreamSerializer by only getting many-to-many
  relationships that are speculatively non-empty
  based on information we have in other fields

We run this every time we create an object as an on_commit action
  so it is expected this will have a major impact on response times for launching jobs
2022-07-19 14:27:51 -04:00
Alan RomingerandElijah DeLee fd671ecc9d Give specific messages if job was killed due to SIGTERM or SIGKILL (#12435)
* Reap jobs on dispatcher startup to increase clarity, replace existing reaping logic

* Exit jobs if receiving SIGTERM signal

* Fix unwanted reaping on shutdown, let subprocess close out

* Add some sanity tests for signal module

* Add a log for an unhandled dispatcher error

* Refine wording of error messages

Co-authored-by: Elijah DeLee <kdelee@redhat.com>
2022-06-30 13:20:08 -04:00
Alan Rominger 783b744bdb Pass combined artifacts from nested workflows into downstream nodes (#12223)
* Track combined artifacts on workflow jobs

* Avoid schema change for passing nested workflow artifacts

* Basic support for nested workflow artifacts, add test

* Forgot that only does not work with polymorphic

* Remove incorrect field

* Consolidate logic and prevent recursion with UJ artifacts method

* Stop trying to do precedence by status, filter for obvious ones

* Review comments about sets

* Fix up bug with convergence node paths and artifacts
2022-06-23 16:54:53 -03:00
Alan Rominger 923cc671db Merge pull request #12391 from AlanCoding/compose_graphs
Do the grafana thing in docker-compose templating itself
2022-06-16 16:23:36 -04:00
Alan Rominger db105c21e4 Set default false values 2022-06-16 15:46:42 -04:00
Alan Rominger 372aa36207 Make the prometheus config file ignored by git 2022-06-16 15:42:10 -04:00
Alan Rominger 173318764b Remove existing yml file for prometheus 2022-06-16 15:37:18 -04:00
Alan Rominger 1dd535a859 Remove old way of doing grafana graphs 2022-06-16 15:31:45 -04:00
Alan Rominger f4ef7d6927 Add volumes to the clean command 2022-06-16 14:03:22 -04:00
Alan Rominger c441db2aab docs workding edits and depends_on 2022-06-16 12:07:26 -04:00
Alan Rominger fb292d9706 Move visualization containers into docker-compose 2022-06-16 10:25:02 -04:00
Alan Rominger 81b8028ea2 Merge pull request #12355 from AlanCoding/autoreload_once
Make awx-autoreloader work faster for large code changes
2022-06-09 15:19:17 -04:00
Alan Rominger a4bfb032ff Make awx-autoreloader work faster for large code changes 2022-06-09 14:52:03 -04:00
Alan Rominger 4543f6935f Only do substitutions for container path conversions with resolved paths (#12313)
* Resolve paths as much as possible before doing replacements

* Move unused method out of main code, test symlink
2022-06-09 11:36:29 -04:00
Alan Rominger 78d3d6dc94 Merge pull request #12219 from AlanCoding/really_skip
Change Demo Project status to successful
2022-06-09 11:19:57 -04:00
Alan Rominger 3af89c1e2b Merge pull request #12307 from AlanCoding/twilio
Upgrade twilio dependency to pick up fix
2022-06-02 13:48:34 -04:00
Alan Rominger d553c37d7d Upgrade twilio dependency to pick up fix 2022-06-01 11:35:43 -04:00
Alan Rominger 8366386126 Merge pull request #12260 from AlanCoding/callback_status
Fix the callback receiver --status command
2022-05-24 15:26:02 -04:00
Alan Rominger 1e6ca01686 Fix the callback receiver --status command 2022-05-19 15:00:49 -04:00
Alan Rominger bca6e00e37 Change Demo Project status to successful 2022-05-12 16:14:09 -04:00
Alan Rominger aaad634483 Only use in-memory cache for database settings, set ttl=5 (#12166)
* Only use in-memory cache for database settings

Make necessary adjustments to monkeypatch
  as it is very vunerable to recursion
  Remove migration exception that is now redundant

Clear cache if a setting is changed

* Use dedicated middleware for setting cache stuff
  Clear cache for each request

* Add tests for in-memory cache
2022-05-10 21:58:22 -04:00
Alan Rominger 452744b67e Delay update of artifacts and error fields until final job save (#11832)
* Delay update of artifacts until final job save

Save tracebacks from receptor module to callback object

Move receptor traceback check up to be more logical

Use new mock_me fixture to avoid DB call with me method

Update the special runner message to the delay_update pattern

* Move special runner message into post-processing of callback fields
2022-05-03 14:42:50 -04:00
Alan Rominger 29d60844a8 Fix notification timing issue by sending in the latter of 2 events (#12110)
* Track host_status_counts and use that to process notifications

* Remove now unused setting

* Back out changes to callback class not needed after all

* Skirt the need for duck typing by leaning on the cached field

* Delete tests for deleted task

* Revert "Back out changes to callback class not needed after all"

This reverts commit 3b8ae350d218991d42bffd65ce4baac6f41926b2.

* Directly hardcode stats_event_type for callback class

* Fire notifications if stats event was never sent

* Remove test content for deleted methods

* Add placeholder for when no hosts matched

* Make field default be None, denote events processed with empty dict

* Make UI process null value for host_status_counts

* Fix tracking of EOF dispatch for system jobs

* Reorganize EVENT_MAP into class properties

* Consolidate conditional I missed from EVENT_MAP refactor

* Give up on the null condition, also applies for empty hosts

* Remove cls position argument not being used

* Move wrapup method out of class, add tests
2022-04-29 13:54:31 -04:00
Alan Rominger cb63d92bbf Remove committed_capacity field, delete supporting code (#12086)
* Remove committed_capacity field, delete supporting code

* Track consumed capacity to solve the negatives problem

* Use more verbose name for IG queryset
2022-04-22 13:41:32 -04:00
Alan Rominger 29702400f1 Avoid parent instance update when status was unchanged 2022-04-22 09:07:03 -04:00
Alan Rominger 44f0609314 Merge pull request #11996 from AlanCoding/blockhead
Remove unnecessary blocks from project update playbook
2022-04-21 13:58:48 -04:00
Alan Rominger 4b45148614 Merge pull request #12016 from Ladas/analytics_collector_should_collect_full_license_data
Analytics collector should collect full license data
2022-04-21 11:12:33 -04:00