Commit Graph

204 Commits

Author SHA1 Message Date
Alan Rominger
61093b2532 Treat instance_groups prompt as template-less 2022-09-22 16:08:22 -04:00
John Westcott IV
e076f1ee2a Making labels additive and not adding a many item to config if already in parent 2022-09-22 15:39:49 -04:00
John Westcott IV
33c0fb79d6 JT param everything (#12646)
* Making almost all fields promptable on job templates and config models
* Adding EE, IG and label access checks
* Changing jobs preferred instance group function to handle the new IG cache field
* Adding new ask fields to job template modules
* Address unit/functional tests
* Adding migration file
2022-09-22 15:16:12 -04:00
Alan Rominger
a3fef27002 Add logs to debug waiting bottlenecking 2022-08-17 11:33:49 -04:00
Seth Foster
e6f8852b05 Cache task_impact
task_impact is now a field on the database
It is calculated and set during create_unified_job

set task_impact on .save for adhoc commands
2022-08-05 14:33:47 -04:00
Seth Foster
0a47d05d26 split schedule_task_manager into 3
each call to schedule_task_manager becomes one of

ScheduleTaskManager
ScheduleDependencyManager
ScheduleWorkflowManager
2022-08-05 14:33:25 -04:00
Seth Foster
431b9370df Split TaskManager into
- DependencyManager spawns dependencies if necessary
- WorkflowManager processes running workflows to see if a new job is
  ready to spawn
- TaskManager starts tasks if unblocked and has execution capacity
2022-08-05 14:29:02 -04:00
Jeff Bradberry
11890f0eee Fix the job event partition alignment
it really should be always aligned to the hour, so that real job
events don't slip through the cracks.
2022-04-18 14:54:06 -04:00
Seth Foster
24152555c5 Handle error for create_partition
Occasionally the create_partition will error with,
relation "main_projectupdateevent_20220323_19" already exists

This change wraps the db command into a try except block with its
own transaction
2022-03-28 16:37:50 -04:00
Jeff Bradberry
a3a216f91f Fix up new Django 3.0 deprecations
Mostly text based: force/smart_text, ugettext_*
2022-03-07 18:11:36 -05:00
Elijah DeLee
799968460d Fixup conversion of memory and cpu settings to support k8s resource request format (#11725)
fix memory and cpu settings to suport k8s resource request format

* fix conversion of memory setting to bytes

This setting has not been getting set by default, and needed some fixing
up to be compatible with setting the memory in the same way as we set it
in the operator, as well as with other changes from last year which
assume that ansible runner is returning memory in bytes.

This way we can start setting this setting in the operator, and get a
more accurate reflection of how much memory is available to the control
pod in k8s.

On platforms where services are all sharing memory, we deduct a
penalty from the memory available. On k8s we don't need to do this
because the web, redis, and task containers each have memory
allocated to them.

* Support CPU setting expressed in units used by k8s

This setting has not been getting set by default, and needed some fixing
up to be compatible with setting the CPU resource request/limits in the
same way as we set it in the resource requests/limits.

This way we can start setting this setting in the
operator, and get a more accurate reflection of how much cpu is
available to the control pod in k8s.

Because cpu on k8s can be partial cores, migrate cpu field to decimal.

k8s does not allow granularity of less than 100m (equivalent to 0.1 cores), so only
store up to 1 decimal place.

fix analytics to deal with decimal cpu

need to use DjangoJSONEncoder when Decimal fields in data passed to
json.dumps
2022-02-15 14:08:24 -05:00
Alan Rominger
9664aed1f2 Remove unused ansible version method 2022-01-14 14:55:35 -05:00
Alan Rominger
82671680e3 Respect linter rule F811 for trivial re-definition 2022-01-14 13:23:04 -05:00
Alan Rominger
424dbe8208 Use ansible-runner imports for cpu and memory calculation (#10954)
* Use ansible-runner imports for cpu and memory calculation

* Fix bug with capacity and memory adjustment
2021-08-27 21:46:53 -04:00
Alan Rominger
daf4310176 Clean up work_type processing and fix execution vs control capacity (#10930)
* Clean up added work_type processing for mesh_code branch

* track both execution and control capacity

* Remove unused execution_capacity property

* Count all forms of capacity to make test pass

* Force jobs to be on execution nodes, updates on control nodes

* Introduce capacity_type property to abstract some details out

* Update test to cover all job types at same time

* Register OpenShift nodes as control types

* Remove unqualified consumed_capacity from task manager and make unit tests work

* Remove unqualified consumed_capacity from task manager and make unit tests work

* Update unit test to execution vs control TM logic changes

* Fix bug, else handling for work_type method
2021-08-26 07:24:14 -04:00
Alan Rominger
928c35ede5 Model changes for instance last_seen field to replace modified (#10870)
* Model changes for instance last_seen field to replace modified

* Break up refresh_capacity into smaller units

* Rename execution node methods, fix last_seen clustering

* Use update_fields to make it clear save only affects capacity

* Restructing to pass unit tests

* Fix bug where a PATCH did not update capacity value
2021-08-24 08:41:35 -04:00
Alan Rominger
3b1e40d227 Use the ansible-runner worker --worker-info to perform execution node capacity checks (#10825)
* Introduce utilities for --worker-info health check integration

* Handle case where ansible-runner is not installed

* Add ttl parameter for health check

* Reformulate return data structure and add lots of error cases

* Move up the cleanup tasks, close sockets

* Integrate new --worker-info into the execution node capacity check

* Undo the raw value override from the PoC

* Additional refinement to execution node check frequency

* Put in more complete network diagram

* Followup on comment to remove modified from from health check responsibilities
2021-08-24 08:41:35 -04:00
Alan Rominger
13300bdbd4 Update rebase to keep old control plane capacity check
Also do some basic work to separate control versus execution capacity
  this is to assure that we don't send jobs to the control node
2021-08-24 08:40:19 -04:00
Ryan Petrello
05cb876df5 implement an initial development environment for receptor-based clusters 2021-08-24 08:40:18 -04:00
Chris Meyers
84116349ab reload settings on fork or new thread
* Reloading settings will re-import cache. Django cache is thread aware,
but keeping an old reference to the cache avoids the awareness.
2021-06-28 13:53:11 -04:00
Christian M. Adams
06b04007a0 Rename managed_by_tower to managed 2021-06-22 10:49:36 -04:00
Alan Rominger
21aa1fc11f Handle inventory types where Automation Hub collection names differ
Move imports added by Bill to be in-line, because utils should not import models at top

Remove more get_licenser inline imports
2021-06-16 13:39:52 -04:00
softwarefactory-project-zuul[bot]
b64f966db1 Merge pull request #10090 from rebeccahhh/custom_venv_command
add a new awx-manage command `custom_venvs`

add an awx-manage command that gets pip freeze data from custom_venv and outputs to command line stdout
SUMMARY

part of #7062  - this command is a glorified pip freeze + some extra stuff, people could navigate to each of their custom virtual environments themselves and run a pip freeze, but this allows them to not, and everyone likes their life to be easier. The extra stuff allows users to see the connections that their existing virtual envs have in awx to things like organizations, jobs, inventory updates, and projects.

ISSUE TYPE


Feature Pull Request

COMPONENT NAME


API

AWX VERSION

awx: 19.1.0

ADDITIONAL INFORMATION

This is built off of existing code and there is a line that gets custom venv paths from the settings module, that line does not seem to be working. I have written around that but want to make a note of it.

Reviewed-by: Alan Rominger <arominge@redhat.com>
Reviewed-by: Rebeccah Hunter <rhunter@redhat.com>
Reviewed-by: Jeff Bradberry <None>
Reviewed-by: Shane McDonald <me@shanemcd.com>
Reviewed-by: Elijah DeLee <kdelee@redhat.com>
2021-06-04 18:44:52 +00:00
Chris Meyers
1a1d66d2a2 bump db partition migration 2021-06-04 09:17:09 -07:00
Chris Meyers
30871bd6cf close db and cache connection in new threads 2021-06-04 09:17:09 -07:00
Jim Ladd
2a23b4c719 bump migration 2021-06-04 09:17:09 -07:00
Jim Ladd
ef9f9129ba bump migration 2021-06-04 09:17:09 -07:00
Chris Meyers
6ce227a6b6 bump migrations 2021-06-04 09:17:09 -07:00
Chris Meyers
137111351c bump migrations after devel rebase 2021-06-04 09:17:09 -07:00
Jim Ladd
0f9f3f58e2 bump migration 2021-06-04 09:17:08 -07:00
Jim Ladd
fb97687d14 lint 2021-06-04 09:17:08 -07:00
Jim Ladd
0f53d9b911 bump db partition migration 2021-06-04 09:17:08 -07:00
Jim Ladd
6b4effc85a bump partition migration to 135 2021-06-04 09:17:08 -07:00
Jim Ladd
a7cabec3d0 bump partition migration to 0132 2021-06-04 09:17:07 -07:00
Jim Ladd
8501a45531 lint fixes 2021-06-04 09:17:07 -07:00
Jim Ladd
14b610dabf bump partition migration 2021-06-04 09:17:07 -07:00
Jim Ladd
373edbf8c0 Update reference to partition migration 2021-06-04 09:17:07 -07:00
Ryan Petrello
08b96a0bd7 correct filter events on the migration event horizon
events that existed *prior* to the partition migration will have
`job_created=1970-01-01` auto-applied at migration time; as such,
queries for these events e.g., /api/v2/job/N/job_events/
use 1970-01-01 in related event searche

events created *after* the partition migration (net-new playbook runs
will have `job_created` values that *exactly match* the related
`UnifiedJob.created` field.
2021-06-04 09:17:07 -07:00
Jim Ladd
ec484f81cf Partition *all* job event tables 2021-06-04 09:17:06 -07:00
Jim Ladd
82df3ebddb add option to create partitions that span a single minute
* turned on by default currently for testing
2021-06-04 09:17:06 -07:00
Jim Ladd
612e91263c auto-create partition 2021-06-04 09:17:06 -07:00
Jim Ladd
445042c0f4 Create partition only if it doesn't exist 2021-06-04 09:17:06 -07:00
Jim Ladd
0c289205de Give new primary key constraint unique name, create first live partition 2021-06-04 09:17:06 -07:00
Jim Ladd
ba45592d93 create helper method to create partitions
* create_partition()
2021-06-04 09:17:06 -07:00
Rebeccah
779ca8b260 split the one command into two for clarity and remove unused imports 2021-06-04 09:11:21 -04:00
Shane McDonald
256a47618f Ignore awx venv
- General usability / readibility improvements
2021-06-04 09:11:20 -04:00
Rebeccah
dfaa69be51 add an awx-manage command that gets pip freeze data from custom_venvs and outputs to command line stdout
remove analytics tests for counts of custom venvs, bump collector version, and remove list of custom venvs from API
2021-06-04 09:11:16 -04:00
Chris Meyers
9b7bc5109a close db and cache connection in new threads 2021-05-13 11:13:26 -04:00
Bill Nottingham
c8cf28f266 Assorted renaming and string changes 2021-04-30 14:32:05 -04:00
Ryan Petrello
300f5a3a1f use flake8 to lint for a few things black doesn't catch
black does *not* warn about missing or extraneous imports,
so let's bring back flake8 in our linting to check for them
2021-04-12 12:55:39 -04:00