We should be consistent about this. Also this takes us from doing a as
many queries to the UnifiedJob table as we have instances to doing 1
query to the UnifiedJob table (and both do 1 query to Instances table)
* add database connection to the metrics endpoint
* bump the counts collector version to 1.2
* check for postgresql as database so to not break the tests
fix memory and cpu settings to suport k8s resource request format
* fix conversion of memory setting to bytes
This setting has not been getting set by default, and needed some fixing
up to be compatible with setting the memory in the same way as we set it
in the operator, as well as with other changes from last year which
assume that ansible runner is returning memory in bytes.
This way we can start setting this setting in the operator, and get a
more accurate reflection of how much memory is available to the control
pod in k8s.
On platforms where services are all sharing memory, we deduct a
penalty from the memory available. On k8s we don't need to do this
because the web, redis, and task containers each have memory
allocated to them.
* Support CPU setting expressed in units used by k8s
This setting has not been getting set by default, and needed some fixing
up to be compatible with setting the CPU resource request/limits in the
same way as we set it in the resource requests/limits.
This way we can start setting this setting in the
operator, and get a more accurate reflection of how much cpu is
available to the control pod in k8s.
Because cpu on k8s can be partial cores, migrate cpu field to decimal.
k8s does not allow granularity of less than 100m (equivalent to 0.1 cores), so only
store up to 1 decimal place.
fix analytics to deal with decimal cpu
need to use DjangoJSONEncoder when Decimal fields in data passed to
json.dumps
add a new awx-manage command `custom_venvs`
add an awx-manage command that gets pip freeze data from custom_venv and outputs to command line stdout
SUMMARY
part of #7062 - this command is a glorified pip freeze + some extra stuff, people could navigate to each of their custom virtual environments themselves and run a pip freeze, but this allows them to not, and everyone likes their life to be easier. The extra stuff allows users to see the connections that their existing virtual envs have in awx to things like organizations, jobs, inventory updates, and projects.
ISSUE TYPE
Feature Pull Request
COMPONENT NAME
API
AWX VERSION
awx: 19.1.0
ADDITIONAL INFORMATION
This is built off of existing code and there is a line that gets custom venv paths from the settings module, that line does not seem to be working. I have written around that but want to make a note of it.
Reviewed-by: Alan Rominger <arominge@redhat.com>
Reviewed-by: Rebeccah Hunter <rhunter@redhat.com>
Reviewed-by: Jeff Bradberry <None>
Reviewed-by: Shane McDonald <me@shanemcd.com>
Reviewed-by: Elijah DeLee <kdelee@redhat.com>
* Before, we would get the min and max pk of the set we are to gather.
This changeset removes that.
* Before, we would, basically, know the size of the set we are to gather
and would query 100,000 of those job event records at a time. That logic
is now gone.
* Now, for unpartitioned job events we gather 4 hours at a time by
created time.
* Now, for partitioned job events we gather 4 hours at a time by
modified time.
* The order by results in an in-memory sort that COULD blow out the
worker mem buffer and result in sorting having to take place on disk.
* This WILL happen with a default postgres 4MB mem buffer. We saw as
much as 20MB used. Note that AWX defaults postgres mem worker buffer to
3% of the DB memory on external installs and 1% on same-node installs.
So for a 16GB remote DB this would not be a problem.
* We are going to avoid this problem all together by NOT doing a sort
when gathering. Instead, we will sort remotely, in analytics.
* Old, _unpartitioned_main_jobevent table does not have the job_created
column
* New, main_jobevent does.
* Always in clude the job_created column. NULL if old, job_created if
new
* Bump events_table schema version from 1.2 to 1.3 because of the
job_created field
* We found that having multiple `::json` casts in a query slows down
queries more and more by =~> 33%.
* This change coerces postgres into only casting once. Micro
benchmarking shows =~ 2-3x performance boost
* pre-migration jobevents live in unpartitioned table
where only created field has index
* post-migration jobevents live in partitions
where modified field has index
(and should be used to ensure no events are missing)
* when collecting job events by creation time
it is possible to miss events that were created
at one point, but actually committed to the db
much later.
* since events' modified time is set when they are
committed to the db, we shouldn't miss any job events
* selecting job events by modified time wasn't possible
beforehand because we didn't have an index for
jobevent's modified field
Isolated removal
SUMMARY
Removal of the isolated nodes feature.
ISSUE TYPE
Feature Pull Request
COMPONENT NAME
API
AWX VERSION
Reviewed-by: Alan Rominger <arominge@redhat.com>
Reviewed-by: Jeff Bradberry <None>
Reviewed-by: Elyézer Rezende <None>
Reviewed-by: Bianca Henderson <beeankha@gmail.com>
- `since` should not be after `until`
- neither `since` nor `until` should be in the future
- `since`, `AUTOMATION_ANALYTICS_LAST_GATHER`, and `AUTOMATION_ANALYTICS_LAST_ENTRIES[key]` should be truncated to 4 weeks prior to `until`
- an explicit `since` parameter should always take precedence over the settings values
With the change to use pk-based interval slicing for the job events
table, we need analytics.gather to be the code that manages all of the
"expensive" collector slicing. While we are at it, let's ship each
chunked tarball file as we produce it.
AWX no longer includes Ansible on the control plane and there is no
"default" version of Ansible aside from what's configured at the
Execution Environment level
see: https://github.com/ansible/awx/issues/9472
