Commit Graph

66 Commits

Author SHA1 Message Date
Mauricio Magnani Jr
7ee0aab856 fix: ensure temp files are cleaned up after failed HCC (#15996) 2025-05-21 13:17:46 -04:00
Seth Foster
95289ff28c Update subscription API to use service accounts
Update code to pull subscriptions from
console.redhat.com instead of
subscription.rhsm.redhat.com

Uses service account client ID and client secret
instead of username/password, which is being
deprecated in July 2025.

Additional changes:

- In awx.awx.subscriptions module, use new service
account params rather than old basic auth params

- Update awx.awx.license module to use subscription_id
instead of pool_id. This is due to using a different API,
which identifies unique subscriptions by subscriptionID
instead of pool ID.

Signed-off-by: Seth Foster <fosterbseth@gmail.com>
Co-authored-by: Chris Meyers <chris.meyers.fsu@gmail.com>
Co-authored-by: Peter Braun <pbraun@redhat.com>
2025-04-30 15:44:38 -04:00
Seth Foster
0f0f5aa289 Pass in private_data_dir when project update is on K8S
In OCP/K8S, projects run in the task pod's ee container. The private_data_dir is not extracted to /runner. Instead, the project update runs directly from the mounted in private_data_dir, e.g. /tmp/awx_1_abcd.

When injecting a credential that uses extra vars, we pass the private_data_dir as as the container_root, so that the correct command line argument is generated, e.g. "-e /tmp/awx_1_abcd/env/extra_var_file".

Signed-off-by: Seth Foster <fosterbseth@gmail.com>
2025-03-11 23:12:10 -04:00
jessicamack
e56752d55b Ship analytics data using service account token (#15812)
Use oidc client to ship analytics data
2025-02-19 16:38:47 -05:00
Alan Rominger
f57a9863d6 Use advisory_lock from DAB (#15676)
* Use advisory_lock from DAB

* Remove the django-pglocks dep

* Re-run updater script

* Move the import in new location
2025-01-15 14:06:59 -05:00
Hao Liu
5a0080658c Fix analytic ship (#15496)
REDHAT_USERNAME and REDHAT_PASSWORD are default to empty string instead of None
2024-09-09 14:14:32 -04:00
Hao Liu
3da9e322b7 Fix subscription username password setting name (#15493)
used in analytic
2024-09-05 19:59:45 +00:00
Hao Liu
ac6c5630f1 Fallback to use subscription cred for analytic upload (#15479)
* Fallback to use subscription cred for analytic

Fall back to use SUBSCRIPTION_USERNAME/PASSWORD to upload analytic to if REDHAT_USERNAME/PASSWORD are not set

* Improve error message

* Guard against request with no query or data

* Add test for _send_to_analytics

Focus on credentials

* Supress sonarcloud warning about password

* Add test for analytic ship
2024-08-30 10:39:53 +02:00
John Westcott IV
71c307ab8a Merge pull request #13808 from ansible/feature_on-premise-analytics
Proxy analytics requests through AWX API
2023-04-07 11:46:14 -04:00
Martin Slemr
44db4587be Analytics upload: HostMetrics hybrid sync 2023-04-07 08:56:03 -04:00
Martin Slemr
0a40b758c3 Analytics API: Paths, headers and Error handling 2023-04-07 08:54:56 -04:00
John Mitchell
bbb9770a97 change back to Automation Analytics name (#12022) 2022-04-12 14:23:13 -04:00
Elijah DeLee
799968460d Fixup conversion of memory and cpu settings to support k8s resource request format (#11725)
fix memory and cpu settings to suport k8s resource request format

* fix conversion of memory setting to bytes

This setting has not been getting set by default, and needed some fixing
up to be compatible with setting the memory in the same way as we set it
in the operator, as well as with other changes from last year which
assume that ansible runner is returning memory in bytes.

This way we can start setting this setting in the operator, and get a
more accurate reflection of how much memory is available to the control
pod in k8s.

On platforms where services are all sharing memory, we deduct a
penalty from the memory available. On k8s we don't need to do this
because the web, redis, and task containers each have memory
allocated to them.

* Support CPU setting expressed in units used by k8s

This setting has not been getting set by default, and needed some fixing
up to be compatible with setting the CPU resource request/limits in the
same way as we set it in the resource requests/limits.

This way we can start setting this setting in the
operator, and get a more accurate reflection of how much cpu is
available to the control pod in k8s.

Because cpu on k8s can be partial cores, migrate cpu field to decimal.

k8s does not allow granularity of less than 100m (equivalent to 0.1 cores), so only
store up to 1 decimal place.

fix analytics to deal with decimal cpu

need to use DjangoJSONEncoder when Decimal fields in data passed to
json.dumps
2022-02-15 14:08:24 -05:00
Chris Meyers
1c97b9a046 no longer get the size of the gather set
* Before, we would get the min and max pk of the set we are to gather.
This changeset removes that.
* Before, we would, basically, know the size of the set we are to gather
and would query 100,000 of those job event records at a time. That logic
is now gone.
* Now, for unpartitioned job events we gather 4 hours at a time by
created time.
* Now, for partitioned job events we gather 4 hours at a time by
modified time.
2021-06-04 09:17:09 -07:00
Bill Nottingham
c9ac18db24 Update branding to match guidelines 2021-05-25 16:25:34 -04:00
Bill Nottingham
8683872927 Update analytics branding 2021-05-11 15:46:56 -04:00
Jeff Bradberry
4857c5edcb Break out the main interval trimming calculation into a new function 2021-04-26 13:31:05 -04:00
Jeff Bradberry
f4d848e596 Log adjustments made to the analytics interval due to the 4-week limit 2021-04-26 13:31:05 -04:00
softwarefactory-project-zuul[bot]
555b25321e Merge pull request #9982 from jladdjr/jladd_explain_setting_gather_no_matter_what_devel
[devel] add clarifying comment re: AUTOMATION_ANALYTICS_LAST_GATHER being set regardless

(the comment in the changeset pretty much says it all)

Reviewed-by: Chris Meyers <None>
2021-04-23 17:07:13 +00:00
Jim Ladd
fb97a79aca add clarifying comment re: AUTOMATION_ANALYTICS_LAST_GATHER being set regardless 2021-04-21 21:28:32 -07:00
Jeff Bradberry
b9389208dd Add in the missing dry-run check for csv analytics collectors 2021-04-20 09:36:29 -04:00
Jeff Bradberry
62c7554ec4 Adjust datetimes to be aware when using awx-manage gather_analytics
Also, make sure that an explicit `since` parameter will win over
default `until=now()` when calculating the 4-week data limit.
2021-04-06 15:59:54 -04:00
Jeff Bradberry
f85e8a44de Properly parse datetimes from AUTOMATION_ANALYTICS_LAST_ENTRIES 2021-04-05 13:58:32 -04:00
Jeff Bradberry
6030c5cf4c Rationalize the interval calculations for analytics gathering
- `since` should not be after `until`
- neither `since` nor `until` should be in the future
- `since`, `AUTOMATION_ANALYTICS_LAST_GATHER`, and `AUTOMATION_ANALYTICS_LAST_ENTRIES[key]` should be truncated to 4 weeks prior to `until`
- an explicit `since` parameter should always take precedence over the settings values
2021-04-05 13:58:32 -04:00
Jeff Bradberry
8ce3a14da5 A couple more fixes:
- stop trying to ship csv slices when one breaks
- only update LAST_ENTRIES if all of the files in a time/pk slice succeed
- don't allow an explicit --until parameter to set the GATHER/ENTRIES values backwards
2021-04-05 13:58:32 -04:00
Jeff Bradberry
1dacd7e8cf Only clean up tarballs if we had all successfully ship 2021-04-05 13:58:32 -04:00
Jeff Bradberry
39886da4b6 Deal with datetimes in AUTOMATION_ANALYTICS_LAST_ENTRIES 2021-04-05 13:58:32 -04:00
Jeff Bradberry
99daa4319e Require the config.json file to be in dry-run tarballs 2021-04-05 13:58:32 -04:00
Jeff Bradberry
3568558571 A couple of bug fixes 2021-04-05 13:58:32 -04:00
Jeff Bradberry
0b31e771b1 Fix the gather_analytics management command
Previously, invoking the command with neither of the --ship or
--dry-run flags would result in effectively doing a dry run.  With the
stricter checks now in place in analytics.core.gather, let's make sure
that we pass the 'dry-run' parameter in to gather() in the no-flags
case.
2021-04-05 13:58:32 -04:00
Jeff Bradberry
772da61980 If a csv collector is successful but results in no files, increment anyway 2021-04-05 13:58:32 -04:00
Jeff Bradberry
9cde10c93a Fix problems with the package() function 2021-04-05 13:58:32 -04:00
Jeff Bradberry
1bf37266b4 Differentiate the log level depending on whether we are run from the task 2021-04-05 13:58:32 -04:00
Jeff Bradberry
77f7e88e68 Address the problems with trying to use a JSONField 2021-04-05 13:58:32 -04:00
Jeff Bradberry
3e4e255d3f Require config collector data for all posts to analytics
We need the cluster ID when consuming all incoming data.
2021-04-05 13:58:32 -04:00
Jeff Bradberry
a448cb17d9 Refactor analytics.gather
With the change to use pk-based interval slicing for the job events
table, we need analytics.gather to be the code that manages all of the
"expensive" collector slicing.  While we are at it, let's ship each
chunked tarball file as we produce it.
2021-04-05 13:58:32 -04:00
Ryan Petrello
c2ef0a6500 move code linting to a stricter pep8-esque auto-formatting tool, black 2021-03-23 09:39:58 -04:00
Alan Rominger
44e6e9344b Remove mercurial support 2020-12-01 14:06:34 -05:00
Ryan Petrello
4445d096f5 reimplement licensing to work with RHSM and entitlement uploads
Co-authored-by: Christian Adams <chadams@redhat.com>
2020-10-30 16:39:57 -04:00
Bill Nottingham
13802fcf2b Don't return error messages for license errors
Just log the exception and return None.
2020-09-10 21:19:07 -04:00
Bill Nottingham
c753324872 Move back to less frequent collections, and split large event tables
This should ensure we stay under 100MB at all times.
2020-09-09 17:42:40 -04:00
Bill Nottingham
9f67b6742c Fail more gracefully if analytics.ship() is called with a bad path,
or it's deleted out from under us.
2020-09-09 17:42:40 -04:00
Bill Nottingham
1a15f18be3 Stop using the TowerAnalyticsState solo model
This is now tracked in the AUTOMATION_ANALYTICS_LAST_GATHER setting.
2020-09-09 17:42:40 -04:00
Bill Nottingham
40309e6f70 Ensure we do not send large bundles, or empty bundles
Collect expensive collectors separately, and in a loop
where we make smaller intermediate dumps.

Don't return a table dump if there are no records, and
don't put that CSV in the manifest.

Fix up unit tests.
2020-09-09 17:42:40 -04:00
Bill Nottingham
1c4b06fe1e Refactor analytics collectors.
- Only have one registration class
- Add description fields
- Add automation collector information to /api/v2/config
2020-09-09 17:10:14 -04:00
Christian M. Adams
4ea648307e Accept all responses <300 from Insights API 2020-08-28 12:45:50 -04:00
Gabe Muniz
80234c5600 rework the tar to always delete the leftover directories 2020-03-16 19:54:15 +00:00
Bill Nottingham
71ef7cdec1 Use AWX_TASK_ENV when connecting to Red Hat services 2020-02-07 14:29:42 -05:00
Christian Adams
05e6f4ab3c Add User-Agent to analytics upload POST 2019-11-25 22:22:05 -05:00
Graham Mainwaring
7dd241fcff Add a --dry-run option to gather analytics locally, even if analytics is disabled in settings. 2019-10-17 13:54:13 -04:00