Commit Graph

68 Commits

Author SHA1 Message Date
Bill Nottingham
5b0bb4939f Allow subsets of table gathering for unit tests.
sqlite does not like some of our PG-isms.
2020-04-21 20:21:20 +02:00
Ladislav Smola
144cffe009 Send job and template nodes to analytics
Sending tables main_workflowjobnode and main_workflowjobtemplatenode
containing arrays of success/failure/always_nodes which is compatible
to what API call for nodes return.
2020-04-21 20:02:30 +02:00
Christian Adams
a899a147e1 Fix new flake8 from pyflakes 2.2.0 release 2020-04-20 09:50:50 -04:00
Bill Nottingham
f99a43ffa6 Collect information on inventory sources
Also remove one minor query from smart inventory collection that will
never return anything.
2020-04-14 19:15:19 -04:00
Bill Nottingham
cd6d2ed53a Move the comma so unit test can filter things properly. 2020-04-14 13:12:03 -04:00
Bill Nottingham
a296f64696 Cast the start/end times with timezone. 2020-04-14 10:53:57 -04:00
softwarefactory-project-zuul[bot]
c414fd68a0 Merge pull request #6176 from Ladas/send_also_workflows_as_part_of_unified_jobs
Send also workflows as part of unified jobs and send all changes to jobs

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul
2020-04-13 18:41:36 +00:00
Bill Nottingham
d651786206 Collect task timing, warnings, and deprecations from job events
Timing information requires ansible-runner >= 1.4.6.
2020-04-09 17:27:19 -04:00
Ladislav Smola
6a86af5b43 Use indexed timestamps
Use created and finished, which are indexed, to try to fetch all
states of jobs. If job is not finished, we might not get the
right terminal status, but that should be ok for now.
2020-04-09 15:20:27 +02:00
Ladislav Smola
6a503e152a Send also workflows as part of unified jobs
Workflows do not have a record in main_job, therefore the JOIN
was ignoring those. We need to do LEFT JOIN to include also
workflows.

It also seems like we are not able to get a link to organizations
from workflows? When looking at:
<tower_url>#/organizations?organization_search=page_size:20;order_by:name

We don't seem to list a relation to workflows. Is it possible to get it from
somewhere?
2020-04-09 15:20:27 +02:00
Ladislav Smola
b7227113be Use modified to check if job should be sent to analytics
It can take several hours for a job to go from pending to
successful/failed state and we need to also send the job with
a changed state, otherwise the analytics will be incorrect.
2020-04-09 15:20:27 +02:00
chris meyers
c00f1505d7 add broadcast websocket status command 2020-04-02 21:46:12 -04:00
Bill Nottingham
96576b0e3d Don't return different fields for smart vs non-smart inventories 2020-03-30 17:15:55 -04:00
chris meyers
d58df0f34a fix sliding window calculation 2020-03-18 16:10:19 -04:00
chris meyers
9e5fe7f5c6 translate Instance hostname to safe analytics name
* More robust translation of Instance hostname to analytics safe name by
replacing all non-alpha numeric characters with _
2020-03-18 16:10:19 -04:00
chris meyers
093d204d19 fix flake8 2020-03-18 16:10:19 -04:00
chris meyers
d6594ab602 add broadcast websocket metrics
* Gather brroadcast websocket metrics and push them into redis every
configurable seconds.
* Pop metrics from redis in web view layer to display via the api on
demand
2020-03-18 16:10:18 -04:00
softwarefactory-project-zuul[bot]
e210ee4077 Merge pull request #6301 from gamuniz/catch_analytics_failure
rework the gather() to always delete the leftover directories

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul
2020-03-16 20:45:36 +00:00
Gabe Muniz
80234c5600 rework the tar to always delete the leftover directories 2020-03-16 19:54:15 +00:00
AlanCoding
7d0b207571 Organization on JT as read-only field
Set JT.organization with value from its project

Remove validation requiring JT.organization

Undo some of the additional org definitions in tests

Revert some tests no longer needed for feature

exclude workflow approvals from unified organization field

revert awxkit changes for providing organization

Roll back additional JT creation permission requirement

Fix up more issues by persisting organization field when project is removed

Restrict project org editing, logging, and testing

Grant removed inventory org admin permissions in migration

Add special validate_unique for job templates
  this deals with enforcing name-organization uniqueness

Add back in special message where config is unknown
  when receiving 403 on job relaunch

Fix logical and performance bugs with data migration

within JT.inventory.organization make-permission-explicit migration

remove nested loops so we do .iterator() on JT queryset

in reverse migration, carefully remove execute role on JT
  held by org admins of inventory organization,
  as well as the execute_role holders

Use current state of Role model in logic, with 1 notable exception
  that is used to filter on ancestors
  the ancestor and descentent relationship in the migration model
    is not reliable
  output of this is saved as an integer list to avoid future
    compatibility errors

make the parents rebuilding logic skip over irrelevant models
  this is the largest performance gain for small resource numbers
2020-03-12 15:45:46 -04:00
Ben Thomasson
fc402aff29 Bumps the version of config.json to 1.1 2020-02-28 15:28:31 -05:00
Christian Adams
f93ca814ac Add total licensed instances to analytics 2020-02-27 10:42:25 -05:00
Bill Nottingham
71ef7cdec1 Use AWX_TASK_ENV when connecting to Red Hat services 2020-02-07 14:29:42 -05:00
Bill Nottingham
95960c8c14 Fix collection of free_instances from /api/v2/config 2020-01-13 15:31:21 -05:00
Nikhil Jain
0d4e6d7e0b do the sum once and store it in a variable which will be reused 2019-12-13 13:21:49 +05:30
Nikhil Jain
9ae038868c adding instance consumed and remaining capacity to metrics 2019-12-13 13:21:49 +05:30
Christian Adams
05e6f4ab3c Add User-Agent to analytics upload POST 2019-11-25 22:22:05 -05:00
Graham Mainwaring
7dd241fcff Add a --dry-run option to gather analytics locally, even if analytics is disabled in settings. 2019-10-17 13:54:13 -04:00
Christian Adams
9d81b00772 have analytics collections verify with system trusted CA list 2019-10-16 13:32:06 -04:00
Christian Adams
844b8a803f update frequency of collection for automation analytics 2019-10-09 21:10:48 -04:00
Christian Adams
6309c0a426 Upload using RH cred settings 2019-09-11 13:24:09 -04:00
Ben Thomasson
10d53637ad Changes uploader to use the insights api directly 2019-09-06 10:58:35 -04:00
softwarefactory-project-zuul[bot]
8ae93848db Merge pull request #4564 from rooftopcellist/manifest_destiny
add collection version tracker & query info

Reviewed-by: https://github.com/apps/softwarefactory-project-zuul
2019-08-27 15:09:17 +00:00
Christian Adams
78c0d531bc Adds versions to analytics collectors and manifest file.
- adds 'query_info.json' to contain collection metadata
  - adds 'manifest.json' to contain collection file version info
2019-08-27 10:14:14 -04:00
Christian Adams
cd3645eb4d Send job & org data 2019-08-26 10:22:07 -04:00
Christian Adams
11a6e98230 Add pending jobs and system level job status to metrics 2019-07-25 14:19:20 -04:00
Ryan Petrello
f2996f1c89 include instance hostnames in metrics endpoint 2019-07-24 13:41:56 -04:00
Christian Adams
e72b2fac6d Delete collection tarball when no longer needed
* Delete after shipping it
   * Delete when ship() fails
2019-07-01 11:11:44 -04:00
Christian Adams
4d7c49372c Fix collection datetime for isolated instance info
* 'last_isolated_check' was a non JSON-serializable object and needed to be a str
2019-06-27 13:31:10 -04:00
Christian Adams
a5c057cc18 Fixes analytics & metrics instance specific job counts 2019-06-19 11:32:05 -04:00
Christian Adams
a727de184b Fix analytics job counts by ordering query
- More Info: https://github.com/ansible/awx/issues/4014
2019-06-06 10:45:26 -04:00
softwarefactory-project-zuul[bot]
bd50e5d6a8 Merge pull request #3896 from rooftopcellist/collection_no_reset
collect events based on last ship, not collection

Reviewed-by: https://github.com/softwarefactory-project-zuul[bot]
2019-05-17 15:32:48 +00:00
Christian Adams
0c4925afe8 rm extra collection error line 2019-05-16 14:48:41 -04:00
Christian Adams
d0e160a037 collect events based on last ship, not collection 2019-05-14 17:14:44 -04:00
Ryan Petrello
27d74528c0 Merge pull request #3494 from rooftopcellist/analytics_csv
analytics table copies formatted as csv now
2019-04-25 14:48:59 -04:00
Ryan Petrello
c9424f9af8 fix a few issues with license counts in /api/v2/metrics/
- switched these to gauges so people can track them over time
- fixed a typo that caused `free_instances` to always be zero
2019-04-24 12:23:07 -04:00
Christian Adams
64ae7a6e45 analytics table copies formatted as csv now 2019-04-23 16:26:14 -04:00
Christian Adams
add8673d7c rename analytics setting 2019-04-17 15:28:15 -04:00
Christian Adams
40393e201f add uuids to ping and metrics 2019-04-12 18:37:40 -04:00
Christian Adams
f49532090b add prom permissions test 2019-04-09 15:12:04 -04:00