[4.6 backport] Feature indirect host counting (#15802) (#6858)

* Feature indirect host counting (#15802)

* AAP-37282 Add parse JQ data and test it for a `job` object in isolation (#15774)

* Add jq dependency

* Add file in progress

* Add license for jq

* Write test and get it passing

* Successfully test collection of `event_query.yml` data (#15761)

* Callback plugin method from cmeyers adapted to global collection list

Get tests passing

Mild rebranding

Put behind feature flag, flip true in dev

Add noqa flag

* Add missing wait_for_events

* feat: try grabbing query files from artifacts directory (#15776)

* Contract changes for the event_query collection callback plugin (#15785)

* Minor import changes to collection processing in callback plugin

* Move agreed location of event_query file

* feat: remaining schema changes for indirect host audits (#15787)

* Re-organize test file and move artifacts processing logic to callback (#15784)

* Rename the indirect host counting test file

* Combine artifacts saving logic

* Connect host audit model to jq logic via new task

* Add unit tests for indirect host counting (#15792)

* Do not get django flags from database (#15794)

* Document, implement, and test remaining indirect host audit fields (#15796)

* Document, implement, and test remaining indirect host audit fields

* Fix hashing

* AAP-39559 Wait for all event processing to finish, add fallback task (#15798)

* Wait for all event processing to finish, add fallback task

* Add flag check to periodic task

* feat: cleanup of old indirect host audit records (#15800)

* By default, do not count indirect hosts (#15801)

* By default, do not count indirect hosts

* Fix copy paste goof

* Fix linter issue from base branch

* prevent multiple tasks from processing the same job events, prevent p… (#15805)

prevent multiple tasks from processing the same job events, prevent periodic task from spawning another task per job

* Fix typos and other bugs found by Pablo review

* fix: rely on resolved_action instead of task, adapt to proposed query… (#15815)

* fix: rely on resolved_action instead of task, adapt to proposed query structure

* tests: update indirect host tests

* update remaining queries to new format

* update live test

* Remove polling loop for job finishing event processing (#15811)

* Remove polling loop for job finishing event processing

* Make awx/main/tests/live dramatically faster (#15780)

* AAP-37282 Add parse JQ data and test it for a `job` object in isolation (#15774)

* Add jq dependency

* Add file in progress

* Add license for jq

* Write test and get it passing

* Successfully test collection of `event_query.yml` data (#15761)

* Callback plugin method from cmeyers adapted to global collection list

Get tests passing

Mild rebranding

Put behind feature flag, flip true in dev

Add noqa flag

* Add missing wait_for_events

* feat: try grabbing query files from artifacts directory (#15776)

* Contract changes for the event_query collection callback plugin (#15785)

* Minor import changes to collection processing in callback plugin

* Move agreed location of event_query file

* feat: remaining schema changes for indirect host audits (#15787)

* Re-organize test file and move artifacts processing logic to callback (#15784)

* Rename the indirect host counting test file

* Combine artifacts saving logic

* Connect host audit model to jq logic via new task

* Document, implement, and test remaining indirect host audit fields (#15796)

* AAP-39559 Wait for all event processing to finish, add fallback task (#15798)

* Wait for all event processing to finish, add fallback task

* Add flag check to periodic task

* feat: cleanup of old indirect host audit records (#15800)

* prevent multiple tasks from processing the same job events, prevent p… (#15805)

prevent multiple tasks from processing the same job events, prevent periodic task from spawning another task per job

* Remove polling loop for job finishing event processing (#15811)

* Make awx/main/tests/live dramatically faster (#15780)

* reorder migrations to allow indirect instances backport

* cleanup for rebase and merge into devel

---------

Co-authored-by: Peter Braun <pbraun@redhat.com>
Co-authored-by: jessicamack <jmack@redhat.com>
Co-authored-by: Peter Braun <pbranu@redhat.com>

requirements/requirements.in

@@ -1,6 +1,7 @@
 aiohttp>=3.11.6 # CVE-2024-52304
 ansiconv==1.0.0  # UPGRADE BLOCKER: from 2013, consider replacing instead of upgrading
 ansible-runner==2.4.0
+jq  # used for indirect host counting feature
 asciichartpy
 asn1
 azure-identity

requirements/requirements.txt

@@ -30,6 +30,7 @@ asgiref==3.7.2
     #   channels-redis
     #   daphne
     #   django
+    #   django-ansible-base
     #   django-cors-headers
 asn1==2.7.0
     # via -r /awx_devel/requirements/requirements.in
@@ -164,7 +165,9 @@ django-crum==0.7.9
 django-extensions==3.2.3
     # via -r /awx_devel/requirements/requirements.in
 django-flags==5.0.13
-    # via -r /awx_devel/requirements/requirements.in
+    # via
+    #   -r /awx_devel/requirements/requirements.in
+    #   django-ansible-base
 django-guid==3.2.1
     # via -r /awx_devel/requirements/requirements.in
 django-oauth-toolkit==1.7.1
@@ -266,6 +269,8 @@ jmespath==1.0.1
     # via
     #   boto3
     #   botocore
+jq==1.8.0
+    # via -r /awx_devel/requirements/requirements.in
 json-log-formatter==0.5.2
     # via -r /awx_devel/requirements/requirements.in
 jsonschema==4.21.1
@@ -412,6 +417,7 @@ pygerduty==0.38.3
 pyjwt[crypto]==2.8.0
     # via
     #   adal
+    #   django-ansible-base
     #   msal
     #   social-auth-core
     #   twilio
@@ -474,6 +480,7 @@ requests==2.32.3
     #   -r /awx_devel/requirements/requirements.in
     #   adal
     #   azure-core
+    #   django-ansible-base
     #   django-oauth-toolkit
     #   kubernetes
     #   msal
@@ -533,6 +540,7 @@ sqlparse==0.5.0
     # via
     #   -r /awx_devel/requirements/requirements.in
     #   django
+    #   django-ansible-base
 tacacs-plus==1.0
     # via -r /awx_devel/requirements/requirements.in
 tempora==5.5.1
@@ -572,6 +580,7 @@ urllib3==1.26.20
     # via
     #   -r /awx_devel/requirements/requirements.in
     #   botocore
+    #   django-ansible-base
     #   kubernetes
     #   requests
 uwsgi==2.0.28

requirements/requirements_dev.txt

@@ -30,3 +30,4 @@ pip>=21.3,<=24.0 # PEP 660 – Editable installs for pyproject.toml based builds
 debugpy
 remote-pdb
 sdb
+

requirements/requirements_git.txt

@@ -1,4 +1,4 @@
 git+https://github.com/ansible/system-certifi.git@devel#egg=certifi
 # Remove pbr from requirements.in when moving ansible-runner to requirements.in
 git+https://github.com/ansible/python3-saml.git@devel#egg=python3-saml
-django-ansible-base @ git+ssh://git@github.com/ansible-automation-platform/django-ansible-base@stable-2.5#egg=django-ansible-base[rest-filters,jwt_consumer,resource-registry,rbac,feature-flags]
+django-ansible-base @ git+https://github.com/alancoding/django-ansible-base@sqlparse#egg=django-ansible-base[rest-filters,jwt_consumer,resource-registry,rbac,feature-flags]