7d30dff075
* AAP-37282 Add parse JQ data and test it for a `job` object in isolation (#15774) * Add jq dependency * Add file in progress * Add license for jq * Write test and get it passing * Successfully test collection of `event_query.yml` data (#15761) * Callback plugin method from cmeyers adapted to global collection list Get tests passing Mild rebranding Put behind feature flag, flip true in dev Add noqa flag * Add missing wait_for_events * feat: try grabbing query files from artifacts directory (#15776) * Contract changes for the event_query collection callback plugin (#15785) * Minor import changes to collection processing in callback plugin * Move agreed location of event_query file * feat: remaining schema changes for indirect host audits (#15787) * Re-organize test file and move artifacts processing logic to callback (#15784) * Rename the indirect host counting test file * Combine artifacts saving logic * Connect host audit model to jq logic via new task * Add unit tests for indirect host counting (#15792) * Do not get django flags from database (#15794) * Document, implement, and test remaining indirect host audit fields (#15796) * Document, implement, and test remaining indirect host audit fields * Fix hashing * AAP-39559 Wait for all event processing to finish, add fallback task (#15798) * Wait for all event processing to finish, add fallback task * Add flag check to periodic task * feat: cleanup of old indirect host audit records (#15800) * By default, do not count indirect hosts (#15801) * By default, do not count indirect hosts * Fix copy paste goof * Fix linter issue from base branch * prevent multiple tasks from processing the same job events, prevent p… (#15805) prevent multiple tasks from processing the same job events, prevent periodic task from spawning another task per job * Fix typos and other bugs found by Pablo review * fix: rely on resolved_action instead of task, adapt to proposed query… (#15815) * fix: rely on resolved_action instead of task, adapt to proposed query structure * tests: update indirect host tests * update remaining queries to new format * update live test * Remove polling loop for job finishing event processing (#15811) * Remove polling loop for job finishing event processing * Make awx/main/tests/live dramatically faster (#15780) * AAP-37282 Add parse JQ data and test it for a `job` object in isolation (#15774) * Add jq dependency * Add file in progress * Add license for jq * Write test and get it passing * Successfully test collection of `event_query.yml` data (#15761) * Callback plugin method from cmeyers adapted to global collection list Get tests passing Mild rebranding Put behind feature flag, flip true in dev Add noqa flag * Add missing wait_for_events * feat: try grabbing query files from artifacts directory (#15776) * Contract changes for the event_query collection callback plugin (#15785) * Minor import changes to collection processing in callback plugin * Move agreed location of event_query file * feat: remaining schema changes for indirect host audits (#15787) * Re-organize test file and move artifacts processing logic to callback (#15784) * Rename the indirect host counting test file * Combine artifacts saving logic * Connect host audit model to jq logic via new task * Document, implement, and test remaining indirect host audit fields (#15796) * Document, implement, and test remaining indirect host audit fields * Fix hashing * AAP-39559 Wait for all event processing to finish, add fallback task (#15798) * Wait for all event processing to finish, add fallback task * Add flag check to periodic task * feat: cleanup of old indirect host audit records (#15800) * prevent multiple tasks from processing the same job events, prevent p… (#15805) prevent multiple tasks from processing the same job events, prevent periodic task from spawning another task per job * Remove polling loop for job finishing event processing (#15811) * Remove polling loop for job finishing event processing * Make awx/main/tests/live dramatically faster (#15780) * temp * remove test * reorder migrations to allow indirect instances backport * cleanup for rebase and merge into devel --------- Co-authored-by: Peter Braun <pbraun@redhat.com> Co-authored-by: jessicamack <jmack@redhat.com> Co-authored-by: Peter Braun <pbranu@redhat.com>
Dependency Management
The requirements.txt file is generated from requirements.in and requirements_git.txt, using pip-tools and pip-compile.
How To Use
Commands should be run in the awx container from inside the ./requirements directory of the awx repository.
Upgrading or Adding Select Libraries
If you need to add or upgrade one targeted library, then modify requirements.in,
then run the script:
./updater.sh run
Upgrading Unpinned Dependency
If you require a new version of a dependency that does not have a pinned version
for a fix or feature, pin a minimum version in requirements.in and run ./updater.sh run. For example,
replace the line asgi-amqp with asgi-amqp>=1.1.4, and consider leaving a
note.
Then next time that a general upgrade is performed, the minimum version specifiers
can be removed, because *.txt files are upgraded to latest.
Upgrading Dependencies
You can upgrade (pip-compile --upgrade) the dependencies by running
./updater.sh upgrade.
Licenses and Source Files
If any library has a change to its license with the upgrade, then the license for that library
inside of licenses needs to be updated.
For libraries that have source distribution requirements (LGPL as an example), a tarball of the library is kept along with the license. To download the PyPI tarball, you can run this command:
pip download <pypi library name> -d licenses/ --no-binary :all: --no-deps
Make sure to delete the old tarball if it is an upgrade.
UPGRADE BLOCKERs
Anything pinned in *.in files involves additional manual work in
order to upgrade. Some information related to that work is outlined here.
django-oauth-toolkit
Versions later than 1.4.1 throw an error about id_token_id, due to the OpenID Connect work that was done in https://github.com/jazzband/django-oauth-toolkit/pull/915. This may be fixable by creating a migration on our end?
pip, setuptools and setuptools_scm
If modifying these libraries make sure testing with the offline build is performed to confirm they are functionally working. Versions need to match the versions used in the pip bootstrapping step in the top-level Makefile.
Verify ansible-runner's build dependency doesn't conflict with the changes made.
cryptography
If modifying this library make sure testing with the offline build is performed to confirm it is functionally working.
Library Notes
pexpect
Version 4.8 makes us a little bit nervous with changes to searchwindowsize https://github.com/pexpect/pexpect/pull/579/files
Pin to pexpect==4.7.x until we have more time to move to 4.8 and test.