9d9c125e47
* feat: 38589 GitHub App Authentication (#15807) * feat: 38589 GitHub App Authentication Allows both git@<personal-token> and x-access-token@<github-access-token> when authenticating using git. This allows GitHub App tokens to work without interfering with existing authentication types. --------- Co-authored-by: Jake Jackson <thedoubl3j@Jakes-MacBook-Pro.local> * revert change made to allow UI to accept x-access-token, just use htt… (#15851) revert change made to allow UI to accept x-access-token, just use https:// instead * Add Github dep for new cred support if used (#15850) * Add pygithub for new app token support * fixed git requirements file with new * added new github dep and relevant deps it needs * add required licenses * Add artifacts to satisfy license check * Remove duplicated license --------- Co-authored-by: Andrea Restle-Lay <arestlel@redhat.com> Co-authored-by: Alan Rominger <arominge@redhat.com> * Remove deps update it came with the cherry-pick and is not needed in this version Remove unneeded deps updates from requirements.in Remove point to awx-plugins as it is not needed in tower * Add a credential plugin that uses GitHub Apps to get tokens * Add github app tests * Ran requirements updater script Ran black on github_app_test to fix formatting issue Add scm_github_app to managed credentials Ran updater script to reflect new deps Added github app info to def build_passwords in jobs.py, cred now appears in credential types Update ManagedCredentialType for GitHub App to match what we have in awx-plugins Update inputs in maManagedCredentialType to github_app_inputs to communicate with awx/main/credential_plugins/github_app.py Revert incorrect change in ManagedCredentialType, change github_app_lookup to call inputs instead of github_app_inputs Updated namespace to github_app_lookup to agree with nomenclature used in the rest of the implementation and to resolve failing API test Remove import pointing to awx plugins and update to point to credential_plugins Remove references to gh_app_plugin_mod and change to github_app Remove from awx_plugins.interfaces._temporary_private_django_api import ( # noqa: WPS436 to resolve failing test Remove flake8 typing & typing references that do not exist in this version of Tower Remove references in jobs.py and __init__.py since this is an external cred type and registered it in setup.cfg instead Remove blank line REvise name in cfg from github_app_lookup to github_app to see if that ifxes module not found error Revise first declaration of github_app to agree with file name to see if that resolves issue Rename line 174 to agree with what's in config Fix reference to github_app_lookup to github_app Linters compliaining about the github_app in __all__ not being defined, renamed to see if that aligns them Fix naming in test to correspond to naming of cred type Update naming to be more specific and add blank line to setup.cfg Remove __all__ from githubapp.py to satisfy linters Revert formatting change since it is not needed in this repository * Add blank line at the end of requirements.in --------- Co-authored-by: Andrea Restle-Lay <andrearestlelay@gmail.com> Co-authored-by: Jake Jackson <thedoubl3j@Jakes-MacBook-Pro.local> Co-authored-by: Jake Jackson <jljacks93@gmail.com> Co-authored-by: Andrea Restle-Lay <arestlel@redhat.com> Co-authored-by: Alan Rominger <arominge@redhat.com>
Dependency Management
The requirements.txt file is generated from requirements.in and requirements_git.txt, using pip-tools and pip-compile.
How To Use
Commands should be run in the awx container from inside the ./requirements directory of the awx repository.
Upgrading or Adding Select Libraries
If you need to add or upgrade one targeted library, then modify requirements.in,
then run the script:
./updater.sh run
Upgrading Unpinned Dependency
If you require a new version of a dependency that does not have a pinned version
for a fix or feature, pin a minimum version in requirements.in and run ./updater.sh run. For example,
replace the line asgi-amqp with asgi-amqp>=1.1.4, and consider leaving a
note.
Then next time that a general upgrade is performed, the minimum version specifiers
can be removed, because *.txt files are upgraded to latest.
Upgrading Dependencies
You can upgrade (pip-compile --upgrade) the dependencies by running
./updater.sh upgrade.
Licenses and Source Files
If any library has a change to its license with the upgrade, then the license for that library
inside of licenses needs to be updated.
For libraries that have source distribution requirements (LGPL as an example), a tarball of the library is kept along with the license. To download the PyPI tarball, you can run this command:
pip download <pypi library name> -d licenses/ --no-binary :all: --no-deps
Make sure to delete the old tarball if it is an upgrade.
UPGRADE BLOCKERs
Anything pinned in *.in files involves additional manual work in
order to upgrade. Some information related to that work is outlined here.
django-split-settings
When we attemed to upgrade past 1.0.0 the build process in GitHub failed on the docker build step with the following error:
#19 [builder 12/12] RUN AWX_SETTINGS_FILE=/dev/null SKIP_SECRET_KEY_CHECK=yes SKIP_PG_VERSION_CHECK=yes /var/lib/awx/venv/awx/bin/awx-manage collectstatic --noinput --clear
#19 sha256:cd5adb08d3aa92504348338475db9f8bb820b4f67ba5b75edf9ae7554175f1d0
#19 0.725 Traceback (most recent call last):
#19 0.725 File \"/var/lib/awx/venv/awx/bin/awx-manage\", line 8, in <module>
#19 0.726 sys.exit(manage())
#19 0.726 File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py\", line 178, in manage
#19 0.726 prepare_env()
#19 0.726 File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/__init__.py\", line 133, in prepare_env
#19 0.726 if not settings.DEBUG: # pragma: no cover
#19 0.726 File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/conf/__init__.py\", line 82, in __getattr__
#19 0.726 self._setup(name)
#19 0.726 File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/conf/__init__.py\", line 69, in _setup
#19 0.726 self._wrapped = Settings(settings_module)
#19 0.726 File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/django/conf/__init__.py\", line 170, in __init__
#19 0.726 mod = importlib.import_module(self.SETTINGS_MODULE)
#19 0.726 File \"/usr/lib64/python3.9/importlib/__init__.py\", line 127, in import_module
#19 0.726 return _bootstrap._gcd_import(name[level:], package, level)
#19 0.726 File \"<frozen importlib._bootstrap>\", line 1030, in _gcd_import
#19 0.726 File \"<frozen importlib._bootstrap>\", line 1007, in _find_and_load
#19 0.726 File \"<frozen importlib._bootstrap>\", line 986, in _find_and_load_unlocked
#19 0.726 File \"<frozen importlib._bootstrap>\", line 680, in _load_unlocked
#19 0.726 File \"<frozen importlib._bootstrap_external>\", line 850, in exec_module
#19 0.726 File \"<frozen importlib._bootstrap>\", line 228, in _call_with_frames_removed
#19 0.726 File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/settings/production.py\", line 74, in <module>
#19 0.726 include(settings_file, optional(settings_files), scope=locals())
#19 0.726 File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/split_settings/tools.py\", line 116, in include
#19 0.726 module = module_from_spec(spec) # type: ignore
#19 0.726 File \"<frozen importlib._bootstrap>\", line 562, in module_from_spec
#19 0.726 AttributeError: 'NoneType' object has no attribute 'loader'
#19 ERROR: executor failed running [/bin/sh -c AWX_SETTINGS_FILE=/dev/null SKIP_SECRET_KEY_CHECK=yes SKIP_PG_VERSION_CHECK=yes /var/lib/awx/venv/awx/bin/awx-manage collectstatic --noinput --clear]: exit code: 1
The various versions past 1.0.0 talk about adding and removing support for different python versions so there may be a mismatch in what the versions of the library support vs what is being built inside the container. Ironically, we did not experience the problem on our local containers when running collectstatic so we think it has something to do specifically with the build process.
This issue was not picked up by any existing QE testing, only when building in GitHub.
social-auth-app-django
django-social keeps a list of backends in memory that it gathers
based on the value of settings.AUTHENTICATION_BACKENDS at import time:
c1e2795b00/social_django/utils.py (L13)
Our settings.AUTHENTICATION_BACKENDS can change
dynamically as settings are changed (i.e., if somebody
configures Github OAuth2 integration), so we need to
overwrite this in-memory value at the top of every request so
that we have the latest version
django-oauth-toolkit
Versions later than 1.4.1 throw an error about id_token_id, due to the OpenID Connect work that was done in https://github.com/jazzband/django-oauth-toolkit/pull/915. This may be fixable by creating a migration on our end?
pip, setuptools and setuptools_scm
If modifying these libraries make sure testing with the offline build is performed to confirm they are functionally working. Versions need to match the versions used in the pip bootstrapping step in the top-level Makefile.
cryptography
If modifying this library make sure testing with the offline build is performed to confirm it is functionally working.
Library Notes
pexpect
Version 4.8 makes us a little bit nervous with changes to searchwindowsize https://github.com/pexpect/pexpect/pull/579/files
Pin to pexpect==4.7.x until we have more time to move to 4.8 and test.