External-Mirrors/awx
2
0
Fork 0
mirror of https://github.com/ansible/awx.git synced 2026-02-25 06:56:00 -03:30
Code Issues Packages Projects Releases Wiki Activity

General upgrade of dependencies

Browse Source 
Update licenses for new versions after dependency upgrades

pin pycurl to version that does not break on install

implement new workflow for py2/3 requirements management

require twisted tls extras, resolve service-identity version

Upgrade celery to resolve importlib DeprecationWarning

use flags to resolve the unsafe and cache problems
This commit is contained in:
AlanCoding
2019-04-11 10:49:00 -04:00
parent f174902bb2
commit 2bec5ddb41
13 changed files with 149 additions and 457 deletions
Download Patch File Download Diff File Expand all files Collapse all files

47
requirements/README.md
View File

@@ -1,23 +1,46 @@
The requirements.txt and requirements_ansible.txt files are generated from requirements.in and requirements_ansible.in, respectively, using `pip-tools` `pip-compile`. The following commands should do this if ran inside the tower_tools container.
The requirements.txt and requirements_ansible.txt files are generated from requirements.in and requirements_ansible.in, respectively, using `pip-tools` `pip-compile`. The following commands should do this if ran inside the tools_awx container.
NOTE: before running `pip-compile`, please copy-paste contents in `requirements/requirements_git.txt` to the top of `requirements/requirements.in` and prepend each copied line with `-e `. Later after `requirements.txt` is generated, don't forget to remove all `git+https://github.com...`-like lines from both `requirements.txt` and `requirements.in` (repeat for `requirements_ansible_git.txt` and `requirements_ansible.in`)
At the end of `requirements/requirements.in`, pip and setuptools need to have their versions pinned.
Run these commands from the root of the awx repo. This will produce python 3 requirements files.
```
python3 -m venvpip install pip-tools /buildit
python3 -m venv /buildit
source /buildit/bin/activate
pip install pip-tools
pip install pip --upgrade
pip-compile requirements/requirements.in > requirements/requirements.txt
pip-compile requirements/requirements_ansible.in > requirements/requirements_ansible.txt
pip-compile -U -r --allow-unsafe --output-file requirements/requirements.txt requirements/requirements.in
pip-compile -U -r --allow-unsafe --output-file requirements/requirements_ansible_py3.txt requirements/requirements_ansible.in
```
## Known Issues
The Ansible venv requirements file needs to start with the python 2 version
as a base. Then we can run the tool again to get the python 3 version.
Consult the output of the `diff` command and add a conditional switch in those cases.
* Remove the `-e` from packages of the form `-e git+https://github.com...` in the generated `.txt`. Failure to do so will result in a "bad" RPM and DEB due to the `pip install` laying down a symbolic link with an absolute path from the virtualenv to the git repository that will differ from when the RPM and DEB are build to when the RPM and DEB are installed on a machine. By removing the `-e` the symbolic egg link will not be created and all is well.
```
virtualenv -p python2 /buildit_py2
source /buildit_py2/bin/activate
pip install pip-tools
pip install pip --upgrade
* As of `pip-tools` `1.8.1` `pip-compile` does not resolve packages specified using a git url. Thus, dependencies for things like `dm.xmlsec.binding` do not get resolved and output to `requirements.txt`. This means that:
* can't use `pip install --no-deps` because other deps WILL be sucked in
* all dependencies are NOT captured in our `.txt` files. This means you can't rely on the `.txt` when gathering licenses.
pip-compile -U -r --allow-unsafe --output-file requirements/requirements_ansible.txt requirements/requirements_ansible.in
diff requirements/requirements_ansible_py3.txt requirements/requirements_ansible.txt
rm requirements/requirements_ansible_py3.txt
```
Python 3 exceptions should be added to relevant `requirements_ansible.txt` lines
after version numbers with the syntax of `; python_version < '3'`.
## Licenses and Source Files
If any library has a change to its license with the upgrade, then the license for that library
inside of `docs/licenses` needs to be updated.
For libraries that have source distribution requirements (LGPL as an example),
a tarball of the library is kept along with the license.
To download the PyPI tarball, you can run this command:
```
pip download <pypi library name> -d docs/licenses/ --no-binary :all: --no-deps
```
Make sure to delete the old tarball if it is an upgrade.

4
requirements/requirements.in
View File

@@ -5,8 +5,9 @@ asgiref==1.1.2
azure-keyvault==1.1.0
boto==2.47.0
channels==1.1.8
celery==4.2.1
celery==4.3.0
daphne==1.3.0 # Last before backwards-incompatible channels 2 upgrade
twisted[tls]>=17.1 # from daphne, see https://github.com/django/daphne/pull/257
defusedxml==0.5.0 # py36 support https://github.com/tiran/defusedxml/pull/4
Django==1.11.20
django-auth-ldap==1.7.0
@@ -43,7 +44,6 @@ social-auth-core==3.0.0
social-auth-app-django==2.1.0
requests==2.21.0
requests-futures==0.9.7
service-identity==17.0.0
slackclient==1.1.2
tacacs_plus==1.0
twilio==6.10.4

72
requirements/requirements.txt
View File

@@ -2,31 +2,31 @@
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile requirements/requirements.in
# pip-compile -U -r --allow-unsafe --output-file requirements/requirements.txt requirements/requirements.in
#
adal==1.2.1 # via msrestazure
amqp==2.3.2 # via kombu
amqp==2.4.2 # via kombu
ansible-runner==1.3.4
appdirs==1.4.2
argparse==1.4.0 # via uwsgitop
asgi-amqp==1.1.3
asgiref==1.1.2
asn1crypto==0.24.0 # via cryptography
attrs==18.2.0 # via automat, service-identity, twisted
autobahn==19.2.1 # via daphne
attrs==19.1.0 # via automat, service-identity, twisted
autobahn==19.3.3 # via daphne
automat==0.7.0 # via twisted
azure-common==1.1.18 # via azure-keyvault
azure-common==1.1.19 # via azure-keyvault
azure-keyvault==1.1.0
azure-nspkg==3.0.2 # via azure-keyvault
billiard==3.5.0.5 # via celery
billiard==3.6.0.0 # via celery
boto==2.47.0
celery==4.2.1
certifi==2018.11.29 # via msrest, requests
cffi==1.12.1 # via cryptography
celery==4.3.0
certifi==2019.3.9 # via msrest, requests
cffi==1.12.3 # via cryptography
channels==1.1.8
chardet==3.0.4 # via requests
constantly==15.1.0 # via twisted
cryptography==2.6.1 # via adal, azure-keyvault, pyopenssl
cryptography==2.6.1 # via adal, azure-keyvault, pyopenssl, service-identity
daphne==1.3.0
defusedxml==0.5.0
django-auth-ldap==1.7.0
@@ -44,79 +44,81 @@ django-taggit==0.22.2
django==1.11.20
djangorestframework-yaml==1.0.3
djangorestframework==3.7.7
docutils==0.14 # via python-daemon
future==0.16.0 # via django-radius
hyperlink==18.0.0 # via twisted
idna==2.8 # via hyperlink, requests
hyperlink==19.0.0 # via twisted
idna==2.8 # via hyperlink, requests, twisted
incremental==17.5.0 # via twisted
inflect==2.1.0 # via jaraco.itertools
irc==16.2
isodate==0.6.0 # via msrest, python3-saml
jaraco.classes==2.0 # via jaraco.collections
jaraco.collections==2.0 # via irc, jaraco.text
jaraco.collections==2.0 # via irc
jaraco.functools==2.0 # via irc, jaraco.text, tempora
jaraco.itertools==4.4.1 # via irc
jaraco.itertools==4.4.2 # via irc
jaraco.logging==2.0 # via irc
jaraco.stream==2.0 # via irc
jaraco.text==2.0 # via irc, jaraco.collections
jaraco.text==3.0 # via irc, jaraco.collections
jinja2==2.10.1
jsonpickle==1.1 # via asgi-amqp
jsonschema==2.6.0
kombu==4.2.1 # via asgi-amqp, celery
lxml==4.3.1 # via xmlsec
kombu==4.5.0 # via asgi-amqp, celery
lockfile==0.12.2 # via python-daemon
lxml==4.3.3 # via xmlsec
markdown==2.6.11
markupsafe==1.1.0 # via jinja2
more-itertools==6.0.0 # via irc, jaraco.functools, jaraco.itertools
markupsafe==1.1.1 # via jinja2
more-itertools==7.0.0 # via irc, jaraco.functools, jaraco.itertools
msgpack-python==0.5.6 # via asgi-amqp
msrest==0.6.4 # via azure-keyvault, msrestazure
msrest==0.6.6 # via azure-keyvault, msrestazure
msrestazure==0.6.0 # via azure-keyvault
netaddr==0.7.19 # via pyrad
oauthlib==2.0.6 # via django-oauth-toolkit, requests-oauthlib, social-auth-core
oauthlib==3.0.1 # via django-oauth-toolkit, requests-oauthlib, social-auth-core
ordereddict==1.1
pexpect==4.6.0
pkgconfig==1.4.0 # via xmlsec
pkgconfig==1.5.1 # via xmlsec
prometheus_client==0.6.0
psutil==5.4.3
psycopg2==2.7.3.2
ptyprocess==0.6.0 # via pexpect
pyasn1-modules==0.2.4 # via python-ldap, service-identity
pyasn1-modules==0.2.5 # via python-ldap, service-identity
pyasn1==0.4.5 # via pyasn1-modules, python-ldap, service-identity
pycparser==2.19 # via cffi
pygerduty==0.37.0
pyhamcrest==1.9.0 # via twisted
pyjwt==1.7.1 # via adal, social-auth-core, twilio
pyopenssl==19.0.0 # via service-identity
pyopenssl==19.0.0 # via twisted
pyparsing==2.2.0
pyrad==2.1 # via django-radius
pysocks==1.6.8 # via twilio
python-daemon==2.2.0 # via ansible-runner
python-daemon==2.2.0
python-dateutil==2.7.2
python-ldap==3.1.0 # via django-auth-ldap
python-ldap==3.2.0 # via django-auth-ldap
python-memcached==1.59
python-radius==1.0
python3-openid==3.1.0 # via social-auth-core
python3-saml==1.4.0
pytz==2018.9 # via celery, django, irc, tempora, twilio
pyyaml==5.1 # via djangorestframework-yaml
pytz==2019.1 # via celery, django, irc, tempora, twilio
pyyaml==5.1 # via ansible-runner, djangorestframework-yaml
requests-futures==0.9.7
requests-oauthlib==1.2.0 # via msrest, social-auth-core
requests[security]==2.21.0
service-identity==17.0.0
requests==2.21.0
service-identity==18.1.0 # via twisted
simplejson==3.16.0 # via uwsgitop
six==1.12.0 # via asgi-amqp, asgiref, autobahn, automat, cryptography, django-extensions, irc, isodate, jaraco.classes, jaraco.collections, jaraco.itertools, jaraco.logging, jaraco.stream, pygerduty, pyhamcrest, pyopenssl, pyrad, python-dateutil, python-memcached, slackclient, social-auth-app-django, social-auth-core, tacacs-plus, tempora, twilio, txaio, websocket-client
six==1.12.0 # via ansible-runner, asgi-amqp, asgiref, autobahn, automat, cryptography, django-extensions, irc, isodate, jaraco.classes, jaraco.collections, jaraco.itertools, jaraco.logging, jaraco.stream, pygerduty, pyhamcrest, pyopenssl, pyrad, python-dateutil, python-memcached, slackclient, social-auth-app-django, social-auth-core, tacacs-plus, tempora, twilio, txaio, websocket-client
slackclient==1.1.2
social-auth-app-django==2.1.0
social-auth-core==3.0.0
tacacs_plus==1.0
tempora==1.14 # via irc, jaraco.logging
tempora==1.14.1 # via irc, jaraco.logging
twilio==6.10.4
twisted==18.9.0 # via daphne
twisted[tls]==19.2.0
txaio==18.8.1 # via autobahn
typing==3.6.6 # via django-extensions
urllib3==1.24.3 # via requests
uwsgi==2.0.17
uwsgitop==0.10.0
vine==1.2.0 # via amqp
websocket-client==0.47.0 # via slackclient
vine==1.3.0 # via amqp, celery
websocket-client==0.56.0 # via slackclient
xmlsec==1.3.3 # via python3-saml
zope.interface==4.6.0 # via twisted

1
requirements/requirements_ansible.in
View File

@@ -46,6 +46,7 @@ ncclient==0.6.3
netaddr
# oVirt/RHV
ovirt-engine-sdk-python==4.2.4 # minimum set inside Ansible facts module requirements
pycurl==7.43.0.1 # higher versions will not install without SSL backend specified
# AWX usage
pexpect==4.6.0 # same as AWX requirement
psutil==5.4.3 # same as AWX requirement

93
requirements/requirements_ansible.txt
View File

@@ -2,12 +2,12 @@
# This file is autogenerated by pip-compile
# To update, run:
#
# pip-compile --output-file requirements/requirements_ansible.txt requirements/requirements_ansible.in
# pip-compile -U -r --allow-unsafe --output-file requirements/requirements_ansible.txt requirements/requirements_ansible.in
#
adal==1.2.1 # via msrestazure
appdirs==1.4.3 # via openstacksdk
applicationinsights==0.11.1 # via azure-cli-core
argcomplete==1.9.4 # via azure-cli-core, knack
applicationinsights==0.11.9 # via azure-cli-core
argcomplete==1.9.5 # via azure-cli-core, knack
asn1crypto==0.24.0 # via cryptography
azure-cli-core==2.0.35
azure-cli-nspkg==3.0.2
@@ -42,82 +42,83 @@ azure-mgmt-web==0.41.0
azure-nspkg==2.0.0
azure-storage==0.35.1
backports.ssl-match-hostname==3.5.0.1
bcrypt==3.1.4 # via paramiko
bcrypt==3.1.6 # via paramiko
boto3==1.6.2
boto==2.47.0
botocore==1.9.3 # via boto3, s3transfer
cachetools==3.0.0 # via google-auth
certifi==2018.1.18 # via msrest, requests
cffi==1.11.5 # via bcrypt, cryptography, pynacl
botocore==1.9.23 # via boto3, s3transfer
cachetools==3.1.0 # via google-auth
certifi==2019.3.9 # via msrest, requests
cffi==1.12.3 # via bcrypt, cryptography, pynacl
chardet==3.0.4 # via requests
colorama==0.3.9 # via azure-cli-core, knack
cryptography==2.6.1 # via adal, azure-keyvault, azure-storage, paramiko, pyopenssl, requests-kerberos, requests-ntlm, secretstorage
decorator==4.2.1 # via openstacksdk
deprecation==2.0 # via openstacksdk
colorama==0.4.1 # via azure-cli-core, knack
cryptography==2.6.1 # via adal, azure-keyvault, azure-storage, openstacksdk, paramiko, pyopenssl, requests-kerberos, requests-ntlm
decorator==4.4.0 # via dogpile.cache, openstacksdk
docutils==0.14 # via botocore
dogpile.cache==0.6.5 # via openstacksdk
dogpile.cache==0.7.1 # via openstacksdk
enum34==1.1.6; python_version < '3' # via cryptography, knack, msrest, ovirt-engine-sdk-python
futures==3.2.0; python_version < '3' # via openstacksdk, s3transfer
google-auth==1.6.2
humanfriendly==4.8 # via azure-cli-core
idna==2.6 # via cryptography, requests
ipaddress==1.0.19 # via cryptography, openstacksdk
humanfriendly==4.18 # via azure-cli-core
idna==2.8 # via requests
ipaddress==1.0.22; python_version < '3' # via cryptography, openstacksdk
iso8601==0.1.12 # via keystoneauth1, openstacksdk
isodate==0.6.0 # via msrest
jinja2==2.10.1
jmespath==0.9.3 # via azure-cli-core, boto3, botocore, knack, openstacksdk
jsonpatch==1.21 # via openstacksdk
jmespath==0.9.4 # via azure-cli-core, boto3, botocore, knack, openstacksdk
jsonpatch==1.23 # via openstacksdk
jsonpointer==2.0 # via jsonpatch
keystoneauth1==3.11.2 # via openstacksdk, os-client-config
keystoneauth1==3.14.0 # via openstacksdk
knack==0.3.3 # via azure-cli-core
lxml==4.1.1 # via ncclient, pyvmomi
monotonic==1.4 # via humanfriendly
lxml==4.3.3 # via ncclient, pyvmomi
markupsafe==1.1.1 # via jinja2
monotonic==1.5; python_version < '3' # via humanfriendly
msrest==0.6.1
msrestazure==0.5.0
munch==2.2.0 # via openstacksdk
munch==2.3.2 # via openstacksdk
ncclient==0.6.3
netaddr==0.7.19
netifaces==0.10.6 # via openstacksdk
ntlm-auth==1.0.6 # via requests-credssp, requests-ntlm
oauthlib==2.0.6 # via requests-oauthlib
netifaces==0.10.9 # via openstacksdk
ntlm-auth==1.3.0 # via requests-credssp, requests-ntlm
oauthlib==3.0.1 # via requests-oauthlib
openstacksdk==0.23.0
os-service-types==1.2.0 # via openstacksdk
os-service-types==1.6.0 # via keystoneauth1, openstacksdk
ovirt-engine-sdk-python==4.2.4
packaging==17.1
packaging==19.0
paramiko==2.4.2 # via azure-cli-core, ncclient
pbr==3.1.1 # via keystoneauth1, openstacksdk, os-service-types, stevedore
pbr==5.2.0 # via keystoneauth1, openstacksdk, os-service-types, stevedore
pexpect==4.6.0
psutil==5.4.3
ptyprocess==0.5.2 # via pexpect
pyasn1-modules==0.2.3 # via google-auth
pyasn1==0.4.2 # via paramiko
pycparser==2.18 # via cffi
pycurl==7.43.0.1 # via ovirt-engine-sdk-python
pygments==2.2.0 # via azure-cli-core, knack
pyjwt==1.6.0 # via adal, azure-cli-core
ptyprocess==0.6.0 # via pexpect
pyasn1-modules==0.2.5 # via google-auth
pyasn1==0.4.5 # via paramiko, pyasn1-modules, rsa
pycparser==2.19 # via cffi
pycurl==7.43.0.1
pygments==2.3.1 # via azure-cli-core, knack
pyjwt==1.7.1 # via adal, azure-cli-core
pykerberos==1.2.1 # via requests-kerberos
pynacl==1.2.1 # via paramiko
pyopenssl==17.5.0 # via azure-cli-core, pyvmomi, requests-credssp
pyparsing==2.2.0 # via packaging
pynacl==1.3.0 # via paramiko
pyopenssl==19.0.0 # via azure-cli-core, pyvmomi, requests-credssp
pyparsing==2.4.0 # via packaging
python-dateutil==2.6.1 # via adal, azure-storage, botocore
pyvmomi==6.5
pywinrm[kerberos]==0.3.0
pyyaml==5.1 # via azure-cli-core, knack, openstacksdk, os-client-config
pyyaml==5.1 # via azure-cli-core, knack, openstacksdk
requests-credssp==1.0.2
requests-kerberos==0.12.0 # via pywinrm
requests-ntlm==1.1.0 # via pywinrm
requests-oauthlib==0.8.0 # via msrest
requests==2.20.0
requestsexceptions==1.4.0 # via openstacksdk, os-client-config
requests-oauthlib==1.2.0 # via msrest
requests==2.21.0
requestsexceptions==1.4.0 # via openstacksdk
rsa==4.0 # via google-auth
s3transfer==0.1.13 # via boto3
selectors2==2.0.1 # via ncclient
six==1.11.0 # via azure-cli-core, bcrypt, cryptography, google-auth, isodate, keystoneauth1, knack, munch, ncclient, ntlm-auth, openstacksdk, ovirt-engine-sdk-python, packaging, pynacl, pyopenssl, python-dateutil, pyvmomi, pywinrm, stevedore
stevedore==1.28.0 # via keystoneauth1
tabulate==0.7.7 # via azure-cli-core, knack
six==1.12.0 # via azure-cli-core, bcrypt, cryptography, google-auth, isodate, keystoneauth1, knack, munch, ncclient, openstacksdk, ovirt-engine-sdk-python, packaging, pynacl, pyopenssl, python-dateutil, pyvmomi, pywinrm, stevedore
stevedore==1.30.1 # via keystoneauth1
tabulate==0.8.2 # via azure-cli-core, knack
typing==3.6.6; python_version < '3' # via msrest
urllib3==1.24.3 # via requests
wheel==0.30.0 # via azure-cli-core
xmltodict==0.11.0 # via pywinrm
xmltodict==0.12.0 # via pywinrm
# The following packages are considered to be unsafe in a requirements file:
pip==9.0.1