Commit Graph
100 Commits
Author SHA1 Message Date
Ryan Petrello bd42dfe474 defer UnifiedJob.result_stdout_text for improved performance
result_stdout_text can be _very_ large - some customers have 5MB+ per
job; querying for this in list contexts results in _very_ large datasets
being read from the database which is very slow.  It's very uncommon to
actually need this column outside of the context of job details, so
defer it.

see: https://github.com/ansible/ansible-tower/issues/7568
2017-09-12 16:35:49 -04:00
Ryan Petrello 4213960ec3 write the scm_revision_output to the project path instead of /tmp
see: https://github.com/ansible/ansible-tower/issues/7558
2017-09-11 17:44:53 -04:00
Ryan Petrello a9c9ecb5ea bind ansible and awx virtualenvs readonly so that jobs can't modify them
see: https://github.com/ansible/ansible-tower/issues/7558
2017-09-11 15:57:35 -04:00
Ryan Petrello a2ca0e6012 add process isolation to project updates
see: https://github.com/ansible/ansible-tower/issues/7506
2017-09-11 15:57:28 -04:00
Ryan Petrello 4cc58a221b fix busted conf unit tests 2017-09-11 11:28:43 -04:00
Ryan Petrello 768c7ba3dc bump azurerm dependencies to support Ansible 2.4
see: https://github.com/ansible/ansible-tower/issues/7470
2017-09-01 15:15:53 -04:00
Ryan Petrello 7192d5c4bb include extra_credentials data in summary fields for JT/Job detail view
see: https://github.com/ansible/ansible-tower/issues/7521
2017-08-30 11:56:24 -04:00
Ryan Petrello beae7ffa20 update the license check command to return license type, not validity
we care about preventing upgradability from AWX installs w/ the open
source license, not general license validity

see: https://github.com/ansible/ansible-tower/issues/6555
2017-08-29 15:25:32 -04:00
Ryan Petrello 0b68ad9b10 properly sanitize conf.settings debug logs
cache.set() and cache.get() arguments are logged when the log level is
DEBUG; this _may_ include plaintext secrets; strip sensitive values
before logging them

see: https://github.com/ansible/ansible-tower/issues/7476
2017-08-21 10:23:39 -04:00
Ryan Petrello b0a1988c29 add required fields for cloudforms credentials
see: https://github.com/ansible/ansible-tower/issues/7462
2017-08-18 16:18:06 -04:00
Ryan Petrello bcd8e13c24 add required fields for gce credentials
see: https://github.com/ansible/ansible-tower/issues/7463
2017-08-18 16:18:06 -04:00
Ryan Petrello fc73bdcc18 add required fields for azure classic credentials
https://github.com/ansible/ansible-tower/issues/7464
2017-08-18 16:18:05 -04:00
Ryan Petrello 5ba76f28ce add required fields for azure credentials
see: https://github.com/ansible/ansible-tower/issues/7465
2017-08-18 16:18:05 -04:00
Ryan Petrello 90b5d98e5c add required fields for network credentials
see: https://github.com/ansible/ansible-tower/issues/7466
2017-08-18 16:18:04 -04:00
Ryan Petrello 50782b9465 add required fields for RHSatellite6 credentials
see: https://github.com/ansible/ansible-tower/issues/7467
2017-08-18 16:18:04 -04:00
Ryan Petrello 438d41c986 make vault_password required for Vault credentials
see: https://github.com/ansible/ansible-tower/issues/7468
2017-08-18 14:10:19 -04:00
Ryan Petrello 9d931e7f7e don't run debug_tree for production inventory imports
the output of the `debug_tree` function is *very* verbose, and is most
useful for people who are debugging inventory import code

pexpect reads large stdout/stderr streams *very slowly*;
when verbosity is set to `DEBUG` for inventory imports, it's not
uncommon for this function to write 50MB+ of data into stderr, causing
pexpect to read over a pseudoterminal for 30+ minutes

see: https://github.com/ansible/ansible-tower/issues/7414#issuecomment-321615104
2017-08-10 16:25:37 -04:00
Ryan Petrello 61aa7c1ad7 return proper stdout size for DB-backed stdout (like system jobs)
see: https://github.com/ansible/ansible-tower/issues/7398
2017-08-07 17:08:25 -04:00
Ryan Petrello 4bc5858ab4 replace PyCrypto usage w/ crytography; ansible no longer bundles it
see: https://github.com/ansible/ansible-tower/issues/7395
2017-08-07 12:19:51 -04:00
Ryan Petrello b4d69cb5c7 don't delete settings that are marked as read_only 2017-08-04 10:06:35 -04:00
Ryan Petrello 7db9b48e9c add a configurable for disabling the auto-generated isolated RSA key
some users won't want to utilize the RSA key we auto-generate for
isolated node SSH access, but will instead want to manage SSH
authentication by hand outside of Tower

see: https://github.com/ansible/ansible-tower/issues/7380
2017-08-03 17:16:28 -04:00
Ryan Petrello b34adcb10d add special case read-only support for isolated public/private keys
We recently made AWX_ISOLATED_PRIVATE_KEY and AWX_ISOLATED_PUBLIC_KEY
read-only so that they're not inadvertently modified and/or deleted
(which would cause isolated task execution to break).  Tower's notion of
a read-only setting, though, is really more like "hard-coded in a .py
file".  What we really need is support for settings that are not
user configurable/changeable, but that we still want to display to users
from the DB.

In leiu of a complicated change to `awx.conf.settings`, this change
provides special support to these two settings so they behave in the
manner we expect.

see: https://github.com/ansible/ansible-tower/issues/7375
2017-08-03 16:41:23 -04:00
Ryan Petrello d09872d13a fix a bug which breaks inventory update stdout downloads
see: https://github.com/ansible/ansible-tower/issues/7363
see: https://github.com/ansible/ansible-tower/issues/7337
2017-08-01 10:48:40 -04:00
Ryan Petrello 63273d8ffd filter Python unicode string prefix from jsonschema validation errors.
see: https://github.com/ansible/ansible-tower/issues/6417
2017-07-31 15:22:19 -04:00
Ryan Petrello 147d795fa0 force python-ldap to create a new SSL context for OPT_X_TLS_REQUIRE_CERT
according to the internet, this is a thing that is necessary for
python-ldap:

https://github.com/rbarrois/python-ldap/commit/71024518dbc9c539a5bab834080e3792ac4dbfe9#diff-1a5a32c6cc1ae9767cb2008703e4534fR41
https://stackoverflow.com/a/29722445
https://stackoverflow.com/a/38136255

see: https://github.com/ansible/ansible-tower/issues/7335
2017-07-31 13:59:36 -04:00
Ryan Petrello 5e15f9e04e add validation errors for certain dependent credential fields
see: https://github.com/ansible/ansible-tower/issues/7323
see: https://github.com/ansible/ansible-tower/issues/7293
see: https://github.com/ansible/ansible-tower/issues/7289
see: https://github.com/ansible/ansible-tower/issues/7292
2017-07-28 16:07:37 -04:00
Ryan Petrello 568986b39f force isolated timeout settings to be positive integers
see: https://github.com/ansible/ansible-tower/issues/7328
2017-07-28 12:24:21 -04:00
Ryan Petrello ad66ff9938 pin kombu to a known working version (from tower 3.1.4)
we're encountering some *really* gnarly bugs with newer version of
celery and kombu; pin to *exactly* what works in 3.1.4 to try to avoid
them, because we can't upgrade celery to 4.0 (where this bug *may* be
fixed)

see: https://github.com/ansible/ansible-tower/issues/6534
see: https://github.com/celery/celery/issues/3712
see: https://github.com/celery/kombu/pull/760
2017-07-28 10:59:06 -04:00
Ryan Petrello 60da24d82f fix a few activity stream bugs related to setting creation/update
* when a setting is created, only create *one* activity stream record
  for the creation, not one for create and another for update (similar
  to https://github.com/ansible/tower/pull/53)
* add code to hide `$encrypted$` activity stream content

see: https://github.com/ansible/ansible-tower/issues/7320
2017-07-27 11:51:10 -04:00
Ryan Petrello cb85038976 filter credential_type__search from related search fields in API v1
see: https://github.com/ansible/ansible-tower/issues/6116
2017-07-26 16:42:16 -04:00
Ryan Petrello c71e8d38b7 improve private passphrase error when it's provided unnecessarily
see: https://github.com/ansible/ansible-tower/issues/7293
2017-07-26 15:04:10 -04:00
Ryan Petrello 2c2e5cadbf don't require a credential for job launch if vault is specified
see: https://github.com/ansible/ansible-tower/issues/7310
2017-07-26 14:35:06 -04:00
Ryan Petrello d3796e81ba disable extraneous activity stream messages for credential creation
see: https://github.com/ansible/ansible-tower/issues/7257
2017-07-25 15:47:55 -04:00
Ryan Petrello 0ce3152e6f fix busted test runs 2017-07-25 12:47:46 -04:00
Ryan Petrello e29492a259 more tower -> awx for task execution and isolated tooling 2017-07-25 10:36:06 -04:00
Ryan Petrello 8ce1421c6a fix tower-expect -> awx-expect for isolated tower builds 2017-07-24 16:03:58 -04:00
Ryan Petrello b0f5d2f82d allow Job Templates to launch with *only* a vault credential
see: https://github.com/ansible/ansible-tower/issues/7252
2017-07-24 11:42:35 -04:00
Ryan Petrello 9f11c008d2 don't allow boolean credential type fields that specify secret
secret doesn't really make sense for boolean values; they can't store
sensitive content because they're just true|false

see: https://github.com/ansible/ansible-tower/issues/6776
2017-07-24 11:41:12 -04:00
Ryan Petrello 6d201c44d9 fix a busted unit test re: tower -> awx 2017-07-24 11:36:11 -04:00
Ryan Petrello 83ac761d81 fix a unit test that periodically fails
if you run this test *a lot*, it fails - seemingly due to a bug in
pexpect itself and how it handles stdout buffering; introducing a bit
of latency seems to make the issue go away

see: #7171
2017-07-21 14:39:55 -04:00
Ryan Petrello a640d6afec improve sanitation of empty credential values to match API v1 behavior
This is mostly backwards compatability to avoid surprises: in 3.1.x
if you submit a field value with `null` or an empty string to
a CharField, it's treated as an empty string (and SSH key validation
is skipped).  For boolean field values (`net.authorize`), `null` and
empty string are coerced to `False`.

see: #7216
see: #7218
2017-07-21 11:25:56 -04:00
Ryan Petrello 198f2e31a4 consolidate tasks.py cleanup for temporary files into a single place
see: #6199
2017-07-20 15:12:06 -04:00
Ryan Petrello 2e659244f7 fix a bug that breaks inventory updates if no credential is supplied
see: #7206
2017-07-20 14:30:41 -04:00
Ryan Petrello d42ea31f75 use a named pipe for isolated secret passthrough (not stdin)
it's not unusual for the secret data we pass into the `run_isolated.yml`
playbook to be quite long, namely because it can contain RSA key
data; by passing this value into the ansible-playbook process using
`vars_prompt`, we're limited by pexpect's tty line limit (which looks
like it caps out around 4k).  Because of this, large payloads are
being truncated and causing job run failures.

this changes the implementation to use a named pipe instead, which
doesn't have the same limitation

see: #7183
2017-07-20 12:42:03 -04:00
Ryan Petrello b98d384671 filter CredentialType OPTIONS *and* help text for kind requirements
see: #7115
see: #6959
2017-07-18 17:51:07 -04:00
Ryan Petrello f4d6c03325 properly copy user owner roles when migrating v1 SSH to SSH + Vault
see: #7050
2017-07-18 15:24:43 -04:00
Ryan Petrello 4b2be9850e copy adhoc events properly for stdout downloads
see: #7100
2017-07-18 12:12:02 -04:00
Ryan Petrello afb307c146 properly validate choices for credential input validation
see: #7119
2017-07-17 16:58:33 -04:00
Ryan Petrello 53259e4d24 properly capture job events for adhoc commands run on isolated instances
see: #7100
2017-07-17 14:51:24 -04:00
Ryan Petrello 9784633728 pin websocket-client to a known working version
websocket-client 0.43.0 has a bug in its https client that breaks
requests:
https://github.com/websocket-client/websocket-client/issues/326

slackclient is affected by this bug:
https://github.com/slackapi/python-slackclient/issues/189

this commits also upgrades slackclient to add proxy support

see: #7088
see: #7093
2017-07-17 12:04:30 -04:00
Ryan Petrello 0b6c43dac0 allow access to insights credentials in /api/v1/
see: #6978
2017-07-17 10:14:29 -04:00
Ryan Petrello 937e90d342 force PENDO_TRACKING_STATE to be readonly for the open source license
see: #6890
2017-07-14 16:39:13 -04:00
Ryan Petrello d8da1dddf5 wrap instance and queue registration in postgres advisory locks
see: #7040
2017-07-14 14:56:14 -04:00
Ryan Petrello 0239cd37fe pin celery to a known version that works
newer celery 3.x seems to a bug which causes forked celery worker
interprocess communication to hang (see: #6534)
2017-07-13 16:24:30 -04:00
Ryan Petrello 58abfc3fde properly copy parent roles when migrating v1 SSH creds to SSH + Vault
see: #7050
2017-07-13 15:02:55 -04:00
Ryan Petrello 2d5c021b72 log task failures when DEBUG = True
this helps a lot for development - otherwise you have to go look an
explanaton in the database for these sorts of failures
2017-07-13 08:24:42 -04:00
Ryan Petrello 80224b791d avoid a race condition in recording deletions in the activity stream
1. You delete something.
2. A signal is generated to record an activity stream deletion.
3. The process of deleting that activity stream deletion attempts to
   look up a related field which has been deleted (in the meantime) via
   a cascade.

see: #6721
see: #7022
2017-07-12 16:39:39 -04:00
Ryan Petrello 5f6ed1bd6c upgrade to the latest version of tacacs_plus 2017-07-12 13:52:37 -04:00
Ryan Petrello 28f44c3ab0 filter Insights credentials from /api/v1/credentials/
see: #6978
see: #6088
2017-07-11 13:55:43 -04:00
Ryan Petrello 09055f9c2f fix a few issues in credential type kind validation
- fix a typo from `network` to `net`
- properly update OPTIONS for CredentialTypes to reflect allowed `kind`
  values for POST/PUT/

see: #6959
2017-07-11 09:43:55 -04:00
Ryan Petrello 6996b16d5a add additional field validation to AWX_TASK_ENV
AWX_TASK_ENV should only allow simple key-value assignment (since we're
using it to set environment variables).

see: #3508
2017-07-10 17:29:38 -04:00
Ryan Petrello 31829038bd prevent creation of custom credential types that != 'cloud || network`
see: #6959
2017-07-10 16:34:25 -04:00
Ryan Petrello 0e29f3617d periodically run orphaned task cleanup as part of the scheduler
Running orphaned task cleanup within its own scheduled task via
celery-beat causes a race-y lock contention between the cleanup task and
the task scheduler.  Unfortunately, the scheduler and the cleanup task
both run at similar intervals, so this race condition is fairly easy to
hit.  At best, it results in situations where the scheduler is
regularly delayed 20s; depending on timing, this can cause situations
where task execution is needlessly delayed a minute+.  At worst, it can
result in situations where the scheduler is never able to schedule
tasks.

This change implements the cleanup as a periodic block of code in the
scheduler itself that tracks its "last run" time in memcached (by
default, it performs a cleanup every 60 seconds)

see: #6534
2017-07-10 15:51:46 -04:00
Ryan Petrello 7d12427497 add a new configurable, PROXY_IP_WHITELIST
implement a whitelist setting that - if populated - will only allow
specific IPs/hostnames to provide custom REMOTE_HOST_HEADERS header
values (i.e., `HTTP_X_FORWARDED_FOR`)

see: #6538
2017-07-10 10:31:45 -04:00
Ryan Petrello 7f286b50b8 don't auto-coerce boolean-like extra_vars for credential injection
ansible itself already does this for you (albeit, with its own set of
quirks): https://github.com/ansible/ansible/issues/11905

see: #6900
2017-07-07 16:11:29 -04:00
Ryan Petrello 5fde6ead42 properly copy prompted vault passwords on job launch
see: #6924
2017-07-07 13:15:17 -04:00
Ryan Petrello 3c2fe5e6db fix a variety of bugs that break vault pass injection for playbook runs
see: #6924
2017-07-07 12:19:31 -04:00
Ryan Petrello 12d41e2deb Support AWX_TASK_ENV injection in task and notification invocations.
This change _only_ injects `AWS_TASK_ENV` into `os.environ`; it's up to
underlying libraries to be good citizens and actually respect things
like `HTTPS_PROXY`.

see: #3508
2017-07-06 13:51:37 -04:00
Ryan Petrello d9fcefa7ad update help text for changes to AWX_PROOT_BASE_PATH
see: #3472
2017-07-05 14:16:14 -04:00
Ryan Petrello 0a5b9c458b standardize tasks.py temporary file paths under a single parameter
see: #3472
2017-07-05 13:50:43 -04:00
Ryan Petrello 7673a6fe49 fix a bug that prevents boolean inputs from being used in injectors
when used as environment variables, boolean credential values are
stringified; when used in extra_vars, they are treated as actual JSON
boolean values (where possible)

see: #6776
2017-07-05 11:21:01 -04:00
Ryan Petrello cd7dc39703 rename "Insights Basic Auth" to "Insights"
see: #6870
2017-07-03 16:25:44 -04:00
Ryan Petrello 1c066d478b substitute "$encrypted$" for actual password on logging config test
see: #6780
2017-07-03 12:14:43 -04:00
Ryan Petrello 2f6d6bfd67 update help text for credential inputs and injectors
see: #6078
2017-07-03 11:42:04 -04:00
Ryan Petrello 0a4c5d9d30 translate labels and help_text for Credential Types (where possible)
see: #6327
2017-07-03 11:42:04 -04:00
Ryan Petrello aaa0ed4c72 add help text for built-in Tower credential types
see: #6561
2017-07-03 11:42:04 -04:00
Ryan Petrello dade5c12a7 fix a bug in the CredentialType field validator that breaks required
see: #6769
2017-06-30 12:27:43 -04:00
Ryan Petrello de84f3cf4a fix a bug in cluster node version comparison 2017-06-30 11:28:04 -04:00
Ryan Petrello 413e8c3bc9 isolated nodes should report their awx version in their heartbeat
see: #6810
2017-06-29 16:55:11 -04:00
Ryan Petrello 611c42f741 fix the make rdb debugging tool 2017-06-29 15:40:09 -04:00
Ryan Petrello ac9bc166a0 properly invoke the byte-compiled isolated job runner in production
see: #5903
2017-06-29 11:11:28 -04:00
Ryan Petrello 405c01a847 more isolated production tinkering
see: #5903
see: #6507
2017-06-29 09:35:26 -04:00
Ryan Petrello a4b00cf485 minor tinkering to isolated key generation 2017-06-28 15:32:31 -04:00
Ryan Petrello b0e51b42d8 fix a bug that prevents unpriveleged users from listing CredentialTypes
see: #6737
2017-06-27 16:47:02 -04:00
Ryan Petrello 65ace1f446 fix a bug in AES -> fernet encryption migration
see: #6747
2017-06-27 15:07:44 -04:00
Ryan Petrello 3000f52a92 install a randomized RSA key for controller -> isolated rampart auth
see: #6507
2017-06-27 10:53:44 -04:00
Ryan Petrello bfb7ad15ae make isolated rampart settings API-configurable
see: #6659
2017-06-26 15:47:14 -04:00
Ryan Petrello 5adc1c603a properly update the heartbeat timestamp for isolated nodes 2017-06-26 11:03:56 -04:00
Ryan Petrello 97562fc4df block job template launches when an inventory is being deleted
see: #4382
2017-06-22 15:33:55 -04:00
Ryan Petrello 14addae813 delete inventories in the background via a celery task
see: #4382
see: #6279
2017-06-22 14:32:33 -04:00
Ryan Petrello 5e908bfc50 more isolated acceptance doc updates 2017-06-22 12:32:20 -04:00
Ryan Petrello 0804b7336c make isolated sdist use the correct filename for unofficial builds 2017-06-22 10:51:29 -04:00
Ryan Petrello 45bdd9f747 improve readability of the honcho console logs in the dev environment
* colorize uwsgi and celery logs; DEBUG lines are green, WARN lines
  are yellow, ERROR lines (and tracebacks) are red
* pretty-print fact callback receiver JSON
* simplify the uwsgi log format so it's more legible
2017-06-22 09:19:32 -04:00
Ryan Petrello 39525316ac prefix all /api/v1/ view descriptions with a deprecation message
additionally, fix a bug in /api/v1/credentials/ view descriptions that
causes v2 fields to display (see: #6116)
2017-06-21 16:27:07 -04:00
Ryan Petrello e832a56506 fix a bug in team-based credential validation
when a credential is created with `team` in the payload, set the
credential's `organization` *prior* to validation so that we don't miss
organization-oriented validators (like the org + name + kind unique
validation)

see: #3303
2017-06-21 14:25:51 -04:00
Ryan Petrello b8e0c91bd4 store stderr on successful inventory syncs
see: #4823
2017-06-21 10:20:50 -04:00
Ryan Petrello c35d83df13 fix a race condition in isolated stdout processing 2017-06-19 16:59:03 -04:00
Ryan Petrello 2e608f4eac remove ANSIBLE_USE_VENV and TOWER_USE_VENV
modern Tower deployments (both production and dev container-based)
always build ansible and Tower in separate, distinct virtualenvs, not
the global site-packages
2017-06-19 16:11:15 -04:00
Ryan Petrello 2ba9e56033 Include the Tower venv in the isolated bubblewrapped arguments.
This moves the container-based code location and venvs.
The goal here is that the paths of Tower source for isolated
vs normal nodes matches (both in prod and local development) so that we
don't have to add a bunch of additional bwrap argument logic for
<location-of-isolated-tower-venv>.
2017-06-19 15:52:54 -04:00
Ryan Petrello 1ea03aa4c9 more isolated task execution tweaking
* set a more reasonable default `AWX_ISOLATED_CHECK_INTERVAL`
* make manual cancellation work for high values of
  `AWX_ISOLATED_CHECK_INTERVAL`
* remove the `/tmp/ansible_tower/jobs/` symlink directory

see: #6616
2017-06-16 15:37:07 -04:00