Commit Graph
100 Commits
Author SHA1 Message Date
Ryan Petrello a15e257b9e fix a unicode bug in the stdout endpoint when ?content_encoding=base64
see: https://github.com/ansible/awx/issues/1042
2018-01-25 15:53:43 -05:00
Ryan Petrello e7ed4811c1 reload the entire celery worker pool when uwsgi reloads the Python app
this is for the development environment only; when uwsgi notices a code
change, it automatically reloads the uwsgi workers; this patch includes
a hook that sends `SIGHUP` to the celery process, causing it to spawn
a new set of workers as well
2018-01-25 14:59:40 -05:00
Ryan Petrello ef80ecd3b6 only import the redbaron library on-demand
redbaron is a library we use to facilitate parsing local settings files;
at _import_ time it generates a parse tree and caches it to disk at
`/tmp`; this process is _really time consuming, and only necessary if
we're actually *using* the library

right now, we're importing this library and paying the penalty
_every_ time we load the awx application
2018-01-25 10:23:44 -05:00
Ryan Petrello 9857c8272e add more tests for weird timezone/DST boundaries in schedules
see: https://github.com/ansible/awx/pull/1024
2018-01-22 14:57:57 -05:00
Ryan Petrello 15906b7e3c support TZID= in schedule rrules
this commit allows schedule `rrule` strings to include local timezone
information via TZID=NNNNN; occurrences are _generated_ in the local
time specific by the user (or UTC, if e.g., DTSTART:YYYYMMDDTHHMMSSZ)
while Schedule.next_run, Schedule.dtstart, and Schedule.dtend will be
stored in the UTC equivalent (i.e., the scheduler will still do math on
"what to run next" based on UTC datetimes).

in addition to this change, there is now a new API endpoint,
`/api/v2/schedules/preview/`, which takes an rrule and shows the next
10 occurrences in local and UTC time.

see: https://github.com/ansible/ansible-tower/issues/823
related: https://github.com/dateutil/dateutil/issues/614
2018-01-22 11:50:00 -05:00
Ryan Petrello ac3f7d0fac fix a bug that breaks workflows w/ a survey password + inventory sync
prior versions of awx did not raise an exception for this scenario
- they simply ignored kwargs that they couldn't accept.  this change is
a sort of middle ground - it ignores them, but gives a clue in the logs
as to why

see: https://github.com/ansible/awx/issues/955
related: https://github.com/ansible/awx/pull/803
2018-01-22 09:41:30 -05:00
Ryan Petrello 43f19cc94b fix another bug that breaks the JT callback process
see: https://github.com/ansible/awx/issues/980
related: 17cd0595d7
2018-01-18 13:17:06 -05:00
Ryan Petrello e5aaeedc43 don't require an IRC password in the notification UI
see: https://github.com/ansible/awx/issues/1010
2018-01-18 09:14:22 -05:00
Ryan Petrello dbb4d2b011 refactor credential injection for builtin types
this cleanups up a _lot_ of code duplication that we have for builtin
credential types. it will allow customers to setup custom inventory
sources that utilize builtin credential types (e.g., a custom inventory
script that could use an AzureRM credential)

see: https://github.com/ansible/ansible-tower/issues/7852
2018-01-17 16:50:28 -05:00
Ryan Petrello 838b793704 fix a minor unicode handling bug in project names
see: https://github.com/ansible/ansible-tower/issues/7853
2018-01-17 13:37:06 -05:00
Ryan Petrello 17cd0595d7 fix a bug which broke the callback plugin launch process
see: https://github.com/ansible/awx/issues/980
2018-01-17 11:28:13 -05:00
Ryan Petrello 51f7907a01 optimize OutputEventFilter for large stdout streams
update our event data search algorithm to be a bit lazier in event data
discovery; this drastically improves processing speeds for stdout >5MB

see: https://github.com/ansible/awx/issues/417
2018-01-16 14:41:35 -05:00
Ryan Petrello 4a2ca20b60 fix a minor bug in the JT launch related to support for zero credentials
see: https://github.com/ansible/ansible-tower/issues/7841
2018-01-12 11:37:33 -05:00
Ryan Petrello a5007ccd41 fix a unicode handling bug
see: https://github.com/ansible/ansible-tower/issues/7843
related: https://github.com/ansible/awx/pull/807
2018-01-10 15:56:31 -05:00
Ryan Petrello b01b229fea add an example for custom virtualenv setup in containers 2018-01-10 13:48:55 -05:00
Ryan Petrello 4bb2b5768e properly compose stdout downloads that contain unicode 2018-01-09 23:52:02 -05:00
Ryan Petrello c0a641ed52 properly handle unicode for isolated job buffers
from: https://docs.python.org/2/library/stringio.html#module-cStringIO
"Unlike the StringIO module, this module is not able to accept Unicode
strings that cannot be encoded as plain ASCII strings."

see: https://github.com/ansible/ansible-tower/issues/7846
2018-01-09 23:46:17 -05:00
Ryan Petrello 1e8c89f536 implement support for per-playbook/project/org virtualenvs
see: https://github.com/ansible/awx/issues/34
2018-01-09 22:47:01 -05:00
Ryan Petrello 60743d6ba6 add the vault_id to the response payload on the JT launch endpoint
see: https://github.com/ansible/awx/issues/914
2018-01-04 15:13:48 -05:00
Ryan Petrello f28f7c6184 refactor job event signal generation code 2018-01-04 12:50:12 -05:00
Ryan Petrello 1494c8395b update websockets docs to reflect new event groups 2018-01-04 11:40:15 -05:00
Ryan Petrello 2691e1d707 make the fact caching plugin fail more gracefully for large payloads
related: https://github.com/ansible/ansible/pull/34424
2018-01-04 11:33:47 -05:00
Ryan Petrello c6b6a3ad89 send and subcribe to summary events for all new events 2018-01-03 09:09:45 -05:00
Ryan Petrello 2bd656e61d calculate stdout download length using the ORM, not raw SQL 2018-01-03 09:09:44 -05:00
Ryan Petrello 35b8e40d3c remove deprecation from the stdout endpoint; text downloads still use it 2018-01-03 09:09:44 -05:00
Ryan Petrello c4d901bf2c add functional API tests for deprecated job event stdout composition
see: https://github.com/ansible/awx/issues/200
2018-01-03 09:09:44 -05:00
Ryan Petrello 1369f72885 add new API endpoints and websocket emit for new job event types
see: https://github.com/ansible/awx/issues/200
2018-01-03 09:09:44 -05:00
Ryan Petrello 0b30e7907b change stdout composition to generate from job events on the fly
this approach totally removes the process of reading and writing stdout
files on the local file system at settings.JOBOUTPUT_ROOT when jobs are
run; now stdout content is only written on-demand as it's fetched for
the deprecated `stdout` endpoint

see: https://github.com/ansible/awx/issues/200
2018-01-03 09:09:43 -05:00
Ryan Petrello fc94b3a943 generalize stdout event processing to emit events for *all* job types
* introduces three new models: `ProjectUpdateEvent`,
  `InventoryUpdateEvent`, and `SystemJobEvent`
* simplifies the stdout callback management in `tasks.py` - now _all_
  job run types capture and emit events to the callback receiver
* supports stdout reconstruction from events for stdout downloads for
  _all_ job types
* configures `ProjectUpdate` runs to configure the awx display callback
  (so we can capture real playbook events for `project_update.yml`)
* ProjectUpdate, InventoryUpdate, and SystemJob runs no longer write
  text blobs to the deprecated `main_unifiedjob.result_stdout_text` column

see: https://github.com/ansible/awx/issues/200
2018-01-03 09:09:42 -05:00
Ryan Petrello ea4cd99003 fix a few tests caused by fallout between 3.2.2 bugs and 3.3 multicred 2017-12-13 14:02:25 -05:00
Ryan Petrello d35bfafcf5 more test cleanup from 3.2.2 merge 2017-12-13 13:14:22 -05:00
Ryan Petrello a978d094b4 fix a number of failing unit tests related to the 3.2.2 merge 2017-12-13 13:03:17 -05:00
Ryan Petrello 202161f090 move legacy UnifiedJob stdout data to a separate unmanaged model
This data often (in the case of inventory updates) represents large data
blobs (5+MB per job run).  Storing it on the polymorphic base class
table, `main_unifiedjob`, causes it to be automatically fetched on every
query (and every polymorphic join) against that table, which can result
in _very_ poor performance for awx across the board.  Django offers
`defer()`, but it's quite complicated to sprinkle this everywhere (and
easy to get wrong/introduce side effects related to our RBAC and usage
of polymorphism).

This change moves the field definition to a separate unmanaged model
(which references the same underlying `main_unifiedjob` table) and adds
a proxy for fetching the data as needed

see https://github.com/ansible/awx/issues/200
2017-12-12 18:16:19 -05:00
Ryan Petrello 8c90d36290 add some more tests and acceptance docs to wrap up multivault support
see: https://github.com/ansible/awx/issues/352
2017-12-11 16:56:02 -05:00
Ryan Petrello ce94ba4c83 change how we detect the current user to avoid a nuanced recursion error
see: https://github.com/ansible/ansible-tower/issues/7802
2017-12-08 15:35:47 -05:00
Ryan Petrello 1fb890f4eb fix a race condition in "host.last_job" when jobs are deleted
see: https://github.com/ansible/ansible-tower/issues/7815
2017-12-06 11:30:19 -05:00
Ryan Petrello fe93ef5488 more unit tests for survey default handling 2017-12-05 09:04:57 -05:00
Ryan Petrello 33dedc88c8 don't fetch stdout when purging jobs - it's slow and causes OOMs
see: https://github.com/ansible/ansible-tower/issues/7751
2017-12-04 15:14:55 -05:00
Ryan Petrello 112757e202 properly handle JT launch POST for required survey fields w/ no default
see: ansible/ansible-tower#7805
2017-12-04 09:45:21 -05:00
Ryan Petrello 12380fe1b1 add more tests for survey default encryption
see: https://github.com/ansible/ansible-tower/issues/7805
see: https://github.com/ansible/ansible-tower/issues/7806
see: https://github.com/ansible/ansible-tower/issues/7807
2017-12-04 09:45:14 -05:00
Ryan Petrello 6c7851b51f fix another survey encryption-related bug 2017-12-01 17:11:00 -05:00
Ryan Petrello 1057b93570 refactor survey spec validation into a separate testable function 2017-12-01 15:34:07 -05:00
Ryan Petrello 8c6a1e348d upgrade survey encryption migration to work around an old survey bug
see: https://github.com/ansible/ansible-tower/issues/7800
2017-12-01 11:34:47 -05:00
Ryan Petrello a1f8f65add support specifying multiple vault IDs for a playbook run
see: https://github.com/ansible/awx/issues/352
2017-11-30 16:55:17 -05:00
Ryan Petrello a3dd9eb4b7 removing some cruft we thought would help us catch bugs (it didn't) 2017-11-29 16:23:07 -05:00
Ryan Petrello 68ada92f3b more survey password encryption bug squashing
the nature of this latest bug is that the WorkflowJob has a *different*
implementation of _accept_or_ignore_job_kwargs, and it wasn't performing
encryption for extra vars provided at launch time; this change places the
encryption mechanism in UJT.create_unified_job so that it works the same
for _all_ UJTs

see: https://github.com/ansible/ansible-tower/issues/7798
see: https://github.com/ansible/ansible-tower/issues/7046
2017-11-29 14:40:41 -05:00
Ryan Petrello 1425021106 fix another encrypted survey password bug
properly encrypt extra_vars that overlap with survey passwords when
`ask_variables_on_launch=true`

see: https://github.com/ansible/ansible-tower/issues/7796
2017-11-28 16:52:47 -05:00
Ryan Petrello 26845642f0 fix a bug which caused v1 cred backwards-compat to apply to v2 requests
see: https://github.com/ansible/ansible-tower/issues/7793
2017-11-28 13:05:13 -05:00
Ryan Petrello 6fa0d9d4ed rename oVirt4 to Red Hat Virtualization
see: https://github.com/ansible/ansible-tower/issues/7790
2017-11-28 11:02:42 -05:00
Ryan Petrello 044c047ac6 fix a bug in survey password default validation
see: https://github.com/ansible/ansible-tower/issues/7046
see: https://github.com/ansible/ansible-tower/issues/7764
see: https://github.com/ansible/ansible-tower/issues/7784
2017-11-27 17:25:45 -05:00
Ryan Petrello a80d5b1b39 reencrypt WFJT.survey_spec too
https://github.com/ansible/ansible-tower/issues/7046
2017-11-16 23:00:22 -05:00
Ryan Petrello 4187d02b8a make settings.AWX_ISOLATED_KEY_GENERATION readonly
see: https://github.com/ansible/ansible-tower/issues/7380
2017-11-16 13:35:37 -05:00
Ryan Petrello 35c374fc79 fix a bug in the survey reencryption migration
see: https://github.com/ansible/ansible-tower/issues/7046
2017-11-15 10:26:46 -05:00
Ryan Petrello fa09d68603 add a few minor optimizations and some refactoring for multi-cred 2017-11-14 16:47:28 -05:00
Ryan Petrello eb140d9e69 fix a permissions bug for credentials specified at JT launch time
hat tip to @alancoding for spotting this one
2017-11-14 16:21:05 -05:00
Ryan Petrello 28ce9b700e replace all Job/JT relations with a single M2M credentials relation
Includes backwards compatibility for now-deprecated .credential,
.vault_credential, and .extra_credentials

This is a building block for multi-vault implementation and Alan's saved
launch configurations (both coming soon)

see: https://github.com/ansible/awx/issues/352
see: https://github.com/ansible/awx/issues/169
2017-11-14 12:49:12 -05:00
Ryan Petrello 38f893c124 undo an optimization in django.conf.settings that breaks awx settings 2017-11-14 11:03:50 -05:00
Ryan Petrello be00b1ca96 render survey_spec for display purposes in a safe manner
survey_spec is a nested dict, so if we don't `deepcopy()` it, updates
to the individual fields could corrupt the original data structure;
this was causing a bug whereby activity stream updates converted
encrypted survey password defaults -> `$encrypted$`, but inadvertently
modified the originating model due to shared references

see: https://github.com/ansible/ansible-tower/issues/7769
2017-11-13 13:01:56 -05:00
Ryan Petrello 78961c8037 properly perform validation on encrypted survey defaults
see: https://github.com/ansible/ansible-tower/issues/7764
2017-11-10 10:52:09 -05:00
Ryan Petrello 91ad0a9f89 add a useful tool for isolated connectivity debugging 2017-11-08 09:27:33 -05:00
Ryan Petrello 8d2ab3de42 improve the callback worker's ability to deal with idle/disconnected DB
if database connectivity is lost, callback workers currently raise an
uncaught exception and hang; this can cause the entire process to stop
handling callback events

see: https://github.com/ansible/ansible-tower/issues/7660
2017-10-31 09:51:13 -04:00
Ryan Petrello 3953366a9e help people avoid mistakenly inputting their version info as a comment 2017-10-27 14:43:20 -04:00
Ryan Petrello c95d7d465a work around an ansible bug that can cause project syncs to fail
https://github.com/ansible/ansible-tower/issues/7746
https://github.com/ansible/ansible/issues/30064
2017-10-25 11:09:39 -04:00
Ryan Petrello b60a30cbd4 fix a unicode handling bug in inventory source name migration
see: https://github.com/ansible/ansible-tower/issues/7740
2017-10-20 12:13:44 -04:00
Ryan Petrello c1e20fe7a0 properly support authentication for ovirt4 ansible modules
see: https://github.com/ansible/ansible-tower/issues/6522
see: https://github.com/ansible/ansible-tower/issues/6522#issuecomment-337909863
2017-10-19 09:47:25 -04:00
Ryan Petrello ea683344f5 remove support for job-scoped auth tokens
When Jobs and Adhoc Commands are launched, awx uses a job-scoped auth
token to dynamically fetch inventory via the awx REST API; this process
is complicated, hard to debug, and likely won't work going forward with
oauth2-based tokens in awx

see: https://github.com/ansible/awx/issues/21
2017-10-18 17:11:47 -04:00
Ryan Petrello 10a7544d68 properly follow symlinks for bwrap'd working directories
see: https://github.com/ansible/ansible-tower/issues/7741
2017-10-18 17:03:10 -04:00
Ryan Petrello 5895654538 store cloudforms inventory cache files in the proper location on disk
with process isolation enabled (which is the awx default), cloudforms
caches inventory script results on disk; awx should direct cloudforms to
store these cache files in a location that's exposed to the isolated
environment

see: ansible/ansible#31760
2017-10-17 17:07:21 -04:00
Ryan Petrello 59f9967dba store cloudforms inventory cache files in the proper location on disk
with process isolation enabled (which is the awx default), cloudforms
caches inventory script results on disk; awx should direct cloudforms to
store these cache files in a location that's exposed to the isolated
environment

see: ansible/ansible#31760
2017-10-17 17:06:48 -04:00
Ryan Petrello 5bb06fdb50 stop hard-coding the awx version in the isolated development environment
see: #296
2017-10-13 12:17:04 -04:00
Ryan Petrello 82d05e0a10 properly sanitize encrypted default passwords in JT.survey_spec
see: https://github.com/ansible/ansible-tower/issues/7259
2017-10-10 14:38:47 -04:00
Ryan Petrello b372cebf8d fix a bug when Tower is integrated with ipsilon SAML server
https://github.com/ansible/ansible-tower/issues/6683
2017-10-10 14:38:46 -04:00
Ryan Petrello f26bdb3e96 migrate existing survey passwords to be encrypted
see: https://github.com/ansible/ansible-tower/issues/7046
2017-10-10 14:38:46 -04:00
Ryan Petrello 4be4e3db7f encrypt job survey data
see: https://github.com/ansible/ansible-tower/issues/7046
2017-10-10 14:38:46 -04:00
Ryan Petrello a0cfbb93e9 fix busted 3.2.2 activity stream migration
see: ansible/ansible-tower#7704
2017-10-10 14:38:45 -04:00
Ryan Petrello f4a252a331 add new credential types in a more stable way in migrations
instead of writing individual migrations for new built-in credential
types, this change makes the "setup_tower_managed_defaults" function
idempotent so that it only adds the credential types you're missing
2017-10-10 14:38:45 -04:00
Ryan Petrello e06d4d7734 don't install pycurl from pypi; use a system package instead
the ovirt4 sdk relies on pycurl, which is complicated to install w/ pip;
rely on pycurl to be provided by a system package instead
2017-10-10 14:38:44 -04:00
Ryan Petrello 7438062b97 add ovirt sdk dependency for ovirt4 support 2017-10-10 14:38:44 -04:00
Ryan Petrello 74f2509482 support ovirt4 as a built-in inventory source
see: https://github.com/ansible/ansible-tower/issues/6522
2017-10-10 14:38:44 -04:00
Ryan Petrello 94b4dabee2 disable GCE inventory caching w/ a .ini file
see: https://github.com/ansible/ansible-tower/issues/7609
see: https://github.com/ansible/tower/pull/451#pullrequestreview-64454393
2017-10-10 14:38:44 -04:00
Ryan Petrello 94d44e8791 disable GCE inventory source cache
by default, the GCE inventory script caches results on disk for
5 minutes; disable this behavior

see: https://github.com/ansible/ansible-tower/issues/7609
2017-10-10 14:38:44 -04:00
Ryan Petrello 7a21a45781 properly encode LDAP DN values on validation
see: https://github.com/ansible/ansible-tower/issues/7554
2017-10-10 14:38:43 -04:00
Ryan Petrello c8f4320b58 allow the credential type to be changed for unused credentials
see: https://github.com/ansible/ansible-tower/issues/7607
2017-10-10 14:38:43 -04:00
Ryan Petrello 96572fe3d4 don't show polymorphic_ctype in unique validation error messaging
see: https://github.com/ansible/ansible-tower/issues/7620
2017-10-10 14:38:43 -04:00
Ryan Petrello 554a9586c6 add awx meta variables to adhoc command extra_vars
see: https://github.com/ansible/ansible-tower/issues/7513
2017-10-10 14:38:37 -04:00
Ryan Petrello f2f42c2c8a don't append to the activity stream on LDAP group disassociate
for organizations w/ a large number of ldap orgs/teams, this results in
a _huge_ number of extraneous activity stream entries

see: https://github.com/ansible/ansible-tower/issues/7655
2017-10-10 14:38:18 -04:00
Ryan Petrello 4c5ec2fb3a work around an ansible 2.4 inventory caching bug
see: https://github.com/ansible/awx/issues/246
2017-10-03 15:45:11 -04:00
Ryan Petrello 328b70ceb8 fix a bug that causes azure inventory syncs w/ region='all' to fail
see: https://github.com/ansible/ansible-tower/issues/7668
2017-09-21 15:08:54 -04:00
Ryan Petrello 1af9c43b5b Fix an issue not populating the version metadata for isolated sdist
see: https://github.com/ansible/ansible-tower/issues/6810
2017-09-19 10:25:21 -04:00
Ryan Petrello cfe1f1e8e4 more legacy azure deprecation cleanup 2017-09-18 10:43:16 -04:00
Ryan Petrello 14b0f9aa24 remove reference to legacy rax credentials 2017-09-18 10:43:02 -04:00
Ryan Petrello 4dd265633e remove legacy azure inventory script
see: ansible/ansible-tower#7629
2017-09-18 10:41:49 -04:00
Ryan Petrello 68b924efe5 flake8 fixup 2017-09-18 09:44:39 -04:00
Ryan Petrello 6a4b4edea3 properly detect deferred ORM objects 2017-09-18 09:08:53 -04:00
Ryan Petrello 7a958a1af1 more result_stdout_text defer optimization for inventory updates
see: https://github.com/ansible/ansible-tower/issues/7568
2017-09-15 09:51:53 -04:00
Ryan Petrello 1bb6c17fe2 trick django-polymorphic into allowing defer() on polymorphic objects
django-polymorphic itself generates queries for polymorphic object
lookups, and these queries for UnifiedJob are *not* properly defering the
`result_stdout_text` column, resulting in more very slow queries.  This
solution is _very_ hacky, and very specific to this specific
version of Django and django-polymorphic, but it works until we can
solve this problem the proper way in 3.3 (by removing large stdout blobs
from the database).

see: https://github.com/ansible/ansible-tower/issues/7568
2017-09-15 09:51:43 -04:00
Ryan Petrello d3df5de0ce build extra_cred related urls for jobs and JTs a less volatile way
see: https://github.com/ansible/ansible-tower/issues/7635
2017-09-13 13:28:26 -04:00
Ryan Petrello bd42dfe474 defer UnifiedJob.result_stdout_text for improved performance
result_stdout_text can be _very_ large - some customers have 5MB+ per
job; querying for this in list contexts results in _very_ large datasets
being read from the database which is very slow.  It's very uncommon to
actually need this column outside of the context of job details, so
defer it.

see: https://github.com/ansible/ansible-tower/issues/7568
2017-09-12 16:35:49 -04:00
Ryan Petrello 4213960ec3 write the scm_revision_output to the project path instead of /tmp
see: https://github.com/ansible/ansible-tower/issues/7558
2017-09-11 17:44:53 -04:00
Ryan Petrello a9c9ecb5ea bind ansible and awx virtualenvs readonly so that jobs can't modify them
see: https://github.com/ansible/ansible-tower/issues/7558
2017-09-11 15:57:35 -04:00