Commit Graph
100 Commits
Author SHA1 Message Date
Ryan Petrello 527594285f more Swagger template markup 2018-02-06 10:12:58 -05:00
Ryan Petrello 07dfab648c add some tests to prove that OpenAPI JSON compilation works 2018-02-06 10:12:58 -05:00
Ryan Petrello 10974159b5 add support for marking Swagger paths deprecated 2018-02-06 10:12:58 -05:00
Ryan Petrello ac7c5f8648 clean up API markdown docs 2018-02-06 10:12:57 -05:00
Ryan Petrello 57c22c20b2 add support for building swagger/OpenAPI JSON
to build, run `make swagger`; a file named `swagger.json` will be
written to the current working directory
2018-02-06 10:12:57 -05:00
Ryan Petrello 35230eded1 run the celery reload in a shell so the uwsgi hook isn't fatal on fail 2018-02-01 22:32:08 -05:00
Ryan Petrello 0a8df7fde2 work around a bug in dateutil that incorrectly parses Z dates
related: https://github.com/dateutil/dateutil/issues/349
2018-02-01 15:51:59 -05:00
Ryan Petrello 81bdbef785 fix a bug which can break the schedules list endpoint
see: https://github.com/ansible/ansible-tower/issues/7881
related: https://github.com/ansible/awx/pull/1095
2018-02-01 14:30:56 -05:00
Ryan Petrello 8ddc1c61ef add related links to the inventory and project for a schedule
see: https://github.com/ansible/awx/issues/276
2018-01-31 15:18:11 -05:00
Ryan Petrello 0aa6c7b83f remove some leaky mock.patch() that were causing sporadic test failures 2018-01-31 15:12:59 -05:00
Ryan Petrello 2a6f6111dc add documentation for how awx uses/interacts with ansible 2018-01-31 14:02:18 -05:00
Ryan Petrello 81af34fce3 fix a bug which causes /api/v2/auth/ to list disabled auth backends
see: https://github.com/ansible/awx/issues/1073
2018-01-30 16:20:29 -05:00
Ryan Petrello 8fed469975 fix a bug in custom venv support that breaks legacy POST /api/v1/jobs
see: https://github.com/ansible/ansible-tower/issues/7876
2018-01-30 15:29:11 -05:00
Ryan Petrello e642af82cc fix a few bugs for scheduled jobs that run without inventories
see: https://github.com/ansible/ansible-tower/issues/7865
see: https://github.com/ansible/ansible-tower/issues/7866
2018-01-29 15:15:57 -05:00
Ryan Petrello 6753f1ca35 adhere to RFC5545 regarding UNTIL timezones
If the "DTSTART" property is specified as a date with UTC time or a date with
local time and time zone reference, then the UNTIL rule part MUST be specified
as a date with UTC time.
2018-01-29 12:42:31 -05:00
Ryan Petrello 6f0c937236 don't allow distant DTSTART values for schedules; it's slow
see: https://github.com/ansible/ansible-tower/issues/7869
2018-01-29 10:16:03 -05:00
Ryan Petrello a15e257b9e fix a unicode bug in the stdout endpoint when ?content_encoding=base64
see: https://github.com/ansible/awx/issues/1042
2018-01-25 15:53:43 -05:00
Ryan Petrello e7ed4811c1 reload the entire celery worker pool when uwsgi reloads the Python app
this is for the development environment only; when uwsgi notices a code
change, it automatically reloads the uwsgi workers; this patch includes
a hook that sends `SIGHUP` to the celery process, causing it to spawn
a new set of workers as well
2018-01-25 14:59:40 -05:00
Ryan Petrello ef80ecd3b6 only import the redbaron library on-demand
redbaron is a library we use to facilitate parsing local settings files;
at _import_ time it generates a parse tree and caches it to disk at
`/tmp`; this process is _really time consuming, and only necessary if
we're actually *using* the library

right now, we're importing this library and paying the penalty
_every_ time we load the awx application
2018-01-25 10:23:44 -05:00
Ryan Petrello 9857c8272e add more tests for weird timezone/DST boundaries in schedules
see: https://github.com/ansible/awx/pull/1024
2018-01-22 14:57:57 -05:00
Ryan Petrello 15906b7e3c support TZID= in schedule rrules
this commit allows schedule `rrule` strings to include local timezone
information via TZID=NNNNN; occurrences are _generated_ in the local
time specific by the user (or UTC, if e.g., DTSTART:YYYYMMDDTHHMMSSZ)
while Schedule.next_run, Schedule.dtstart, and Schedule.dtend will be
stored in the UTC equivalent (i.e., the scheduler will still do math on
"what to run next" based on UTC datetimes).

in addition to this change, there is now a new API endpoint,
`/api/v2/schedules/preview/`, which takes an rrule and shows the next
10 occurrences in local and UTC time.

see: https://github.com/ansible/ansible-tower/issues/823
related: https://github.com/dateutil/dateutil/issues/614
2018-01-22 11:50:00 -05:00
Ryan Petrello ac3f7d0fac fix a bug that breaks workflows w/ a survey password + inventory sync
prior versions of awx did not raise an exception for this scenario
- they simply ignored kwargs that they couldn't accept.  this change is
a sort of middle ground - it ignores them, but gives a clue in the logs
as to why

see: https://github.com/ansible/awx/issues/955
related: https://github.com/ansible/awx/pull/803
2018-01-22 09:41:30 -05:00
Ryan Petrello 43f19cc94b fix another bug that breaks the JT callback process
see: https://github.com/ansible/awx/issues/980
related: 17cd0595d7
2018-01-18 13:17:06 -05:00
Ryan Petrello e5aaeedc43 don't require an IRC password in the notification UI
see: https://github.com/ansible/awx/issues/1010
2018-01-18 09:14:22 -05:00
Ryan Petrello dbb4d2b011 refactor credential injection for builtin types
this cleanups up a _lot_ of code duplication that we have for builtin
credential types. it will allow customers to setup custom inventory
sources that utilize builtin credential types (e.g., a custom inventory
script that could use an AzureRM credential)

see: https://github.com/ansible/ansible-tower/issues/7852
2018-01-17 16:50:28 -05:00
Ryan Petrello 838b793704 fix a minor unicode handling bug in project names
see: https://github.com/ansible/ansible-tower/issues/7853
2018-01-17 13:37:06 -05:00
Ryan Petrello 17cd0595d7 fix a bug which broke the callback plugin launch process
see: https://github.com/ansible/awx/issues/980
2018-01-17 11:28:13 -05:00
Ryan Petrello 51f7907a01 optimize OutputEventFilter for large stdout streams
update our event data search algorithm to be a bit lazier in event data
discovery; this drastically improves processing speeds for stdout >5MB

see: https://github.com/ansible/awx/issues/417
2018-01-16 14:41:35 -05:00
Ryan Petrello 4a2ca20b60 fix a minor bug in the JT launch related to support for zero credentials
see: https://github.com/ansible/ansible-tower/issues/7841
2018-01-12 11:37:33 -05:00
Ryan Petrello a5007ccd41 fix a unicode handling bug
see: https://github.com/ansible/ansible-tower/issues/7843
related: https://github.com/ansible/awx/pull/807
2018-01-10 15:56:31 -05:00
Ryan Petrello b01b229fea add an example for custom virtualenv setup in containers 2018-01-10 13:48:55 -05:00
Ryan Petrello 4bb2b5768e properly compose stdout downloads that contain unicode 2018-01-09 23:52:02 -05:00
Ryan Petrello c0a641ed52 properly handle unicode for isolated job buffers
from: https://docs.python.org/2/library/stringio.html#module-cStringIO
"Unlike the StringIO module, this module is not able to accept Unicode
strings that cannot be encoded as plain ASCII strings."

see: https://github.com/ansible/ansible-tower/issues/7846
2018-01-09 23:46:17 -05:00
Ryan Petrello 1e8c89f536 implement support for per-playbook/project/org virtualenvs
see: https://github.com/ansible/awx/issues/34
2018-01-09 22:47:01 -05:00
Ryan Petrello 60743d6ba6 add the vault_id to the response payload on the JT launch endpoint
see: https://github.com/ansible/awx/issues/914
2018-01-04 15:13:48 -05:00
Ryan Petrello f28f7c6184 refactor job event signal generation code 2018-01-04 12:50:12 -05:00
Ryan Petrello 1494c8395b update websockets docs to reflect new event groups 2018-01-04 11:40:15 -05:00
Ryan Petrello 2691e1d707 make the fact caching plugin fail more gracefully for large payloads
related: https://github.com/ansible/ansible/pull/34424
2018-01-04 11:33:47 -05:00
Ryan Petrello c6b6a3ad89 send and subcribe to summary events for all new events 2018-01-03 09:09:45 -05:00
Ryan Petrello 2bd656e61d calculate stdout download length using the ORM, not raw SQL 2018-01-03 09:09:44 -05:00
Ryan Petrello 35b8e40d3c remove deprecation from the stdout endpoint; text downloads still use it 2018-01-03 09:09:44 -05:00
Ryan Petrello c4d901bf2c add functional API tests for deprecated job event stdout composition
see: https://github.com/ansible/awx/issues/200
2018-01-03 09:09:44 -05:00
Ryan Petrello 1369f72885 add new API endpoints and websocket emit for new job event types
see: https://github.com/ansible/awx/issues/200
2018-01-03 09:09:44 -05:00
Ryan Petrello 0b30e7907b change stdout composition to generate from job events on the fly
this approach totally removes the process of reading and writing stdout
files on the local file system at settings.JOBOUTPUT_ROOT when jobs are
run; now stdout content is only written on-demand as it's fetched for
the deprecated `stdout` endpoint

see: https://github.com/ansible/awx/issues/200
2018-01-03 09:09:43 -05:00
Ryan Petrello fc94b3a943 generalize stdout event processing to emit events for *all* job types
* introduces three new models: `ProjectUpdateEvent`,
  `InventoryUpdateEvent`, and `SystemJobEvent`
* simplifies the stdout callback management in `tasks.py` - now _all_
  job run types capture and emit events to the callback receiver
* supports stdout reconstruction from events for stdout downloads for
  _all_ job types
* configures `ProjectUpdate` runs to configure the awx display callback
  (so we can capture real playbook events for `project_update.yml`)
* ProjectUpdate, InventoryUpdate, and SystemJob runs no longer write
  text blobs to the deprecated `main_unifiedjob.result_stdout_text` column

see: https://github.com/ansible/awx/issues/200
2018-01-03 09:09:42 -05:00
Ryan Petrello ea4cd99003 fix a few tests caused by fallout between 3.2.2 bugs and 3.3 multicred 2017-12-13 14:02:25 -05:00
Ryan Petrello d35bfafcf5 more test cleanup from 3.2.2 merge 2017-12-13 13:14:22 -05:00
Ryan Petrello a978d094b4 fix a number of failing unit tests related to the 3.2.2 merge 2017-12-13 13:03:17 -05:00
Ryan Petrello 202161f090 move legacy UnifiedJob stdout data to a separate unmanaged model
This data often (in the case of inventory updates) represents large data
blobs (5+MB per job run).  Storing it on the polymorphic base class
table, `main_unifiedjob`, causes it to be automatically fetched on every
query (and every polymorphic join) against that table, which can result
in _very_ poor performance for awx across the board.  Django offers
`defer()`, but it's quite complicated to sprinkle this everywhere (and
easy to get wrong/introduce side effects related to our RBAC and usage
of polymorphism).

This change moves the field definition to a separate unmanaged model
(which references the same underlying `main_unifiedjob` table) and adds
a proxy for fetching the data as needed

see https://github.com/ansible/awx/issues/200
2017-12-12 18:16:19 -05:00
Ryan Petrello 8c90d36290 add some more tests and acceptance docs to wrap up multivault support
see: https://github.com/ansible/awx/issues/352
2017-12-11 16:56:02 -05:00
Ryan Petrello ce94ba4c83 change how we detect the current user to avoid a nuanced recursion error
see: https://github.com/ansible/ansible-tower/issues/7802
2017-12-08 15:35:47 -05:00
Ryan Petrello 1fb890f4eb fix a race condition in "host.last_job" when jobs are deleted
see: https://github.com/ansible/ansible-tower/issues/7815
2017-12-06 11:30:19 -05:00
Ryan Petrello fe93ef5488 more unit tests for survey default handling 2017-12-05 09:04:57 -05:00
Ryan Petrello 33dedc88c8 don't fetch stdout when purging jobs - it's slow and causes OOMs
see: https://github.com/ansible/ansible-tower/issues/7751
2017-12-04 15:14:55 -05:00
Ryan Petrello 112757e202 properly handle JT launch POST for required survey fields w/ no default
see: ansible/ansible-tower#7805
2017-12-04 09:45:21 -05:00
Ryan Petrello 12380fe1b1 add more tests for survey default encryption
see: https://github.com/ansible/ansible-tower/issues/7805
see: https://github.com/ansible/ansible-tower/issues/7806
see: https://github.com/ansible/ansible-tower/issues/7807
2017-12-04 09:45:14 -05:00
Ryan Petrello 6c7851b51f fix another survey encryption-related bug 2017-12-01 17:11:00 -05:00
Ryan Petrello 1057b93570 refactor survey spec validation into a separate testable function 2017-12-01 15:34:07 -05:00
Ryan Petrello 8c6a1e348d upgrade survey encryption migration to work around an old survey bug
see: https://github.com/ansible/ansible-tower/issues/7800
2017-12-01 11:34:47 -05:00
Ryan Petrello a1f8f65add support specifying multiple vault IDs for a playbook run
see: https://github.com/ansible/awx/issues/352
2017-11-30 16:55:17 -05:00
Ryan Petrello a3dd9eb4b7 removing some cruft we thought would help us catch bugs (it didn't) 2017-11-29 16:23:07 -05:00
Ryan Petrello 68ada92f3b more survey password encryption bug squashing
the nature of this latest bug is that the WorkflowJob has a *different*
implementation of _accept_or_ignore_job_kwargs, and it wasn't performing
encryption for extra vars provided at launch time; this change places the
encryption mechanism in UJT.create_unified_job so that it works the same
for _all_ UJTs

see: https://github.com/ansible/ansible-tower/issues/7798
see: https://github.com/ansible/ansible-tower/issues/7046
2017-11-29 14:40:41 -05:00
Ryan Petrello 1425021106 fix another encrypted survey password bug
properly encrypt extra_vars that overlap with survey passwords when
`ask_variables_on_launch=true`

see: https://github.com/ansible/ansible-tower/issues/7796
2017-11-28 16:52:47 -05:00
Ryan Petrello 26845642f0 fix a bug which caused v1 cred backwards-compat to apply to v2 requests
see: https://github.com/ansible/ansible-tower/issues/7793
2017-11-28 13:05:13 -05:00
Ryan Petrello 6fa0d9d4ed rename oVirt4 to Red Hat Virtualization
see: https://github.com/ansible/ansible-tower/issues/7790
2017-11-28 11:02:42 -05:00
Ryan Petrello 044c047ac6 fix a bug in survey password default validation
see: https://github.com/ansible/ansible-tower/issues/7046
see: https://github.com/ansible/ansible-tower/issues/7764
see: https://github.com/ansible/ansible-tower/issues/7784
2017-11-27 17:25:45 -05:00
Ryan Petrello a80d5b1b39 reencrypt WFJT.survey_spec too
https://github.com/ansible/ansible-tower/issues/7046
2017-11-16 23:00:22 -05:00
Ryan Petrello 4187d02b8a make settings.AWX_ISOLATED_KEY_GENERATION readonly
see: https://github.com/ansible/ansible-tower/issues/7380
2017-11-16 13:35:37 -05:00
Ryan Petrello 35c374fc79 fix a bug in the survey reencryption migration
see: https://github.com/ansible/ansible-tower/issues/7046
2017-11-15 10:26:46 -05:00
Ryan Petrello fa09d68603 add a few minor optimizations and some refactoring for multi-cred 2017-11-14 16:47:28 -05:00
Ryan Petrello eb140d9e69 fix a permissions bug for credentials specified at JT launch time
hat tip to @alancoding for spotting this one
2017-11-14 16:21:05 -05:00
Ryan Petrello 28ce9b700e replace all Job/JT relations with a single M2M credentials relation
Includes backwards compatibility for now-deprecated .credential,
.vault_credential, and .extra_credentials

This is a building block for multi-vault implementation and Alan's saved
launch configurations (both coming soon)

see: https://github.com/ansible/awx/issues/352
see: https://github.com/ansible/awx/issues/169
2017-11-14 12:49:12 -05:00
Ryan Petrello 38f893c124 undo an optimization in django.conf.settings that breaks awx settings 2017-11-14 11:03:50 -05:00
Ryan Petrello be00b1ca96 render survey_spec for display purposes in a safe manner
survey_spec is a nested dict, so if we don't `deepcopy()` it, updates
to the individual fields could corrupt the original data structure;
this was causing a bug whereby activity stream updates converted
encrypted survey password defaults -> `$encrypted$`, but inadvertently
modified the originating model due to shared references

see: https://github.com/ansible/ansible-tower/issues/7769
2017-11-13 13:01:56 -05:00
Ryan Petrello 78961c8037 properly perform validation on encrypted survey defaults
see: https://github.com/ansible/ansible-tower/issues/7764
2017-11-10 10:52:09 -05:00
Ryan Petrello 91ad0a9f89 add a useful tool for isolated connectivity debugging 2017-11-08 09:27:33 -05:00
Ryan Petrello 8d2ab3de42 improve the callback worker's ability to deal with idle/disconnected DB
if database connectivity is lost, callback workers currently raise an
uncaught exception and hang; this can cause the entire process to stop
handling callback events

see: https://github.com/ansible/ansible-tower/issues/7660
2017-10-31 09:51:13 -04:00
Ryan Petrello 3953366a9e help people avoid mistakenly inputting their version info as a comment 2017-10-27 14:43:20 -04:00
Ryan Petrello c95d7d465a work around an ansible bug that can cause project syncs to fail
https://github.com/ansible/ansible-tower/issues/7746
https://github.com/ansible/ansible/issues/30064
2017-10-25 11:09:39 -04:00
Ryan Petrello b60a30cbd4 fix a unicode handling bug in inventory source name migration
see: https://github.com/ansible/ansible-tower/issues/7740
2017-10-20 12:13:44 -04:00
Ryan Petrello c1e20fe7a0 properly support authentication for ovirt4 ansible modules
see: https://github.com/ansible/ansible-tower/issues/6522
see: https://github.com/ansible/ansible-tower/issues/6522#issuecomment-337909863
2017-10-19 09:47:25 -04:00
Ryan Petrello ea683344f5 remove support for job-scoped auth tokens
When Jobs and Adhoc Commands are launched, awx uses a job-scoped auth
token to dynamically fetch inventory via the awx REST API; this process
is complicated, hard to debug, and likely won't work going forward with
oauth2-based tokens in awx

see: https://github.com/ansible/awx/issues/21
2017-10-18 17:11:47 -04:00
Ryan Petrello 10a7544d68 properly follow symlinks for bwrap'd working directories
see: https://github.com/ansible/ansible-tower/issues/7741
2017-10-18 17:03:10 -04:00
Ryan Petrello 5895654538 store cloudforms inventory cache files in the proper location on disk
with process isolation enabled (which is the awx default), cloudforms
caches inventory script results on disk; awx should direct cloudforms to
store these cache files in a location that's exposed to the isolated
environment

see: ansible/ansible#31760
2017-10-17 17:07:21 -04:00
Ryan Petrello 59f9967dba store cloudforms inventory cache files in the proper location on disk
with process isolation enabled (which is the awx default), cloudforms
caches inventory script results on disk; awx should direct cloudforms to
store these cache files in a location that's exposed to the isolated
environment

see: ansible/ansible#31760
2017-10-17 17:06:48 -04:00
Ryan Petrello 5bb06fdb50 stop hard-coding the awx version in the isolated development environment
see: #296
2017-10-13 12:17:04 -04:00
Ryan Petrello 82d05e0a10 properly sanitize encrypted default passwords in JT.survey_spec
see: https://github.com/ansible/ansible-tower/issues/7259
2017-10-10 14:38:47 -04:00
Ryan Petrello b372cebf8d fix a bug when Tower is integrated with ipsilon SAML server
https://github.com/ansible/ansible-tower/issues/6683
2017-10-10 14:38:46 -04:00
Ryan Petrello f26bdb3e96 migrate existing survey passwords to be encrypted
see: https://github.com/ansible/ansible-tower/issues/7046
2017-10-10 14:38:46 -04:00
Ryan Petrello 4be4e3db7f encrypt job survey data
see: https://github.com/ansible/ansible-tower/issues/7046
2017-10-10 14:38:46 -04:00
Ryan Petrello a0cfbb93e9 fix busted 3.2.2 activity stream migration
see: ansible/ansible-tower#7704
2017-10-10 14:38:45 -04:00
Ryan Petrello f4a252a331 add new credential types in a more stable way in migrations
instead of writing individual migrations for new built-in credential
types, this change makes the "setup_tower_managed_defaults" function
idempotent so that it only adds the credential types you're missing
2017-10-10 14:38:45 -04:00
Ryan Petrello e06d4d7734 don't install pycurl from pypi; use a system package instead
the ovirt4 sdk relies on pycurl, which is complicated to install w/ pip;
rely on pycurl to be provided by a system package instead
2017-10-10 14:38:44 -04:00
Ryan Petrello 7438062b97 add ovirt sdk dependency for ovirt4 support 2017-10-10 14:38:44 -04:00
Ryan Petrello 74f2509482 support ovirt4 as a built-in inventory source
see: https://github.com/ansible/ansible-tower/issues/6522
2017-10-10 14:38:44 -04:00
Ryan Petrello 94b4dabee2 disable GCE inventory caching w/ a .ini file
see: https://github.com/ansible/ansible-tower/issues/7609
see: https://github.com/ansible/tower/pull/451#pullrequestreview-64454393
2017-10-10 14:38:44 -04:00
Ryan Petrello 94d44e8791 disable GCE inventory source cache
by default, the GCE inventory script caches results on disk for
5 minutes; disable this behavior

see: https://github.com/ansible/ansible-tower/issues/7609
2017-10-10 14:38:44 -04:00
Ryan Petrello 7a21a45781 properly encode LDAP DN values on validation
see: https://github.com/ansible/ansible-tower/issues/7554
2017-10-10 14:38:43 -04:00
Ryan Petrello c8f4320b58 allow the credential type to be changed for unused credentials
see: https://github.com/ansible/ansible-tower/issues/7607
2017-10-10 14:38:43 -04:00
Ryan Petrello 96572fe3d4 don't show polymorphic_ctype in unique validation error messaging
see: https://github.com/ansible/ansible-tower/issues/7620
2017-10-10 14:38:43 -04:00