Commit Graph
100 Commits
Author SHA1 Message Date
chris meyers c8eeacacca POC channels 2 2020-03-18 16:10:12 -04:00
chris meyers 0db0f81e53 allow external pg connections
* Postgres containers now, by default, do not allow passwordless users
to connect remotely. This change explicitly allows that case.
2020-02-17 10:16:20 -05:00
chris meyers 0b3e2cc7e3 pin virtualenv < 20 for awx_web builds 2020-02-11 08:43:26 -05:00
chris meyers a36bf4af64 fix instance migration is_isolated() issue
* Older versions of Instance model code may not contain the
is_isolated() method. This change accounts for that fact.
2020-02-05 09:16:31 -05:00
chris meyers 0291c476d4 use existing version of Instance
* Without this change, future modifications to the Instance object may
result in migration fails (i.e. adding a field to the Instance model)
2020-02-03 14:25:06 -05:00
chris meyers 129374a1c2 keep *-devel package in the dev container
* requirements/updater.sh does pip magic. In doing this magic, devel
system packages are required to download/install/build. This change
ensures those dev packages are available.
2019-12-03 14:31:09 -05:00
chris meyers 9fa4dac847 do not expose the notication secret fields 2019-08-14 13:58:47 -04:00
chris meyers d785145c59 force proj sync when collections/requirements.yml
* Similar to roles/requirements.yml sync optimization logic.
2019-08-12 22:16:02 -04:00
chris meyers cc6413c44c use ansible nightly
* ansible:devel now has ansible-galaxy collection support
2019-08-12 22:14:31 -04:00
chris meyers 4be65a0879 collections/requirements.yml support
* just like we support ansible-galaxy role install, support
ansible-galaxy collection install
2019-08-12 22:14:30 -04:00
chris meyers 9236fd2a53 fake it till you make it!
* The user awx is passed to the launch of our dev docker container. The
docker system automagically creates that user for us and sets the home
dir to /tmp in /etc/passwd. Many methods of detecting the user home dir
don't use that. Instead, they use the HOME env var. This is a half-way
solution that solves the problem of python expanding the ~ dir.
* If other things break because they determine the users home dir via
/etc/passwd entry then a more in-depth fix will be needed.
2019-07-29 09:58:47 -04:00
chris meyers 9b95cc27c4 map in a global collections path 2019-07-17 15:36:09 -04:00
chris meyers acb6d9c4d1 wrap smart inv cache update w/ advisory lock
* Two job templates that use the same smart inventory running at the
same time can easily race to recompute the smart inventory <-> host
mapping. In this case, bulk_create() can throw an error when racing.
* The per-smart-inventory advisory lock ensures that the state of the
system is consistent & that bulk_create() runs in isolation.
2019-06-21 08:52:34 -04:00
chris meyers 8aa28092ff disabled instance does not mean offline instance
* Disabling an instance is used to stop and instance from being the
target of new jobs to run.
* The instance should still perform it's heartbeat so that it isn't
considered offline.
* If the instance was allowed to go offline on an openshift cluster it
would be deleted from the database.
2019-05-13 11:44:47 -04:00
chris meyers 84c09a19d1 instantiate dispatcher once per job run
* Instantiating the callback dispatch queue on each job event callback
is expensive. Instead, instantiate it only once. Note, we do not need to
instantiate the callback queue in the iso case so we do not.
2019-04-17 14:03:00 -04:00
chris meyers 0c6a522813 remove system tracking
* Leave the artisanal handcrafted migration from vendored fact scan to
external fact gathering.
2019-04-12 14:30:29 -04:00
chris meyers 531d97d3b3 pickup insights system_id from ansible facts
* Continue to pick up facts from scan_insights.py
* This PR adds the ability to pickup facts from
/etc/ansible/facts.d/insights.facts
* Log what transport the insights system_id was found via
2019-04-05 10:35:56 -04:00
chris meyers 71fcb1a82c process host facts for iso runs
* Move isolated clean to our final run hook
* ISO and non-iso code path now share the post-fact-processing code
2019-03-29 16:16:22 -04:00
chris meyers 926d6a6525 include tower header to insights on plan fetch
* We include a special header value in the user agent when tower proxies
requests to get per-host rules.
* This extends that header logic to when we fetch plans (playbooks)
2019-03-28 10:56:56 -04:00
chris meyers cb4d55b47a fixes inventory update deadlock
* all inventory updates continue to occur in parallel up to the point
that they update the database with their results.
* the "funnel" is achieved by using a global per-inventory postgres
named lock
2019-03-27 16:52:07 -04:00
chris meyers 2329079326 runner expects process isolation flags in settings
* Towards the goal of converging the iso code path w/ the non-iso code
path. More process isolation control flags into settings.
2019-03-27 11:08:41 -04:00
chris meyers 87e1ba4dea do not generate a random ident
* instead, set the ident passed to ansible runner to be the job id. That
way, on we know what directory to look in for results when the directory
structure is created.
2019-03-22 12:19:42 -04:00
chris meyers b7b97dd58d doc update fix 2019-03-21 09:26:59 -04:00
chris meyers 0a670e8db1 change from runner master to runner 1.3 2019-03-21 07:46:11 -04:00
chris meyers 60ef160e85 flake8 fix 2019-03-20 16:12:45 -04:00
chris meyers 8c26f20188 add license files for python modules
* python-daemon
* ansible-runner
2019-03-20 14:51:41 -04:00
chris meyers 2c52a7d9a8 fix more unit tests for runner
* isolated will be fixed in the future so pytest skip those
* fact cache moved one directory level up, account for that
2019-03-20 14:32:52 -04:00
chris meyers b006510035 do not save sensitive env vars
* job_env gets exposed via the api. Sensitive env variables should be
redacted before saved into job_env.
2019-03-20 14:00:22 -04:00
chris meyers 1a6ae6e107 allow for runner setting parent_uuid
* Previously, parent_uuid was expected only on events generated for a
Job run. Now, there maybe a parent_uuid for any job type. AWX does not
support parenting events for any job type other than Job.
2019-03-20 11:05:01 -04:00
chris meyers 060585434a update tests 2019-03-20 09:44:38 -04:00
chris meyers 5135b8a969 fixup unit tests for tasks 2019-03-18 14:21:47 -04:00
chris meyers 8a04c22b2b point at another runner branch
* revert parent_uuid because it causes problems with unexpected
parameter on event creation for some event type.
2019-03-18 14:21:47 -04:00
chris meyers f7842cf283 refactor and fix unit tests
* fixup task TestGenericRun
* make runner callback functions accessable to testing
* reduce isintance() usage in run() by using build_ pattern
* move process_isolation param building to build_ function so it can be
tested
2019-03-18 14:21:47 -04:00
chris meyers 827ad0fa75 remove safe_args and add status_handler
* safe_args no longer makes sense. We have moved extra_vars to a file
and thus do not pass sensitive content on the cmdline
2019-03-18 14:21:47 -04:00
chris meyers 8fb65b40de use ansible runner to run playbooks
* Project Updates
* Jobs
* Inventory Updates
* System Jobs
* AdHoc Commands

* Notifications
* Fact Cache
* proot
2019-03-18 14:21:47 -04:00
chris meyers a7cda95803 init ansible-runner requirements 2019-03-18 14:21:47 -04:00
chris meyers 723f581fd0 fix handling of serial strategy
* v2_playbook_on_play_start is called multiple times for the same UUID.
Specifically, once for each host in the play. This changes makes the
uuid unique before going to the dispatcher.
2019-02-22 13:14:35 -05:00
chris meyers c121565209 add tests for host_filter
* Ensure that building the smart inventory query string doesn't invoke
the database.
2019-02-12 16:11:54 -05:00
chris meyers 63e3e733e0 AWX 3.0.1 2019-02-08 14:17:33 -05:00
chris meyers ed78978b5f do not observe queries when constructing them
* While parsing host_filter in the smart inventory code it was
triggering sql queries. This changset avoids executing the query that is
being constructed.
2019-02-08 12:30:51 -05:00
chris meyers d4c3c089df fix scheduled jobs race condition
* The periodic scheduler that runs and spawns jobs from Schedule()'s can
end up spawning more jobs than intended, for a single Schedule.
Specifically, when tower clustering is involed. This change adds a
"global" database lock around this critical code. If another process is
already doing the scheduling, short circuit.
2019-02-08 08:43:11 -05:00
chris meyers db2bb19d65 add docs for task manager node decider 2019-01-02 12:17:28 -05:00
chris meyers d1705dd0cc more sane default log handlers
* Removed the emailing of admins on request error. When turned on, the
handler will include all django settings in the email. This is not
desirable from a security standpoint.
2018-12-05 09:38:27 -05:00
chris meyers 21fce00102 python3 compliance
* This ones for you rydog
2018-11-29 14:07:43 -05:00
chris meyers d347a06e3d do not deny existing workflow node relationships 2018-11-29 13:29:16 -05:00
chris meyers 916c92ffc7 save state 2018-11-29 08:53:46 -05:00
chris meyers 09dff99340 enforce 1 edge between 2 nodes constraint 2018-11-28 16:57:50 -05:00
chris meyers 83760deb9d align tests with new replay get_job interface 2018-11-28 13:33:44 -05:00
chris meyers 228e412478 simplify workflow job failure reason
* Log the more detailed reason for a workflow job failing but expose a
simplified reason to users via job_explanation
2018-11-27 16:12:42 -05:00
chris meyers f8f2e005ba better comment for deciding parent's status 2018-11-27 16:12:42 -05:00
chris meyers d8bf82a8cb add help_text to do_not_run workflow field 2018-11-27 16:12:41 -05:00
chris meyers 2eeca3cfd7 add example workflow run to docs 2018-11-27 16:12:41 -05:00
chris meyers 0c8dde9718 fix dfs_run_nodes()
* Tried to re-use the topological sort order to crawl the graph to find
the next node(s) to run. This is incorrect, we need to take into account
the fail/success of jobs and directionally crawl the graph.
2018-11-27 16:12:41 -05:00
chris meyers febf051748 do not mark ujt None nodes dnr
* Leave workflow nodes with no related unified job template nodes
do_not_run = False. If we mark it True, we can't differentiate between
the actual want to not take that path vs. do not run this because I do
not have a valid related unified job template.
2018-11-27 16:12:41 -05:00
chris meyers 7b087d4a6c loop over dnr nodes by topological sort
* Perform topological sort on graph nodes before looping over them to
mark do not run. This guarantees that parent nodes will be processed
before calling dependent child nodes. The complexity of the sorting is
N. The complexity of marking the the nodes is N*V
2018-11-27 16:12:41 -05:00
chris meyers cfa098479e Revert "optimize mark dnr nodes algorithm"
This reverts commit 6372c52772.
2018-11-27 16:12:41 -05:00
chris meyers 4c9a1d6b90 optimize mark dnr nodes algorithm
* Compute largest depth of each node and traverse graph by depth. This
allows us to check a node once, and only once, to determine if it needs
to be marked for do not run.
2018-11-27 16:12:41 -05:00
chris meyers d1aa52a2a6 fix up mark dnr logic 2018-11-27 16:12:41 -05:00
chris meyers f30f52a0a8 handle missing unified job template in workflow
* Workflow Node without unified_job_template is treated as a job marked
as failure; when deciding what path to execute.
* Remove optimization of marking dnr nodes due to it making the
algorithm incorrect.
2018-11-27 16:12:41 -05:00
chris meyers 676c068b71 add job_description to failed workflow node
* When workflow job fails because a workflow job node doesn't have a
related unified_job_template note that with an error on the workflow
job's job_description
* When a workflow job fails because a failure path isn't defined, note
that on the workflow job job_description
2018-11-27 16:12:41 -05:00
chris meyers 00d71cea50 detect workflow nodes without job templates
* Fail workflow job run when encountering a Workflow Job Nodes with
no related job templates.
2018-11-27 16:12:41 -05:00
chris meyers 281345dd67 flake8 fix 2018-11-27 16:12:41 -05:00
chris meyers 1a85fcd2d5 update docs to include workflow failure semantic 2018-11-27 16:12:41 -05:00
chris meyers c1171fe4ff treat canceled nodes as failed when processing wf
* When deciding what jobs to run next, treat canceled as failed.
* Also add tests.
2018-11-27 16:12:41 -05:00
chris meyers d6a8ad0b33 treat canceled jobs in wf the same as failed jobs
* Also fix spelling mistake that caused workflows to be falsely marked
successful in the case of a canceled job.
2018-11-27 16:12:41 -05:00
chris meyers 266831e26d add cycle unit test 2018-11-27 16:12:41 -05:00
chris meyers a6e20eeaaa update wf done and failed tests 2018-11-27 16:12:41 -05:00
chris meyers 6529c1bb46 update done and fail detection for workflow
* Instead of traversing the workflow graph to determine if a workflow is
done or has failed; instead, loop through all the nodes in the graph and
grab only the relevant nodes.
2018-11-27 16:12:41 -05:00
chris meyers b81d795c00 fix up dot graph generator
* Update graph dot generator to use the new efficient graph
2018-11-27 16:12:41 -05:00
chris meyers 1b87e11d8f flake8 2018-11-27 16:12:41 -05:00
chris meyers 8bb9cfd62a add dag tests 2018-11-27 16:12:41 -05:00
chris meyers a176a4b8cf remove unused code 2018-11-27 16:12:41 -05:00
chris meyers 3f4d14e48d crawl entire graph when marking DNR
* From the root, the code was only going down the did run path to find
nodes to mark DNR. This is incorrect, Now, we traverse the entire graph
each time to find nodes to mark DNR.
2018-11-27 16:12:41 -05:00
chris meyers 0499d419c3 more efficient graph processing
* Getting parent nodes from child was inefficient. Optimize it with a
hash table like we did for the getting of children.
* Getting leaf nodes was inefficient. Optimize it like we did getting
root nodes. A node is assumed to be a leaf node until it gets a child.
2018-11-27 16:12:41 -05:00
chris meyers 3dadeb3037 remove print statements 2018-11-27 16:12:41 -05:00
chris meyers 16a60412cf optimization fix
* WorkflowDAG accepts workflow job template and workflow jobs for which
to build a graph out of the nodes. The optimized query for each is
different. This changeset adds the differing queries for a workflow job.
2018-11-27 16:12:41 -05:00
chris meyers 9f3e272665 optimize cycle detection 2018-11-27 16:12:41 -05:00
chris meyers e1e8d3b372 bump migration 2018-11-27 16:12:40 -05:00
chris meyers 07db7a41b3 more flake8 2018-11-27 16:12:40 -05:00
chris meyers 1120f8b1e1 try2 at the devil flake8 2018-11-27 16:12:40 -05:00
chris meyers 17b3996568 fix flake8 anyway I can 2018-11-27 16:12:40 -05:00
chris meyers 584b3f4e3d remove workflow test
* We now handle workflows with jobs that have errored. We treat them the
same as a failure result. Before, we would abort the workflow when we
encountered an error.
2018-11-27 16:12:40 -05:00
chris meyers f8c53f4933 handle job error state in convergence 2018-11-27 16:12:40 -05:00
chris meyers 6e40e9c856 handle edge case ring cycle 2018-11-27 16:12:40 -05:00
chris meyers 2f9dc4d075 remove relationship in view if cycle detected 2018-11-27 16:12:40 -05:00
chris meyers 9afc38b714 fixup migrations 2018-11-27 16:12:40 -05:00
chris meyers dfccc9e07d rework wf cycle detection for convergence 2018-11-27 16:12:40 -05:00
chris meyers 7b22d1b874 cycle detection when multiple parents 2018-11-27 16:12:40 -05:00
chris meyers 1e10d4323f update docs 2018-11-27 16:12:40 -05:00
chris meyers 4111e53113 correctly name migration to align with 3.4.0 2018-11-27 16:12:40 -05:00
chris meyers 02df0c29e9 merge artifacts deterministically 2018-11-27 16:12:40 -05:00
chris meyers 475c90fd00 prevent job launching twice 2018-11-27 16:12:40 -05:00
chris meyers 2742b00a65 flake8 2018-11-27 16:12:40 -05:00
chris meyers ea29e66a41 fix workflow finish state detector
* Take into account the new do_not_run field when finding if a workflow
is finished. If do_not_run is True then the node is considered finished.
2018-11-27 16:12:40 -05:00
chris meyers 6ef6b649e8 cleaner code 2018-11-27 16:12:40 -05:00
chris meyers 9bf2a49e0f save state 2018-11-27 16:12:40 -05:00
chris meyers 914892c3ac all parents should finish before start child 2018-11-27 16:12:40 -05:00
chris meyers 77661c6032 short circuit performance optimization 2018-11-27 16:12:40 -05:00
chris meyers b4fc585495 stop DNR propogation on always path
* This makes sure DNR propogation stops when a job is successful, down
an always path
2018-11-27 16:12:40 -05:00
chris meyers ff6db37a95 correct stop DNR propogation
* If a child has a parent that is not in the finished state then do not
propogate the DNR to the child in question.
* If a parent is in a finished state; do not propogate the DNR to the
child if the path to the child is traversed (based on the parent job
status).
2018-11-27 16:12:40 -05:00