* If a child has a parent that is not in the finished state then do not
propogate the DNR to the child in question.
* If a parent is in a finished state; do not propogate the DNR to the
child if the path to the child is traversed (based on the parent job
status).
* If scheduler inv exists then the inv summary will be filled in with
our generic summary filler inner. Else, if the related unified job has
an inventory, fill in the inv summary with that, explicitly.
* Use the same logic that related inventory uses. If there is an
inventory that overrides the inventory on the unified job template then
summarize that field. Else, use the inventory on the unified job template
being scheduled.
* Previously, we have logic in the API to ensure that ldap group type
params, when changed, align with ldap group type Class init
expectations. However, we did not have this logic in the migrations.
This PR adds the validation check to migrations.
* celery workers have internal queue names that are named after the
system hostname. This may differ from what tower knows the host by,
Instance.hostname
This adds a mapping so we can convert internal celery names to Instance
names for purposes of reaping jobs.
* We now target Instances in the task manager when transitioning jobs
from pending to waiting; whereas before we submitted jobs to Instance
Groups to be picked up by Instance's in those Instance Groups.
Subscribing Instances to their Instance Groups is no longer needed. This
change removes the Instance Group queue subscription.
* Instead of passing around the isolated host that the task is to
execute on; grab the isolated execution host from the instance further
down the call stack. Without passing the isolated hostname around.
* Randomly chose an instance in the controller instance group for which
to control the isolated node run. Note the chosen instance via a job
controller_node field
* Deciding the Instance that a Job runs on at celery task run-time makes
it hard to evenly distribute tasks among Instnaces. Instead, the task
manager will look at the world of running jobs and choose an instance
node to run on; applying a deterministic job distribution algo.
* The "should we cancel?" callback check is only called while the
subprocess (child) is alive. If the child executes quickly, the cancel
callback has no chance to run.
* call a subprocess that takes longer to run like sleep 2
* A disabled node needs to stay subscribed to the broadcast queue
because the work to re-subscribe the node to queues when the node is
re-enabled is done over the broadcast queue.
* Each time a route is needed (i.e. when a task is sumitted to celery).
The router will be queried. This is ideal. With the previous method we
had to consider how a change in the routes would propogate to all celery
workers and nodes.
* fully describe the default awx queue
* Our dynamic queue registration would correct awx_private_queue.
However, we don't want celery to even create an "invalid"/extra
queue-exchange-route. This change makes sure we don't create extranious
things in rabbitmq.
* reduce the cluster queue registration output. Only output when the
queue registration list changes.
* Using Kombu's default Broadcast() constructor requires only 1
parameter. That parameter defines the exchange name and the queue name
is randomly generated per-node.
* This caused problems if/when celery enters an infinite restart loop
because too many rabbit queues get created and rabbit OOM's
(gracefully).
* To remedy this we tell Broadcast the queue name to use, which is
derived from some constant + the node name so that the per-node queue
name is stable.
* They were previously not required until a min/max was enforced. This
caused the fields to, unintentionally, be required.
* This fix makes the policy fields not required and provides sane defaults.
* Before, we had a special group, tower, that ran any async work that
tower needed done. This allowed users fine grain control over which
nodes did background work. However, this granularity was too complicated
for users. So now, all tower system work goes to a special non-user
exposed celery queue. Tower remains the fallback instance group to
execute jobs on. The tower group will be created upon install and
protected from deletion.
* Fancy url finding regex can result in infinite loop for malformed ipv6
urls so replace it with a more nieve regex that can overmatch.
* regex's that find malformed ipv6 urls will be passed to urlparse. This
can result in a parsing/ValueError. For these cases we redact the entire
found URI.
* LDAP params is a new field. It contains the kwargs that will be passed
to the python class specified by group type. The default for group type
is MemberDNGroupType. The required params are now those in the defaults.
* It's problematic to delete an instance that is referenced by a foreign
key; where the referening model is one that has a Polymorphic parent.
* Specifically, when Django goes to nullify the relationship it relies
on the related instances[0] class type to issue a query to decide what
to nullify. So if the foreignkey references multiple different types
(i.e. ProjectUpdate, Job) then only 1 of those class types will get
nullified. The end result is an IntegrityError when delete() is called.
* This changeset ensures that the parent Polymorphic class is queried so
that all the foreignkey entries are nullified
* Also remove old Django "hack" that doesn't work with Django 1.11
* Was considering an isolated instance: any instance that has at least 1
group with no controller. This is technically correct since an iso node
can not be a part of a non-iso group.
* The query is now more robust and considers a node an iso node if ALL
groups that a node belong to ALL have a controller.
* Also added better debugging for the special tower instance group
* Added a check for the existance of the special tower group so that
logs are less "messy" during the install process.
* The Instance Group list of instances was getting over-written with
every call to the register_instance management command.
* This changeset appends --hostnames to the Instance Group policy list.
* It's possible to have an exception raised in BaseTask.run() before the
stdout handler gets defined. This is problematic when the exception
handler tries to access that undefined var .. causing another exception.
Note that the second exception is caught also but it's not desirable to
lose the first exception.
* This fix checks to see if the stdout handler var is defined before
calling it's methods. Thus, we retain the original error message.
* Endpoint exposes all jobs associated with an Instance. This is what we
want. Align the endpoint description with this behavior by removing the
word running.
* Nodes are marked offline, then deleted; given enough time. Nodes can
come back for various reasions (i.e. netsplit). When they come back,
have them recreate the node Instance if AWX_AUTO_DEPROVISION_INSTANCES
is True. Otherwise, do nothing. The do nothing case will show up in the
logs as celery job tracebacks as they fail to be self aware.
* I had thought that setting the settings was required. But carefully
selected defaults for the settings is the correct way to deal with
errors I was seeing early in developing this feature.
chris meyers