* celery workers have internal queue names that are named after the
system hostname. This may differ from what tower knows the host by,
Instance.hostname
This adds a mapping so we can convert internal celery names to Instance
names for purposes of reaping jobs.
* We now target Instances in the task manager when transitioning jobs
from pending to waiting; whereas before we submitted jobs to Instance
Groups to be picked up by Instance's in those Instance Groups.
Subscribing Instances to their Instance Groups is no longer needed. This
change removes the Instance Group queue subscription.
* Instead of passing around the isolated host that the task is to
execute on; grab the isolated execution host from the instance further
down the call stack. Without passing the isolated hostname around.
* Randomly chose an instance in the controller instance group for which
to control the isolated node run. Note the chosen instance via a job
controller_node field
* Deciding the Instance that a Job runs on at celery task run-time makes
it hard to evenly distribute tasks among Instnaces. Instead, the task
manager will look at the world of running jobs and choose an instance
node to run on; applying a deterministic job distribution algo.
* The "should we cancel?" callback check is only called while the
subprocess (child) is alive. If the child executes quickly, the cancel
callback has no chance to run.
* call a subprocess that takes longer to run like sleep 2
* A disabled node needs to stay subscribed to the broadcast queue
because the work to re-subscribe the node to queues when the node is
re-enabled is done over the broadcast queue.
* Each time a route is needed (i.e. when a task is sumitted to celery).
The router will be queried. This is ideal. With the previous method we
had to consider how a change in the routes would propogate to all celery
workers and nodes.
* fully describe the default awx queue
* Our dynamic queue registration would correct awx_private_queue.
However, we don't want celery to even create an "invalid"/extra
queue-exchange-route. This change makes sure we don't create extranious
things in rabbitmq.
* reduce the cluster queue registration output. Only output when the
queue registration list changes.
* Using Kombu's default Broadcast() constructor requires only 1
parameter. That parameter defines the exchange name and the queue name
is randomly generated per-node.
* This caused problems if/when celery enters an infinite restart loop
because too many rabbit queues get created and rabbit OOM's
(gracefully).
* To remedy this we tell Broadcast the queue name to use, which is
derived from some constant + the node name so that the per-node queue
name is stable.
* They were previously not required until a min/max was enforced. This
caused the fields to, unintentionally, be required.
* This fix makes the policy fields not required and provides sane defaults.
* Before, we had a special group, tower, that ran any async work that
tower needed done. This allowed users fine grain control over which
nodes did background work. However, this granularity was too complicated
for users. So now, all tower system work goes to a special non-user
exposed celery queue. Tower remains the fallback instance group to
execute jobs on. The tower group will be created upon install and
protected from deletion.
* Fancy url finding regex can result in infinite loop for malformed ipv6
urls so replace it with a more nieve regex that can overmatch.
* regex's that find malformed ipv6 urls will be passed to urlparse. This
can result in a parsing/ValueError. For these cases we redact the entire
found URI.
* LDAP params is a new field. It contains the kwargs that will be passed
to the python class specified by group type. The default for group type
is MemberDNGroupType. The required params are now those in the defaults.
* It's problematic to delete an instance that is referenced by a foreign
key; where the referening model is one that has a Polymorphic parent.
* Specifically, when Django goes to nullify the relationship it relies
on the related instances[0] class type to issue a query to decide what
to nullify. So if the foreignkey references multiple different types
(i.e. ProjectUpdate, Job) then only 1 of those class types will get
nullified. The end result is an IntegrityError when delete() is called.
* This changeset ensures that the parent Polymorphic class is queried so
that all the foreignkey entries are nullified
* Also remove old Django "hack" that doesn't work with Django 1.11
* Was considering an isolated instance: any instance that has at least 1
group with no controller. This is technically correct since an iso node
can not be a part of a non-iso group.
* The query is now more robust and considers a node an iso node if ALL
groups that a node belong to ALL have a controller.
* Also added better debugging for the special tower instance group
* Added a check for the existance of the special tower group so that
logs are less "messy" during the install process.
* The Instance Group list of instances was getting over-written with
every call to the register_instance management command.
* This changeset appends --hostnames to the Instance Group policy list.
* It's possible to have an exception raised in BaseTask.run() before the
stdout handler gets defined. This is problematic when the exception
handler tries to access that undefined var .. causing another exception.
Note that the second exception is caught also but it's not desirable to
lose the first exception.
* This fix checks to see if the stdout handler var is defined before
calling it's methods. Thus, we retain the original error message.
* Endpoint exposes all jobs associated with an Instance. This is what we
want. Align the endpoint description with this behavior by removing the
word running.
* Nodes are marked offline, then deleted; given enough time. Nodes can
come back for various reasions (i.e. netsplit). When they come back,
have them recreate the node Instance if AWX_AUTO_DEPROVISION_INSTANCES
is True. Otherwise, do nothing. The do nothing case will show up in the
logs as celery job tracebacks as they fail to be self aware.
* I had thought that setting the settings was required. But carefully
selected defaults for the settings is the correct way to deal with
errors I was seeing early in developing this feature.
* Use unicode InstanceGroup and queue names up until the point we
actually create the queue
* kombu add_consumers returns a dict with a value that contians the
passed in queue name. Trouble is, the returned dict value is a string
and not a unicode string and this results in an error.
* Adds pattern to easy add django-auth-ldap group types classes and to
pass parameters via AUTH_LDAP_GROUP_TYPE_PARAMS
* Adds new group type PosixUIDGroupType that accepts the attribute,
ldap_group_user_attr, on which to search for the user(s) in the group.
* related to https://github.com/ansible/ansible-tower/issues/7931
* The Tower Instance group is special. It should always exist, so
prevent any delete to it.
* Only allow super users to associate/disassociate instances the 'tower'
instance group.
* Do not allow fields of tower instance group to be changed.
Related to https://github.com/ansible/ansible-tower/issues/7957
* Problem presented itself as Instances falling out of Instance Groups.
This was due to the cluster membership policy decider erroring out on a
logger message with unicode.
* Fixed up potential other unicode logger unicode issues in tasks.py
* invoke the first heartbeat as early as possible. Results in a much
better user experience where when a user scales up an awx node, the node
appears with capacity earlier.
* invoke the first heartbeat as early as possible. Results in a much
better user experience where when a user scales up an awx node, the node
appears with capacity earlier.
* awx task container uses postgres port to wait for postgres to become
available before the container init continues. The () are problematic
and are removed.
* () was originally added to fix an openshift issues. That error does
NOT occur with this fix.
* use embedded beat rather than standalone
* dynamically set celeryd hostname at runtime
* add embeded beat flag to celery startup
* Embedded beat mode routes will piggyback off of celery worker setup
signal
* Based on the tower topology (Instance and InstanceGroup
relationships), have celery dyamically listen to queues on boot
* Add celery task capable of "refreshing" what queues each celeryd
worker listens to. This will be used to support changes in the topology.
* Cleaned up some celery task definitions.
* Converged wrongly targeted job launch/finish messages to 'tower'
queue, rather than a 1-off queue.
* Dynamically route celery tasks destined for the local node
* separate beat process
add support for separate beat process
related to https://github.com/ansible/awx/issues/217
* Adds a configure tower in tower setting for users to configure a saml
attribute that tower will use to put users into teams and orgs.
* Celery + json pickling do not handle custom Exceptions (and may never
do so). Mentioning of, if handling custom Exceptions then the code would
be susceptible to same arbitrary code execution that python pickle is
vulnerable to.
* So don't use custom Exceptions.
* celery error callback signature isn't well defined. Thus, our error
callback signature is made to handle just about any call signature and
depend on only 1 attribute, id, existing.
See https://github.com/celery/celery/issues/3709
* Check the reason for a dependent project update failure. If it's
because of a cancel, then let the normal cancel mechanisms update the
jobs status and explanation. Do not update the dependent job's status
for a project update that was canceled, in the run code.
* python-social-auth has SOCIAL_AUTH_SAML_SECURITY_CONFIG, which is
forwarded to python-saml settings configuration. This commit exposes
SOCIAL_AUTH_SAML_SECURITY_CONFIG to configure tower in tower to allow
users to set requestedAuthnContext, which will disable the requesting of
password type auth from the idp. Thus, it's up to the idp to choose
which auth to use (i.e. 2-factor).
chris meyers