* AWX_ISOLATION_SHOW_PATHS will be shared between containers. Strange
file not found error can crop up when concurrently accessing shared
directories between multiple containers that are bind mounted with big
Z. So make sure we use little z.
* The cron ran logrotate will now rotate our log files instead of python
* If not error log file is specified in the config then do not include
it as a paremter to rsyslog omhttp module. This is useful for
containers.
* The namespace for isolated logging was not enabled. Add a handler and
logger so that it's enabled. This is particularly useful when the
logging level is switched to DEBUG
* Knowing how long check_isolated.yml ran can be helpful in debuging the
isolated execution path. Especially if you suspect the connection speed
or reliability of the control node -> execution node
* It's hard/impossible to know what job a check_isolated.yml playbook
runs for by just looking at the logs.
* Forward the job id for which an iso management playbook is running for
and output that job id so it can be found in the logs.
* We batch logging isolated management playbook output. This results in
the timestamp of the log being useless when trying to determine when
each task in the playbook ran.
* To fix this, we enable timestamp logging at the playbook level via
ansible `profile_tasks` callback plugin.
* Task manager fit_ optimization code caused problems with container
group code.
* Note that we don't actually get the benefit of the optimization for
container groups. We just make it so that the code doesn't blow up. It
will take another pass to apply optimizations to the container group
task manager path.
* Tried to fill in application_name in awx/__init__.py but I think that
is too late
* Fill in database application_name with enough information to easily
trace the connection from postgres back to the node and pid that
initiated the connection.
* Set application_name in django settings so that application_name is
set _before_ the first postgres connection is established.
* Do not query the database for the set of Instance that belong to the
group for which we are trying to fit a job on, for each job.
* Instead, cache the set of instances per-instance group.
* We update the parent unified job template to point at new jobs
created. We also update a similar foreign key when the job finishes
running. This causes lock contention when the job template is
allow_simultaneous and there are a lot of jobs from that job template
running in parallel. I've seen as bad as 5 minutes waiting for the lock
when a job finishes.
* This change moves the parent->child update to OUTSIDE of the
transaction if the job is allow_simultaneous (inherited from the parent
unified job). We sacrafice a bit of correctness for performance. The
logic is, if you are launching 1,000 parallel jobs do you really care
that the job template contains a pointer to the last one you launched?
Probably not. If you do, you can always query jobs related to the job
template sorted by created time.
* Do not query the database for the set of Instance that belong to the
group for which we are trying to fit a job on, for each job.
* Instead, cache the set of instances per-instance group.
* We update the parent unified job template to point at new jobs
created. We also update a similar foreign key when the job finishes
running. This causes lock contention when the job template is
allow_simultaneous and there are a lot of jobs from that job template
running in parallel. I've seen as bad as 5 minutes waiting for the lock
when a job finishes.
* This change moves the parent->child update to OUTSIDE of the
transaction if the job is allow_simultaneous (inherited from the parent
unified job). We sacrafice a bit of correctness for performance. The
logic is, if you are launching 1,000 parallel jobs do you really care
that the job template contains a pointer to the last one you launched?
Probably not. If you do, you can always query jobs related to the job
template sorted by created time.
* Do not write out inventory source_vars to a file on disk as they _may_
contain sensitive information. This also removes support for backwards
migrations. This is fine, backwards migration is really only useful
during development.
* Before, we were re-writing `plugin:` when users updated the
InventorySource via the API. Now, we just override at run-time. This
makes for a more sane API interaction
* broadcast websockets have stats tracked (i.e. connection status,
number of messages total, messages per minute, etc). Previous to this
change, stats were tracked by ip address, if it was defined on the
instance, XOR hostname. This changeset tracks stats by hostname.
* The websocket backplane interconnect is done via ip address for
Kubernetes and OpenShift. On init run_wsbroadcast reads all Instances
from the DB and makes a decision to use the ip address or the hostname
based, with preference given to the ip address if defined. For
Kubernetes and OpenShift the nodes can load the Instance before the
ip_address is set. This would cause the connection to be tried by
hostname rather than ip address. This changeset ensures that an ip
address set after an Instance record is created will be detected and
used.
* Not sure how or when, but ours crawling tests set the cluster node id
to NOT start with `awx-`. That is fine, the schema checker just needs to
account for that.
* The websocket backplane interconnect is done via ip address for
Kubernetes and OpenShift. On init run_wsbroadcast reads all Instances
from the DB and makes a decision to use the ip address or the hostname
based, with preference given to the ip address if defined. For
Kubernetes and OpenShift the nodes can load the Instance before the
ip_address is set. This would cause the connection to be tried by
hostname rather than ip address. This changeset ensures that an ip
address set after an Instance record is created will be detected and
used.
* broadcast websockets have stats tracked (i.e. connection status,
number of messages total, messages per minute, etc). Previous to this
change, stats were tracked by ip address, if it was defined on the
instance, XOR hostname. This changeset tracks stats by hostname.
* There are 2 data-structures that django channels redis uses: (1) zset
and (2) list. (1) is used for group membership where the key is the
logic user group and the value(s) are websocket clients. The score of
the zset entry is used for group expiration. We can not rely on group
expiration for clean-up because there is no interface privided by redis
channels to refresh the expiration. Choosing a small value for
group_expiry could result on our websocket backplane group expiring,
which would result in job events not being delivered. Instead, we
increase the group expiration to 5 years and clean up on daphne service
start.
* The list (2) data-structure is used by django channels redis to queue
websocket events per-websocket-client as needed. The need arises to
queue per-websocket-client events when the consumer can not keep up with
the producer. The consumer here is daphne, the producer is AWX.
* When AWX is operating healthy group membership in Redis is reflective
of the real-world. When AWX is unhealthy i.e. daphne cycles, the zset
will contain stale websocket client entries. This can be observed by
running `zrange asgi::group:jobs-status_changed 0 -1`. If the entries
returned look like:
specific.fUkXXpYj!DKOIfwPICNgw
specific.fUkXXpYj!FQcdopZeiRdG
specific.lpTSAgnk!IOKldfzcfdDp
specific.lpTSAgnk!NbvRUZsDpIQx
The entries with `fUkXXpYj` are stale. Note that this changeset fixes
this by removing all `asgi:*` entries on daphne start.
* Also note that individual message themselves have an expiration that
is configurable and defaults to 60.
* Also note that zset's tracking group membership will be deleted by
django channels redis when they are empty.
* Sleep before trying to reconnect
Most common reason for entering this reconnect loop is when Redis
service stops before the callback receiver when stopping tower services.
* Make quiter the daphne logs by raising the level to INFO instead of
DEBUG
* Output the django channels name of broadcast clients. This way, if the
queue gets backed up, we can find it in redis.
* zcard asgi::group:jobs-status_changed <-- to see a group set that
continues to grow. Issue this command in a loop while refreshing the
browser page on the jobs list. Before this change the set size would
continue to grow as daphne channel names are added to the group. After
this change the set size stays stable at the expected, 1.
* Replying to websocket group membership with the previous state, delta,
and new state has shown to be quite stable. This debug message is not
very helpful and is noisy in the dev env. This change removes the debug
message.
* We log stats using a safe hostname because of prometheus requirements.
However, when we display users the hostname we should use the Instance
hostname. This change outputs the Instance.hostname instead of the safe
prometheus name.
* We strip out the json select fields in our tests since it is an sql
lite database underneath. Ideally, we would do something fancier, but we
aren't. In doing this stipping, we could strip the last element in the
projection list. This would result in an extra dangling comma. This
commit removes the danging comma in the projection list after the
removal of JSON projections.
* OPTIONS response descritpion for workflow job template node identifier
value was an ever changing uuid4(). This is telling the user the wrong
thing. We can not know what uuid4() is going to be in the docs. Instead,
for the OPTIONS response description, tell the user the form that the
uuid4() takes, ie. xxx-xxxx...
* Note that the API browser still populates a uuid4 for the user when it
generates the sample POST data. This is nice.
* BROKER_URL now describes how to connect to redis. We use a unix socket
to connect to redis. Therefore, no longer need to support fancy uri's
that contain fancy characters in the password.
* Add the end of the redis PR we rebased devel a bunch. requirements
snuck into requirements.txt this way. This PR removes those requirements
(i.e. kombu) and bumps other requirements.
* The heartbeat of an instance is determined to be the last modified
time of the Instance object. Therefore, we want to be careful to only
update very specific fields of the Instance object.
* postgres notify/listen channel names have size limitations as well as
character limitations. Respect those limitations while at the same time
generate a unique channel name.
Chris Meyers