Commit Graph
100 Commits
Author SHA1 Message Date
Jeff Bradberry 42701f32fe Build the source distribution bundle to also upload to PyPI
This has been broken since 20.0.1.
2024-01-11 15:49:07 -05:00
Jeff Bradberry 4822dd79fc Revert "Improve performance for awx cli export (#13182)" 2023-08-15 15:55:10 -04:00
Jeff Bradberry 14992cee17 Add in an async task to migrate the data over 2023-08-10 13:48:58 -04:00
Jeff Bradberry 6db663eacb Modify main/0185 to set aside the json fields that might be a problem
Rename them, then create a new clean field of the new jsonb type.
We'll use a task to do the data conversion.
2023-08-10 13:48:58 -04:00
Jeff Bradberry 8f6849fc22 Include listener_port in the defaults for Instance.objects.register (#13328) 2022-12-19 14:16:05 -03:00
Jeff Bradberry e87e041a2a Break up and conditionally add the RBAC checks for ActivityStream (#13279)
This should vastly improve the queries executed when accessing any of
the activity stream endpoints as a normal user, in many cases.
2022-12-16 15:11:14 -03:00
Jeff Bradberry 7b8dcc98e7 Merge pull request #13308 from jbradberry/rebuild-org-ee-admin-roles
Ensure that the Organization.execution_environment_admin_role always gets built
2022-12-16 11:29:20 -05:00
Jeff Bradberry e524d3df3e Replace the role fixup post_migrate handler with a data migration 2022-12-12 10:20:56 -05:00
Jeff Bradberry 15b7ad3570 Add a post_migrate signal handler to rebuild the Org roles
particularly, the execution_environment_admin_role.
2022-12-07 15:57:20 -05:00
Jeff Bradberry adbcb5c5e4 Merge pull request #13289 from jbradberry/improve-psql-paging
Make sure that the psql pager does not clear the screen afterwards
2022-12-06 13:17:24 -05:00
Jeff Bradberry 8054c6aedc Make sure that the psql pager does not clear the screen afterwards
Also, avoid paging if there is a single page.
2022-12-06 10:46:47 -05:00
Jeff Bradberry bb047baeba Merge pull request #13277 from jbradberry/psql-pager
Configure dbshell to use the 'less' pager
2022-12-02 13:27:46 -05:00
Jeff Bradberry 57430afc55 Configure dbshell to use the 'less' pager 2022-12-02 12:31:49 -05:00
Jeff Bradberry 95e4b2064f Merge pull request #13264 from jbradberry/fix-awxkit-field-removals
Remove update_on_project_update from awxkit
2022-12-01 14:11:03 -05:00
Jeff Bradberry 48eba60be4 Remove update_on_project_update from awxkit
This was removed from the InventorySource model a bit ago.
2022-12-01 13:44:10 -05:00
Jeff Bradberry 65c3db8cb8 Merge pull request #13250 from jbradberry/fix-inventorysource-migration
Fix a problem with migration main/0164
2022-11-30 15:07:54 -05:00
Jeff Bradberry 7fa9dcbc2a Merge pull request #13168 from ArtsiomMusin/feature_13055_add_multiple_assert_export
Add multiple asset export for awx cli
2022-11-30 15:01:42 -05:00
Jeff Bradberry 7cfb957de3 Add the elements: str type to the lists 2022-11-30 14:15:57 -05:00
Jeff Bradberry d0d467e863 Fix a problem with migration main/0164
Since the original version of the migration a) invoked the .save()
method, and b) involved a model with a custom field that had a
post_save handler attached, this migration had a side-effect that
caused the codebase's version of the model to be used when the table
involved wasn't yet up to date.  This triggers an UndefinedColumn error.

This change works around the problem by making use of queryset
.update() methods instead, which should avoid the post_save signal
trigger.
2022-11-30 11:06:08 -05:00
Jeff Bradberry 721e19e1c8 Merge pull request #13181 from jbradberry/remove-qsstats
Replace the querysets provided by django-qsstats-magic
2022-11-11 10:58:51 -05:00
Jeff Bradberry e029cf7196 Remove the django-qsstats-magic dependency 2022-11-10 15:37:44 -05:00
Jeff Bradberry 477a63d1b4 Replace the querysets provided by django-qsstats-magic
with ones that are constructed to do the same thing, without any
extraneous functionality.
2022-11-10 11:00:39 -05:00
Jeff Bradberry 86b0a3d4f1 Merge pull request #13174 from jbradberry/fix-3.11-awxkit
Avoid adding the 'config' subparser twice
2022-11-09 13:37:03 -05:00
Jeff Bradberry 3d3e4ad150 Avoid adding the 'config' subparser twice
Once since it is defined as a CustomCommand subclass, and once because
it is an endpoint at the /api/v2/ level.  With Python 3.11 argparse
has become more strict and will raise an exception when you try to
inject duplicate subparsers.
2022-11-08 15:13:35 -05:00
Jeff Bradberry c5cf39abb7 Merge pull request #13132 from jbradberry/import-project-schedules
Expand the dependencies of the Schedule page type
2022-10-31 16:29:04 -04:00
Jeff Bradberry b57ae592ed Expand the dependencies of the Schedule page type
Really these could get any of the unified job template types, not just
system job templates, so importing e.g. a project with a schedule was
doing them in the wrong order.

Also, bump the timeout of the project update and make sure that we
stash it in the page cache even if it doesn't finish in 5 minutes.
2022-10-28 12:56:35 -04:00
Jeff Bradberry 153a197fad Merge pull request #13125 from jbradberry/improve-staticfile-management
UI static files are now directly copied to the proper static dir
2022-10-27 09:25:09 -04:00
Jeff Bradberry 8f4c329c2a UI static files are now directly copied to the proper static dir
when running `make ui-devel`.  Previously they were going to
/awx_devel/awx/public/static, but that directory is no longer being
served up by nginx, which forced us to have to run `make
collectstatic` (or equivalent) to get the files to the right place.
2022-10-26 17:51:29 -04:00
Jeff Bradberry eeb9d61488 Merge pull request #13069 from jbradberry/hostname-validation-regression
Add back in the uniqueness validation on Instance.hostname
2022-10-17 11:40:54 -04:00
Jeff Bradberry 234ce529fc Add back in the uniqueness validation on Instance.hostname 2022-10-17 10:55:38 -04:00
Jeff Bradberry 5b5aac675b Merge pull request #12959 from ansible/new-health-check-started
Add a new Instance.health_check_started field
2022-09-28 10:58:43 -04:00
Jeff Bradberry 6b0618b244 Merge pull request #12968 from ansible/instance-serializer-defaults
Make sure to include field defaults for Instance node_type and node_state
2022-09-28 10:53:31 -04:00
Jeff Bradberry 5aa55d7347 Make sure to include field defaults for Instance node_type and node_state 2022-09-27 17:15:45 -04:00
Jeff Bradberry 65179d9cd0 Add a new Instance.health_check_started field
This will enable us to provide more useful information for the user,
now that all user-triggered health checks are async.

Also, de-bounce the health check endpoint to not allow additional
health check tasks to be triggered when one is already in progress.
2022-09-27 17:09:41 -04:00
Jeff Bradberry b879cbc2ec Prevent any edits to hop nodes
to retain the behavior that they had pre-mesh-scaling.
2022-09-23 09:46:15 -04:00
Jeff Bradberry 0e578534fa Update the instance install bundle requirements.yml
to point to the 0.1.0 release of ansible.receptor.
2022-09-23 09:46:14 -04:00
Jeff Bradberry 08c18d71bf Move InstanceLink creation and updating to the async tasks
So that they get applied in situations that do not go through the API.
2022-09-23 09:46:14 -04:00
Jeff Bradberry 68a44529b6 Register pages for the Instance peers and install bundle endpoints
This includes exposing a new interface for Page objects, Page.bytes,
to return the full bytestring contents of the response.
2022-09-23 09:46:14 -04:00
Jeff Bradberry f3a9d4db07 Assign a default queue to wait_for_jobs() 2022-09-23 09:46:14 -04:00
Jeff Bradberry ebd200380a Resolve a deadlock in write_receptor_config() 2022-09-23 09:46:13 -04:00
Jeff Bradberry 1b650d6927 When deprovisioning a node, kick off a task that waits on running jobs
After all jobs on the node are complete, delete the node then
broadcast the write_receptor_config task.

Also, make sure that write_receptor_config updates the state of links
that are in 'adding' state.
2022-09-23 09:46:13 -04:00
Jeff Bradberry b6946c7e35 Update API to support setting instances to Deprovisioning
- allow the node_state to be set to deprovisioning
- set the links that touch the instance to removing
- only allow on K8S
- only allow to be done to execution nodes
2022-09-23 09:46:13 -04:00
Jeff Bradberry 3bc86ca8cb Follow up on new execution node creation
- hop nodes are descoped
- links need to be created on execution node creation
- expose the 'edit' capabilities on the instance serializer
2022-09-23 09:46:13 -04:00
Jeff Bradberry 0465a10df5 Deal with exceptions when running execution_node_health_check (#12733) 2022-09-23 09:46:12 -04:00
Jeff Bradberry e4518f7b13 Changes in posting constraints due to rescoping to OCP/K8S-only
- node_state is now read only
- node_state gets set automatically to Installed in the create view
- raise a validation error when creating on non-K8S
- allow SystemAdministrator the 'add' permission for Instances
- expose the new listener_port field
2022-09-23 09:46:12 -04:00
Jeff Bradberry 604fac2295 Update task management to only do things with ready instances 2022-09-23 09:46:11 -04:00
Jeff Bradberry 24bfacb654 Check state when processing receptorctl advertisements
Nodes that show up and were in one of the unready states need to be
transitioned to ready, even if the logic in Instance.is_lost was not
met.
2022-09-23 09:46:11 -04:00
Jeff Bradberry 3bcd539b3d Make sure that the health checks handle the state transitions properly
- nodes with states Provisioning, Provisioning Fail, Deprovisioning,
  and Deprovisioning Fail should bypass health checks and should never
  transition due to the existing machinery
- nodes with states Unavailable and Installed can transition to Ready
  if they check out as healthy
- nodes in the Ready state should transition to Unavailable if they
  fail a check
2022-09-23 09:46:11 -04:00
Jeff Bradberry 81e68cb9bf Update node and link registration to put them in the right state
'Installed' for the nodes, 'Established' for the links.
2022-09-23 09:46:11 -04:00
Jeff Bradberry a575f17db5 Add the state fields and the peer relationships to the serializers 2022-09-23 09:46:11 -04:00
Jeff Bradberry 2fba3db48f Add state fields to Instance and InstanceLink
Also, listener_port to Instance.
2022-09-23 09:46:11 -04:00
Jeff Bradberry 5290c692c1 Merge pull request #12620 from jbradberry/even-narrower-reload
Restrict files that trigger a reload
2022-08-04 09:21:31 -04:00
Jeff Bradberry 90a19057d5 Restrict files that trigger a reload
to files explicitly ending in '.py' that do not start with a dot.
This will avoid Emacs lockfiles from triggering the restart.
2022-08-03 18:23:48 -04:00
Jeff Bradberry 1bcfc8f28e Merge pull request #12544 from jbradberry/awxkit-fix-no-content
Suppress 204 No Content results causing an error during import
2022-07-20 10:48:02 -04:00
Jeff Bradberry fe1a767f4f Suppress 204 No Content results causing an error during import 2022-07-19 12:25:24 -04:00
Jeff Bradberry d59b6f834c Merge pull request #12431 from jbradberry/fix-ugettext-deprecation
Fix a ugettext deprecation that snuck back in
2022-06-27 13:58:07 -04:00
Jeff Bradberry ae7be525e1 Fix a ugettext deprecation that snuck back in
at some point after the Django 3.2 upgrade.
2022-06-27 13:27:35 -04:00
Jeff Bradberry b44442c460 Merge pull request #12351 from AlexSCorey/5673-t-importExportSchedules
Adds import export to awx cli for schedules as a top level object
2022-06-22 10:13:56 -04:00
Jeff Bradberry 48a2ebd48c Merge pull request #12271 from HampusLundqvist/gitlab-webhooks-fixes-#12268
return event_status on push, tag push, and merge gitlab webhook events
2022-06-14 17:12:27 -04:00
Jeff Bradberry 3bc0c53e37 Merge pull request #12368 from jbradberry/narrower-autoreload
Narrow down the inotifywait criteria for reloading the dev environment
2022-06-14 10:13:41 -04:00
Jeff Bradberry e8b2998578 Narrow down the inotifywait criteria for reloading the dev environment
- listen specifically within awx/awx, so that changes in awxkit or
  awx_collection don't trigger spurious reloads
- expand the exclude pattern to ignore the test directories
2022-06-13 16:08:20 -04:00
Jeff Bradberry 9191079dda Merge pull request #11921 from jbradberry/fix-export-reconstruct-endpoint
Look up the correct top-level resource name when reconstructing foreign keys
2022-06-06 17:08:02 -04:00
Jeff Bradberry faa5df19ca Merge pull request #12252 from jbradberry/fix-analytics-unicode
Double escape all unicode escape sequences in job events data
2022-06-06 16:41:06 -04:00
Jeff Bradberry 127016d36b Allow for multiple --extra_vars or --variables flags in awx-cli
This is particularly useful when you are using the @filepath version
of the flag, since otherwise there would be no way to issue the
command with multiple vars files.

Also, add `-e` as an alias to `--extra_vars`
2022-06-01 13:24:24 -04:00
Jeff Bradberry 973facebba Double escape all unicode escape sequences in job events data
when collecting it for analytics.
2022-05-18 12:00:03 -04:00
Jeff Bradberry 4ca9e9577b Merge pull request #12216 from jangel97/devel
add param all_pages to method export_assets
2022-05-11 14:51:50 -04:00
Jeff Bradberry f3725c714a Merge pull request #12119 from kimbernator/devel
Remove hardcoded public schema in cleanup_jobs.py
2022-05-10 17:14:11 -04:00
Jeff Bradberry 2bef5ce09b Merge pull request #12099 from jbradberry/add-content-type-option-header
Add the X-Content-Type-Options nosniff header
2022-04-28 14:41:02 -04:00
Jeff Bradberry 68221cdcbe Merge pull request #12106 from jbradberry/django-bump
Bump Django to 3.2.13
2022-04-26 15:07:52 -04:00
Jeff Bradberry d64c457b3d Bump Django to 3.2.13 2022-04-26 10:34:28 -04:00
Jeff Bradberry 47d5a89f40 Add the X-Content-Type-Options nosniff header 2022-04-25 13:45:16 -04:00
Jeff Bradberry 3a1268de1e Merge pull request #12068 from jbradberry/fix-event-partition-alignment-devel
Fix the job event partition alignment
2022-04-19 10:36:48 -04:00
Jeff Bradberry 11890f0eee Fix the job event partition alignment
it really should be always aligned to the hour, so that real job
events don't slip through the cracks.
2022-04-18 14:54:06 -04:00
Jeff Bradberry 6c1adade25 Merge pull request #11947 from jbradberry/django-3.2-upgrade
Remove the out-of-band JSONField migration
2022-03-28 12:02:53 -04:00
Jeff Bradberry d54838cd94 Remove the out-of-band migration
that was turning all old JSONFields into a jsonb type database column.
The use of JSONBlob makes this unnecessary.
2022-03-24 15:21:59 -04:00
Jeff Bradberry e3f3ab224a Replace all previously text-based json fields with JSONBlob
This JSONBlob field type is a wrapper around Django's new generic
JSONField, but with the database column type forced to be text.  This
should behave close enough to our old wrapper around
django-jsonfield's JSONField and will avoid needing to do the
out-of-band database migration.
2022-03-24 15:21:54 -04:00
Jeff Bradberry aa46a7fe06 Merge pull request #11924 from jbradberry/awxkit-support-inventory-labels
Support the new Inventory.labels in awxkit / awx-cli
2022-03-21 10:19:42 -04:00
Jeff Bradberry 51021f380b Support the new Inventory.labels in awxkit / awx-cli 2022-03-18 14:24:49 -04:00
Jeff Bradberry b562d5cc88 Look up the correct top-level resource name when reconstructing foreign keys
during an awx-cli export.
2022-03-18 10:32:33 -04:00
Jeff Bradberry ac6a82eee4 Merge pull request #11654 from jbradberry/django-3.2-upgrade
Django 3.2 upgrade
2022-03-17 10:34:22 -04:00
Jeff Bradberry 574e3ed6ef Fix the signature of RADIUSBackend.get_django_user
to match what django-radius is now expecting.
2022-03-14 13:19:57 -04:00
Jeff Bradberry 584514766d Disable our wrapping of the Django settings during migrations
This will hopefully get us past the unfortunate check against the
HostMetric table, which doesn't exist when you are upgrading from 3.8
to 4.x.

Additionally, guard against AUTH_LDAP_GROUP_TYPE not being in settings
for conf migration 0006.
2022-03-14 13:19:57 -04:00
Jeff Bradberry eb8a1fec49 Make API queries involving json fields cast the fields to text
This restores compatibility with the way it previously worked -- we
never supported proper json operator queries through the API.
2022-03-14 13:19:57 -04:00
Jeff Bradberry 38ccea0f1f Fix up warnings
- the default auto-increment primary key field type is now
  configurable, and Django's check command issues a warning if you are
  just assuming the historical behavior of using AutoField.

- Django 3.2 brings in automatic AppConfig discovery, so all of our
  explicit `default_app_config = ...` assignments in __init__.py
  modules are no longer needed, and raise a RemovedInDjango41Warning.
2022-03-14 13:19:57 -04:00
Jeff Bradberry 5d000c37d6 Deal with breaking tests for 3.2
- Instantiating an abstract model raises a TypeError
2022-03-14 13:19:57 -04:00
Jeff Bradberry 1803c5bdb4 Fix up usage of django-guid
It has replaced the class-based middleware, everything is
function-based now.
2022-03-14 13:19:57 -04:00
Jeff Bradberry 23e700a1ef Upgrade to Django 3.2
- upgrades
  - Django 3.2.12
  - pytz 2021.3 (from 2019.3)
  - oauthlib 3.2.0 (from 3.1.0)
  - requests-oauthlib 1.3.1 (from 1.3.0)
  - django-guid 3.2.1 (from 2.2.1)
  - django-solo 2.0.0 (from 1.1.3)
  - django-taggit 2.1.0 (from 1.2.0)
  - netaddr 0.8.0 (from 0.7.19)
  - pyrad 2.4 (from 2.3)
  - django-radius devel (from 1.3.3)
  - future devel (from 0.16.0)

- django-guid, django-solo, and django-taggit are upgraded to fix the
  AppConfig deprecation warning.  FIXME: django-guid devel has the
  fix, but it hasn't been released yet.

- Released versions of django-radius have a hard-coded pin to
  future==0.16.0, which has a Python warning due to an improperly
  escaped character.  This is fixed in future devel, so for now we are
  pinning to references to the git repos.

- netaddr had a bunch of Python syntax and deprecation warnings
2022-03-14 13:19:53 -04:00
Jeff Bradberry 676b8f6d8f Implement an out-of-band migration to change the json fields 2022-03-07 18:11:36 -05:00
Jeff Bradberry 028f09002f Fix the cleanup_jobs management command
It previously depended on a private Django internal class that changed
with Django 3.1.

I've switched here instead to disabling the django-polymorphic
accessors to get the underlying UnifiedJob object for a Job, which due
to the way they implement those was resulting in N+1 behavior on
deletes.  This gets us back most of the way to the performance gains
we achieved with the custom collector class.  See
https://github.com/django-polymorphic/django-polymorphic/issues/198.
2022-03-07 18:11:36 -05:00
Jeff Bradberry 0500512c3c Fix up deprecation warning about get_response for middleware
It will no longer be allowed to be None by Django 4.0
2022-03-07 18:11:36 -05:00
Jeff Bradberry 1e625ed58b Fix up deprecation warning about re_path() replacing urls() 2022-03-07 18:11:36 -05:00
Jeff Bradberry e620bef2a5 Fix Django 3.1 deprecation removal problems
- FieldDoesNotExist now has to be imported from django.core.exceptions
- Django docs specifically say not to import
  django.conf.global_settings, which now has the side-effect of
  triggering one of the check errors
2022-03-07 18:11:36 -05:00
Jeff Bradberry 05142a779d Replace all usage of customized json fields with the Django builtin
The event_data field on event models, however, is getting an
overridden version that retains the underlying text data type for the
column, to avoid a heavy data migration on those tables.

Also, certain of the larger tables are getting these fields with the
NOT NULL constraint turned off, to avoid a long migration.

Remove the django.utils.six monkey patch we did at the beginning of
the upgrade.
2022-03-07 18:11:36 -05:00
Jeff Bradberry 65d17fb316 Upgrade to Django 3.1
- upgrades
  - Django 3.1.14
  - djangorestframework 3.13.1 (from 3.12.1)
  - djangorestframework-yaml 2.0.0 (from 1.0.3)
  - django-polymorphic 3.1.0 (from 2.1.2)
  - social-auth-core 4.2.0 (from 3.3.1)
  - social-auth-app-django 5.0.0 (from 3.1.0)
  - django-auth-ldap 4.0.0 (from 2.1.0)
  - django-crum 0.7.9 (from 0.7.5)

- django-jsonfield is removed
2022-03-07 18:11:36 -05:00
Jeff Bradberry faa12880a9 Squash a few deprecation warnings
- inspect.getargspec() -> inspect.getfullargspec()
- register pytest.mark.fixture_args
- replace use of DRF's deprecated NullBooleanField
- fix some usage of naive datetimes in the tests
- fix some strings with backslashes that ought to be raw strings
2022-03-07 18:11:36 -05:00
Jeff Bradberry 9b6fa55433 Deal with breaking tests for 3.1
- Django's PostgreSQL JSONField wraps values in a JsonAdapter, so deal
  with that when it happens.  This goes away in Django 3.1.
- Setting related *_id fields clears the actual relation field, so
  trying to fake objects for tests is a problem
- Instance.objects.me() was inappropriately creating stub objects
  every time while running tests, but some of our tests now create
  real db objects. Ditch that logic and use a proper fixture where needed.
- awxkit tox.ini was pinned at Python 3.8
2022-03-07 18:11:36 -05:00
Jeff Bradberry b852baaa39 Fix up logger .warn() calls to use .warning() instead
This is a usage that was deprecated in Python 3.0.
2022-03-07 18:11:36 -05:00
Jeff Bradberry a3a216f91f Fix up new Django 3.0 deprecations
Mostly text based: force/smart_text, ugettext_*
2022-03-07 18:11:36 -05:00
Jeff Bradberry df61d1a59c Upgrade to Django 3.0
- upgrades
  - Django 3.0.14
  - django-jsonfield 1.4.1 (from 1.2.0)
  - django-oauth-toolkit 1.4.1 (from 1.1.3)
    - Stopping here because later versions have changes to the
      underlying model to support OpenID Connect.  Presumably this can
      be dealt with via a migration in our project.
  - django-guid 2.2.1 (from 2.2.0)
  - django-debug-toolbar 3.2.4 (from 1.11.1)
  - python3-saml 1.13.0 (from 1.9.0)
  - xmlsec 1.3.12 (from 1.3.3)

- Remove our project's use of django.utils.six in favor of directly
  using six, in awx.sso.fields.

- Temporarily monkey patch six back in as django.utils.six, since
  django-jsonfield makes use of that import, and is no longer being
  updated.  Hopefully we can do away with this dependency with the new
  generalized JSONField brought in with Django 3.1.

- Force a json decoder to be used with all instances of JSONField
  brought in by django-jsonfield.  This deals with the 'cast to text'
  problem noted previously in our UPGRADE_BLOCKERS.

- Remove the validate_uris validator from the OAuth2Application in
  migration 0025, per the UPGRADE_BLOCKERS, and remove that note.

- Update the TEMPLATES setting to satisfy Django Debug Toolbar.  It
  requires at least one entry that has APP_DIRS=True, and as near as I
  can tell our custom OPTIONS.loaders setting was effectively doing
  the same thing as Django's own machinery if this setting is set.
2022-03-07 18:11:36 -05:00
Jeff Bradberry 32cc8e1a63 Merge pull request #11845 from jbradberry/awxkit-import-role-precedence
Expand out the early membership role assignment
2022-03-07 11:21:48 -05:00