Use shared lock for project copy when no sync is needed
When a job template runs and no project sync is required (the common
case for slice jobs), multiple jobs were serializing behind an exclusive
lock even though they were only reading the shared project directory to
copy it.
Introduce read/write lock semantics using fcntl LOCK_SH/LOCK_EX:
- acquire_lock gains an exclusive= parameter (default True, preserving
existing behavior for all callers that don't pass it)
- sync_and_copy always acquires LOCK_SH first, then refreshes DB state
and calls get_sync_needs() under the lock — preserving the invariant
that the sync check always happens under a lock (no TOCTOU window)
- If sync is needed, LOCK_SH is released and LOCK_EX is acquired before
proceeding; sync_and_copy_without_lock re-checks internally
- project.refresh_from_db() is guarded by project.pk to handle unsaved
model instances used in unit tests
This allows slice jobs backed by the same project to copy the project
tree concurrently instead of sequentially. Measured improvement on a
200MB/54k-file project with 8 slice jobs: 140.5s total lock wait → 0s,
wall time 42.6s → 16.9s (2.5x).
Assisted-by: Claude Code via Google Vertex AI <cmeyers+claude@redhat.com>
Django allows passing update_fields=None to model.save() to mean 'update all fields'.
However, kwargs.get('update_fields', []) returns None when the key exists with a None value,
rather than the default empty list.
This caused crashes in mark_field_for_save() and other code that assumes update_fields is
a list when doing membership tests ('field' not in update_fields) or append operations.
Changed pattern from:
update_fields = kwargs.get('update_fields', [])
To:
update_fields = kwargs.get('update_fields') or []
This correctly handles:
- Key missing: get() returns None → or [] gives []
- Key present with None: get() returns None → or [] gives []
- Key present with list: get() returns list → or [] keeps the list
Fixed in 12 locations across 8 model files:
- awx/main/models/base.py (3 instances)
- awx/main/models/unified_jobs.py (2)
- awx/main/models/jobs.py (2)
- awx/main/models/ad_hoc_commands.py
- awx/main/models/inventory.py
- awx/main/models/notifications.py
- awx/main/models/projects.py
- awx/main/models/workflow.py
Fixes task manager crashes with:
TypeError: argument of type 'NoneType' is not iterable
* git worktree friendly precomit install
* worktrees don't have a .git directory. Before, docker-compose would
trigger pre-commit install and fail.
* make docker-compose work in git worktree
* AWX tries to discover the version via info stored in .git/ dir.
setuptools-scm is capable of finding the .git/ dir, starting from a
worktree, but is unable because only the worktree is mapped into the
container, not the .git/ dir itself. Thus, we have to detect and pass
the version into the container from outside. That is why this change
landed in the Makefile.
* Add test to recreate the error
* Also begin to add detection for empty event
* Remove breakpoint
* fix: ignore events with missing event types
* run linter and apply changes
---------
Co-authored-by: AlanCoding <arominge@redhat.com>
Co-authored-by: Peter Braun <pbraun@redhat.com>
* Support <collection_namespace>.<collection_name>.* indirect host query
to match ANY module in the <collection_namespace>.<collection_name>
* Add tests for new wildcard indirect host count
* error checking of ansible event name
* error checking of ansible event query
* Support <collection_namespace>.<collection_name>.* indirect host query
to match ANY module in the <collection_namespace>.<collection_name>
* Add tests for new wildcard indirect host count
* error checking of ansible event name
* error checking of ansible event query
* Align Orign and Host header
* Before this change the Host: header was runserver. Seems to be set by
nginx upstream flow.
* After this change we explicitly set the Host: header
* More about CSRF checks ...
CSRF checks that Origin == Host. Think about how the browser works.
<browser goes to awx.com>
"I'm executing javascript that I downloaded from awx.com (ORIGIN) and
I'm making an XHR POST request to awx.com (HOST)"
Server verifies; Host: header == Origin: header; OK!
vs. the malicious case.
<hacker injects javascript code into google.com>
<browser goes to google.com>
"I'm executing javascript that I downloaded from google.com (ORIGIN)
and I'm making an XHR POST request to awx.com (HOST)"
Server verifies; Host: header != Origin: header; NOT OK!
* Update awx/settings/development.py
---------
Co-authored-by: Hao Liu <44379968+TheRealHaoLiu@users.noreply.github.com>
Tried to dig as to why we ever needed this and could not find the answer. We removed it and ran all the tests and the tests passed so assuming it's no longer needed.
* allow pytest --migrations to succeed
* We actually subvert migrations from running in test via pytest.ini
--no-migrations option. This has led to bit rot for the sqlite
migrations happy path. This changeset pays off that tech debt and
allows for an sqlite migration happy path.
* This paves the way for programatic invocation of individual migrations
and weaving of the creation of resources (i.e. Instance, Job Template,
etc). With this, a developer can instantiate various database states,
trigger a migration, assert the state of the db, and then have pytest
rollback all of that.
* I will note that in practice, running these migrations is dog shit
slow BUT this work also opens up the possibility of saving and
re-using sqlite3 database files. Normally, caching is not THE answer
and causes more harm than good. But in this case, our migrations are
mostly write-once (I say mostly because this change set violates
that :) so cache invalidation isn't a major issue.
* functional test for migrations on sqlite
* We commonly subvert running migrations in test land. Test land uses
sqlite. By not constantly exercising this code path it atrophies. The
smoke test here is to continuously exercise that code path.
* Add ci test to run migration tests separately, they take =~ 2-3
minutes each on my laptop.
* The smoke tests also serves as an example of how to write migration
tests.
* run migration tests in ci
Getting all connections that have been alive > 5 minutes is noisy. Instead, we want only queries that are active and have been alive for more than 5 minutes. This will filter out any idle connections.
Chris Meyers
