mirror of
https://github.com/ansible/awx.git
synced 2026-10-29 22:03:20 -02:30
Dirk JulichandClaude Opus 4.6
5613b5ac6c
* AAP-96085: Decouple collection cache from project cache_id Introduce a requirements-hash-keyed cache (reqcache_<hash>/) so that ansible-galaxy collection/role installs are reused across project commits when requirements files haven't changed. Previously, any commit to the project repo invalidated the entire cache, triggering a full reinstall of all collections — a single collection like community.general 11.0.0 (~3,500 files) adds 40-90s of I/O on SAN-backed storage. * Address review feedback: atomic cache publish, bounded eviction, path-aware hash - Atomic publish: _sync_requirements_cache() copies into a temp directory and renames into place, so a failed copytree cannot leave a partial reqcache directory that looks valid to subsequent cache-hit checks - Bounded eviction: clear_project_cache() evicts the oldest reqcache_* directories when the count exceeds max_reqcache (default 10), preventing unbounded storage growth from repeated requirements changes - Path-aware hash: requirements_hash now includes file paths alongside checksums via zip(), so swapping content between requirements files correctly invalidates the cache * Protect current reqcache from eviction during cleanup Read the requirements hash from stage/.requirements_hash and exclude the matching reqcache_* entry from age-based eviction. Without this, a cache hit on the oldest entry could be evicted before the job uses it. * AAP-96176: Cache tarball for make_local_copy() to reduce small-file I/O After populating reqcache_<hash>/, create a tarball of each subfolder (requirements_collections.tar, requirements_roles.tar). make_local_copy() prefers the tarball when available — copies and extracts one file instead of shutil.copytree() across potentially thousands of individual collection files. Falls back to copytree when no tarball exists (upgrade compatibility). On SAN-backed storage, this significantly reduces per-job copy I/O by eliminating per-file source reads. * Fix direct-cache tarball cross-reference and partial extraction fallback Remove reqcache tarball lookup on direct cache hits — the direct cache could diverge from the reqcache (e.g. galaxy server changes not in the requirements hash), serving stale content. The tarball optimization still works for reqcache fallback hits where cache_subpath + '.tar' resolves directly. Clean up partially extracted destination before copytree fallback so a failed tarball extraction doesn't leave behind a directory that causes FileExistsError. --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>