Add ability for a restarted controller to reconnect to in-progress work units
on the execution environment and replay their events, eliminating the need
to re-run jobs after controller restart in hybrid/mesh deployments.
Key changes:
1. Extract _process_phase() so it can be called standalone without re-transmitting
artifacts. Unified per-job loops consolidate startup/heartbeat processing
with explicit adoption/error branches for each job.
2. Add reattach_to_work_unit() to reconnect and stream events from a completed
receptor work unit. Uses hybrid memory-safe dedup (safe_threshold + collision_zone)
to avoid re-sending events already in the database to the callback receiver.
3. Unify job finalization: extract _finalize_job_run() shared between normal
execution and adoption paths. Extract _configure_runner_callback() as the
canonical callback factory used by both paths.
4. Move _receptor_release_work() after terminal status is committed to DB,
eliminating the race where a kill between release and commit left jobs
stuck in 'running' without adoptable work units.
5. Add adopt_job_async @task to run adoption in background, enabling real-time
event streaming from the EE during controller restart recovery.
Cluster-verified across 9 scenarios with all event stream data checked.
Unit and functional tests cover all adoption paths and edge cases.
Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
* fix: AAP-92631 Only defer control nodes in mesh stability gate
When the mesh routing table is empty, the stability gate was blocking
cleanup of ALL lost instances (both control and execution nodes). However,
execution nodes don't need the mesh and can be safely reaped immediately.
This fix separates lost instances by node type:
- Control nodes: defer cleanup (they need mesh consensus)
- Execution/hop nodes: reap immediately (they don't need mesh)
The previous behavior caused stale execution node instances to accumulate
in the database indefinitely, preventing job cleanup and degrading
controller performance.
Fixes: AAP-92631
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* fix: AAP-92631 Handle Kubernetes mesh gate differently from cluster deployments
The mesh stability gate checks KnownConnectionCosts to detect if receptor
routing is established. In cluster/mesh deployments, empty routing means
the mesh is re-establishing after a controller restart, so the gate blocks
cleanup of lost instances to prevent cluster fragmentation.
However, in Kubernetes (IS_K8S=True), controller pods are stateless and
independent — they don't need peer connections. Empty routing is the
expected steady state, not a sign of instability. The gate was treating
both deployments identically, causing lost instance rows to accumulate
indefinitely in Kubernetes.
This fix adds an IS_K8S check: when Kubernetes mode is enabled, the gate
always passes (returns True), allowing normal cleanup. In cluster mode, the
gate continues to check KnownConnectionCosts.
Fixes: AAP-92631
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* fix: [AAP-92631] Return execution/hop nodes when mesh gate blocks cleanup
## Issue
When the non-K8s mesh gate blocked cleanup (empty routing table), the code
returned an empty list, preventing execution and hop node cleanup. This was
unintended—these nodes don't depend on mesh consensus and should be reaped
immediately.
## Fix
- Filter lost_instances to return only execution/hop nodes when mesh gate blocks
- Control nodes (which require mesh consensus) continue to be deferred
- K8s deployments bypass the gate entirely (empty routing is steady state)
## Test Coverage
Added three tests per reviewer request:
1. test_kubernetes_mesh_gate_allows_empty_routing (parametrized)
- Unit test verifying gate logic with different mesh_status shapes
- Confirms _mesh_all_ready_nodes_visible() returns True in K8s
2. test_non_kubernetes_mesh_gate_blocks_empty_routing
- Unit test confirming non-K8s gate blocks with empty routing
3. test_mesh_gate_defers_control_but_cleans_execution_hop
- E2E integration test with all three node types
- Verifies exec/hop nodes reach cleanup while control nodes defer
- Matches production failure scenario from baremetal-a cluster
All 18 heartbeat/mesh tests pass.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
AAP-90120: add _mesh_all_ready_nodes_visible() to block peer-judgment
while KnownConnectionCosts is empty after a controller restart. Gate
fails open when mesh_status is unavailable. Eliminates double receptor
status call per heartbeat.
AAP-90121: wrap each instance's reap+mark_offline block in
advisory_lock('task_manager_lock', wait=False) so the task manager
cannot dispatch to a node being concurrently marked offline. Extracts
_reap_and_mark_lost_instance() helper.
Tests: rewrite new test classes to use real Django DB (functional/)
instead of MagicMock objects.
---------
Signed-off-by: Hui Song <hsong@redhat.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Hui SongandClaude Sonnet 4.6