* fix: AAP-92631 Only defer control nodes in mesh stability gate
When the mesh routing table is empty, the stability gate was blocking
cleanup of ALL lost instances (both control and execution nodes). However,
execution nodes don't need the mesh and can be safely reaped immediately.
This fix separates lost instances by node type:
- Control nodes: defer cleanup (they need mesh consensus)
- Execution/hop nodes: reap immediately (they don't need mesh)
The previous behavior caused stale execution node instances to accumulate
in the database indefinitely, preventing job cleanup and degrading
controller performance.
Fixes: AAP-92631
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* fix: AAP-92631 Handle Kubernetes mesh gate differently from cluster deployments
The mesh stability gate checks KnownConnectionCosts to detect if receptor
routing is established. In cluster/mesh deployments, empty routing means
the mesh is re-establishing after a controller restart, so the gate blocks
cleanup of lost instances to prevent cluster fragmentation.
However, in Kubernetes (IS_K8S=True), controller pods are stateless and
independent — they don't need peer connections. Empty routing is the
expected steady state, not a sign of instability. The gate was treating
both deployments identically, causing lost instance rows to accumulate
indefinitely in Kubernetes.
This fix adds an IS_K8S check: when Kubernetes mode is enabled, the gate
always passes (returns True), allowing normal cleanup. In cluster mode, the
gate continues to check KnownConnectionCosts.
Fixes: AAP-92631
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
* fix: [AAP-92631] Return execution/hop nodes when mesh gate blocks cleanup
## Issue
When the non-K8s mesh gate blocked cleanup (empty routing table), the code
returned an empty list, preventing execution and hop node cleanup. This was
unintended—these nodes don't depend on mesh consensus and should be reaped
immediately.
## Fix
- Filter lost_instances to return only execution/hop nodes when mesh gate blocks
- Control nodes (which require mesh consensus) continue to be deferred
- K8s deployments bypass the gate entirely (empty routing is steady state)
## Test Coverage
Added three tests per reviewer request:
1. test_kubernetes_mesh_gate_allows_empty_routing (parametrized)
- Unit test verifying gate logic with different mesh_status shapes
- Confirms _mesh_all_ready_nodes_visible() returns True in K8s
2. test_non_kubernetes_mesh_gate_blocks_empty_routing
- Unit test confirming non-K8s gate blocks with empty routing
3. test_mesh_gate_defers_control_but_cleans_execution_hop
- E2E integration test with all three node types
- Verifies exec/hop nodes reach cleanup while control nodes defer
- Matches production failure scenario from baremetal-a cluster
All 18 heartbeat/mesh tests pass.
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Haiku 4.5 <noreply@anthropic.com>
AAP-90120: add _mesh_all_ready_nodes_visible() to block peer-judgment
while KnownConnectionCosts is empty after a controller restart. Gate
fails open when mesh_status is unavailable. Eliminates double receptor
status call per heartbeat.
AAP-90121: wrap each instance's reap+mark_offline block in
advisory_lock('task_manager_lock', wait=False) so the task manager
cannot dispatch to a node being concurrently marked offline. Extracts
_reap_and_mark_lost_instance() helper.
Tests: rewrite new test classes to use real Django DB (functional/)
instead of MagicMock objects.
---------
Signed-off-by: Hui Song <hsong@redhat.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Hui SongandClaude Haiku 4.5