Re-sync orchestrator tracked reservations with executor usage.
There are several state to balance for individual hosts. From the executor perspective the host can either be reserved
for it or not. From the orchestrator perspective, the host can be idle, reserved for a given executor or missing
completely - presumably because of recent removal from config repo. There are several situations resulting from that:
Agreement
- A1: All nodes agree the host is idle,
- A2: Orchestrator agree with the only executor that claims the host.
No action needed here as grid is in sync.
No collision
- NC1: Orchestrator tracks a reservation for executor that does not report the host being reserved. UC: Executor failover or Missed returnNode call
- NC2: Orchestrator tracks no reservation for the host yet one executor claims it. UC: Orchestrator failover.
- NC3: Orchestrator tracks reservation of a host for executor X that does not report it while executor Y does. Bug or Race Condition
State representation of the Orchestrator is adjusted to match the one of the grid.
Collision
- C1: Orchestrator tracks reservation but extra executors report usage of the host. Bug or Race condition.
- C2: Multiple executors report reservation but orchestrator tracks none. Bug or Race condition.