Skip to main content

ADR-013: Adaptive Consensus for 2-Node Clusters

Status​

Accepted

Context​

Many users will deploy 2-node clusters (primary + backup), but traditional consensus algorithms (Raft/Paxos) require odd numbers for majority quorum. Need a solution that provides good failover characteristics for 2-node deployments while scaling to larger clusters.

Decision​

Implement adaptive consensus that automatically switches modes based on cluster size:

  • 1 node: Standalone mode (no consensus needed)
  • 2 nodes: Primary/Replica with lease-based arbitration
  • 3+ nodes: Full Raft consensus

For 2-node clusters, provide multiple options based on environment:

  1. Conservative mode (default): 30s lease, 35s failover - safe for WAN/unreliable networks
  2. Aggressive LAN mode: 5s lease, 6s failover - requires reliable LAN + NTP
  3. Quorum disk: <1s failover - requires shared storage (NFS/SAN)
  4. Witness node: <100ms failover - requires 3rd device (can be tiny)
  5. Eventual consistency: CRDTs only - for availability over consistency

Consequences​

Positive​

  • First-class support for common 2-node deployments
  • Automatic mode selection reduces configuration complexity
  • Multiple options for different infrastructure constraints
  • LAN users can achieve <1s failover with quorum disk
  • Witness node option provides full Raft benefits with minimal resources
  • Graceful degradation during arbitrator failures
  • Clear upgrade path from 2 to 3+ nodes

Negative​

  • Default 35s failover time for safety (configurable down to 6s for LAN)
  • Requires external dependency (arbitrator/witness/storage) for strong consistency
  • Additional complexity in consensus layer
  • Different operational characteristics based on cluster size
  • Users must understand trade-offs between safety and speed

Alternatives Considered​

  1. Force 3+ nodes: Too restrictive for many users
  2. No 2-node support: Would lose significant user base
  3. Split-brain acceptance: Data consistency issues
  4. Blockchain consensus: Overkill and too slow
  5. Vector clocks only: No strong consistency option

Implementation​

The consensus layer detects cluster size at startup and runtime:

  • 1 node: Bypasses consensus entirely
  • 2 nodes: Activates lease-based primary election with chosen arbitrator
  • 3+ nodes: Standard Raft with leader election

Configuration example:

[cluster]
mode = "auto"  # auto-detect based on size

[cluster.two_node]
arbitrator = "quorum_disk"  # or "witness", "lease"
lease_duration = "5s"       # for LAN mode
quorum_disk_path = "/mnt/shared/quorum"
witness_url = "http://witness.local:8080"