Incident response

​

Severity levels

​

P0: Node not signing blocks

# Check validator power (0 = not signing)
curl -s localhost:26660/metrics | grep tendermint_consensus_validator_power

# Check signing info
autheod query slashing signing-info \
  $(autheod tendermint show-validator --home /path/to/node-home)

1. Check if the service is running: sudo systemctl status autheod
2. If stopped, restart: sudo systemctl start autheod
3. Check logs: sudo journalctl -u autheod -n 200 --no-pager
4. Check sync status: autheod status | jq '.SyncInfo'
5. If out of sync, restore from snapshot (see Backups and restore)

​

P0: Validator tombstoned

autheod query slashing signing-info \
  $(autheod tendermint show-validator --home /path/to/node-home) \
  | grep tombstoned

1. Stop the tombstoned node immediately
2. Do not attempt unjail — it will fail
3. Commission a new server
4. Generate a new consensus key: autheod init new-validator --chain-id autheo_2127-1
5. Register a new validator with MsgCreateValidator and a new consensus key
6. Bind your Sovereign license to the new validator address

​

P1: Validator jailed (liveness)

autheod query staking validator <autheovaloper-address> | grep status
# Look for: BOND_STATUS_UNBONDED (jailed)

sudo journalctl -u autheod --since "2 hours ago" | grep -E "error|panic|missed"

autheod status | jq '.SyncInfo.catching_up'
# Must return false before unjailing

autheod query license license <license-id>

autheod tx slashing unjail \
  --from mykey \
  --chain-id autheo_2127-1 \
  --keyring-backend file

autheod tx staking delegate <autheovaloper-address> <amount>aauth \
  --from mykey --chain-id autheo_2127-1 --keyring-backend file

​

P1: Node not syncing (catching_up: true)

autheod status | jq '.SyncInfo.catching_up'
# Returns true if behind

1. Check peer count: curl -s localhost:26657/net_info | jq '.result.n_peers'
2. If peers < 3, add persistent peers in config/config.toml
3. If syncing is extremely slow (hours behind), restore from snapshot:
   sudo systemctl stop autheod
   rm -rf /path/to/node-home/data/
   wget https://snapshot.autheo.com/data_backup_latest.tar.gz
   tar xzvf data_backup_latest.tar.gz
   mv data/ /path/to/node-home/data/
   cp /secure/backup/priv_validator_state.json /path/to/node-home/data/priv_validator_state.json
   sudo systemctl start autheod
   

​

P1: Hardware failure — migrate to new host

​

Post-incident review

1. Document the timeline (when detected, when resolved)
2. Identify the root cause
3. Update monitoring thresholds if the incident was not caught early enough
4. Review the monitoring checklist
5. Update runbooks if the incident revealed a gap