Configuration Guide

4.7.2 Configuration Guide

To configure active standby and arbitration, use the following settings on each replication-manager.

4.7.2.1 Arbitration settings

All arbitration settings are server-scoped (global): they apply to the whole replication-manager instance and can be changed at runtime from the Arbitration card in the global Settings page of the dashboard.

arbitration-external (1.0)
Item Value
Description Enable external arbitration on split brain. When enabled, the instance starts in standby mode and contacts the arbitrator for election.
Type Boolean
Default Value false

Enabling external arbitration through the API or the dashboard requires a registered Cloud18 account with a support, support-services or partner subscription plan. Free or unregistered instances cannot enable it (the API returns an error, and automatic split-brain resolution reports ERR00104).

arbitration-external-hosts (1.0)
Item Value
Description Address of the external arbitrator service. Supports https:// prefix for TLS.
Type String
Default Value "https://arbitrator.cloud18.io" (was "88.191.151.84:80" before 3.1)

When using a TLS reverse proxy in front of the arbitrator, prefix the address with https://:

arbitration-external-hosts = "https://arbitrator.cloud18.io"

Without a scheme prefix, http:// is used by default — no breaking change for existing configurations.

arbitration-external-secret (1.0)
Item Value
Description Shared secret used by the arbitrator to identify your cluster. Should be unique across all users of the arbitrator. Use your organization name combined with random alphanumeric characters.
Type String
Default Value ""

The secret is not an authentication credential: together with the cluster name it forms the primary key under which the arbitrator stores your heartbeats and arbitration decisions. Its purpose is to prevent another (possibly malicious) user of a shared arbitrator from overwriting your cluster's records.

When the secret is stored encrypted in the configuration (hash_... value produced by replication-manager password), the encrypted form is what is sent to the arbitrator — the plain-text secret never leaves the server, consistent with the general secret handling described in the security chapter. This is safe for arbitration because both instances of an active/standby pair share the same encryption key anyway (a shared key is required for configuration exchange via git), so both peers send the identical value and register under the same primary key.

arbitration-external-unique-id (1.0)
Item Value
Description Unique identifier for each replication-manager instance. Each instance in the pair must have a different value.
Type Integer
Default Value 0

When both instances are standby, the one with the lower id becomes active. With the same id on both sides (the default 0 left on both) neither does: the pair stays standby, and since 3.1.44 the state GWARN020 names the setting to change.

arbitration-peer-hosts (1.0)
Item Value
Description Address where the peer replication-manager answers /api/heartbeat: its http-port, or an https:// URL in front of it.
Type String
Default Value "127.0.0.1:10001"

The heartbeat is served by the peer's http-port (10001 by default, plain HTTP), not by its API port api-port (10005), which answers 404 on /api/heartbeat. An address without a scheme is called over http://.

# peer reached directly on the local network or VPN
arbitration-peer-hosts = "192.168.1.20:10001"
# peer reached through a TLS reverse proxy or gateway in front of its http-port
arbitration-peer-hosts = "https://repman-b.example.com"

Check from each host that the address answers: curl <address>/api/heartbeat returns {"uuid":…,"status":"A"} (or "S"). Any other answer reads as a split brain (GWARN006). Since 3.1.44:

  • every failed heartbeat keeps its reason, the URL called and the error, in the state GWARN018 (a 404 says the address is the API port instead of the http-port);
  • an address written without a scheme is retried once over https:// when the peer refuses plain HTTP (or over http:// the other way round); the scheme that answers is kept for that peer and the state GWARN019 gives the value to write;
  • an address written with a scheme is never changed: when the scheme is wrong, GWARN018 says so.

Over https:// the peer's certificate is verified against the system's trusted authorities.

arbitration-failed-master-script (2.1)
Item Value
Description External script executed when a master loses arbitration during split brain.
Type String
Default Value ""
arbitration-read-timeout (2.0)
Item Value
Description Read timeout for arbitrator responses in milliseconds. Should be shorter than monitoring-ticker (in seconds) to avoid overloading the monitoring loop.
Type Integer
Default Value 800
arbitration-verdict-streak (3.1.43)
Item Value
Description Consecutive arbitrator answers of the same kind, in monitoring ticks, before a cluster changes status: a cluster goes standby after that many looser verdicts in a row, and the minority fail-safe (yield, read-only master) fires after that many unreachable answers in a row, an arbitrator answering that it has no store counting as unreachable. A winner resets the count. One answer, or two arbitrator instances disagreeing, never moves a cluster. The trade-off: a real loss, and the minority fail-safe, act that many ticks later than with 1 (6 seconds with the default ticker), in exchange for immunity to one bad answer.
Type Integer
Default Value 3 (1 = act on every answer)

4.7.2.2 Setup example

Give each instance a different arbitration-external-unique-id and point each to its peer:

Instance A (192.168.1.10, http-port 10001):

arbitration-external = true
arbitration-external-hosts = "https://arbitrator.signal18.io"
arbitration-external-secret = "myorg-a7x9k2m"
arbitration-external-unique-id = 1
arbitration-peer-hosts = "192.168.1.20:10001"

Instance B (192.168.1.20, http-port 10001):

arbitration-external = true
arbitration-external-hosts = "https://arbitrator.signal18.io"
arbitration-external-secret = "myorg-a7x9k2m"
arbitration-external-unique-id = 2
arbitration-peer-hosts = "192.168.1.10:10001"

Both instances must use the same arbitration-external-secret and arbitration-external-hosts, a different arbitration-external-unique-id, and a peer address that answers /api/heartbeat (the http-port, or an https:// URL in front of it).

4.7.2.3 Active/standby toggle

The active/standby status can be toggled via the REST API or the dashboard.

API:

GET /api/clusters/{clusterName}/actions/set-active-status

This endpoint requires the cluster-settings ACL grant. It toggles the cluster between active and standby mode:

  • Without arbitration — the toggle is a local-only operation. Useful for testing.
  • With arbitration — the toggle forces an arbitrator election so the peer instance can take over as active.

Dashboard:

The active/standby status is displayed as a clickable pill in the cluster detail header. Clicking it opens a confirmation dialog before toggling. The pill is only clickable for users with the cluster-settings grant.

4.7.2.4 Running a private arbitrator

The arbitrator is a separate binary (replication-manager-arb) that can be deployed independently.

arbitrator-bind-address (2.0 arb)
Item Value
Description Arbitrator listen address and port.
Type String
Default Value "0.0.0.0:10001"
arbitrator-driver (2.0 arb)
Item Value
Description Arbitrator backend storage type.
Type Enum
List Values sqlite, mysql
Default Value "sqlite"

SQLite backend (default, simplest):

[arbitrator]
title = "arbitrator"
arbitrator-bind-address = "0.0.0.0:10001"
arbitrator-driver = "sqlite"

[default]
monitoring-datadir = "/var/lib/replication-manager"

Start:

replication-manager-arb arbitrator

MySQL backend:

[arbitrator]
title = "arbitrator"
arbitrator-bind-address = "0.0.0.0:10001"
arbitrator-driver = "mysql"
db-servers-hosts = "192.168.0.201:3306"
db-servers-credential = "user:password"

[default]
Health check

The arbitrator exposes a GET /health endpoint that returns HTTP 200 with {"status":"ok"} when the backend database is reachable, or HTTP 503 with {"status":"failed"} otherwise. Since 3.1.43 an election request the arbitrator cannot decide, because its store is unreachable, answers HTTP 503 {"arbitration":"error"} instead of a looser verdict, and replication-manager treats it as an unreachable arbitrator. When the arbitrator is an app behind a Cloud18 gateway, give its route a monitor on /health: the gateway then checks it and removes an instance without its store from the rotation. Use this for load balancer health checks when running behind a reverse proxy.

Multi-tenant

A single arbitrator can serve multiple replication-manager pairs. Each pair is identified by its arbitration-external-secret and cluster name. There is no configuration needed on the arbitrator side — it stores heartbeats and arbitration decisions keyed by these identifiers.

TLS reverse proxy

The arbitrator itself listens on plain HTTP. To expose it securely over the internet, place it behind a TLS reverse proxy (nginx, HAProxy, Caddy, etc.). The replication-manager instances then use https:// in arbitration-external-hosts to reach it.

4.7.2.5 Bootstrap fallback to the standby (3.1.44)

Every database and proxy fetches its configuration from replication-manager when it starts: the OpenSVC init container, the on-premise provisioning scripts and the Kubernetes init container. They call the instance that provisioned the service. Without a fallback, a restart during a takeover, while that instance is down, cannot fetch anything, and the service starts on the configuration already on its volume.

With arbitration-external = true, the two instances of the pair offer each other as a fallback, with nothing to configure:

  • Each instance advertises its API URL (https://<monitoring-address>:<api-port>) in its /api/heartbeat answer. Its peer keeps that URL only when it is a plain https://host:port whose host is the one configured in arbitration-peer-hosts.
  • The URLs of the pair are passed to every bootstrap as REPLICATION_MANAGER_URL_DR:
    • OpenSVC: a key of the namespace env config;
    • on-premise: the SSH environment;
    • Kubernetes: written into the init container command.
  • A bootstrap tries REPLICATION_MANAGER_URL first, then each URL of the pair not tried yet. It then uses the one that answered for the login, the configuration and the CLI.

Requirements:

  • both instances run 3.1.44 or later;
  • monitoring-address of each instance resolves from the database and proxy hosts;
  • both instances accept the same API credentials.

A dead main instance adds about 10 seconds per step to a start, because each attempt is bounded by a 10-second timeout.