From 3 to 5 Nodes: Adding a Third Region to a Multi-Cloud CockroachDB Cluster, Online

 



Our multi-cloud CockroachDB test cluster started with 3 nodes in 2 regions: two in Azure Canada Central and one in AWS. It worked, but it had a gap: if Azure Central went down, the cluster went down with it. So we grew it to 5 nodes in 3 regions, adding Azure Canada East and a second AWS node, without rebuilding and without downtime.

This post is the expansion runbook, step by step. Everything follows the CockroachDB production checklist and deployment docs; names, address ranges and IDs are examples.

Part of a series: the leaseholder latency case study (on the original 3-node cluster) and replication factor 3 across three regions (how the expanded cluster places its copies).


STEP 01 / 10

Why expand, and what changes

Before: 3 nodesAfter: 5 nodes
Azure Canada Centraln1, n2n1, n2
Azure Canada East–n4 NEW
AWS ca-central-1n3n3, n5 NEW
Replication factor33
Where the 3 copies live2 in Azure Central, 1 in AWS1 per region
Azure Central region downCluster down (2 of 3 copies lost)Keeps running (East + AWS)
AWS region downKeeps runningKeeps running
One AWS VM down for maintenanceAWS has no copy until it returnsn5 still holds AWS's copies

With only two regions, one of them must hold two of the three copies, and losing that region loses the majority. A third region is what turns "survives a node" into "survives a region". The second AWS node adds capacity and keeps AWS's copy available during maintenance. Replication factor stays at 3.

STEP 02 / 10

Plan the new nodes

ItemRule
VM sizeSame as the existing nodes. CockroachDB spreads data and leases evenly, so a smaller node becomes the bottleneck
Checklist minimumsAt least 4 GiB RAM per vCPU; disks with 500 IOPS and 30 MB/s per vCPU; no burstable VM families
DisksAzure Premium SSD v2 / Premium SSD; AWS gp3 with IOPS and throughput raised above the 3,000 IOPS / 125 MiB/s default for 8 vCPU nodes
VersionExactly the version the cluster runs now (step 05)
LocalitySame keys in the same order as the existing nodes, e.g. cloud=…,region=…,zone=…
Canada East has no availability zones. Give n4 a zone label anyway (for example zone=1), because every node must use the same locality keys.
STEP 03 / 10

Extend the network to the new region

The new region must reach every existing node on TCP 26257, and the existing nodes must reach it. Nothing about the existing VMs changes; only routes and rules are added.

ChangeExample
New VNet in Canada East NEW10.20.0.0/16, not overlapping Central (10.10/16) or AWS (10.30/16)
Global VNet peering Central ↔ East NEWWith gateway transit on Central and use remote gateways on East, so East reaches AWS through Central's existing VPN gateway
AWS route tablesAdd 10.20.0.0/16 via the VPN, next to the existing 10.10.0.0/16 route
VPN connectionAdd 10.20.0.0/16 to the Azure side's address space / traffic selectors
NSG and security groupsAllow TCP 26257 from the new subnet on every node, and from all node subnets on the new nodes; TCP 8080 for ops and probes
$ on n4
# from the new East node: every existing node must answer
for ip in 10.10.1.11 10.10.2.11 10.30.1.11; do
  nc -zv -w 3 $ip 26257
  ping -c 5 $ip | tail -1
done
✓ Check: East ↔ Central a few ms, East ↔ AWS similar to Central ↔ AWS, no loss. Run the same check from n5 to all Azure nodes.
STEP 04 / 10

Prepare the new VMs and clocks

# on n4 and n5
sudo mkfs.ext4 -F /dev/disk/azure/scsi1/lun0          # AWS: /dev/nvme1n1
sudo mkdir -p /mnt/data1
echo '/dev/disk/azure/scsi1/lun0 /mnt/data1 ext4 defaults,nofail 0 2' | sudo tee -a /etc/fstab
sudo mount -a
sudo useradd -r -m -d /var/lib/cockroach cockroach
sudo chown cockroach:cockroach /mnt/data1

chronyc tracking | grep -E 'Reference|System time'
timedatectl | grep synchronized

Clock sync first. A node shuts itself down if its clock drifts too close to --max-offset (500 ms by default) from the rest of the cluster. On AWS use the Amazon Time Sync Service (169.254.169.123); on Azure keep chrony on the host clock that current Ubuntu images configure.

STEP 05 / 10

Install the same CockroachDB version

SQL on the existing cluster
SELECT version();
# on n4 and n5
VER=v26.2.0                      # the version returned above
curl -sSL https://binaries.cockroachdb.com/cockroach-$VER.linux-amd64.tgz | tar -xz
sudo cp cockroach-$VER.linux-amd64/cockroach /usr/local/bin/
cockroach version | head -2
STEP 06 / 10

Create certificates from the existing CA

The new nodes must trust, and be trusted by, the existing cluster, so their certificates are signed by the same CA (ca.crt + ca.key) you used for n1–n3. The existing nodes keep their certificates.

$ admin workstation
# n4 (Azure East): own IP and hostname (no load balancer in East)
cockroach cert create-node 10.20.1.11 crdb-az-east-0 localhost 127.0.0.1 \
  --certs-dir=certs --ca-key=my-safe-directory/ca.key
mkdir -p n4 && mv certs/node.crt certs/node.key n4/ && cp certs/ca.crt n4/

# n5 (AWS 1b): own IP and hostname + the AWS internal NLB DNS name
cockroach cert create-node 10.30.2.11 crdb-aws-1b-0 crdb.internal.example localhost 127.0.0.1 \
  --certs-dir=certs --ca-key=my-safe-directory/ca.key
mkdir -p n5 && mv certs/node.crt certs/node.key n5/ && cp certs/ca.crt n5/
No CA key, no expansion. If the original ca.key is lost, you cannot sign certificates the cluster trusts, and adding nodes means rotating the CA on every node first. Keep the CA key in a vault, off the database servers.
STEP 07 / 10

Join the new nodes (no init)

This is the step that differs most from a new build. The new nodes join the running cluster: there is no cockroach init. Point --join at the existing nodes; the cluster accepts them, and they start receiving copies right away.

Node--advertise-addr--locality--join
n4 NEW10.20.1.11cloud=azure,region=canadaeast,zone=110.10.1.11,10.10.2.11,10.30.1.11
n5 NEW10.30.2.11cloud=aws,region=ca-central-1,zone=ca-central-1b10.10.1.11,10.10.2.11,10.30.1.11
/etc/systemd/system/cockroach.service (n4, [Unit]/[Install] as on the other nodes)
[Service]
Type=notify
User=cockroach
WorkingDirectory=/var/lib/cockroach
ExecStart=/usr/local/bin/cockroach start \
  --certs-dir=/var/lib/cockroach/certs \
  --store=/mnt/data1 \
  --advertise-addr=10.20.1.11 \
  --join=10.10.1.11,10.10.2.11,10.30.1.11 \
  --locality=cloud=azure,region=canadaeast,zone=1 \
  --cache=.25 --max-sql-memory=.25
TimeoutStopSec=300
Restart=always
RestartSec=10
LimitNOFILE=65536
$ start, then check from the admin workstation
sudo systemctl daemon-reload && sudo systemctl enable --now cockroach
cockroach node status --certs-dir=certs --host=10.10.1.100:26257
✓ Check: 5 rows, all live, n4 and n5 with the new localities. Then, at your next maintenance window, update the existing nodes' --join to name one node per region (for example n1, n4, n3) so any node can always find the cluster. That is a rolling restart, one node at a time.
Never run cockroach init here. The cluster is already initialized. init is only for the very first start of a brand-new cluster.
STEP 08 / 10

Add the new AWS node to its load balancer

Load balancerChange
AWS internal NLBRegister n5 in the target group (TCP 26257, HTTPS health check on 8080 /health?ready=1). Applications keep the same address
Azure internal LBNo change: Central still has n1 and n2. East has no application, so n4 gets no load balancer

Because applications connect through the load balancers, they need no configuration change for the expansion.

STEP 09 / 10

Set placement and watch the rebalance

Without constraints, CockroachDB would already spread copies across the new region. Make it explicit, so the layout is a rule and not an outcome:

SQL
ALTER RANGE default CONFIGURE ZONE USING
  num_replicas = 3,
  constraints = '{"+region=canadacentral": 1, "+region=canadaeast": 1, "+cloud=aws": 1}',
  lease_preferences = '[[+region=canadacentral], [+region=canadaeast]]';

CockroachDB now moves one copy of each range from Central (which had two) to East, online, by sending snapshots in the background. Leases stay in Central. Watch progress:

SQL
-- repeat until has_east equals ranges
SELECT count(*) AS ranges,
       count(*) FILTER (WHERE array_to_string(replica_localities, ' ') LIKE '%region=canadacentral%') AS has_central,
       count(*) FILTER (WHERE array_to_string(replica_localities, ' ') LIKE '%region=canadaeast%')    AS has_east,
       count(*) FILTER (WHERE array_to_string(replica_localities, ' ') LIKE '%cloud=aws%')            AS has_aws
FROM [SHOW RANGES FROM DATABASE appdb];

The DB Console Replication dashboard shows replicas per node rising on n4 and n5 and the snapshot traffic. On a small test database this takes minutes; on terabytes it takes hours, so start it outside peak load.

STEP 10 / 10

Verify, test failures, rerun the workload

$ admin workstation
cockroach node status --ranges --certs-dir=certs --host=10.10.1.100:26257
SQL
SHOW ZONE CONFIGURATION FROM RANGE default;
SELECT lease_holder_locality, count(*) FROM [SHOW RANGES FROM DATABASE appdb WITH DETAILS] GROUP BY 1;
✓ Check: ranges_underreplicated and ranges_unavailable 0 on all 5 nodes; has_central = has_east = has_aws = ranges; leaseholders in canadacentral.
TestExpected
Stop both Central VMsCluster keeps running; leases move to East; Azure application slower but working
Stop n4 (East)Keeps running; writes now need an AWS acknowledgement, so measure write latency
Stop both AWS VMsKeeps running; Azure application unaffected
Rerun the API testsCompare with the 3-node results: reads should match (leases unchanged); writes now commit across two Azure regions
Rollback, if you ever need it: decommission the new nodes (cockroach node decommission 4 5 --certs-dir=certs --host=…) after removing the constraints. CockroachDB moves their copies back before the nodes leave, again online.

Expansion checklist

ItemDone
New VMs same size and disk performance as existing ones☐
East VNet peered with gateway transit; AWS routes and VPN selectors updated☐
26257 open both ways between all five nodes☐
Clocks synchronized on the new nodes☐
Same CockroachDB version as the cluster☐
New certificates signed by the existing CA☐
New nodes started with --join and matching locality keys; no init☐
n5 registered in the AWS load balancer☐
Zone config applied; rebalance finished (has_east = ranges)☐
Failure tests passed and API tests rerun☐
Existing nodes' --join updated to one node per region (rolling restart)☐
© 2026 Tamilselvan Perumal · TechStarAdmin. Based on the CockroachDB production checklist and deployment documentation; names, addresses and identifiers are examples. Views are my own. Test in a non-production environment first.
Originally published at techstaradmin.blogspot.com.