Our multi-cloud CockroachDB test cluster started with 3 nodes in 2 regions: two in Azure Canada Central and one in AWS. It worked, but it had a gap: if Azure Central went down, the cluster went down with it. So we grew it to 5 nodes in 3 regions, adding Azure Canada East and a second AWS node, without rebuilding and without downtime.
This post is the expansion runbook, step by step. Everything follows the CockroachDB production checklist and deployment docs; names, address ranges and IDs are examples.
Why expand, and what changes
| Before: 3 nodes | After: 5 nodes | |
|---|---|---|
| Azure Canada Central | n1, n2 | n1, n2 |
| Azure Canada East | – | n4 NEW |
| AWS ca-central-1 | n3 | n3, n5 NEW |
| Replication factor | 3 | 3 |
| Where the 3 copies live | 2 in Azure Central, 1 in AWS | 1 per region |
| Azure Central region down | Cluster down (2 of 3 copies lost) | Keeps running (East + AWS) |
| AWS region down | Keeps running | Keeps running |
| One AWS VM down for maintenance | AWS has no copy until it returns | n5 still holds AWS's copies |
With only two regions, one of them must hold two of the three copies, and losing that region loses the majority. A third region is what turns "survives a node" into "survives a region". The second AWS node adds capacity and keeps AWS's copy available during maintenance. Replication factor stays at 3.
Plan the new nodes
| Item | Rule |
|---|---|
| VM size | Same as the existing nodes. CockroachDB spreads data and leases evenly, so a smaller node becomes the bottleneck |
| Checklist minimums | At least 4 GiB RAM per vCPU; disks with 500 IOPS and 30 MB/s per vCPU; no burstable VM families |
| Disks | Azure Premium SSD v2 / Premium SSD; AWS gp3 with IOPS and throughput raised above the 3,000 IOPS / 125 MiB/s default for 8 vCPU nodes |
| Version | Exactly the version the cluster runs now (step 05) |
| Locality | Same keys in the same order as the existing nodes, e.g. cloud=…,region=…,zone=… |
zone=1), because every node must use the same locality keys.Extend the network to the new region
The new region must reach every existing node on TCP 26257, and the existing nodes must reach it. Nothing about the existing VMs changes; only routes and rules are added.
| Change | Example |
|---|---|
| New VNet in Canada East NEW | 10.20.0.0/16, not overlapping Central (10.10/16) or AWS (10.30/16) |
| Global VNet peering Central ↔ East NEW | With gateway transit on Central and use remote gateways on East, so East reaches AWS through Central's existing VPN gateway |
| AWS route tables | Add 10.20.0.0/16 via the VPN, next to the existing 10.10.0.0/16 route |
| VPN connection | Add 10.20.0.0/16 to the Azure side's address space / traffic selectors |
| NSG and security groups | Allow TCP 26257 from the new subnet on every node, and from all node subnets on the new nodes; TCP 8080 for ops and probes |
# from the new East node: every existing node must answer for ip in 10.10.1.11 10.10.2.11 10.30.1.11; do nc -zv -w 3 $ip 26257 ping -c 5 $ip | tail -1 done
Prepare the new VMs and clocks
sudo mkfs.ext4 -F /dev/disk/azure/scsi1/lun0 # AWS: /dev/nvme1n1 sudo mkdir -p /mnt/data1 echo '/dev/disk/azure/scsi1/lun0 /mnt/data1 ext4 defaults,nofail 0 2' | sudo tee -a /etc/fstab sudo mount -a sudo useradd -r -m -d /var/lib/cockroach cockroach sudo chown cockroach:cockroach /mnt/data1 chronyc tracking | grep -E 'Reference|System time' timedatectl | grep synchronized
Clock sync first. A node shuts itself down if its clock drifts too close to --max-offset (500 ms by default) from the rest of the cluster. On AWS use the Amazon Time Sync Service (169.254.169.123); on Azure keep chrony on the host clock that current Ubuntu images configure.
Install the same CockroachDB version
SELECT version();
VER=v26.2.0 # the version returned above curl -sSL https://binaries.cockroachdb.com/cockroach-$VER.linux-amd64.tgz | tar -xz sudo cp cockroach-$VER.linux-amd64/cockroach /usr/local/bin/ cockroach version | head -2
Create certificates from the existing CA
The new nodes must trust, and be trusted by, the existing cluster, so their certificates are signed by the same CA (ca.crt + ca.key) you used for n1–n3. The existing nodes keep their certificates.
# n4 (Azure East): own IP and hostname (no load balancer in East) cockroach cert create-node 10.20.1.11 crdb-az-east-0 localhost 127.0.0.1 \ --certs-dir=certs --ca-key=my-safe-directory/ca.key mkdir -p n4 && mv certs/node.crt certs/node.key n4/ && cp certs/ca.crt n4/ # n5 (AWS 1b): own IP and hostname + the AWS internal NLB DNS name cockroach cert create-node 10.30.2.11 crdb-aws-1b-0 crdb.internal.example localhost 127.0.0.1 \ --certs-dir=certs --ca-key=my-safe-directory/ca.key mkdir -p n5 && mv certs/node.crt certs/node.key n5/ && cp certs/ca.crt n5/
ca.key is lost, you cannot sign certificates the cluster trusts, and adding nodes means rotating the CA on every node first. Keep the CA key in a vault, off the database servers.Join the new nodes (no init)
This is the step that differs most from a new build. The new nodes join the running cluster: there is no cockroach init. Point --join at the existing nodes; the cluster accepts them, and they start receiving copies right away.
| Node | --advertise-addr | --locality | --join |
|---|---|---|---|
| n4 NEW | 10.20.1.11 | cloud=azure,region=canadaeast,zone=1 | 10.10.1.11,10.10.2.11,10.30.1.11 |
| n5 NEW | 10.30.2.11 | cloud=aws,region=ca-central-1,zone=ca-central-1b | 10.10.1.11,10.10.2.11,10.30.1.11 |
[Service] Type=notify User=cockroach WorkingDirectory=/var/lib/cockroach ExecStart=/usr/local/bin/cockroach start \ --certs-dir=/var/lib/cockroach/certs \ --store=/mnt/data1 \ --advertise-addr=10.20.1.11 \ --join=10.10.1.11,10.10.2.11,10.30.1.11 \ --locality=cloud=azure,region=canadaeast,zone=1 \ --cache=.25 --max-sql-memory=.25 TimeoutStopSec=300 Restart=always RestartSec=10 LimitNOFILE=65536
sudo systemctl daemon-reload && sudo systemctl enable --now cockroach cockroach node status --certs-dir=certs --host=10.10.1.100:26257
--join to name one node per region (for example n1, n4, n3) so any node can always find the cluster. That is a rolling restart, one node at a time.init is only for the very first start of a brand-new cluster.Add the new AWS node to its load balancer
| Load balancer | Change |
|---|---|
| AWS internal NLB | Register n5 in the target group (TCP 26257, HTTPS health check on 8080 /health?ready=1). Applications keep the same address |
| Azure internal LB | No change: Central still has n1 and n2. East has no application, so n4 gets no load balancer |
Because applications connect through the load balancers, they need no configuration change for the expansion.
Set placement and watch the rebalance
Without constraints, CockroachDB would already spread copies across the new region. Make it explicit, so the layout is a rule and not an outcome:
ALTER RANGE default CONFIGURE ZONE USING
num_replicas = 3,
constraints = '{"+region=canadacentral": 1, "+region=canadaeast": 1, "+cloud=aws": 1}',
lease_preferences = '[[+region=canadacentral], [+region=canadaeast]]';
CockroachDB now moves one copy of each range from Central (which had two) to East, online, by sending snapshots in the background. Leases stay in Central. Watch progress:
-- repeat until has_east equals ranges
SELECT count(*) AS ranges,
count(*) FILTER (WHERE array_to_string(replica_localities, ' ') LIKE '%region=canadacentral%') AS has_central,
count(*) FILTER (WHERE array_to_string(replica_localities, ' ') LIKE '%region=canadaeast%') AS has_east,
count(*) FILTER (WHERE array_to_string(replica_localities, ' ') LIKE '%cloud=aws%') AS has_aws
FROM [SHOW RANGES FROM DATABASE appdb];
The DB Console Replication dashboard shows replicas per node rising on n4 and n5 and the snapshot traffic. On a small test database this takes minutes; on terabytes it takes hours, so start it outside peak load.
Verify, test failures, rerun the workload
cockroach node status --ranges --certs-dir=certs --host=10.10.1.100:26257
SHOW ZONE CONFIGURATION FROM RANGE default; SELECT lease_holder_locality, count(*) FROM [SHOW RANGES FROM DATABASE appdb WITH DETAILS] GROUP BY 1;
| Test | Expected |
|---|---|
| Stop both Central VMs | Cluster keeps running; leases move to East; Azure application slower but working |
| Stop n4 (East) | Keeps running; writes now need an AWS acknowledgement, so measure write latency |
| Stop both AWS VMs | Keeps running; Azure application unaffected |
| Rerun the API tests | Compare with the 3-node results: reads should match (leases unchanged); writes now commit across two Azure regions |
cockroach node decommission 4 5 --certs-dir=certs --host=…) after removing the constraints. CockroachDB moves their copies back before the nodes leave, again online.Expansion checklist
| Item | Done |
|---|---|
| New VMs same size and disk performance as existing ones | ☐ |
| East VNet peered with gateway transit; AWS routes and VPN selectors updated | ☐ |
| 26257 open both ways between all five nodes | ☐ |
| Clocks synchronized on the new nodes | ☐ |
| Same CockroachDB version as the cluster | ☐ |
| New certificates signed by the existing CA | ☐ |
| New nodes started with --join and matching locality keys; no init | ☐ |
| n5 registered in the AWS load balancer | ☐ |
| Zone config applied; rebalance finished (has_east = ranges) | ☐ |
| Failure tests passed and API tests rerun | ☐ |
| Existing nodes' --join updated to one node per region (rolling restart) | ☐ |
Originally published at techstaradmin.blogspot.com.

