"We set replication factor 3 and we have three regions, so there is one copy in each region, right?" I was asked exactly this while handing over a CockroachDB cluster stretched across Azure and AWS. The honest answer is "not by definition, but in our case yes, and here is how we proved it."
This post explains how CockroachDB decides where the copies go, how to check it with two queries, what survives each kind of outage (including one that surprised the team), and how to pause a multi-cloud test cluster overnight without breaking it.
01/ The question
Replication factor 3 (num_replicas = 3) means CockroachDB keeps 3 copies of every range. That's all it says: it does not say where the copies go. Placement is decided by the allocator, which tries to spread copies across as many different localities as it can. With three regions and three copies, that normally means one copy per region, but "normally" is not something you sign off a DR design on. You check it.
02/ The cluster
| Region | Nodes | Example --locality flag |
|---|---|---|
| Azure Canada Central | n1, n2 (zones 1 and 2) | cloud=azure,region=canadacentral,zone=1 |
| Azure Canada East | n4 | cloud=azure,region=canadaeast |
| AWS ca-central-1 | n3, n5 (1a and 1b) | cloud=aws,region=ca-central-1,zone=ca-central-1a |
Five nodes, replication factor 3, leaseholders preferred in Azure Central (next to the main application). Each cloud's application connects through an internal load balancer to the database nodes in its own cloud, over TLS.
03/ How CockroachDB places the copies
- Locality tiers are ordered, most general first (
cloud→region→zone). CockroachDB spreads copies across the first tier first, then the next. - With the localities above, the 3 copies of a range end up as one in AWS and two in Azure, and the two Azure copies go to different Azure regions. Result: one copy in each of the three regions.
- Adding a node in the same region (n2, n5) adds capacity, not copies. A range never gets a 4th copy just because there is a 5th node.
- The default is 3 copies for user data. CockroachDB's own system ranges (such as node liveness and metadata) default to 5 copies, so on a 5-node cluster they live on every node.
04/ Check it on your cluster
1. Confirm every node reports the locality you expect. A typo here silently changes placement:
SELECT node_id, locality FROM crdb_internal.gossip_nodes ORDER BY node_id;
2. Count how many ranges have a copy in each region. Every column should equal the total:
SELECT count(*) AS ranges,
count(*) FILTER (WHERE array_to_string(replica_localities, ' ') LIKE '%region=canadacentral%') AS has_central,
count(*) FILTER (WHERE array_to_string(replica_localities, ' ') LIKE '%region=canadaeast%') AS has_east,
count(*) FILTER (WHERE array_to_string(replica_localities, ' ') LIKE '%cloud=aws%') AS has_aws
FROM [SHOW RANGES FROM DATABASE appdb];
ranges | has_central | has_east | has_aws
--------+-------------+----------+---------
26 | 26 | 26 | 26
All 26 ranges of the application database had a copy in every region. 3. Check where the leaseholders are (needs admin or the ZONECONFIG privilege):
SELECT lease_holder_locality, count(*) AS ranges FROM [SHOW RANGES FROM DATABASE appdb WITH DETAILS] GROUP BY 1 ORDER BY 2 DESC;
05/ What survives which outage
| Outage | Copies left | Available? | What changes |
|---|---|---|---|
| Azure Central down | East + AWS = 2 of 3 | Yes | Leases move to East or AWS; the Azure application now reads from another region, so it gets slower |
| Azure East down | Central + AWS = 2 of 3 | Yes | Reads unchanged, but every write now needs an acknowledgement from AWS, so writes get slower |
| AWS down | Central + East = 2 of 3 | Yes | Azure clients see no change; AWS clients are down with their region |
| All of Azure down (2 regions) | AWS only = 1 of 3 | No | No majority: ranges cannot serve reads or writes until Azure returns |
Note also: "survives the loss of any one region" is not the same as "survives the loss of a cloud". Two of our three regions are Azure, so an Azure-wide incident takes out two copies.
06/ Taking control of placement
If you want placement to be a rule rather than an outcome, write it into the zone configuration:
-- exactly one copy per region, enforced
ALTER DATABASE appdb CONFIGURE ZONE USING
num_replicas = 3,
constraints = '{"+region=canadacentral": 1, "+region=canadaeast": 1, "+cloud=aws": 1}',
lease_preferences = '[[+region=canadacentral], [+region=canadaeast]]';
lease_preferences is an ordered list: keep leases in Central, and if Central is down, prefer East over AWS so the Azure application stays within Azure. Use the exact locality values from step 1.
| If you wanted… | Zone config | Trade-off |
|---|---|---|
| All 3 copies inside Azure | constraints = '[+cloud=azure]' | Faster Azure writes, but AWS holds no copy: an Azure outage loses the database |
| Two copies in Central | constraints = '{"+region=canadacentral": 2, "+cloud=aws": 1}' | Writes commit inside Central (fast), but losing Central loses the majority |
| Survive any one region (our choice) | one copy per region, as above | Write latency includes a second region; East failure pushes writes to AWS |
07/ RF 3 vs SURVIVE REGION FAILURE
CockroachDB's multi-region SQL abstractions offer a survival goal instead of hand-written constraints:
ALTER DATABASE appdb SET PRIMARY REGION "canadacentral"; ALTER DATABASE appdb ADD REGION "canadaeast"; ALTER DATABASE appdb ADD REGION "ca-central-1"; ALTER DATABASE appdb SURVIVE REGION FAILURE;
Region survival needs at least 3 database regions, and on a 3-region cluster it raises the replication factor from 3 to 5. That gives more headroom (the cluster also survives an extra node failure during a region outage) at the cost of more storage and replication traffic. With plain RF 3 you get region survival as long as placement is one-per-region, which is what we verified. Note that the region names in these statements come from the region= tier of the node localities.
08/ Pausing a multi-cloud test cluster
A 5-node, 2-cloud test cluster costs money every hour it runs. Between test cycles we stop all nodes and start them again on the next working day. Deallocating keeps disks, data and zone configurations; CockroachDB starts again when the VMs boot if it is installed as a service.
# Azure: deallocate (stops compute billing, keeps disks)
az vm deallocate -g rg-crdb-lab -n crdb-az-central-0 --no-wait
az vm deallocate -g rg-crdb-lab -n crdb-az-central-1 --no-wait
az vm deallocate -g rg-crdb-lab -n crdb-az-east-0 --no-wait
# AWS: stop
aws ec2 stop-instances --region ca-central-1 --instance-ids i-AAAA i-BBBB
# a few minutes later: Azure should show "VM deallocated", AWS "stopped"
az vm list -g rg-crdb-lab -d --query "[].{name:name, state:powerState}" -o table
az vm start -g rg-crdb-lab -n crdb-az-central-0 --no-wait az vm start -g rg-crdb-lab -n crdb-az-central-1 --no-wait az vm start -g rg-crdb-lab -n crdb-az-east-0 --no-wait aws ec2 start-instances --region ca-central-1 --instance-ids i-AAAA i-BBBB # 3-5 minutes later: all nodes live, no unavailable or under-replicated ranges cockroach node status --ranges --certs-dir=certs --host=<any-node>:26257
| Gotcha | What to do |
|---|---|
Azure returns ZonalAllocationFailed when starting a deallocated VM | The zone is short of capacity for that VM size. Wait 10–15 minutes and start it again, or keep a capacity reservation for test windows |
| Node certificates or join flags use IP addresses | Use static private IPs or DNS names, so a restart can never change an address the cluster depends on |
| Starting only some nodes | Ranges come back once a majority of their copies is up; start all nodes together and check node status --ranges before testing |
| Expired cloud CLI login | Re-authenticate (az login, AWS SSO) before the start script, not halfway through it |
09/ Checklist
| Before you sign off a multi-region design | Done |
|---|---|
Every node has the intended --locality (query 1) | ☐ |
| Every range has a copy in every region (query 2) | ☐ |
| Leaseholders sit where the main application runs (query 3) | ☐ |
| Placement is enforced with constraints, not left to chance | ☐ |
| Write latency measured with each region down, not just reads | ☐ |
| Stop/start runbook tested with all nodes, including a capacity failure | ☐ |
Originally published at techstaradmin.blogspot.com.

