Replication Factor 3 Across Three Regions: Where CockroachDB Really Puts the Copies



"We set replication factor 3 and we have three regions, so there is one copy in each region, right?" I was asked exactly this while handing over a CockroachDB cluster stretched across Azure and AWS. The honest answer is "not by definition, but in our case yes, and here is how we proved it."

This post explains how CockroachDB decides where the copies go, how to check it with two queries, what survives each kind of outage (including one that surprised the team), and how to pause a multi-cloud test cluster overnight without breaking it.

Companion post: this is the same multi-cloud test environment as my case study on why the same API ran 6× slower from the other cloud, now grown to 5 nodes. Names, addresses and identifiers are removed.

01/ The question

Replication factor 3 (num_replicas = 3) means CockroachDB keeps 3 copies of every range. That's all it says: it does not say where the copies go. Placement is decided by the allocator, which tries to spread copies across as many different localities as it can. With three regions and three copies, that normally means one copy per region, but "normally" is not something you sign off a DR design on. You check it.

02/ The cluster

RegionNodesExample --locality flag
Azure Canada Centraln1, n2 (zones 1 and 2)cloud=azure,region=canadacentral,zone=1
Azure Canada Eastn4cloud=azure,region=canadaeast
AWS ca-central-1n3, n5 (1a and 1b)cloud=aws,region=ca-central-1,zone=ca-central-1a

Five nodes, replication factor 3, leaseholders preferred in Azure Central (next to the main application). Each cloud's application connects through an internal load balancer to the database nodes in its own cloud, over TLS.

03/ How CockroachDB places the copies

  • Locality tiers are ordered, most general first (cloud → region → zone). CockroachDB spreads copies across the first tier first, then the next.
  • With the localities above, the 3 copies of a range end up as one in AWS and two in Azure, and the two Azure copies go to different Azure regions. Result: one copy in each of the three regions.
  • Adding a node in the same region (n2, n5) adds capacity, not copies. A range never gets a 4th copy just because there is a 5th node.
  • The default is 3 copies for user data. CockroachDB's own system ranges (such as node liveness and metadata) default to 5 copies, so on a 5-node cluster they live on every node.

04/ Check it on your cluster

1. Confirm every node reports the locality you expect. A typo here silently changes placement:

SQL
SELECT node_id, locality FROM crdb_internal.gossip_nodes ORDER BY node_id;

2. Count how many ranges have a copy in each region. Every column should equal the total:

SQL
SELECT count(*) AS ranges,
       count(*) FILTER (WHERE array_to_string(replica_localities, ' ') LIKE '%region=canadacentral%') AS has_central,
       count(*) FILTER (WHERE array_to_string(replica_localities, ' ') LIKE '%region=canadaeast%')    AS has_east,
       count(*) FILTER (WHERE array_to_string(replica_localities, ' ') LIKE '%cloud=aws%')            AS has_aws
FROM [SHOW RANGES FROM DATABASE appdb];
Our result
 ranges | has_central | has_east | has_aws
--------+-------------+----------+---------
     26 |          26 |       26 |      26

All 26 ranges of the application database had a copy in every region. 3. Check where the leaseholders are (needs admin or the ZONECONFIG privilege):

SQL
SELECT lease_holder_locality, count(*) AS ranges
FROM [SHOW RANGES FROM DATABASE appdb WITH DETAILS]
GROUP BY 1 ORDER BY 2 DESC;
Write it down this way: "Replication factor 3: three copies of every range, one in each region (Azure Central, Azure East, AWS), verified per range. The cluster survives the loss of any one region." That sentence is accurate, and it is backed by a query anyone can rerun.

05/ What survives which outage



OutageCopies leftAvailable?What changes
Azure Central downEast + AWS = 2 of 3YesLeases move to East or AWS; the Azure application now reads from another region, so it gets slower
Azure East downCentral + AWS = 2 of 3YesReads unchanged, but every write now needs an acknowledgement from AWS, so writes get slower
AWS downCentral + East = 2 of 3YesAzure clients see no change; AWS clients are down with their region
All of Azure down (2 regions)AWS only = 1 of 3NoNo majority: ranges cannot serve reads or writes until Azure returns
The one that surprised us: losing the small region (Azure East, one node) does not stop anything, but it does slow every write. In normal operation a write is committed by Central + East, two Azure regions. With East gone, the only other copy is in AWS, so each write waits for a cross-cloud round trip. Plan your write-latency SLAs for this case, not just for the happy path.

Note also: "survives the loss of any one region" is not the same as "survives the loss of a cloud". Two of our three regions are Azure, so an Azure-wide incident takes out two copies.

06/ Taking control of placement

If you want placement to be a rule rather than an outcome, write it into the zone configuration:

SQL
-- exactly one copy per region, enforced
ALTER DATABASE appdb CONFIGURE ZONE USING
  num_replicas = 3,
  constraints = '{"+region=canadacentral": 1, "+region=canadaeast": 1, "+cloud=aws": 1}',
  lease_preferences = '[[+region=canadacentral], [+region=canadaeast]]';

lease_preferences is an ordered list: keep leases in Central, and if Central is down, prefer East over AWS so the Azure application stays within Azure. Use the exact locality values from step 1.

If you wanted…Zone configTrade-off
All 3 copies inside Azureconstraints = '[+cloud=azure]'Faster Azure writes, but AWS holds no copy: an Azure outage loses the database
Two copies in Centralconstraints = '{"+region=canadacentral": 2, "+cloud=aws": 1}'Writes commit inside Central (fast), but losing Central loses the majority
Survive any one region (our choice)one copy per region, as aboveWrite latency includes a second region; East failure pushes writes to AWS

07/ RF 3 vs SURVIVE REGION FAILURE

CockroachDB's multi-region SQL abstractions offer a survival goal instead of hand-written constraints:

SQL
ALTER DATABASE appdb SET PRIMARY REGION "canadacentral";
ALTER DATABASE appdb ADD REGION "canadaeast";
ALTER DATABASE appdb ADD REGION "ca-central-1";
ALTER DATABASE appdb SURVIVE REGION FAILURE;

Region survival needs at least 3 database regions, and on a 3-region cluster it raises the replication factor from 3 to 5. That gives more headroom (the cluster also survives an extra node failure during a region outage) at the cost of more storage and replication traffic. With plain RF 3 you get region survival as long as placement is one-per-region, which is what we verified. Note that the region names in these statements come from the region= tier of the node localities.

08/ Pausing a multi-cloud test cluster

A 5-node, 2-cloud test cluster costs money every hour it runs. Between test cycles we stop all nodes and start them again on the next working day. Deallocating keeps disks, data and zone configurations; CockroachDB starts again when the VMs boot if it is installed as a service.

shell (stop)
# Azure: deallocate (stops compute billing, keeps disks)
az vm deallocate -g rg-crdb-lab -n crdb-az-central-0 --no-wait
az vm deallocate -g rg-crdb-lab -n crdb-az-central-1 --no-wait
az vm deallocate -g rg-crdb-lab -n crdb-az-east-0    --no-wait

# AWS: stop
aws ec2 stop-instances --region ca-central-1 --instance-ids i-AAAA i-BBBB

# a few minutes later: Azure should show "VM deallocated", AWS "stopped"
az vm list -g rg-crdb-lab -d --query "[].{name:name, state:powerState}" -o table
shell (start)
az vm start -g rg-crdb-lab -n crdb-az-central-0 --no-wait
az vm start -g rg-crdb-lab -n crdb-az-central-1 --no-wait
az vm start -g rg-crdb-lab -n crdb-az-east-0    --no-wait
aws ec2 start-instances --region ca-central-1 --instance-ids i-AAAA i-BBBB

# 3-5 minutes later: all nodes live, no unavailable or under-replicated ranges
cockroach node status --ranges --certs-dir=certs --host=<any-node>:26257
GotchaWhat to do
Azure returns ZonalAllocationFailed when starting a deallocated VMThe zone is short of capacity for that VM size. Wait 10–15 minutes and start it again, or keep a capacity reservation for test windows
Node certificates or join flags use IP addressesUse static private IPs or DNS names, so a restart can never change an address the cluster depends on
Starting only some nodesRanges come back once a majority of their copies is up; start all nodes together and check node status --ranges before testing
Expired cloud CLI loginRe-authenticate (az login, AWS SSO) before the start script, not halfway through it

09/ Checklist

Before you sign off a multi-region designDone
Every node has the intended --locality (query 1)☐
Every range has a copy in every region (query 2)☐
Leaseholders sit where the main application runs (query 3)☐
Placement is enforced with constraints, not left to chance☐
Write latency measured with each region down, not just reads☐
Stop/start runbook tested with all nodes, including a capacity failure☐
© 2026 Tamilselvan Perumal · TechStarAdmin. Based on a test environment; resource names, addresses and identifiers are examples. Views are my own. Please link back instead of copying.
Originally published at techstaradmin.blogspot.com.