MongoDB Replica Set Deployment Runbook: 3 Nodes, Secured and Failover-Tested (MongoDB 8.0 on RHEL 9)

My first MongoDB replica set build worked, eventually. Along the way it refused to start three different ways, accepted logins it should have refused (or so it seemed), and created an application user in the wrong database. This runbook is the clean version: twelve steps, in the right order, with every command tested.

It targets MongoDB 8.0 on RHEL 9 (notes for 7.0 where it differs). I rebuilt the whole setup in a lab with MongoDB 8.0.23 and mongosh to check each step, including failover; the outputs below come from that lab. Host names, users and passwords are examples.

01 Plan the hosts 02 Prepare the operating system 03 Install MongoDB and mongosh 04 Create the keyFile 05 Write mongod.conf 06 Start with systemd 07 Initiate the replica set 08 Create the first admin user 09 Create the application user 10 Verify replication and the oplog 11 Test transactions, reads and failover 12 Operate: profiler, monitoring, backups

STEP 01 / 12

Plan the hosts

Host	Role at start	Priority	Example address
mongo1.lab.local	Primary (preferred)	2	10.0.1.11
mongo2.lab.local	Secondary	1	10.0.1.12
mongo3.lab.local	Secondary	1	10.0.1.13

Item	Decision
Members	3 data-bearing nodes (an odd number, so a majority always exists). Avoid arbiters in production: they vote but hold no data
Version	Same MongoDB and mongosh version on every node
Port	27017 between the three nodes and from the application subnet only
Storage	Separate data volume, XFS (recommended for WiredTiger)
Names	Use DNS names or /etc/hosts entries in the replica set config, not IP addresses

STEP 02 / 12

Prepare the operating system

# on ALL 3 nodes (root)

# 1. Names: every node must resolve every other node
cat >> /etc/hosts <<'EOF'
10.0.1.11  mongo1.lab.local mongo1
10.0.1.12  mongo2.lab.local mongo2
10.0.1.13  mongo3.lab.local mongo3
EOF

# 2. Firewall: 27017 only from the cluster and app subnets
firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="10.0.1.0/24" port port="27017" protocol="tcp" accept'
firewall-cmd --reload

# 3. Data volume on XFS
mkfs.xfs /dev/sdb && mkdir -p /data/mongodb
echo '/dev/sdb /data/mongodb xfs defaults,noatime 0 0' >> /etc/fstab && mount -a

# 4. Kernel: memory map areas (must exceed 2 x max connections)
echo 'vm.max_map_count = 262144' > /etc/sysctl.d/90-mongodb.conf && sysctl --system

Transparent Huge Pages (THP): the advice changed in MongoDB 8.0. Our lab's 8.0 node logged exactly this at startup:

db.adminCommand({ getLog: 'startupWarnings' }) on 8.0

sysfsFile: /sys/kernel/mm/transparent_hugepage/enabled   currentValue: never    desiredValue: always
sysfsFile: /sys/kernel/mm/transparent_hugepage/defrag    currentValue: madvise  desiredValue: defer+madvise
"We suggest setting the contents of sysfsFile to 0": .../khugepaged/max_ptes_none (currentValue: 511)

Version	THP setting
MongoDB 8.0 and later	Enable: enabled=always, defrag=defer+madvise, khugepaged/max_ptes_none=0 (new TCMalloc with per-CPU caches)
MongoDB 7.0 and earlier	Disable: enabled=never (the warning my 7.0 build showed: "We suggest setting it to 'never'")

/etc/systemd/system/enable-transparent-huge-pages.service

# /etc/systemd/system/enable-transparent-huge-pages.service  (MongoDB 8.0)
[Unit]
Description=Enable Transparent Hugepages (THP)
DefaultDependencies=no
After=sysinit.target local-fs.target
Before=mongod.service

[Service]
Type=oneshot
ExecStart=/bin/sh -c 'echo always > /sys/kernel/mm/transparent_hugepage/enabled && echo defer+madvise > /sys/kernel/mm/transparent_hugepage/defrag && echo 0 > /sys/kernel/mm/transparent_hugepage/khugepaged/max_ptes_none && echo 1 > /proc/sys/vm/overcommit_memory'

[Install]
WantedBy=basic.target

# on ALL 3 nodes (root)

systemctl daemon-reload && systemctl enable --now enable-transparent-huge-pages
cat /sys/kernel/mm/transparent_hugepage/enabled          # [always] madvise never

RHEL tuned profiles can reset THP after boot. If you use tuned, create a custom profile with transparent_hugepages=always (8.0) as described in MongoDB's TCMalloc performance notes.

SELinux (enforcing): the packages ship a policy for the default paths only. For /data/mongodb label it: semanage fcontext -a -t mongod_var_lib_t "/data/mongodb(/.*)?" then restorecon -Rv /data/mongodb. Never fix a start failure by disabling SELinux.

STEP 03 / 12

Install MongoDB and mongosh

# on ALL 3 nodes (root)

cat > /etc/yum.repos.d/mongodb-org-8.0.repo <<'EOF'
[mongodb-org-8.0]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/9/mongodb-org/8.0/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://pgp.mongodb.com/server-8.0.asc
EOF

dnf install -y mongodb-org
mongod --version | head -1

mongosh and OpenSSL on RHEL 9: the default shell package can fail with "OpenSSL configuration error … rh-allow-sha1-signatures". I worked around it with OPENSSL_CONF=/dev/null mongosh …; the proper fix is to install the shell build that uses the system OpenSSL 3: dnf swap mongodb-mongosh mongodb-mongosh-shared-openssl3.

The legacy mongo shell no longer exists; use mongosh for everything.

STEP 04 / 12

Create the keyFile

Replica set members prove to each other that they belong to the cluster with a shared keyFile. Without it, MongoDB refuses to start once authorization is on:

error

BadValue: security.keyFile is required when authorization is enabled with replica sets

# on mongo1, then copy to mongo2 and mongo3

mkdir -p /etc/mongodb
openssl rand -base64 756 > /etc/mongodb/keyfile
chmod 400 /etc/mongodb/keyfile && chown mongod:mongod /etc/mongodb/keyfile
scp -p /etc/mongodb/keyfile mongo2:/etc/mongodb/keyfile
scp -p /etc/mongodb/keyfile mongo3:/etc/mongodb/keyfile
for h in mongo2 mongo3; do ssh $h 'chown mongod:mongod /etc/mongodb/keyfile && chmod 400 /etc/mongodb/keyfile'; done

what you get with chmod 644

"Read security file failed" … "permissions on /etc/mongodb/keyfile are too open"

✓ Check: identical file on all three nodes (sha256sum /etc/mongodb/keyfile), owner mongod, mode 400. For production, x.509 certificates are the stronger alternative to a keyFile.

STEP 05 / 12

Write mongod.conf

/etc/mongod.conf (all 3 nodes)

# /etc/mongod.conf  (mongo1; change bindIp on each node)
storage:
  dbPath: /data/mongodb
systemLog:
  destination: file
  path: /var/log/mongodb/mongod.log
  logAppend: true
  logRotate: reopen
net:
  port: 27017
  bindIp: 127.0.0.1,mongo1.lab.local     # localhost is needed for the first admin user
processManagement:
  timeZoneInfo: /usr/share/zoneinfo
replication:
  replSetName: rs0
  oplogSizeMB: 20480                     # 20 GB; size it for hours of writes, not MB
security:
  keyFile: /etc/mongodb/keyfile
  authorization: enabled
setParameter:
  transactionLifetimeLimitSeconds: 600   # optional: default is 60 s

Do NOT add	Why
`storage.journal.enabled`	Removed in MongoDB 6.1. Journaling is always on; the option now stops startup with Unrecognized option: storage.journal.enabled
`processManagement.fork: true`	The packaged systemd unit runs mongod in the foreground; let systemd manage the process
`bindIp: 0.0.0.0`	Listen only on localhost and the node's own private name
An oplog of 128 MB	My first build used `--oplogSize 128`: minutes of history. A secondary that falls behind further needs a full resync

YAML needs spaces, not tabs, and consistent indentation. If pasting breaks it, check with python3 -c 'import yaml; yaml.safe_load(open("/etc/mongod.conf"))'.

STEP 06 / 12

Start with systemd

# on ALL 3 nodes (root)

mkdir -p /data/mongodb /var/log/mongodb
chown -R mongod:mongod /data/mongodb /var/log/mongodb
systemctl enable --now mongod
systemctl status mongod --no-pager | head -5
cat /proc/$(pidof mongod)/limits | grep -E 'open files|processes'

Never mix systemd and hand-started mongod. In my first attempt a mongod started by hand with nohup ran next to the systemd one. The service then failed with exit code 48 (port already in use), and the hand-started process ran with the shell's 1,024 open-file limit, which later produced Too many open files. The packaged unit already sets LimitNOFILE=64000; start mongod only through systemd.

systemd status / log shows	Meaning
status=2/INVALIDARGUMENT	Bad option in mongod.conf (typo, tab, removed option)
status=48	Port 27017 already in use: another mongod is running (`ss -ltnp \| grep 27017`)
status=14, "Permission denied" in the log	Often ownership or the SELinux label on the data or log directory

STEP 07 / 12

Initiate the replica set

Run this once, on mongo1, connected through localhost. No users exist yet, and the localhost exception allows exactly this first setup.

# on mongo1 only

mongosh --quiet "mongodb://127.0.0.1:27017/?directConnection=true" --eval '
rs.initiate({ _id: "rs0", members: [
  { _id: 0, host: "mongo1.lab.local:27017", priority: 2 },
  { _id: 1, host: "mongo2.lab.local:27017", priority: 1 },
  { _id: 2, host: "mongo3.lab.local:27017", priority: 1 } ] })'

output

{ ok: 1 }

The higher priority makes mongo1 the preferred primary; it takes the role back after a failover once it is healthy and caught up.

STEP 08 / 12

Create the first admin user

# on mongo1, still via localhost

mongosh --quiet "mongodb://127.0.0.1:27017/?directConnection=true" --eval '
db.getSiblingDB("admin").createUser({
  user: "dbadmin",
  pwd:  passwordPrompt(),
  roles: [ { role: "root", db: "admin" } ] })'

That closes the localhost exception. Everything now needs a login. What an unauthenticated user sees:

output

$ mongosh "mongodb://mongo1.lab.local:27017/"
rs0 [primary] test> show dbs
MongoServerError: Command listDatabases requires authentication

"It still lets me log in without a password!" This confused me on the first build. mongosh connects without credentials, but every command that reads data or settings fails, as above. Connecting is not the same as being authorized; this is the expected behaviour.

log in as admin (prompts for the password)

mongosh "mongodb://dbadmin@mongo1.lab.local:27017,mongo2.lab.local:27017,mongo3.lab.local:27017/?replicaSet=rs0&authSource=admin"

STEP 09 / 12

Create the application user

mongosh

use bankdb
db.createUser({
  user: "bankapp",
  pwd:  passwordPrompt(),
  roles: [ { role: "readWrite", db: "bankdb" } ] })

Rule	Why
Give the application `readWrite` on its own database only	Not `root`, not `readWriteAnyDatabase`, not `userAdmin` (which lets it create users)
Note where you created the user: that is its `authSource`	Created after `use bankdb` → log in with `authSource=bankdb`
Use a separate migration/DDL user if needed	`dbAdmin` for index builds, kept out of the app

My first build created the user while connected to admin, then logged in with authSource set to the application database: authentication failed. Same thing in the lab:

output

mongodb://bankapp:…@…/bankdb?replicaSet=rs0&authSource=admin   →  MongoServerError: Authentication failed.
mongodb://bankapp:…@…/bankdb?replicaSet=rs0&authSource=bankdb  →  connected

as bankapp: other databases are off limits

db.getSiblingDB("admin").system.users.countDocuments()
MongoServerError: not authorized on admin to execute command …

STEP 10 / 12

Verify replication and the oplog

mongosh

rs.status().members.map(m => m.name + "  " + m.stateStr)

output

[ 'mongo1.lab.local:27017  PRIMARY',
  'mongo2.lab.local:27017  SECONDARY',
  'mongo3.lab.local:27017  SECONDARY' ]

mongosh

rs.printSecondaryReplicationInfo()     // lag per secondary
rs.printReplicationInfo()              // oplog size and time window
db.hello().primary                     // who is primary (replaces isMaster)

output

source: mongo2.lab.local:27017
{ syncedTo: 'Sat Oct 31 2026 14:24:25 GMT+0530', replLag: '0 secs (0 hrs) behind the primary ' }

Oplog window: the "log length start to end" from rs.printReplicationInfo() is how long a secondary can be offline and still catch up. Aim for at least a day at peak write load. Resize it online, one member at a time; no restart is needed:

mongosh

db.adminCommand({ replSetResizeOplog: 1, size: 20480 })    // MB, run on EACH member

lab: resized from 2,048 to 4,096 MB on each member

mongo1.lab.local:27017 ok=1 maxSizeMB=4096
mongo2.lab.local:27017 ok=1 maxSizeMB=4096
mongo3.lab.local:27017 ok=1 maxSizeMB=4096

--oplogSize only works once: it applies when the replica set is first initiated. On a running replica set use replSetResizeOplog.

STEP 11 / 12

Test transactions, reads and failover

multi-document transaction as bankapp

const s = db.getMongo().startSession();
const acc = s.getDatabase("bankdb").accounts;
s.startTransaction({ readConcern: { level: "snapshot" }, writeConcern: { w: "majority" } });
acc.updateOne({ _id: "ACC-1" }, { $inc: { balance: -200 } });
acc.updateOne({ _id: "ACC-2" }, { $inc: { balance:  200 } });
s.commitTransaction(); s.endSession();

output

[ { _id: 'ACC-1', balance: 800 }, { _id: 'ACC-2', balance: 700 } ]

Read preference goes in the connection string. Transactions always run on the primary; secondary reads can be slightly behind.

readPreference	Use for
`primary` (default)	Anything that must read its own writes
`primaryPreferred`	Reads that should survive a short primary outage
`secondaryPreferred`	Reports and analytics that can be a few seconds stale
`nearest`	Lowest latency in multi-site setups

connection string (one line)

mongodb://bankapp:<password>@mongo1.lab.local:27017,mongo2.lab.local:27017,mongo3.lab.local:27017/bankdb
  ?replicaSet=rs0&authSource=bankdb&retryWrites=true&w=majority&readPreference=primary

In mongosh, rs.slaveOk() no longer exists and rs.secondaryOk() is deprecated; use db.getMongo().setReadPref('secondaryPreferred').

Planned switchover (patching): step the primary down instead of killing it.

mongosh

rs.stepDown(60)     // on the primary: steps down, may not run for election for 60 s

seconds later

mongo1.lab.local:27017  SECONDARY
mongo2.lab.local:27017  PRIMARY
mongo3.lab.local:27017  SECONDARY

Unplanned failover: we killed the primary's process outright. The secondaries elected mongo3 within seconds, and an application insert with retryWrites=true succeeded. The killed node rejoined as a secondary when restarted:

output

mongo1.lab.local:27017  SECONDARY
mongo2.lab.local:27017  (not reachable/healthy)      ← killed
mongo3.lab.local:27017  PRIMARY                      ← elected
# after: systemctl start mongod on mongo2
mongo2.lab.local:27017  SECONDARY

Stop MongoDB cleanly: systemctl stop mongod or db.adminCommand({ shutdown: 1 }). Never kill -9 in normal operation, and never delete files from the journal directory.

STEP 12 / 12

Operate: profiler, monitoring, backups

Find slow operations with the profiler (per database, per node; level 1 records operations slower than slowms):

mongosh

db.setProfilingLevel(1, { slowms: 100 })
db.system.profile.find().sort({ ts: -1 }).limit(5)
db.setProfilingLevel(0)        // switch off when done

lab: a missing index, found and fixed

{ ns: 'bankdb.big', millis: 80, planSummary: 'COLLSCAN', docsExamined: 200000, nreturned: 40 }
// after db.big.createIndex({ acct: 1 }):
{ executionTimeMillis: 1, docsExamined: 40, nReturned: 40 }

Check	Command
Members and health	`rs.status()`
Replication lag	`rs.printSecondaryReplicationInfo()`
Oplog window	`rs.printReplicationInfo()`
Connections	`db.serverStatus().connections`
Live load	mongostat / mongotop (database tools package)
Startup warnings	`db.adminCommand({ getLog: 'startupWarnings' })`

Backups: for small databases, mongodump --uri '…' --oplog --gzip --archive=/backup/rs0.gz from a secondary; for larger ones, filesystem or cloud-disk snapshots of a secondary. Whatever you choose, test a restore.

Errors from my first build, and the fixes

Symptom	Cause	Fix
BadValue: security.keyFile is required…	Authorization on a replica set without a keyFile	Step 04
Unrecognized option: storage.journal.enabled	Option removed in 6.1	Delete it; journaling is always on
systemd: status=48	A hand-started mongod already had the port	Kill the stray process; use only systemd
mongosh: OpenSSL configuration error	RHEL 9 OpenSSL config vs bundled OpenSSL	mongodb-mongosh-shared-openssl3
Connects without a password	Connecting ≠ authorized	Expected; commands are refused
App user: Authentication failed	User created in admin, logged in with authSource=appdb	Match authSource to where the user lives
Too many open files	mongod started from a shell (ulimit 1024)	systemd unit (LimitNOFILE=64000)
Warnings: THP, vm.max_map_count	OS not tuned for MongoDB	Step 02 (THP setting depends on version)

Go-live checklist

Item	Done
Same version on all nodes; mongosh works without workarounds	☐
THP set for your version; vm.max_map_count raised; XFS; SELinux labels	☐
Port 27017 open only to the cluster and app subnets	☐
keyFile identical on all nodes, mode 400 (or x.509)	☐
mongod.conf: no journal or fork options; oplog sized for a day+	☐
Started only by systemd, enabled at boot	☐
Admin user created via localhost; app user readWrite on its own DB	☐
Unauthenticated commands refused	☐
Transaction, switchover and failover tested	☐
Backups scheduled and one restore tested	☐

© 2026 Tamilselvan Perumal · TechStarAdmin. Commands tested on MongoDB 8.0.23 with mongosh in a lab; host names, users and passwords are examples. Test in a non-production environment first.
Originally published at techstaradmin.blogspot.com.

MongoDB Replica Set Deployment Runbook: 3 Nodes, Secured and Failover-Tested (MongoDB 8.0 on RHEL 9)

Plan the hosts

Prepare the operating system

Install MongoDB and mongosh

Create the keyFile

Write mongod.conf

Start with systemd

Initiate the replica set

Create the first admin user

Create the application user

Verify replication and the oplog

Test transactions, reads and failover

Operate: profiler, monitoring, backups

Errors from my first build, and the fixes

Go-live checklist

Posted by: Tamilselvan Perumal

Search This Blog

About Me

Labels

Popular Posts

Same API, 2 s in One Cloud and 12 s in the Other: CockroachDB Leaseholders in a Multi-Cloud Cluster

Priority Transactions in Oracle AI Database 26ai: When a Batch Blocks a Payment, Measured

JSON Duality Views in Oracle AI Database 26ai: Two Apps, One Account, No Lost Updates

Menu Footer Widget

Contact form

MongoDB Replica Set Deployment Runbook: 3 Nodes, Secured and Failover-Tested (MongoDB 8.0 on RHEL 9)

Plan the hosts

Prepare the operating system

Install MongoDB and mongosh

Create the keyFile

Write mongod.conf

Start with systemd

Initiate the replica set

Create the first admin user

Create the application user

Verify replication and the oplog

Test transactions, reads and failover

Operate: profiler, monitoring, backups

Errors from my first build, and the fixes

Go-live checklist

Posted by: Tamilselvan Perumal

You may like these posts

Search This Blog

About Me

Labels

Popular Posts

Same API, 2 s in One Cloud and 12 s in the Other: CockroachDB Leaseholders in a Multi-Cloud Cluster

Priority Transactions in Oracle AI Database 26ai: When a Batch Blocks a Payment, Measured

JSON Duality Views in Oracle AI Database 26ai: Two Apps, One Account, No Lost Updates

Menu Footer Widget

Contact form