Oracle RAC Startup Sequence Explained: How Grid Infrastructure 19c Boots, Step by Step



When a RAC node does not come back after a reboot, the first question is always the same: how far did the startup get? To answer it you need a clear picture of the order in which Oracle Clusterware starts its daemons, which daemon starts which, and which OS user each one runs as.

This post walks through the Grid Infrastructure 19c startup in four phases, then shows the commands and log files I use to find exactly where a startup stopped.

The big picture

Clusterware starts in two layers. The lower stack is owned by OHASD: everything a node needs to join the cluster and reach shared storage. The upper stack is owned by CRSD: the resources your users care about, such as VIPs, listeners, databases and services.



Startup hierarchy, colour-coded by the OS user each component runs as. Click to enlarge.
PHASE 1
OS boot starts OHASD

When the server boots, the operating system starts the Grid Infrastructure entry point. On Oracle Linux / RHEL 7, 8 and 9 this is the systemd unit oracle-ohasd.service, which runs /etc/init.d/init.ohasd run. That wrapper starts OHASD (ohasd.bin), the Oracle High Availability Services daemon, as root.

OHASD is the first Clusterware daemon and the root of the whole tree. It does not start everything at once. It reads its local configuration from the OLR (Oracle Local Registry, a file on each node's local disk), then starts its agents. The cluster-wide OCR is not used yet, because it usually lives on ASM and ASM is not running yet.

Will it start automatically? Only if autostart is enabled. Check with crsctl config crs; change it with crsctl enable crs / crsctl disable crs as root.
PHASE 2
OHASD agents start the lower stack

OHASD spawns four agents. Each one starts and monitors a set of init resources, the "lower stack" you see with crsctl stat res -t -init.

OHASD agentRuns asStarts (resource)
cssdagentrootocssd.bin (ora.cssd): runs as grid
cssdmonitorrootWatches CSSD; reboots the node if CSSD hangs (to protect data)
orarootagentrootcrsd.bin (ora.crsd), ctssd (ora.ctssd), HAIP (ora.cluster_interconnect.haip), ACFS/ADVM drivers (ora.drivers.acfs), diskmon (ora.diskmon), Cluster Health Monitor (ora.crf)
oraagentgridmdnsd (ora.mdnsd), gpnpd (ora.gpnpd), gipcd (ora.gipcd), evmd (ora.evmd), ASM instance (ora.asm)

These daemons have dependencies on each other, so in practice they come up in roughly this order:

mdnsd → gpnpd → gipcd → cssd → ctssd / HAIP → ASM → crsd

  • gpnpd serves the GPnP profile: the cluster name, the network interfaces for public and private traffic, and the ASM discovery string.
  • CSSD reads that profile to find the voting files and the interconnect. Then it joins the cluster through network and disk heartbeats. If it cannot reach the voting files, nothing above it can start.
  • ASM starts in this lower stack on purpose: the OCR is usually stored in an ASM disk group, so ASM must be up before CRSD can read it.
PHASE 3
CRSD reads the OCR

Once ASM is up, the OHASD orarootagent starts CRSD (crsd.bin). It runs as root, not as the grid user. CRSD reads the Oracle Cluster Registry (OCR), which holds every cluster resource with its dependencies, its placement and the state it should be in.

CRSD is the resource manager for the whole cluster. It does not start databases or VIPs itself; it starts its own set of agents to do that.

PHASE 4
CRSD agents start the upper stack
CRSD agentRuns asStarts and monitors
orarootagentrootNetwork resource (ora.net1.network), node VIPs, SCAN VIPs, GNS (if used)
oraagentgridNode listeners, SCAN listeners, ASM disk groups, ONS, ACFS file systems
oraagentoracleDatabase instances and database services. This agent runs as the database home owner when it differs from the grid owner (role separation)
scriptagentresource ownerCustom application resources registered with crsctl add resource

These are the "upper stack" resources you see with crsctl stat res -t. Each instance then starts its usual background processes (pmon, smon, lgwr, dbw0 and, for RAC, lmon, lms0, lmd0).

Setting the record straight

Many startup diagrams online, including AI-generated ones, repeat a few mistakes. Here is how they compare with what you can see on a real 19c cluster:

Common claimWhat actually happens (11gR2 / 12c / 19c)
OHASD is started by init.crsinit.crs was 10g / 11gR1. From 11gR2 the entry point is init.ohasd, started by systemd (oracle-ohasd.service) on modern Linux.
A "GPNPD root agent" starts GPNPDThere is no GPNPD root agent. GPNPD is started by OHASD's oraagent and runs as the grid owner.
CRSD runs as the Grid Infrastructure ownercrsd.bin runs as root. It is started by OHASD's orarootagent.
OHASD's root agent starts CSSD directlyCSSD is started by cssdagent (root), and ocssd.bin itself runs as grid. cssdmonitor watches it.
ASM instances are started by CRSD's oraagentThe ASM instance is started in the lower stack by OHASD's oraagent, before CRSD, because the OCR lives on ASM. CRSD's oraagent manages ASM disk groups.
Database instances are started by "the" oraagentWith role separation there are two CRSD oraagents: one as grid (listeners, disk groups) and one as oracle (databases, services).
Diskmon monitors the voting disksdiskmon is for Exadata storage cells. On non-Exadata systems it is usually OFFLINE. Voting files are handled by CSSD.

See it on your own cluster

1. Overall health

crsctl check crs                 # OHASD, CRS, CSS, EVM on this node
crsctl check cluster -all        # CRS, CSS, EVM on every node
olsnodes -n -s -t                # node list with status

2. Lower stack (init resources): this is where to look if a node will not join.

crsctl stat res -t -init
Name                     Target  State        Server   State details
ora.asm                  ONLINE  ONLINE       node1    Started
ora.cluster_interconnect.haip  ONLINE  ONLINE  node1
ora.crf                  ONLINE  ONLINE       node1
ora.crsd                 ONLINE  ONLINE       node1
ora.cssd                 ONLINE  ONLINE       node1
ora.cssdmonitor          ONLINE  ONLINE       node1
ora.ctssd                ONLINE  ONLINE       node1    OBSERVER
ora.diskmon              OFFLINE OFFLINE               (normal outside Exadata)
ora.evmd                 ONLINE  ONLINE       node1
ora.gipcd                ONLINE  ONLINE       node1
ora.gpnpd                ONLINE  ONLINE       node1
ora.mdnsd                ONLINE  ONLINE       node1

Abbreviated sample. OBSERVER for ctssd means NTP/chrony is keeping time and CTSS is only watching, which is normal.

3. Upper stack (cluster resources)

crsctl stat res -t
srvctl status database -d ORCL -v
srvctl status scan_listener
srvctl status nodeapps

4. Which user runs what

ps -eo user,pid,args | grep -E "ohasd.bin|ocssd.bin|crsd.bin|orarootagent|oraagent|cssdagent|gpnpd|evmd" | grep -v grep

Where the logs are

From 12.1 onwards, Clusterware logs are in the ADR under the Grid Infrastructure owner's ORACLE_BASE:

cd $ORACLE_BASE/diag/crs/$(hostname -s)/crs/trace     # as the grid user

alert.log          # Clusterware alert log: start here
ohasd.trc          # OHASD
ocssd.trc          # CSSD: voting files, heartbeats, evictions
crsd.trc           # CRSD: OCR access, resource management
ohasd_orarootagent_root.trc / ohasd_oraagent_grid.trc     # lower-stack agents
crsd_orarootagent_root.trc  / crsd_oraagent_grid.trc / crsd_oraagent_oracle.trc
11gR2? Logs are under $GRID_HOME/log/<hostname>/ instead, for example alert<hostname>.log, cssd/ocssd.log and crsd/crsd.log.

Troubleshooting by layer

Work from the bottom up. A higher layer cannot start until the one below it is healthy.

Stuck atTypical causeCheck
OHASD not runningAutostart disabled, systemd unit failed, GI home/permissions issuesystemctl status oracle-ohasd, crsctl config crs, ohasd.trc
mdnsd / gpnpd / gipcdNetwork interface renamed or down, corrupt GPnP profilegpnptool get, oifcfg getif, gpnpd and gipcd traces
ora.cssdVoting files not reachable (disk paths, permissions, multipath), private interconnect downcrsctl query css votedisk, ocssd.trc, ping over the private network
ora.asmASM disks not discovered (asm_diskstring, udev/AFD rules, ownership)ASM alert log, kfod disks=all, ls -l /dev/oracleasm or AFD labels
ora.crsdOCR not readable: disk group not mounted or OCR corruptocrcheck, crsd.trc, ocrconfig -showbackup
Upper-stack resource OFFLINEResource-level problem: listener config, DB spfile/password file, VIP networkcrsctl stat res <name> -f, the matching agent trace, srvctl

Start and stop commands

ActionCommandRun as
Stop / start the whole stack on this nodecrsctl stop crs / crsctl start crsroot
Stop / start the cluster stack on all nodes (OHASD stays up)crsctl stop cluster -all / crsctl start cluster -allroot
Start one lower-stack resourcecrsctl start res ora.crsd -initroot
Stop / start a databasesrvctl stop database -d ORCL / srvctl start database -d ORCLoracle
Enable / disable autostart at bootcrsctl enable crs / crsctl disable crsroot
Never kill Clusterware daemons with kill -9. Killing ocssd.bin can make cssdmonitor reboot the node to protect data. Use crsctl.

Summary

  1. systemd starts OHASD as root.
  2. OHASD's agents start the lower stack: GPnP, interconnect, CSSD (cluster membership through voting files), CTSS, and the ASM instance.
  3. With ASM up, CRSD (root) starts and reads the OCR.
  4. CRSD's agents start the upper stack: VIPs and SCAN VIPs (root), listeners and disk groups (grid), databases and services (oracle).

Keep this order in mind and a failed startup becomes a simple question: which is the lowest layer that is not ONLINE? Start there.

Tamilselvan Perumal · Oracle ACE Apprentice · Lead Product Engineer & Senior DBA