When a RAC node does not come back after a reboot, the first question is always the same: how far did the startup get? To answer it you need a clear picture of the order in which Oracle Clusterware starts its daemons, which daemon starts which, and which OS user each one runs as.
This post walks through the Grid Infrastructure 19c startup in four phases, then shows the commands and log files I use to find exactly where a startup stopped.
The big picture
Clusterware starts in two layers. The lower stack is owned by OHASD: everything a node needs to join the cluster and reach shared storage. The upper stack is owned by CRSD: the resources your users care about, such as VIPs, listeners, databases and services.
Setting the record straight
Many startup diagrams online, including AI-generated ones, repeat a few mistakes. Here is how they compare with what you can see on a real 19c cluster:
| Common claim | What actually happens (11gR2 / 12c / 19c) |
|---|---|
OHASD is started by init.crs | init.crs was 10g / 11gR1. From 11gR2 the entry point is init.ohasd, started by systemd (oracle-ohasd.service) on modern Linux. |
| A "GPNPD root agent" starts GPNPD | There is no GPNPD root agent. GPNPD is started by OHASD's oraagent and runs as the grid owner. |
| CRSD runs as the Grid Infrastructure owner | crsd.bin runs as root. It is started by OHASD's orarootagent. |
| OHASD's root agent starts CSSD directly | CSSD is started by cssdagent (root), and ocssd.bin itself runs as grid. cssdmonitor watches it. |
| ASM instances are started by CRSD's oraagent | The ASM instance is started in the lower stack by OHASD's oraagent, before CRSD, because the OCR lives on ASM. CRSD's oraagent manages ASM disk groups. |
| Database instances are started by "the" oraagent | With role separation there are two CRSD oraagents: one as grid (listeners, disk groups) and one as oracle (databases, services). |
| Diskmon monitors the voting disks | diskmon is for Exadata storage cells. On non-Exadata systems it is usually OFFLINE. Voting files are handled by CSSD. |
See it on your own cluster
1. Overall health
crsctl check crs # OHASD, CRS, CSS, EVM on this node crsctl check cluster -all # CRS, CSS, EVM on every node olsnodes -n -s -t # node list with status
2. Lower stack (init resources): this is where to look if a node will not join.
crsctl stat res -t -init
Name Target State Server State details ora.asm ONLINE ONLINE node1 Started ora.cluster_interconnect.haip ONLINE ONLINE node1 ora.crf ONLINE ONLINE node1 ora.crsd ONLINE ONLINE node1 ora.cssd ONLINE ONLINE node1 ora.cssdmonitor ONLINE ONLINE node1 ora.ctssd ONLINE ONLINE node1 OBSERVER ora.diskmon OFFLINE OFFLINE (normal outside Exadata) ora.evmd ONLINE ONLINE node1 ora.gipcd ONLINE ONLINE node1 ora.gpnpd ONLINE ONLINE node1 ora.mdnsd ONLINE ONLINE node1
Abbreviated sample. OBSERVER for ctssd means NTP/chrony is keeping time and CTSS is only watching, which is normal.
3. Upper stack (cluster resources)
crsctl stat res -t srvctl status database -d ORCL -v srvctl status scan_listener srvctl status nodeapps
4. Which user runs what
ps -eo user,pid,args | grep -E "ohasd.bin|ocssd.bin|crsd.bin|orarootagent|oraagent|cssdagent|gpnpd|evmd" | grep -v grep
Where the logs are
From 12.1 onwards, Clusterware logs are in the ADR under the Grid Infrastructure owner's ORACLE_BASE:
cd $ORACLE_BASE/diag/crs/$(hostname -s)/crs/trace # as the grid user alert.log # Clusterware alert log: start here ohasd.trc # OHASD ocssd.trc # CSSD: voting files, heartbeats, evictions crsd.trc # CRSD: OCR access, resource management ohasd_orarootagent_root.trc / ohasd_oraagent_grid.trc # lower-stack agents crsd_orarootagent_root.trc / crsd_oraagent_grid.trc / crsd_oraagent_oracle.trc
$GRID_HOME/log/<hostname>/ instead, for example alert<hostname>.log, cssd/ocssd.log and crsd/crsd.log.Troubleshooting by layer
Work from the bottom up. A higher layer cannot start until the one below it is healthy.
| Stuck at | Typical cause | Check |
|---|---|---|
| OHASD not running | Autostart disabled, systemd unit failed, GI home/permissions issue | systemctl status oracle-ohasd, crsctl config crs, ohasd.trc |
| mdnsd / gpnpd / gipcd | Network interface renamed or down, corrupt GPnP profile | gpnptool get, oifcfg getif, gpnpd and gipcd traces |
| ora.cssd | Voting files not reachable (disk paths, permissions, multipath), private interconnect down | crsctl query css votedisk, ocssd.trc, ping over the private network |
| ora.asm | ASM disks not discovered (asm_diskstring, udev/AFD rules, ownership) | ASM alert log, kfod disks=all, ls -l /dev/oracleasm or AFD labels |
| ora.crsd | OCR not readable: disk group not mounted or OCR corrupt | ocrcheck, crsd.trc, ocrconfig -showbackup |
| Upper-stack resource OFFLINE | Resource-level problem: listener config, DB spfile/password file, VIP network | crsctl stat res <name> -f, the matching agent trace, srvctl |
Start and stop commands
| Action | Command | Run as |
|---|---|---|
| Stop / start the whole stack on this node | crsctl stop crs / crsctl start crs | root |
| Stop / start the cluster stack on all nodes (OHASD stays up) | crsctl stop cluster -all / crsctl start cluster -all | root |
| Start one lower-stack resource | crsctl start res ora.crsd -init | root |
| Stop / start a database | srvctl stop database -d ORCL / srvctl start database -d ORCL | oracle |
| Enable / disable autostart at boot | crsctl enable crs / crsctl disable crs | root |
Summary
- systemd starts OHASD as root.
- OHASD's agents start the lower stack: GPnP, interconnect, CSSD (cluster membership through voting files), CTSS, and the ASM instance.
- With ASM up, CRSD (root) starts and reads the OCR.
- CRSD's agents start the upper stack: VIPs and SCAN VIPs (root), listeners and disk groups (grid), databases and services (oracle).
Keep this order in mind and a failed startup becomes a simple question: which is the lowest layer that is not ONLINE? Start there.
Tamilselvan Perumal · Oracle ACE Apprentice · Lead Product Engineer & Senior DBA

