Oracle Clusterware and RAC Startup Sequence

Oracle Clusterware and RAC Startup Sequence

The Four-Phased Startup Process: A Methodical Orchestration

Phase 1: Initial System Boot and Root Agent Trigger -

This phase marks the very beginning of the Clusterware startup, typically initiated by the operating system's boot process. At this point, no specific Clusterware daemons are yet running; the system is merely preparing the ground.

• Operating System Boot: The server powers on, and the operating system (OS) initializes its core components.
• `init.crs` script (or equivalent systemd service): The OS executes the primary Clusterware startup script. On Linux systems, this is historically /etc/init.d/init.crs or, in modern Linux distributions, a systemd unit like oracle-ohasd.service. This script, running with root privileges, is responsible for launching the Oracle High Availability Services Daemon (OHASD).

Phase 2: Core Cluster Services Bootstrap -

This is where the true core of the Clusterware comes to life. The essential daemons responsible for high availability and cluster synchronization are initiated, forming the foundational layer without which no other cluster component can operate.

• `OHASD` (Oracle High Availability Services Daemon): This is the first Oracle Clusterware daemon to start. It runs as the root user and acts as the parent process for numerous other critical Clusterware components.
• Purpose: OHASD orchestrates the startup and shutdown of other vital daemons, constantly monitors their health, and automatically attempts restarts if a failure is detected, thereby ensuring high availability at the Clusterware level.
• Associated Processes (launched by OHASD's root agent):
• `CSSD` (Cluster Synchronization Services Daemon): Critically important for managing node membership, processing heartbeats (signals exchanged between nodes to confirm their availability), and handling node evictions (fencing) if communication is lost to prevent data corruption. CSSD primarily communicates over the private interconnect and utilizes quorum disks (voting disks).
• `Diskmon` (Disk Monitoring Service): Monitors the health and accessibility of shared storage, particularly the voting disks.
• `CRSD Root Agent`: A child process of OHASD specifically tasked with managing root-owned cluster resources. This agent will later initiate the main CRSD process.
• `GPNPD Root Agent`: Another child process of OHASD that is responsible for starting the GPNPD daemon.

Phase 3: Grid Infrastructure Daemons and Resource Management Core

Once the foundational high-availability and synchronization services are online, the cluster proceeds to launch the daemons responsible for dynamic configuration and the central management of all cluster resources.

• `GPNPD` (Grid Plug and Play Daemon): This daemon is launched by the GPNPD Root Agent, which is itself managed by OHASD.
• Purpose: Plays a vital role in managing the cluster's dynamic configuration. It facilitates easier addition or removal of nodes and manages network configurations within the cluster, supporting a "plug and play" functionality for grid components.
• `CRSD` (Cluster Ready Services Daemon):
• This central daemon is launched by the CRSD Root Agent, under OHASD's management.
• Purpose: CRSD is the primary resource manager for the entire cluster. It reads the Oracle Cluster Registry (OCR) to determine which resources (e.g., databases, listeners, VIPs, services) need to be managed, their dependencies, and their desired states. CRSD then uses various agents to ensure these resources are brought online or managed appropriately. It's essentially the cluster's orchestration engine.

Phase 4: All Cluster Services Online (RAC Database and Application Startup)

In this final phase, with the foundational daemons and resource management in place, the cluster proceeds to bring up all remaining services, including the Oracle RAC database instances, listeners, and any user-defined applications or services, making the environment fully operational.

Primary Agents and Processes Launched by CRSD:

• `Oracle Agent` (oraagent): A child process of CRSD.
• Purpose: Specifically responsible for starting and managing Oracle-specific resources defined in the OCR. This includes:
• Oracle Net Listeners: Processes that await incoming database connection requests.
• Database Instances (`smon`, pmon, dbw0, lgwr, etc.): The actual Oracle database background processes for each instance on the node.
• ASM Instances (`asmb`, rbal, etc.): If Oracle Automatic Storage Management (ASM) is being used for storage, these instances are brought online.
• Database Services:Any specific application services defined within the database that need to be managed by Clusterware.
• ACFS (ASM Cluster File System) Resources: If ACFS is deployed and managed by Clusterware.
• `System Agent` (orarootagent):

Another critical child process of CRSD.

• Purpose:Responsible for starting and managing root-owned cluster resources. Its primary responsibilities include:
• Virtual IP Addresses (VIPs): These are logical IP addresses that can transparently failover to another node if the current node fails, ensuring continuous client connectivity.
• Any other OS-level or network resources that are managed by Clusterware and require root privileges.

---

The Startup Flow: A Hierarchical View

To summarize the intricate flow, here's a hierarchical representation of the processes and their dependencies:

Operating System Boot

1. Triggers the init.crs script (or systemd service).

init.crs (or systemd service)

1. Starts OHASD (running as root).

OHASD (Oracle High Availability Services Daemon)

Launches:

1. Launches CSSD (Cluster Synchronization Services Daemon)
2. Launches Diskmon (Disk Monitoring Service)
3. Launches CRSD Root Agent
4. Launches GPNPD Root Agent

CRSD Root Agent

1. Starts CRSD (Cluster Ready Services Daemon - running as the Oracle Grid Infrastructure owner).

GPNPD Root Agent

1. Starts GPNPD (Grid Plug and Play Daemon - running as the Oracle Grid Infrastructure owner).

CRSD (Cluster Ready Services Daemon)

1. Launches Oracle Agent (oraagent)
2. Launches System Agent (orarootagent)

Oracle Agent (oraagent)

1. Starts Oracle Net Listeners
2. Starts Oracle Database Instances (e.g., smon, pmon, dbw0)
3. Starts ASM Instances (if applicable)
4. Starts Database Services
5. Starts ACFS Resources (if applicable)

System Agent (orarootagent)

1. Starts Virtual IP Addresses (VIPs)
2. Starts Other root-managed network resources

 

Setting Up Logical Replication in PostgreSQL: A Step-by-Step Guide

Setting Up Logical Replication in PostgreSQL: A Step-by-Step Guide

March 26, 2025

---

Introduction

Imagine managing a high-traffic PostgreSQL database and suddenly needing to migrate data, perform a version upgrade, or set up a real-time reporting system. However, downtime is not an option.

This is where logical replication becomes a lifesaver. Unlike traditional replication, which blindly copies everything, logical replication offers complete control over what gets replicated. You can sync only specific tables, making it perfect for large-scale applications, real-time analytics, and seamless migrations.

In this guide, we will walk through every step, from enabling replication to monitoring and troubleshooting. Whether you’re a DBA, data engineer, or developer, you’ll find this guide practical, easy to follow, and packed with real-world use cases.

By the end of this guide, you’ll not only understand logical replication but also be able to set it up like a pro. Let’s dive in and unlock the full potential of PostgreSQL’s logical replication.

---

1. What is Logical Replication in PostgreSQL?

Understanding the Publisher-Subscriber Model

PostgreSQL's logical replication follows a publisher-subscriber model:

• Publisher → The main server that sends out changes.

• Subscriber → The receiving server that applies those changes.

Unlike physical replication, which copies the entire database, logical replication operates at the table level. This allows you to choose exactly what data to replicate, making it efficient and flexible.

Use Cases of Logical Replication

Logical replication is widely used in production environments. Some real-world applications include:

• Netflix → Syncs data globally for real-time video analytics.

• Uber → Tracks live ride requests across different regions.

• Banking & Fintech → Ensures zero downtime during software upgrades.

When to Use Logical Replication

Use logical replication if:

• You need to sync specific tables instead of the whole database.

• You want to upgrade PostgreSQL without downtime.

• You need real-time analytics from production data.

• You want to migrate data without locking tables.

When NOT to Use Logical Replication

Avoid logical replication if:

• You need full database backups → Use physical replication instead.

• You have very high write workloads → Logical replication may lag.

• You expect schema changes (DDL statements) → These are not replicated.

Now that you have a solid understanding, let’s set up logical replication!

---

2. Step-by-Step: Setting Up Logical Replication in PostgreSQL

Step 1: Enable Logical Replication on Your Server

Before setting up replication, configure your PostgreSQL server.

Open the postgresql.conf file:

sudo nano /etc/postgresql/15/main/postgresql.conf

Modify (or add) these settings:

wal_level = logical         # Enables logical replication
max_replication_slots = 5   # Number of replication slots
max_wal_senders = 5         # Number of concurrent replication connections

Restart PostgreSQL to apply changes:

sudo systemctl restart postgresql

Pro Tip: If replication lags, increase max_wal_senders to 10 or more.

---

Step 2: Set Up the Publisher (Main Server)

Define the tables to replicate. Connect to your PostgreSQL Publisher server and execute:

CREATE PUBLICATION my_publication FOR TABLE employees;

This creates a publication named my_publication for the employees table.

---

Step 3: Allow the Subscriber to Connect

Grant permissions so that the subscriber can connect.

Run this command on the Publisher:

ALTER ROLE replication_user WITH REPLICATION;

Modify pg_hba.conf to allow remote replication connections:

host    replication     replication_user    192.168.1.100/24   md5

Restart PostgreSQL:

sudo systemctl restart postgresql

Now, the publisher is ready.

---

Step 4: Set Up the Subscriber (Replica Server)

On the Subscriber server, create the same table:

CREATE TABLE employees (
    id SERIAL PRIMARY KEY,
    name VARCHAR(100),
    salary NUMERIC
);

Connect it to the publisher:

CREATE SUBSCRIPTION my_subscription
CONNECTION 'host=192.168.1.10 dbname=company user=replication_user password=secret'
PUBLICATION my_publication;

Check if the subscription is active:

SELECT * FROM pg_subscription;

If the status is active, logical replication is successfully set up! 🎯

---

3. Testing Logical Replication

To verify replication:

1. Insert test data on the Publisher:

   INSERT INTO employees (name, salary) VALUES ('John Doe', 70000);

2. Check the Subscriber:

   SELECT * FROM employees;

If John Doe appears in the subscriber, replication is working!

---

4. Monitoring and Troubleshooting

Checking Replication Status

Run this on the Subscriber to check if replication is lagging:

SELECT * FROM pg_stat_subscription;

If lag is high, increase max_wal_senders on the Publisher.

Checking Replication Slots

On the Publisher, view active replication slots:

SELECT * FROM pg_replication_slots;

If a slot is inactive, restart the Subscriber.

Handling Conflicts

If data was inserted manually into the Subscriber, replication may stop. Fix it by:

DELETE FROM employees WHERE id = 5;

Then restart the Subscriber.

---

5. What’s Next? Advanced Use Cases

Once logical replication is working, you can:

• Replicate multiple tables → Add them to the publication.

• Stream data to Apache Kafka → Combine PostgreSQL replication with Kafka for real-time analytics.

• Sync with Cloud Databases → AWS RDS, Google Cloud SQL, or Azure Database for PostgreSQL.

---

Conclusion

Logical replication is a must-know skill for any DBA, data engineer, or developer working with PostgreSQL. Whether you’re handling real-time analytics, seamless migrations, or zero-downtime upgrades, mastering this feature will enhance your PostgreSQL expertise and make you stand out in the industry.

By following this guide, you now have the knowledge to set up and manage logical replication like a pro! 🚀

PL/SQL PACKAGE SYS.DBMS_BACKUP_RESTORE VERSION IS NOT CURRENTPL/SQL PACKAGE SYS.DBMS_RCVMAN VERSION 19.03.00.00 IS TOO OLD

 

Recently, I undertook the task of upgrading an Oracle database from version 19.03 to 19.22. As the process progressed, I began to encounter a series of errors that were consistently appearing in the alert log.

ARC3 (PID:3646): Archived Log entry 3829 added for T-1.S-6 ID 0xbc253a92 LAD:1

2024-06-27T06:12:14.930888+02:00

PL/SQL package SYS.DBMS_BACKUP_RESTORE version  is not current

PL/SQL package SYS.DBMS_RCVMAN version 19.03.00.00 is too old

PL/SQL package SYS.DBMS_BACKUP_RESTORE version  is not current

PL/SQL package SYS.DBMS_RCVMAN version 19.03.00.00 is too old

After diving deep into the issue, it was discovered that running a series of scripts after the upgrade would effectively solve the problem. Furthermore, it was necessary to recompile the invalid packages. Without any delay, I proceeded to execute the specified scripts and recompile the invalid objects.

1 2 3 4

SQL> @$ORACLE_HOME/rdbms/admin/dbmsrman.sql SQL> @$ORACLE_HOME/rdbms/admin/dbmsbkrs.sql SQL> @$ORACLE_HOME/rdbms/admin/prvtrmns.plb SQL> @$ORACLE_HOME/rdbms/admin/prvtbkrs.plb SQL*Plus: Release 19.0.0.0.0 - Production on Tue May 23 06:15:14 2023 Version 19.03.0.0.0   Copyright (c) 1982, 2022, Oracle.  All rights reserved.     Connected to: Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production Version 19.22.0.0.0   SQL> @$ORACLE_HOME/rdbms/admin/dbmsrman.sql   Session altered.     Package created.     PL/SQL procedure successfully completed.     PL/SQL procedure successfully completed.     PL/SQL procedure successfully completed.     Grant succeeded.     Grant succeeded.     Grant succeeded.     Grant succeeded.     Grant succeeded.     PL/SQL procedure successfully completed.     Synonym dropped.     View dropped.     Function dropped.     Type dropped.     Type dropped.     Type dropped.     Type created.     Type created.     Type created.     Type body created.     Function created.     View created.     Synonym created.     Synonym created.     Grant succeeded.     Synonym dropped.     View dropped.     Synonym dropped.     View dropped.     Synonym dropped.     View dropped.     Function dropped.     Type dropped.     Type dropped.     Type dropped.     Type created.     Type created.     Type created.     Type body created.     Function created.     View created.     View created.     Synonym created.     Synonym created.     View created.     View created.     Synonym created.     Synonym created.     View created.     View created.     Synonym created.     Synonym created.     Grant succeeded.     Grant succeeded.     Grant succeeded.     View created.     Synonym created.     Grant succeeded.     View created.     Synonym created.     Grant succeeded.     View created.     Synonym created.     Grant succeeded.     View created.     Synonym created.     Grant succeeded.     View created.     Synonym created.     Grant succeeded.     View created.     Synonym created.     Grant succeeded.     View created.     Synonym created.     Grant succeeded.     View created.     Synonym created.     Grant succeeded.     View created.     Synonym created.     Grant succeeded.     View created.     Synonym created.     Grant succeeded.     View created.     Synonym created.     Grant succeeded.     View created.     Synonym created.     Grant succeeded.     View created.     Synonym created.     Grant succeeded.     View created.     Synonym created.     Grant succeeded.     View created.     Synonym created.     Grant succeeded.     View created.     Synonym created.     Grant succeeded.     View created.     Synonym created.     Grant succeeded.     View created.     Synonym created.     Grant succeeded.     View created.     Synonym created.     Grant succeeded.     View created.     Synonym created.     Grant succeeded.     View created.     Synonym created.     Grant succeeded.     View created.     Synonym created.     Grant succeeded.     Session altered.   SQL> @$ORACLE_HOME/rdbms/admin/dbmsbkrs.sql   Session altered.     Package created.     Session altered.   SQL> @$ORACLE_HOME/rdbms/admin/prvtrmns.plb   Session altered.     Package body created.     Session altered.   SQL> @$ORACLE_HOME/rdbms/admin/prvtbkrs.plb   Session altered.     Package body created.     Session altered.   SQL>    set pagesize 1000           select 'alter '||decode(object_type,'PACKAGE BODY','PACKAGE',object_type)||' '||owner||'.'||object_name||' compile '||           decode(object_type,'PACKAGE BODY','body','PACKAGE','BODY')||';'           from  dba_objects           where  status      =  'INVALID' AND object_type <>'SYNONYM'; SQL>   2    3    4 'ALTER'||DECODE(OBJECT_TYPE,'PACKAGEBODY','PACKAGE',OBJECT_TYPE)||''||OWNER||'.' -------------------------------------------------------------------------------- alter VIEW BACKUP_MGR.RMAN_BACKUP_ARCHIVELOG_DETAILS compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_ARCHIVELOG_SUMMARY compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_CONTROLFL_DETAILS compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_CONTROLFL_SUMMARY compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_DATAFILE_DETAILS compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_DATAFILE_SUMMARY compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_FILES compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_JOB_DETAILS compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_SET_DETAILS compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_SET_SUMMARY compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_SPFILE_DETAILS compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_SPFILE_SUMMARY compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_SUBJOB_DETAILS compile ; alter VIEW BACKUP_MGR.RMAN_UNUSABLE_BACKUP_DETAILS compile ;   14 rows selected.   SQL> alter VIEW BACKUP_MGR.RMAN_BACKUP_ARCHIVELOG_DETAILS compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_ARCHIVELOG_SUMMARY compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_CONTROLFL_DETAILS compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_CONTROLFL_SUMMARY compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_DATAFILE_DETAILS compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_DATAFILE_SUMMARY compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_FILES compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_JOB_DETAILS compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_SET_DETAILS compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_SET_SUMMARY compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_SPFILE_DETAILS compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_SPFILE_SUMMARY compile ; alter VIEW BACKUP_MGR.RMAN_BACKUP_SUBJOB_DETAILS compile ; alter VIEW BACKUP_MGR.RMAN_UNUSABLE_BACKUP_DETAILS compile ;   View altered.   SQL> View altered.   SQL> View altered.   SQL> View altered.   SQL> View altered.   SQL> View altered.   SQL> View altered.   SQL> View altered.   SQL> View altered.   SQL> View altered.   SQL> View altered.   SQL> View altered.   SQL> View altered.   SQL> View altered. And voila, problem solved! I hope this post comes in handy for you! References: MOS note: PL/SQL package SYS.DBMS_BACKUP_RESTORE Version is Not Current (Doc ID 2741760.1)