Recreate Original Primary RAC database as standby database (following an activation)
In this section we will discuss how you can recreate the original primary Oracle RAC datafile following a disaster when you had to fail-over (activate) the standby.
To illustrate this process an example will be used.
This example does assume that the Original Primary RAC environment is still in tact and it will not cover the re-install of the Oracle Software etc.
Example: In this test case below a 3 Node RAC database is protected by a single instance standby database system.
Prior to Fail-over (activation) this was the configuration
RAC Database Name: DEV
Primary RAC Nodes:
Node 1: dbvrlin51 - running database instance DEV1 (ASM instance is +ASM1)
Node 2: dbvrlin52 - running database instance DEV2 (ASM instance is +ASM2)
Node 3: dbvrlin53 - running database instance DEV3 (ASM instance is +ASM3)
Standby Server:
Node 1: dbvrlin54 - running database instance DEV (which is the standby database)
Dbvisit Standby configuration:
Node 1: DDC file - dbv_DEV1.env
Node 2: DDC file - dbv_DEV2.env
Node 3: DDC file - dbv_DEV3.env
Standby Node: all DDC files will be located.
IMPORTANT: Make sure all Dbvisit Standby schedules are stopped
Failover:
During failover the standby database is activated using the "dbv_oraStartStop activate <DDC>" command, which is executed on the standby server.
Example: ./dbv_oraStartStop activate DEV1
oracle@dbvrlin54[/usr/dbvisit/standby]: ./dbv_oraStartStop activate DEV1 ============================================================= Dbvisit Standby Database Technology (7.0.22.12958) (pid 12965) dbv_oraStartStop started on dbvrlin54: Fri Sep 19 15:57:28 2014 () ============================================================= Activating means this database will become a Primary Database. It will no longer be a Standby Database for DEV on dbvrlin51. Activation cannot be reversed. =>Activate Standby Database on dbvrlin54? <Yes/No> [No]: Yes select inst_id, thread#, status, eanbled, instance from v$thread Are you sure? <Yes/No> [No]: Yes Activating now... Activate Standby Database DEV... Standby Database DEV activated. Shutting down standby Database DEV... Standby Database DEV shutdown successfully. Starting Activated Standby Database DEV... Activated Standby Database DEV started . File /usr/dbvisit/standby/conf/dbv_DEV1.env copied to /usr/dbvisit/standby/conf/dbv_DEV1.env.201409191557. Dbvisit Database configuration (DDC) file /usr/dbvisit/standby/conf/dbv_DEV1.env has been updated and variables have been reversed between primary and standby server. select inst_id, thread#, status, enabled, instance from v$thread SOURCE=dbvrlin54 DESTINATION=dbvrlin51. Activation complete. Please ensure a backup of this Database is made. Old archives from before the activation should be removed to avoid mix-up between new and old archive logs. Database has no tempfiles defined. You may need to add tempfiles to this database using ALTER TABLESPACE ADD TEMPFILE command. If the Dbvisit Standby process is to be reversed, then Database on dbvrlin51 will need to be rebuilt as a Standby Database. select inst_id, thread#, status, enabled, instance from gv$thread ============================================================= dbv_oraStartStop ended on dbvrlin54: Fri Sep 19 15:58:01 2014 =============================================================
IMPORTANT
IMPORTANT: When performing an activation the DDC file values will be reversed automatically. This is done so that when you want to recreate the standby (meaning rebuild the old primary as a standby) you can just use the DDC file used during activation without any required changes.
But to make sure, please review the SOURCE and DESTINATION values in the DDC file and make sure they match the required config.
Example in this test case the SOURCE is now dbvrlin54 and the DESTINATION is dbvrlin51
Now that the fail-over was performed the original RAC environment needs to be rebuild as a standby database once it is restored to full capacity.
The following steps can be following:
- Stop the Oracle RAC database, in case:
From node 1, use the srvctl command to stop the database: oracle@dbvrlin51[/home/oracle]: . oraenv ORACLE_SID = [oracle] ? DEV1 The Oracle base remains unchanged with value /u01/app/oracle oracle@dbvrlin51[/home/oracle]: srvctl stop database -d DEV
- Make sure you clean up the ASM location where the old primary database files were located.
- Ensure that the FRA / recovery area is cleaned and that no old files are in place.
Start the Create Standby Database (CSD) Procedure from the new primary (old standby):
- Execute the dbvisit_setup command and select option 7 to start the create standby database process. See example below:
oracle@dbvrlin54[/usr/dbvisit/standby]: ./dbvisit_setup
=========================================================
Dbvisit Standby Database Technology (7.0.22.12419)
http://www.dbvisit.com
=========================================================
=>dbvisit_setup only needs to be run on the primary server.
Is this the primary server? <Yes/No> [Yes]:
=========================================================
Dbvisit Standby Database Technology (7.0.22.12419)
http://www.dbvisit.com
Dbvisit Database setup
Default values will be shown in []
Options:
1) New Dbvisit Database setup (combines options 2,3,4)
1a) New Dbvisit RAC Instance setup (combines options 2,3,4)
2) New Dbvisit Database configuration (DDC) file setup
3) New Dbvisit Archive Management Module (AMM) setup
4) New Dbvisit Database repository (DDR) setup
5) Manage Dbvisit Database repository (DDR)
6) Update Dbvisit Database configuration (DDC) file
7) Create Standby Database (and template)
8) Synchronize Standby Database
9) Uninstall Dbvisit Database repository (DDR)
E) Exit
=========================================================
Please enter choice : 7
------------------------------------------------------------------------------
=>Creating Standby Database.
Primary database will NOT be shutdown.
Choose database:
The following Oracle database(s) have a Dbvisit Database configuration (DDC)
file on this server:
DDC
===
1) DEV1
2) DEV2
3) DEV3
4) DEVDR1
5) Return to menu
Please enter choice : 1
Is this correct? <Yes/No> [Yes]:
Database DEV is up.
>>> Checking Dbvisit Standby for configurational differences between dbvrlin54 and
dbvrlin51...
Dbvisit Standby configurational differences found between dbvrlin54 and dbvrlin51...
> Transferring 'dbv_DEV1.env' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [9849 KB/s] - done.
>>> Checking Oracle installation on dbvrlin51 in
/u01/app/oracle/product/11.2.0/db_1/bin/oracle...
>>> Checking primary datafiles on dbvrlin54...
>>> Validating Dbvisit configuration file on dbvrlin51...
>>> Checking if a database is up and running on dbvrlin51...
Checks completed OK.
>>> Total database size for DEV is 3.05GB
What would you like to do:
1 - Create standby database (and optionally save settings in template)
2 - Help
3 - Terminate processing
Please enter your choice [1]: 1
-------------------------------------------------------------------------------
Do you want to create an ASM standby database (with all or some database files
in ASM storage)? [Yes]:
Standby database will be ASM: Y
Is this correct? <Yes/No> [Yes]:
-------------------------------------------------------------------------------
Use network compression to compress the database files during transfer? [No]: No
Database files will be compressed during transfer: N
Is this correct? <Yes/No> [Yes]: Yes
-------------------------------------------------------------------------------
Dbvisit Standby provides functionality to create missing filesystem directories
on the standby server automatically.
Do you want Dbvisit Standby to create missing filesystem directories on the
standby server automatically? [Yes]: Yes
Dbvisit Standby will create missing filesystem directories automatically: Y
Is this correct? <Yes/No> [Yes]: Yes
-------------------------------------------------------------------------------
Primary database contains Oracle Managed Files (OMF). These files will be
created as OMF files on the standby server. Please make sure creating of OMF
files is enabled on the standby by setting up parameters db_create_file_dest,
db_create_online_log_dest_n and db_recovery_file_dest to point to valid ASM
locations. Refer to Oracle documentation for information how to enable creating
of OMF files.
The following oracle database parameters will be set in the standby database pfile or spfile:
-------------------------------------------------------------------------------
audit_file_dest /u01/app/oracle/admin/DEV/adump
compatible 11.2.0.4.0
db_block_size 8192
db_create_file_dest +DATA
db_name DEV
db_recovery_file_dest +FRA
db_recovery_file_dest_size 19327352832
diagnostic_dest /u01/app/oracle
dispatchers (PROTOCOL=TCP) (SERVICE=DEVXDB)
log_archive_max_processes 4
memory_target 1073741824
open_cursors 300
processes 150
remote_login_passwordfile EXCLUSIVE
spfile +DATA
undo_tablespace UNDOTBS1
-------------------------------------------------------------------------------
What would you like to do:
1 - Proceed with creating the standby database
2 - Edit oracle database parameters for the standby database pfile/spfile
3 - Terminate processing
Please enter your choice [1]:
Validating oracle database parameters... please wait
SUCCEEDED
-------------------------------------------------------------------------------
Do you want to use TRANSPORTABLE MEDIA to transfer the database backup to the
standby server? Transportable media is an external device such as a USB drive
that is first plugged into the primary server and then manually transferred to
the standby site and plugged into the standby server to continue the process.
It can be used for large databases or slow networks.
Specifying No means the network will be used to transfer the database backup.
[No]: No
Transportable media will be used: N
Is this correct? <Yes/No> [Yes]:
-------------------------------------------------------------------------------
A temporary location must be specified on dbvrlin54 where the database will be
backed up to first.
This location must be big enough to hold RMAN backup of the whole database
(3.05GB).
Specify the location on this server: [/usr/tmp]:
Directory for temporary location will be: /usr/tmp
Is this correct? <Yes/No> [Yes]:
-------------------------------------------------------------------------------
A temporary location must be specified on dbvrlin51 where the database backup
will be copied to before moving to specified locations.
Specify location on remote server: [/usr/tmp]:
Directory for temporary will be: /usr/tmp
Is this correct? <Yes/No> [Yes]:
-------------------------------------------------------------------------------
The creation of the standby database consists of 2 parts:
1) Backup database to temporary location.
2) Transfer backup from temporary location to standby server.
The process can be stopped after step 1 and then continued with step 2 at a
later date.
Do you want to the process to stop between these 2 steps? [No]: No
Process will stop between steps: N
Is this correct? <Yes/No> [Yes]:
Dbvisit Standby will automatically create standby OMF ASM datafiles and
tempfiles.
The files will be created under <ASM Disk
Group>/<db_unique_name>/datafile|tempfile.
Review and confirm ASM disk groups for standby datafiles and tempfiles.
-------------------------------------------------------------------------------
Primary ===> Standby
-------------------------------------------------------------------------------
+DATA ===> +DATA
-------------------------------------------------------------------------------
What would you like to do:
1 - Confirm standby location(s) are correct and continue
2 - Provide different standby location(s)
Please enter your choice [1]:
Standby locations validated.
Primary database contains non-OMF redo logs.
Non-OMF ASM redo logs will be created under: <ASM Disk
Group>/<db_unique_name>/onlinelog.
Make sure these locations exist on the standby server or create them manually.
Review and confirm ASM disk group names for non-OMF standby redo logs.
-------------------------------------------------------------------------------
Primary ===> Standby
-------------------------------------------------------------------------------
+DATA ===> +DATA
-------------------------------------------------------------------------------
What would you like to do:
1 - Confirm standby location(s) are correct and continue
2 - Provide different standby location(s)
Please enter your choice [1]:
Standby locations validated.
-------------------------------------------------------------------------------
=>Create standby database template for DEV using provided answers? <Yes/No>
[Yes]: '
>>> Dbvisit will now run a pre-flight check for standby database creation. An attempt will
be made to create a standby (s)pfile using oracle standby database parameters, followed
by trying to start the standby instance. If this step fails, then please double-check
the following items before re-running Dbvisit again:
1) Review the standby database parameters you have supplied and provide valid values
unless a template is used.
2) Recreate the template to provide valid values for standby database parameters if a
template is used.
'
Running pre-flight check for standby creation, please wait... - done.
>>> Backing up primary database...
Backing up datafile 1... - done.
Backing up datafile 2... - done.
Backing up datafile 3... - done.
Backing up datafile 4... - done.
Backing up datafile 5... - done.
Backing up datafile 6... - done.
Backing up datafile 7... - done.
Backing up datafile 8... - done.
|
>>> Creating standby control file... - done.
Backup of primary database completed.
>>> Transferring backup from dbvrlin54 to dbvrlin51...
> Transferring 'dbv_DEV1_csd_dbf_1_3ppitg2e_1_1.rman' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [11677 KB/s] - done.
> Transferring 'dbv_DEV1_csd_dbf_1_3qpitg3h_1_1.rman' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [11985 KB/s] - done.
> Transferring 'dbv_DEV1_csd_dbf_2_3rpitg41_1_1.rman' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [11951 KB/s] - done.
> Transferring 'dbv_DEV1_csd_dbf_3_3spitg59_1_1.rman' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [11603 KB/s] - done.
> Transferring 'dbv_DEV1_csd_dbf_4_3tpitg5k_1_1.rman' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [11858 KB/s] - done.
> Transferring 'dbv_DEV1_csd_dbf_5_3upitg6j_1_1.rman' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [12035 KB/s] - done.
> Transferring 'dbv_DEV1_csd_dbf_6_3vpitg6t_1_1.rman' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [11180 KB/s] - done.
> Transferring 'dbv_DEV1_csd_dbf_7_40pitg77_1_1.rman' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [11877 KB/s] - done.
> Transferring 'dbv_DEV1_csd_dbf_8_41pitg7s_1_1.rman' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [11976 KB/s] - done.
>>> Restoring standby control files... - done.
>>> Starting standby database DEV1 on dbvrlin51 mount... - done.
>>> Restoring datafiles on dbvrlin51...
Restoring datafile 1 - done.
Restoring datafile 2 - done.
Restoring datafile 3 - done.
Restoring datafile 4 - done.
Restoring datafile 5 - done.
Restoring datafile 6 - done.
Restoring datafile 7 - done.
Restoring datafile 8 - done.
>>> Renaming standby redo logs and tempfiles on dbvrlin51... - done.
>>> Performing checkpoint and archiving logs... - done.
>>> Finishing standby database creation... - done.
Standby database created.
Please execute the following commands to complete the database creation process:
1) Run dbvisit on all primary nodes that are up and running.
2) After having run dbvisit on all primary nodes, execute it on the standby node (this
will ensure the standby is in sync with the primary).
If a primary node was down during standby creation, then you need to run Dbvisit with
the '-R' option once this node has been started.
Please press <Enter> to continue...=========================================================
Standby Database creation on dbvrlin51 completed.
Next steps:
1) Exit out of dbvisit_setup.
2) Run Dbvisit on this server with command:
dbvisit database_name
Run Dbvisit on all other primary nodes with command:
dbvisit -R instance_name
3) Run Dbvisit on standby server with command:
dbvisit database_name
Once Dbvisit has been tested, schedule Dbvisit on primary and standby servers
using cron or other scheduling tool.
=========================================================
Please press <Enter> to continue...
- No Run Dbvisit Standby as normal first
On Primary Server (dbvlin54)
oracle@dbvrlin54[/usr/dbvisit/standby]: ./dbvisit DEV1
=============================================================
Dbvisit Standby Database Technology (7.0.22.12963) (pid 14229)
dbvisit started on dbvrlin54: Fri Sep 19 16:30:44 2014 ()
=============================================================
Dbvisit Standby license will expire in 2 days. Please contact Dbvisit Standby (www.dbvisit.com) to obtain license.
>>> Obtaining information from standby database (RUN_INSPECT=Y)...
>>> Checking Dbvisit Standby for configurational differences between dbvrlin54 and
dbvrlin51...
No configurational differences found between dbvrlin54 and dbvrlin51.
>>> Log file(s) for DEV will be transferred from dbvrlin54 to dbvrlin51...
> Transferring 'thread_1_seq_2.276.858702483.gz' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [12241 KB/s] - done.
> Transferring 'thread_1_seq_3.275.858702491.gz' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [11184 KB/s] - done.
> Transferring 'thread_1_seq_4.274.858702501.gz' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [11157 KB/s] - done.
> Transferring 'thread_1_seq_5.273.858702511.gz' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [12243 KB/s] - done.
> Transferring 'thread_1_seq_6.272.858702519.gz' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [12320 KB/s] - done.
> Transferring 'thread_1_seq_7.261.858702523.gz' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [10256 KB/s] - done.
> Transferring 'thread_1_seq_8.271.858702529.gz' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [12430 KB/s] - done.
> Transferring 'thread_1_seq_9.288.858702537.gz' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [10123 KB/s] - done.
> Transferring 'thread_1_seq_10.287.858702549.gz' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [12586 KB/s] - done.
9 archive log transfers to dbvrlin51 for DEV completed.
Last sequence was 10 thread 1.
No Mail sent as SEND_MAIL_FLAG = N
>>> Dbvisit Archive Management Module (AMM)
Config: number of archives to keep = 0
Config: number of days to keep archives = 7
Config: archive backup count = 0
Config: diskspace full threshold = 80%
Current disk percent full (+FRA/dev/archivelog/2014_09_19) = 3%
Number of archive logs deleted = 0
Contacting Standby Database DEV1 on dbvrlin51...
Next standby sequence required for recovery (2) for thread 2.
201409191630 - No new logs to transfer to dbvrlin51 for DEV. LOGSWITCH=N
No Mail sent as SEND_MAIL_FLAG = N
=============================================================
dbvisit ended on dbvrlin54: Fri Sep 19 16:32:35 2014
=============================================================
- On Standby Server (RAC Node 1 - dbvrlin51), now run "./dbvisit <DDC>" to apply logs. In this example: ./dbvisit DEV1
oracle@dbvrlin51[/usr/dbvisit/standby]: ./dbvisit DEV1
=============================================================
Dbvisit Standby Database Technology (7.0.22.12963) (pid 21843)
dbvisit started on dbvrlin51: Fri Sep 19 16:36:17 2014 ()
=============================================================
>>> Log file(s) for DEV1 from dbvrlin54 will be applied to dbvrlin51
201409191636 - Log seq 1 thread 1 applied to standby database DEV1.
201409191636 - Log seq 2 thread 1 applied to standby database DEV1.
201409191636 - Log seq 3 thread 1 applied to standby database DEV1.
201409191636 - Log seq 4 thread 1 applied to standby database DEV1.
201409191636 - Log seq 5 thread 1 applied to standby database DEV1.
201409191636 - Log seq 6 thread 1 applied to standby database DEV1.
201409191636 - Log seq 7 thread 1 applied to standby database DEV1.
201409191636 - Log seq 8 thread 1 applied to standby database DEV1.
201409191636 - Log seq 9 thread 1 applied to standby database DEV1.
201409191636 - Log seq 10 thread 1 applied to standby database DEV1.
201409191636 - Log seq 11 thread 1 applied to standby database DEV1.
No Mail sent as SEND_MAIL_FLAG_DR = N
>>> Dbvisit Archive Management Module (AMM)
Config: number of archives to keep = 0
Config: number of days to keep archives = 7
Config: diskspace full threshold = 80%
Processing /u01/app/oracle/archive/DEV...
Archive log dir: /u01/app/oracle/archive/DEV
Total number of archive files : 11
Number of archive logs deleted = 0
Current Disk percent full : 64%
=============================================================
dbvisit ended on dbvrlin51: Fri Sep 19 16:37:04 2014
=============================================================
If you do get this error on the primary RAC node 1 when you run it the first time, go back to the primary node and run "./dbvisit -R <DDC>" in this case "./dbvisit -R DEV1" to resend all logs.
oracle@dbvrlin51[/usr/dbvisit/standby]: ./dbvisit DEV1
=============================================================
Dbvisit Standby Database Technology (7.0.22.12963) (pid 21444)
dbvisit started on dbvrlin51: Fri Sep 19 16:32:09 2014 ()
=============================================================
>>> Sending heartbeat message... - done.
>>> Log file(s) for DEV1 from dbvrlin54 will be applied to dbvrlin51
201409191632 - Current required archive log with sequence 1 and thread 1 not found. However subsequent archive log with sequence 2 is available.
Please run command: dbvisit -r 1 <DDC>
on primary node for thread 1 to resolve.
Dbvisit Standby terminated.
Return code = 447
(Tracefile required if contacting Dbvisit Standby support: /usr/dbvisit/standby/trace/21444_dbvisit_DEV1_201409191632.trc (server:dbvrlin51))
No Mail sent as SEND_MAIL_FLAG_DR = N
- Now at this stage the spfile might not have all the correct details for all the RAC instances. This can be added. Example, these parameters must be set:
*.cluster_database=true DEV1.instance_number=1 DEV2.instance_number=2 DEV3.instance_number=3 *.remote_listener='dbvr5123-scan.dbvisit.co.nz:1521' DEV1.thread=1 DEV2.thread=2 DEV3.thread=3 DEV1.undo_tablespace='UNDOTBS1' DEV2.undo_tablespace='UNDOTBS2' DEV3.undo_tablespace='UNDOTBS3'
Important: you will need to update values to match your environment.
To update these values in the spfile, login to the database instance from node 1 (dbvrlin51 in this case) and run the following commands:
oracle@dbvrlin51[/usr/dbvisit/standby]: sqlplus /nolog SQL*Plus: Release 11.2.0.4.0 Production on Fri Sep 19 16:42:59 2014 Copyright (c) 1982, 2013, Oracle. All rights reserved. SQL> connect / as sysdba Connected. sql> alter system set instance_number=1 scope=spfile sid='DEV1'; sql> alter system set instance_number=2 scope=spfile sid='DEV2'; sql> alter system set instance_number=3 scope=spfile sid='DEV3'; sql> alter system set remote_listener='dbvr5123-scan.dbvisit.co.nz:1521' scope=spfile sid='*'; sql> alter system set thread=1 scope=spfile sid='DEV1'; sql> alter system set thread=2 scope=spfile sid='DEV2'; sql> alter system set thread=3 scope=spfile sid='DEV3'; sql> alter system set undo_tablespace='UNDOTBS1' scope=spfile sid='DEV1'; sql> alter system set undo_tablespace='UNDOTBS2' scope=spfile sid='DEV2'; sql> alter system set undo_tablespace='UNDOTBS3' scope=spfile sid='DEV3';
- No run a log gap report to make sure that there is no gap for thread 1 (run this from the primary database - dbvrlin54)
oracle@dbvrlin54[/usr/dbvisit/standby]: ./dbvisit -i DEV1
=============================================================
Dbvisit Standby Database Technology (7.0.22.12963) (pid 15599)
dbvisit started on dbvrlin54: Fri Sep 19 16:44:02 2014 ()
=============================================================
Dbvisit Standby license will expire in 2 days. Please contact Dbvisit Standby (www.dbvisit.com) to obtain license.
Dbvisit Standby log gap report for DEV at 201409191644 (thread 1):
-------------------------------------------------------------
Standby database on dbvrlin51 is at sequence: 11.
Primary database on dbvrlin54 is at log sequence: 12.
Primary database on dbvrlin54 is at archived log sequence: 11.
Dbvisit Standby last transfer log sequence: 11.
Dbvisit Standby last transfer at: 201409191632.
Archive log gap for DEV: 0.
Transfer log gap for DEV: 0.
Standby database time lag (HH:MI:SS): 00:11:21.
No Mail sent as SEND_MAIL_FLAG = N
Dbvisit Standby log gap report for DEV at 201409191644 (thread 2):
-------------------------------------------------------------
Standby database on dbvrlin51 is at sequence: 1.
Primary database on dbvrlin54 is at log sequence: 2.
Primary database on dbvrlin54 is at archived log sequence: 1.
Dbvisit Standby last transfer log sequence: 1.
Dbvisit Standby last transfer at: 201409191632.
Archive log gap for DEV: 0.
Transfer log gap for DEV: 0.
Standby database time lag (HH:MI:SS): 00:11:58.
No Mail sent as SEND_MAIL_FLAG = N
Dbvisit Standby log gap report for DEV at 201409191644 (thread 3):
-------------------------------------------------------------
Standby database on dbvrlin51 is at sequence: 1.
Primary database on dbvrlin54 is at log sequence: 2.
Primary database on dbvrlin54 is at archived log sequence: 1.
Dbvisit Standby last transfer log sequence: 0.
Dbvisit Standby last transfer at: .
Archive log gap for DEV: 0.
Transfer log gap for DEV: 1.
Standby database time lag (HH:MI:SS): 00:12:34.
No Mail sent as SEND_MAIL_FLAG = N
=============================================================
dbvisit ended on dbvrlin54: Fri Sep 19 16:45:25 2014
=============================================================
- Make sure you have good backups of the primary database. This is just general good practice.
- Perform a Graceful Switchover Back using the dbv_oraStartStop command once ready.
Example, execute this on the primary node (dbvrlin54) and important, also on the standby node (RAC node 1 - dbvrlin51) : ./dbv_oraStartStop switchover 911
Example output from primary node (dbvrlin54):
oracle@dbvrlin54[/usr/dbvisit/standby]: ./dbv_oraStartStop switchover DEV1 911
=============================================================
Dbvisit Standby Database Technology (7.0.22.12958) (pid 15807)
dbv_oraStartStop started on dbvrlin54: Fri Sep 19 16:49:38 2014 ()
=============================================================
=============================================================
Graceful Switchover starting on Primary Database DEV.
Timestamp: 201409191649.
>>> RAC Database DEV will be shutdown and restarted as single instance <<<
Ensure Dbvisit is no longer scheduled.
>>> Graceful Switchover will attempt to reset Oracle parameters db_file_name_convert and log_file_name_convert to default values (null strings) in the spfile <<<
Obtaining archive log gap....
Contacting Standby Database DEV on dbvrlin51...
Next standby sequence required for recovery (12) for thread 1.
Archive Log Gap for thread 1 is: 0. This is correct to continue.
Contacting Standby Database DEV on dbvrlin51...
Next standby sequence required for recovery (2) for thread 2.
Archive Log Gap for thread 2 is: 0. This is correct to continue.
Contacting Standby Database DEV on dbvrlin51...
Next standby sequence required for recovery (2) for thread 3.
Archive Log Gap for thread 3 is: 0. This is correct to continue.
Please enter unique key to begin graceful switchover for database DEV.
The same key must be entered on both primary and standby server.
Please start command: dbv_oraStartStop switchover DEV
on dbvrlin51 if not already started.
Batch key will be used: 911
Key 911 entered.
Contacting dbvrlin51 to ensure the same unique key is entered for DEV.
Waiting for Key 1 on dbvrlin51...
Checkpoint 1 completed. Key found on dbvrlin51
Waiting for Checkpoint 2 on dbvrlin51...
Checkpoint 2 completed. Key found on dbvrlin51
Waiting for Checkpoint 3 on dbvrlin51...
Checkpoint 3 completed. Key found on dbvrlin51
Shutting down regular Database DEV...
Regular Database DEV shutdown successfully.
Starting Regular Database DEV...
Regular Database DEV started restrict.
Performing Oracle Checkpoint.
Waiting 3 seconds for log switch completion...
Creating standby control file as '/usr/dbvisit/standby/tmp/GS/DEV1/X.dbvisit.911.DEV1.standbycontrolfile'...
Waiting for Checkpoint 4 on dbvrlin51...
Checkpoint 4 completed. Key found on dbvrlin51
Copying new archives for DEV to dbvrlin51...
Compressing thread_1_seq_12.285.858703845...
> Transferring 'thread_1_seq_12.285.858703845.gz' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [14451 KB/s] - done.
Compressing thread_1_seq_13.284.858703901...
> Transferring 'thread_1_seq_13.284.858703901.gz' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [10618 KB/s] - done.
Shutting down regular Database DEV...
Regular Database DEV shutdown successfully.
Copying subsequent new archives for DEV to dbvrlin51...
Compressing thread_1_seq_2.276.858702483...
> Transferring 'thread_1_seq_2.276.858702483.gz' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [11647 KB/s] - done.
Compressing thread_1_seq_5.273.858702511...
> Transferring 'thread_1_seq_5.273.858702511.gz' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [11598 KB/s] - done.
Copying redo logs ... this may take a while...
Compressing X.dbvisit.911.DEV1.redo_1.log...
> Transferring 'X.dbvisit.911.DEV1.redo_1.log.gz' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [12943 KB/s] - done.
Compressing X.dbvisit.911.DEV1.redo_2.log...
> Transferring 'X.dbvisit.911.DEV1.redo_2.log.gz' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [12331 KB/s] - done.
Compressing X.dbvisit.911.DEV1.redo_3.log...
> Transferring 'X.dbvisit.911.DEV1.redo_3.log.gz' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [12539 KB/s] - done.
Compressing X.dbvisit.911.DEV1.redo_4.log...
> Transferring 'X.dbvisit.911.DEV1.redo_4.log.gz' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [11773 KB/s] - done.
Compressing X.dbvisit.911.DEV1.redo_5.log...
> Transferring 'X.dbvisit.911.DEV1.redo_5.log.gz' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [11991 KB/s] - done.
Compressing X.dbvisit.911.DEV1.redo_6.log...
> Transferring 'X.dbvisit.911.DEV1.redo_6.log.gz' to server dbvrlin51:7890
Progress: 0%...20%...40%...60%...80%...100% [12114 KB/s] - done.
Waiting for Checkpoint 5 on dbvrlin51...
Checkpoint 5 completed. Key found on dbvrlin51
Backing up current control files for DEV dbvrlin54...
Database DEV on dbvrlin54 is already down. No action taken.
Starting Regular Database DEV...
Regular Database DEV started nomount.
Control file backed up as /usr/dbvisit/standby/tmp/GS/DEV1/X.dbvisit.911.DEV1.controlfile.
Shutting down standby Database DEV...
Standby Database DEV shutdown successfully.
Waiting for Checkpoint 6 on dbvrlin51...
Checkpoint 6 completed. Key found on dbvrlin51
Waiting for Checkpoint 7 on dbvrlin51...
Checkpoint 7 completed. Key found on dbvrlin51
Waiting for Checkpoint 8 on dbvrlin51...
Checkpoint 8 completed. Key found on dbvrlin51
Waiting for Checkpoint 9 on dbvrlin51...
Checkpoint 9 completed. Key found on dbvrlin51
Waiting for Checkpoint 10 on dbvrlin51...
Checkpoint 10 completed. Key found on dbvrlin51
Waiting for Checkpoint 11 on dbvrlin51...
Checkpoint 11 completed. Key found on dbvrlin51
Waiting for Checkpoint 12 on dbvrlin51...
Checkpoint 12 completed. Key found on dbvrlin51
Database DEV on dbvrlin54 is already down. No action taken.
Starting Regular Database DEV...
Regular Database DEV started nomount.
STANDBY control file(s) restored from /usr/dbvisit/standby/tmp/GS/DEV1/X.dbvisit.911.DEV1.standbycontrolfile.
Shutting down standby Database DEV...
Standby Database DEV shutdown successfully.
Starting Standby Database DEV...
Standby Database DEV started .
Waiting for Checkpoint 13 on dbvrlin51...
Checkpoint 13 completed. Key found on dbvrlin51
Waiting for Checkpoint 14 on dbvrlin51...
Checkpoint 14 completed. Key found on dbvrlin51
File /usr/dbvisit/standby/conf/dbv_DEV1.env copied to /usr/dbvisit/standby/conf/dbv_DEV1.env.201409191649.
Dbvisit Database configuration (DDC) file /usr/dbvisit/standby/conf/dbv_DEV1.env has been updated and variables have been reversed between primary and standby server.
SOURCE=dbvrlin51 DESTINATION=dbvrlin54.
Waiting for Checkpoint 15 on dbvrlin51...
Checkpoint 15 completed. Key found on dbvrlin51
Waiting for Checkpoint 16 on dbvrlin51...
Checkpoint 16 completed. Key found on dbvrlin51
Uncompressing thread_1_seq_12.285.858703845.gz...
Uncompressing thread_1_seq_13.284.858703901.gz...
Uncompressing thread_1_seq_2.276.858702483.gz...
Uncompressing thread_1_seq_5.273.858702511.gz...
Tempfiles dropped.
Waiting for Checkpoint 17 on dbvrlin51...
Checkpoint 17 completed. Key found on dbvrlin51
Waiting for Checkpoint 18 on dbvrlin51...
Checkpoint 18 completed. Key found on dbvrlin51
Graceful switchover completed.
This database (DEV) is now a standby database.
To keep this new standby database in synch,
reschedule Dbvisit as per normal:
dbvisit DEV1
=============================================================
dbv_oraStartStop ended on dbvrlin54: Fri Sep 19 16:56:08 2014
=============================================================
oracle@dbvrlin54[/usr/dbvisit/standby]:
Example output from the New primary node (RAC Node 1 - dbvrlin51) oracle@dbvrlin51[/usr/dbvisit/standby]: ./dbv_oraStartStop switchover DEV1 911 ============================================================= Dbvisit Standby Database Technology (7.0.22.12958) (pid 23059) dbv_oraStartStop started on dbvrlin51: Fri Sep 19 16:49:51 2014 () ============================================================= ============================================================= Graceful Switchover starting on Standby Database DEV1. Timestamp: 201409191649. Database DEV1 will be shutdown and restarted! Ensure Dbvisit is no longer scheduled. Please enter unique key to begin graceful switchover for database DEV1. The same key must be entered on both primary and standby server. Please start command: dbv_oraStartStop switchover DEV1 on dbvrlin54 if not already started. Batch key will be used: 911 Key 911 entered. Contacting dbvrlin54 to ensure the same unique key is entered for DEV1. Waiting for Key 1 on dbvrlin54... Checkpoint 1 completed. Key found on dbvrlin54 Waiting for Checkpoint 2 on dbvrlin54... Checkpoint 2 completed. Key found on dbvrlin54 Waiting for Checkpoint 3 on dbvrlin54... Checkpoint 3 completed. Key found on dbvrlin54 Waiting for Checkpoint 4 on dbvrlin54... Checkpoint 4 completed. Key found on dbvrlin54 Shutting down standby Database DEV1... Standby Database DEV1 shutdown successfully. Waiting for Checkpoint 5 on dbvrlin54... Checkpoint 5 completed. Key found on dbvrlin54 Uncompressing 1_12_858700653.arc.gz... Uncompressing 1_13_858700653.arc.gz... Uncompressing 2_2_858700653.arc.gz... Uncompressing 2_3_858700653.arc.gz... Uncompressing X.dbvisit.911.DEV1.redo_1.log.gz... Uncompressing X.dbvisit.911.DEV1.redo_2.log.gz... Uncompressing X.dbvisit.911.DEV1.redo_3.log.gz... Uncompressing X.dbvisit.911.DEV1.redo_4.log.gz... Uncompressing X.dbvisit.911.DEV1.redo_5.log.gz... Uncompressing X.dbvisit.911.DEV1.redo_6.log.gz... Waiting for Checkpoint 6 on dbvrlin54... Checkpoint 6 completed. Key found on dbvrlin54 Waiting for Checkpoint 7 on dbvrlin54... Checkpoint 7 completed. Key found on dbvrlin54 Backing up current control files for DEV1 dbvrlin51... Database DEV1 on dbvrlin51 is already down. No action taken. Starting Standby Database DEV1... Standby Database DEV1 started nomount. Control file backed up as /usr/dbvisit/standby/tmp/GS/DEV1/X.dbvisit.911.DEV1.stdby_ctl_bck. Shutting down standby Database DEV1... Standby Database DEV1 shutdown successfully. Waiting for Checkpoint 8 on dbvrlin54... Checkpoint 8 completed. Key found on dbvrlin54 Starting Standby Database DEV1... Standby Database DEV1 started nomount. PRIMARY control file(s) restored from /usr/dbvisit/standby/tmp/GS/DEV1/X.dbvisit.911.DEV1.controlfile. Mount Database DEV1... Database DEV1 mounted. Copying redo log /usr/dbvisit/standby/tmp/GS/DEV1/X.dbvisit.911.DEV1.redo_1.log to +DATA/DEV/onlinelog/redo_1_1.log Copying redo log /usr/dbvisit/standby/tmp/GS/DEV1/X.dbvisit.911.DEV1.redo_2.log to +DATA/DEV/onlinelog/redo_2_1.log Copying redo log /usr/dbvisit/standby/tmp/GS/DEV1/X.dbvisit.911.DEV1.redo_3.log to +DATA/DEV/onlinelog/redo_3_1.log Copying redo log /usr/dbvisit/standby/tmp/GS/DEV1/X.dbvisit.911.DEV1.redo_4.log to +DATA/DEV/onlinelog/redo_4_1.log Copying redo log /usr/dbvisit/standby/tmp/GS/DEV1/X.dbvisit.911.DEV1.redo_5.log to +DATA/DEV/onlinelog/redo_5_1.log Copying redo log /usr/dbvisit/standby/tmp/GS/DEV1/X.dbvisit.911.DEV1.redo_6.log to +DATA/DEV/onlinelog/redo_6_1.log Waiting for Checkpoint 9 on dbvrlin54... Checkpoint 9 completed. Key found on dbvrlin54 Recovering new primary database... Catalog archivelog '/u01/app/oracle/archive/DEV/1_12_858700653.arc'... Catalog archivelog '/u01/app/oracle/archive/DEV/1_13_858700653.arc'... Catalog archivelog '/u01/app/oracle/archive/DEV/2_2_858700653.arc'... Catalog archivelog '/u01/app/oracle/archive/DEV/2_3_858700653.arc'... Completed. Waiting for Checkpoint 10 on dbvrlin54... Checkpoint 10 completed. Key found on dbvrlin54 Archive log all for Database DEV1... Completed. Tempfiles dropped. Waiting for Checkpoint 11 on dbvrlin54... Checkpoint 11 completed. Key found on dbvrlin54 Open Database DEV1... Database DEV1 opened. Waiting for Checkpoint 12 on dbvrlin54... Checkpoint 12 completed. Key found on dbvrlin54 Waiting for Checkpoint 13 on dbvrlin54... Checkpoint 13 completed. Key found on dbvrlin54 Waiting for Checkpoint 14 on dbvrlin54... Checkpoint 14 completed. Key found on dbvrlin54 File /usr/dbvisit/standby/conf/dbv_DEV1.env copied to /usr/dbvisit/standby/conf/dbv_DEV1.env.201409191649. Dbvisit Database configuration (DDC) file /usr/dbvisit/standby/conf/dbv_DEV1.env has been updated and variables have been reversed between primary and standby server. SOURCE=dbvrlin51 DESTINATION=dbvrlin54. Waiting for Checkpoint 15 on dbvrlin54... Checkpoint 15 completed. Key found on dbvrlin54 Tempfiles created. Waiting for Checkpoint 16 on dbvrlin54... Checkpoint 16 completed. Key found on dbvrlin54 Compressing 1_12_858700653.arc... Compressing 1_13_858700653.arc... Compressing 2_2_858700653.arc... Compressing 2_3_858700653.arc... Waiting for Checkpoint 17 on dbvrlin54... Checkpoint 17 completed. Key found on dbvrlin54 FORCE LOGGING is left turned off in the primary database DEV1 on dbvrlin51. Waiting for Checkpoint 18 on dbvrlin54... Checkpoint 18 completed. Key found on dbvrlin54 Graceful switchover completed. This database (DEV1) is now the primary database. You may consider re-creating a password file to make sure all sysdba passwords are preserved. To keep the standby database on dbvrlin54 in synch with this primary database, reschedule Dbvisit as per normal: dbvisit DEV1 ============================================================= dbv_oraStartStop ended on dbvrlin51: Fri Sep 19 16:56:13 2014 =============================================================
- No enable the cluster database option on the primary node 1 database instance.
This is done by running setting cluster_database=true and updating the cluster_database_instances to the total RAC instances you will run.
Also rename/copy the spfile in ASM to the correct name for the RAC database
Then shutdown the database and start the RAC database using the srvctl command: srvctl start database -d DEV
Example:
oracle@dbvrlin51[/usr/dbvisit/standby]: sqlplus /nolog SQL*Plus: Release 11.2.0.4.0 Production on Fri Sep 19 16:59:16 2014 Copyright (c) 1982, 2013, Oracle. All rights reserved. 16:59:16 SQL> connect / as sysdba Connected. 16:59:19 SQL> show parameter cluster NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ cluster_database boolean FALSE cluster_database_instances integer 1 cluster_interconnects string 16:59:23 SQL> alter system set cluster_database=true scope=spfile; System altered. 16:59:34 SQL> alter system set cluster_database_instances=3 scope=spfile; System altered. 16:59:46 SQL> shutdown immediate; Database closed. Database dismounted. ORACLE instance shut down. 17:00:04 SQL> exit grid@dbvrlin51[/home/grid]: . oraenv ORACLE_SID = [grid] ? +ASM1 The Oracle base remains unchanged with value /u01/app/grid grid@dbvrlin51[/home/grid]: asmcmd ASMCMD> cd +DATA ASMCMD> ls ASM/ DB_UNKNOWN/ DEV/ ASMCMD> cd DEV ASMCMD> ls CONTROLFILE/ DATAFILE/ onlinelog/ spfiledev1.ora tempfile/ ASMCMD> cp spfiledev1.ora spfiledev.ora copying +DATA/DEV/spfiledev1.ora -> +DATA/DEV/spfiledev.ora ASMCMD> exit grid@dbvrlin51[/home/grid]: exit oracle@dbvrlin51[/u01/app/oracle/product/11.2.0/db_1/dbs]: srvctl status database -d DEV Instance DEV1 is running on node dbvrlin51 Instance DEV2 is running on node dbvrlin52 Instance DEV3 is running on node dbvrlin53 oracle@dbvrlin51[/u01/app/oracle/product/11.2.0/db_1/dbs]:
- You should not have a fully funcational primary RAC enable database.
- Run Dbvisit Standby on all nodes:
It is important that on first run of Dbvisit Standby on node 2 and 3 (thread 2 and 3) that you use the "dbvisit -R <DDC>" option.
Example:
node 1: ./dbvisit DEV1 node 2: ./dbvisit -R DEV2 node 3: ./dbvisit -R DEV3 standby node: ./dbvisit DEV1
- Additional Information
If you did have to re-install the RAC environment, and the cluster is not aware of the RAC database you might need to execute these commands: (again you will need to update to match your environment)
srvctl add database -d DEV -o /u01/app/oracle/product/11.2.0/db_1 -c RAC -a "DATA,FRA" -p +DATA/dev/spfileDEV.ora srvctl add instance -d DEV -i DEV1 -n dbvrlin51 srvctl add instance -d DEV -i DEV2 -n dbvrlin52 srvctl add instance -d DEV -i DEV3 -n dbvrlin53 srvctl config database -d DEV