Graceful Switchover (GS) Examples

This section will show examples on running Graceful Switchover (GS) process.

Before running Graceful Switchover, it is important to make sure that you have executed Dbvisit Standby to send/apply logs followed by reviewing the log gap report to ensure the Primary and Standby databases are in sync.

During the graceful switchover (which is a controlled role reversal between the primary and standby, the redo logs from the primary database is copied to the standby database. It is important to keep in mind that if the redo logs are large, this process can take time.

This process can also be run from the Web Based (GUI) interface - Performing a Graceful Switchover via GUI

Example: Running Graceful Switchover in a Single Instance Linux environment:

In this example the primary server is called "dbvlin101" and the standby database server is called "dbvlin102" with the database being called "testdb".

The DDC name for this database is also called "testdb".

Step 1: Make sure logs are sending and applying

Send Logs from Primary:

oracle@dbvlin101[/usr/dbvisit/standby]: ./dbvisit testdb
=============================================================
Dbvisit Standby Database Technology (7.0.34.13526) (pid 698)
dbvisit started on dbvlin101: Wed Jan 28 08:00:57 2015 ()
=============================================================
>>> Obtaining information from standby database (RUN_INSPECT=Y)...
    Waiting 3 seconds for log switch completion...


>>> Sending heartbeat message... - done.


>>> Checking Dbvisit Standby for configurational differences between dbvlin101 and
    dbvlin102...
    No configurational differences found between dbvlin101 and dbvlin102.


>>> Log file(s) for testdb will be transferred from dbvlin101 to dbvlin102...


  > Transferring 'o1_mf_1_44_bdhr3hwg_.arc.gz' to server dbvlin102:7890
    Progress: 0%...20%...40%...60%...80%...100% [82372 KB/s] - done.
    1 archive log transfer to dbvlin102 for testdb completed.
    Last sequence was 44.


>>> Dbvisit Archive Management Module (AMM)
    Config: number of archives to keep      = 0
    Config: number of days to keep archives = 7
    Config: archive backup count            = 0
    Config: diskspace full threshold        = 80%
    Current disk percent full (/u01/app/oracle/fast_recovery_area) = 52%
    Number of archive logs deleted = 6
=============================================================
dbvisit ended on dbvlin101: Wed Jan 28 08:01:19 2015
=============================================================

Apply logs from the Standby (making sure all available archive logs are applied)

oracle@dbvlin102[/usr/dbvisit/standby]: ./dbvisit testdb
=============================================================
Dbvisit Standby Database Technology (7.0.34.13526) (pid 2618)
dbvisit started on dbvlin102: Wed Jan 28 08:02:14 2015 ()
=============================================================
>>> Sending heartbeat message... - done.


>>> Log file(s) for testdb from dbvlin101 will be applied to dbvlin102


201501280802 - Log seq 44 thread 1 applied to standby database testdb.
    No Mail sent as SEND_MAIL_FLAG_DR = N


>>> Dbvisit Archive Management Module (AMM)
    Config: number of archives to keep      = 0
    Config: number of days to keep archives = 7
    Config: diskspace full threshold        = 80%


Processing /u01/app/oracle/archive/testdb...
    Archive log dir: /u01/app/oracle/archive/testdb
    Total number of archive files   : 6
    Exceeded minimum files to leave(3). Files will not be deleted. To increase minimum set
    MIN_ARCH_TO_KEEP variable in Dbvisit Database configuration (DDC) file dbv_testdb.env
    on dbvlin101
    Number of archive logs deleted = 3
    Current Disk percent full       : 48%
=============================================================
dbvisit ended on dbvlin102: Wed Jan 28 08:02:27 2015
=============================================================

Step 2: Review the Log Gap Report

For more details on the log gap report please see: Dbvisit Standby Log Gap Report

This is executed from the Primary Server: ./dbvisit -i DDC

oracle@dbvlin101[/usr/dbvisit/standby]: ./dbvisit -i testdb
=============================================================
Dbvisit Standby Database Technology (7.0.34.13526) (pid 1595)
dbvisit started on dbvlin101: Wed Jan 28 08:40:51 2015 ()
=============================================================
Dbvisit Standby log gap report for testdb at 201501280840:
-------------------------------------------------------------
Standby database on dbvlin102 is at sequence: 44.
Primary database on dbvlin101 is at log sequence: 45.
Primary database on dbvlin101 is at archived log sequence: 44.
Dbvisit Standby last transfer log sequence: 44.
Dbvisit Standby last transfer at: 201501280800.
Archive log gap for testdb:  0.
Transfer log gap for testdb: 0.
Standby database time lag (HH:MI:SS): 00:39:49.
=============================================================
dbvisit ended on dbvlin101: Wed Jan 28 08:40:54 2015
=============================================================

In summary, the two key lines here is to review the archive and transfer gaps. These values should be 0. (When using Oracle RAC, the archive gap must be 0 or 1)

...
Archive log gap for testdb:  0.
Transfer log gap for testdb: 0.
...

Step 3: Starting the Graceful Switchover Process

The Graceful Switchover process, is a two-step process. You have to execute the same command on both the primary and the standby database server:

./dbv_oraStartStop switchover DDC <unique_number>

The <unique_number> should be the same one on both primary and standby, (this number is not used for anything else and is just a random number)

In this case the commands executed are

On Primary: ./dbv_oraStartStop switchover testdb 1001

On Standby: ./dbv_oraStartStop switchover testdb 1001

This process can run for a few minutes, (depending on the size of redo logs and network capacity).

It is important to test the Graceful Switchover process in your test environment to make sure you are familiar with the process.

You should also document and take note of timings so that when it is executed in a production scenario, you are familiar with the process and the timing

Example output:

oracle@dbvlin101[/usr/dbvisit/standby]: ./dbv_oraStartStop switchover testdb 1001
=============================================================
Dbvisit Standby Database Technology (7.0.34.13503) (pid 1851)
dbv_oraStartStop started on dbvlin101: Wed Jan 28 08:55:13 2015 ()
=============================================================
=============================================================
Graceful Switchover starting on Primary Database testdb.
Timestamp: 201501280855.
>>> Database testdb will be shutdown and restarted <<<
Ensure Dbvisit is no longer scheduled.
>>> Graceful Switchover will attempt to reset Oracle parameters db_file_name_convert and log_file_name_convert to default values (null strings) in the spfile <<<
Obtaining archive log gap....
Contacting Standby Database testdb on dbvlin102...
Next standby sequence required for recovery (45) for thread 1.
Archive Log Gap for thread 1 is: 0. This is correct to continue.
Please enter unique key to begin graceful switchover for database testdb.
The same key must be entered on both primary and standby server.
Please start command: dbv_oraStartStop switchover testdb
on dbvlin102 if not already started.
Batch key will be used: 1001
Key 1001 entered.
Contacting dbvlin102 to ensure the same unique key is entered for testdb.


Waiting for Key 1 on dbvlin102...
Checkpoint 1 completed. Key found on dbvlin102
Waiting for Checkpoint 2 on dbvlin102...
Checkpoint 2 completed. Key found on dbvlin102
Waiting for Checkpoint 3 on dbvlin102...
Checkpoint 3 completed. Key found on dbvlin102
Shutting down regular Database testdb...
Regular Database testdb shutdown successfully.
Starting Regular Database testdb...
Regular Database testdb started restrict.
Performing Oracle Checkpoint.
 Waiting 3 seconds for log switch completion...
Creating standby control file as '/usr/dbvisit/standby/tmp/GS/testdb/X.dbvisit.1001.testdb.standbycontrolfile'...
Waiting for Checkpoint 4 on dbvlin102...
Checkpoint 4 completed. Key found on dbvlin102
Copying new archives for testdb to dbvlin102...
Compressing o1_mf_1_45_bdhvc07p_.arc...
 > Transferring 'o1_mf_1_45_bdhvc07p_.arc.gz' to server dbvlin102:7890
 Progress: 0%...20%...40%...60%...80%...100% [63321 KB/s] - done.
Compressing o1_mf_1_46_bdhvdd07_.arc...
 > Transferring 'o1_mf_1_46_bdhvdd07_.arc.gz' to server dbvlin102:7890
 Progress: 0%...20%...40%...60%...80%...100% [34852 KB/s] - done.
Shutting down regular Database testdb...
Regular Database testdb shutdown successfully.
Copying redo logs ... this may take a while...
Compressing X.dbvisit.1001.testdb.redo_1.log...
 > Transferring 'X.dbvisit.1001.testdb.redo_1.log.gz' to server dbvlin102:7890
 Progress: 0%...20%...40%...60%...80%...100% [89258 KB/s] - done.
Compressing X.dbvisit.1001.testdb.redo_2.log...
 > Transferring 'X.dbvisit.1001.testdb.redo_2.log.gz' to server dbvlin102:7890
 Progress: 0%...20%...40%...60%...80%...100% [121003 KB/s] - done.
Compressing X.dbvisit.1001.testdb.redo_3.log...
 > Transferring 'X.dbvisit.1001.testdb.redo_3.log.gz' to server dbvlin102:7890
 Progress: 0%...20%...40%...60%...80%...100% [120021 KB/s] - done.
Waiting for Checkpoint 5 on dbvlin102...
Checkpoint 5 completed. Key found on dbvlin102
Backing up current control files for testdb dbvlin101...
Database testdb on dbvlin101 is already down. No action taken.
Starting Regular Database testdb...
Regular Database testdb started nomount.
Control file backed up as /usr/dbvisit/standby/tmp/GS/testdb/X.dbvisit.1001.testdb.controlfile.
Shutting down standby Database testdb...
Standby Database testdb shutdown successfully.
Waiting for Checkpoint 6 on dbvlin102...
Checkpoint 6 completed. Key found on dbvlin102
Waiting for Checkpoint 7 on dbvlin102...
Checkpoint 7 completed. Key found on dbvlin102
Waiting for Checkpoint 8 on dbvlin102...
Checkpoint 8 completed. Key found on dbvlin102
Waiting for Checkpoint 9 on dbvlin102...
Checkpoint 9 completed. Key found on dbvlin102
Waiting for Checkpoint 10 on dbvlin102...
Checkpoint 10 completed. Key found on dbvlin102
Waiting for Checkpoint 11 on dbvlin102...
Checkpoint 11 completed. Key found on dbvlin102
Waiting for Checkpoint 12 on dbvlin102...
Checkpoint 12 completed. Key found on dbvlin102
Database testdb on dbvlin101 is already down. No action taken.
Starting Regular Database testdb...
Regular Database testdb started nomount.
STANDBY control file(s) restored from /usr/dbvisit/standby/tmp/GS/testdb/X.dbvisit.1001.testdb.standbycontrolfile.
Shutting down standby Database testdb...
Standby Database testdb shutdown successfully.
Starting Standby Database testdb...
Standby Database testdb started .
Waiting for Checkpoint 13 on dbvlin102...
Checkpoint 13 completed. Key found on dbvlin102
Waiting for Checkpoint 14 on dbvlin102...
Checkpoint 14 completed. Key found on dbvlin102
File /usr/dbvisit/standby/conf/dbv_testdb.env copied to /usr/dbvisit/standby/conf/dbv_testdb.env.201501280855.
Dbvisit Database configuration (DDC) file /usr/dbvisit/standby/conf/dbv_testdb.env has been updated and variables
have been reversed between primary and standby server.
SOURCE=dbvlin102 DESTINATION=dbvlin101.
Waiting for Checkpoint 15 on dbvlin102...
Checkpoint 15 completed. Key found on dbvlin102
Uncompressing o1_mf_1_45_bdhvc07p_.arc.gz...
Uncompressing o1_mf_1_46_bdhvdd07_.arc.gz...
Waiting for Checkpoint 16 on dbvlin102...
Checkpoint 16 completed. Key found on dbvlin102
Waiting for Checkpoint 17 on dbvlin102...
Checkpoint 17 completed. Key found on dbvlin102
Waiting for Checkpoint 18 on dbvlin102...
Checkpoint 18 completed. Key found on dbvlin102


Graceful switchover completed.


This database (testdb) is now a standby database.
To keep this new standby database in synch,
reschedule Dbvisit as per normal:
 dbvisit testdb
=============================================================
dbv_oraStartStop ended on dbvlin101: Wed Jan 28 09:00:10 2015
=============================================================

oracle@dbvlin102[/usr/dbvisit/standby]: ./dbv_oraStartStop switchover testdb 1001
=============================================================
Dbvisit Standby Database Technology (7.0.34.13503) (pid 3277)
dbv_oraStartStop started on dbvlin102: Wed Jan 28 08:55:32 2015 ()
=============================================================
=============================================================
Graceful Switchover starting on Standby Database testdb.
Timestamp: 201501280855.
Database testdb will be shutdown and restarted!
Ensure Dbvisit is no longer scheduled.
Please enter unique key to begin graceful switchover for database testdb.
The same key must be entered on both primary and standby server.
Please start command: dbv_oraStartStop switchover testdb
on dbvlin101 if not already started.
Batch key will be used: 1001
Key 1001 entered.
Contacting dbvlin101 to ensure the same unique key is entered for testdb.


Waiting for Key 1 on dbvlin101...
Checkpoint 1 completed. Key found on dbvlin101
Waiting for Checkpoint 2 on dbvlin101...
Checkpoint 2 completed. Key found on dbvlin101
Waiting for Checkpoint 3 on dbvlin101...
Checkpoint 3 completed. Key found on dbvlin101
Tempfiles dropped.
Waiting for Checkpoint 4 on dbvlin101...
Checkpoint 4 completed. Key found on dbvlin101
Shutting down standby Database testdb...
Standby Database testdb shutdown successfully.
Waiting for Checkpoint 5 on dbvlin101...
Checkpoint 5 completed. Key found on dbvlin101
Uncompressing 1_45_868280481.arc.gz...
Uncompressing 1_46_868280481.arc.gz...
Uncompressing X.dbvisit.1001.testdb.redo_1.log.gz...
Uncompressing X.dbvisit.1001.testdb.redo_2.log.gz...
Uncompressing X.dbvisit.1001.testdb.redo_3.log.gz...
Waiting for Checkpoint 6 on dbvlin101...
Checkpoint 6 completed. Key found on dbvlin101
Waiting for Checkpoint 7 on dbvlin101...
Checkpoint 7 completed. Key found on dbvlin101
Backing up current control files for testdb dbvlin102...
Database testdb on dbvlin102 is already down. No action taken.
Starting Standby Database testdb...
Standby Database testdb started nomount.
Control file backed up as /usr/dbvisit/standby/tmp/GS/testdb/X.dbvisit.1001.testdb.stdby_ctl_bck.
Shutting down standby Database testdb...
Standby Database testdb shutdown successfully.
Waiting for Checkpoint 8 on dbvlin101...
Checkpoint 8 completed. Key found on dbvlin101
Starting Standby Database testdb...
Standby Database testdb started nomount.
PRIMARY control file(s) restored from /usr/dbvisit/standby/tmp/GS/testdb/X.dbvisit.1001.testdb.controlfile.
Mount Database testdb...
Database testdb mounted.

Copying redo log /usr/dbvisit/standby/tmp/GS/testdb/X.dbvisit.1001.testdb.redo_1.log to /u01/app/oracle/fast_recovery_area/TESTDB/onlinelog/redo_1_1.log
Copying redo log /usr/dbvisit/standby/tmp/GS/testdb/X.dbvisit.1001.testdb.redo_2.log to /u01/app/oracle/fast_recovery_area/TESTDB/onlinelog/redo_2_1.log
Copying redo log /usr/dbvisit/standby/tmp/GS/testdb/X.dbvisit.1001.testdb.redo_3.log to /u01/app/oracle/fast_recovery_area/TESTDB/onlinelog/redo_3_1.log
Waiting for Checkpoint 9 on dbvlin101...
Checkpoint 9 completed. Key found on dbvlin101
Recovering new primary database...
Catalog archivelog '/u01/app/oracle/archive/testdb/1_45_868280481.arc'...
Catalog archivelog '/u01/app/oracle/archive/testdb/1_46_868280481.arc'...
Completed.

Waiting for Checkpoint 10 on dbvlin101...
Checkpoint 10 completed. Key found on dbvlin101
Archive log all for Database testdb...
Completed.
Tempfiles dropped.
Waiting for Checkpoint 11 on dbvlin101...
Checkpoint 11 completed. Key found on dbvlin101
Open Database testdb...
Database testdb opened.
Waiting for Checkpoint 12 on dbvlin101...
Checkpoint 12 completed. Key found on dbvlin101
Waiting for Checkpoint 13 on dbvlin101...
Checkpoint 13 completed. Key found on dbvlin101
Waiting for Checkpoint 14 on dbvlin101...
Checkpoint 14 completed. Key found on dbvlin101
File /usr/dbvisit/standby/conf/dbv_testdb.env copied to /usr/dbvisit/standby/conf/dbv_testdb.env.201501280855.
Dbvisit Database configuration (DDC) file /usr/dbvisit/standby/conf/dbv_testdb.env has been updated and variables
have been reversed between primary and standby server.
SOURCE=dbvlin102 DESTINATION=dbvlin101.
Waiting for Checkpoint 15 on dbvlin101...
Checkpoint 15 completed. Key found on dbvlin101
Tempfiles created.
Compressing  1_45_868280481.arc...
Compressing  1_46_868280481.arc...
FORCE LOGGING is turned on in the primary database testdb on dbvlin102.
Waiting for Checkpoint 16 on dbvlin101...
Checkpoint 16 completed. Key found on dbvlin101
Waiting for Checkpoint 17 on dbvlin101...
Checkpoint 17 completed. Key found on dbvlin101
Waiting for Checkpoint 18 on dbvlin101...
Checkpoint 18 completed. Key found on dbvlin101


Graceful switchover completed.


This database (testdb) is now the primary database.
You may consider re-creating a password file to make sure all sysdba passwords are preserved.
To keep the standby database on dbvlin101 in synch with this primary database,
reschedule Dbvisit as per normal:
	dbvisit testdb
=============================================================
dbv_oraStartStop ended on dbvlin102: Wed Jan 28 09:00:10 2015
=============================================================

Once the process is complete, there is one important step required. Run Dbvisit Standby on the new primary, followed by running it on the standby to make sure logs are shipped from the new primary and applied to the standby.

Step 4. Run Dbvisit Standby

First run dbvisit on the new primary:

oracle@dbvlin102[/usr/dbvisit/standby]: ./dbvisit testdb
=============================================================
Dbvisit Standby Database Technology (7.0.34.13526) (pid 4619)
dbvisit started on dbvlin102: Wed Jan 28 09:05:17 2015 ()
=============================================================
>>> Obtaining information from standby database (RUN_INSPECT=Y)...
    Waiting 3 seconds for log switch completion...


>>> Checking Dbvisit Standby for configurational differences between dbvlin102 and
    dbvlin101...
    No configurational differences found between dbvlin102 and dbvlin101.


>>> Log file(s) for testdb will be transferred from dbvlin102 to dbvlin101...
  > Transferring 'o1_mf_1_47_bdhvw3sr_.arc.gz' to server dbvlin101:7890
    Progress: 0%...20%...40%...60%...80%...100% [56733 KB/s] - done.
    1 archive log transfer to dbvlin101 for testdb completed.
    Last sequence was 47.


>>> Dbvisit Archive Management Module (AMM)
    Config: number of archives to keep      = 0
    Config: number of days to keep archives = 7
    Config: archive backup count            = 0
    Config: diskspace full threshold        = 80%
    Current disk percent full (/u01/app/oracle/fast_recovery_area) = 48%
    Number of archive logs deleted = 1
=============================================================
dbvisit ended on dbvlin102: Wed Jan 28 09:05:33 2015
=============================================================

Followed by running it on the new standby database:

oracle@dbvlin101[/usr/dbvisit/standby]: ./dbvisit testdb
=============================================================
Dbvisit Standby Database Technology (7.0.34.13526) (pid 3694)
dbvisit started on dbvlin101: Wed Jan 28 09:06:20 2015 ()
=============================================================


>>> Log file(s) for testdb from dbvlin102 will be applied to dbvlin101
201501280906 - Log seq 47 thread 1 applied to standby database testdb.
    No Mail sent as SEND_MAIL_FLAG_DR = N


>>> Dbvisit Archive Management Module (AMM)
    Config: number of archives to keep      = 0
    Config: number of days to keep archives = 7
    Config: diskspace full threshold        = 80%


Processing /u01/app/oracle/archive/testdb...
    Archive log dir: /u01/app/oracle/archive/testdb
    Total number of archive files   : 13
    Number of archive logs deleted = 0
    Current Disk percent full       : 52%
=============================================================
dbvisit ended on dbvlin101: Wed Jan 28 09:06:23 2015
=============================================================

Graceful Switchover is now complete.

When using Oracle RAC take the following into account:

The Graceful Switchover (GS) is only executed from one of the primary nodes (the one which you want to become the standby database). The other instances can stay running, they will be shutdown as part of the Graceful Switchover process.
You still need to execute the GS command on the standby node as well.
Make sure you are able to ship logs from both primary nodes to the standby node
Run the log gap report and make sure that the standby database is up to date. When using Oracle RAC, the Archive log gap on one of the nodes may be 1. Values more than 1 will not be allowed and the GS process will request that you first run Dbvisit Standby to send/apply logs to reduce the gap to 0 (or 1).

Example, if you have a two node Oracle RAC environment, and a single instance standby database, and you want to perform a switch to make primary node 1 the new standby node and the standby node becomes the primary node, you have to execute the graceful switchover command on node 1 and the standby node. The instance on node 2 will be shutdown as part of the graceful switchover process.

Example running Graceful Switchover on a 2 Node RAC with a Single Instance Standby

The RAC environment consist out of:

Node 1 - dbvrlin301
Node 2 - dbvrlin302

The RAC database is called TSTRAC with instance TSTRAC1 on node 1 (dbvrlin301) and TSTRAC2 instance on node 2 (dbvrlin302)

The standby database environment is a Single instance standby server called dbvrlin305 with standby database.

Primary RAC Node 1	Standby Server (Single Instance)
Step 1 - run log gap report - make sure gap is no more than 1.
oracle@dbvrlin301[/usr/dbvisit/standby]: ./dbvisit -i TSTRAC1 ============================================================= Dbvisit Standby Database Technology (7.0.36.13688) (pid 25828) dbvisit started on dbvrlin301: Tue Mar 3 14:39:36 2015 () ============================================================= Dbvisit Standby log gap report for TSTRAC1 at 201503031439 (thread 1): ------------------------------------------------------------- Standby database on dbvrlin305 is at sequence: 18. Primary database on dbvrlin301 is at log sequence: 19. Primary database on dbvrlin301 is at archived log sequence: 18. Dbvisit Standby last transfer log sequence: 18. Dbvisit Standby last transfer at: 201503031437. Archive log gap for TSTRAC1: 0. Transfer log gap for TSTRAC1: 0. Standby database time lag (HH:MI:SS): 00:01:20. No Mail sent as SEND_MAIL_FLAG = N Dbvisit Standby log gap report for TSTRAC2 at 201503031439 (thread 2): ------------------------------------------------------------- Standby database on dbvrlin305 is at sequence: 10. Primary database on dbvrlin302 is at log sequence: 12. Primary database on dbvrlin302 is at archived log sequence: 11. Dbvisit Standby last transfer log sequence: 11. Dbvisit Standby last transfer at: 201503031437. Archive log gap for TSTRAC2: 1. Transfer log gap for TSTRAC2: 0. Standby database time lag (HH:MI:SS): 00:01:25. No Mail sent as SEND_MAIL_FLAG = N ============================================================= dbvisit ended on dbvrlin301: Tue Mar 3 14:39:46 2015 =============================================================
Step 2 - start Graceful Switchover on Node 1	Step 2 - start Graceful Switchover on the Standby Node (dbvrlin305)
oracle@dbvrlin301[/usr/dbvisit/standby]: ./dbv_oraStartStop switchover TSTRAC1 2 ============================================================= Dbvisit Standby Database Technology (7.0.36.13688) (pid 25957) dbv_oraStartStop started on dbvrlin301: Tue Mar 3 14:40:02 2015 () ============================================================= ============================================================= Graceful Switchover starting on Primary Database TSTRAC1. Timestamp: 201503031440. >>> RAC Database TSTRAC will be shutdown and restarted as single instance <<< Ensure Dbvisit is no longer scheduled. >>> Graceful Switchover will attempt to reset Oracle parameters db_file_name_convert and log_file_name_convert to default values (null strings) in the spfile <<< Obtaining archive log gap.... Contacting Standby Database TSTRAC1 on dbvrlin305... Next standby sequence required for recovery (19) for thread 1. Archive Log Gap for thread 1 is: 0. This is correct to continue. Contacting Standby Database TSTRAC1 on dbvrlin305... Next standby sequence required for recovery (11) for thread 2. Archive Log Gap for thread 2 is: 1. This is correct to continue. Please enter unique key to begin graceful switchover for database TSTRAC1. The same key must be entered on both primary and standby server. Please start command: dbv_oraStartStop switchover TSTRAC1 on dbvrlin305 if not already started. Batch key will be used: 2 Key 2 entered. Contacting dbvrlin305 to ensure the same unique key is entered for TSTRAC1. Waiting for Key 1 on dbvrlin305... Checkpoint 1 completed. Key found on dbvrlin305 Waiting for Checkpoint 2 on dbvrlin305... Checkpoint 2 completed. Key found on dbvrlin305 Waiting for Checkpoint 3 on dbvrlin305... Checkpoint 3 completed. Key found on dbvrlin305 Shutting down regular Database TSTRAC1... Regular Database TSTRAC1 shutdown successfully. Starting Regular Database TSTRAC1... Regular Database TSTRAC1 started restrict. Performing Oracle Checkpoint. Waiting 3 seconds for log switch completion... Creating standby control file as '/usr/dbvisit/standby/tmp/GS/TSTRAC1/X.dbvisit.2.TSTRAC1.standbycontrolfile'... Waiting for Checkpoint 4 on dbvrlin305... Checkpoint 4 completed. Key found on dbvrlin305 Copying new archives for TSTRAC1 to dbvrlin305... Compressing 1_19_872079635.dbf... > Transferring '1_19_872079635.dbf.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [6739 KB/s] - done. Compressing 1_20_872079635.dbf... > Transferring '1_20_872079635.dbf.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [554 KB/s] - done. Compressing 1_21_872079635.dbf... > Transferring '1_21_872079635.dbf.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [11879 KB/s] - done. Compressing 1_22_872079635.dbf... > Transferring '1_22_872079635.dbf.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [538 KB/s] - done. Compressing 1_23_872079635.dbf... > Transferring '1_23_872079635.dbf.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [13146 KB/s] - done. Compressing 1_24_872079635.dbf... > Transferring '1_24_872079635.dbf.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [6206 KB/s] - done. Compressing 1_25_872079635.dbf... > Transferring '1_25_872079635.dbf.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [6763 KB/s] - done. Compressing 2_11_872079635.dbf... > Transferring '2_11_872079635.dbf.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [17506 KB/s] - done. Compressing 2_12_872079635.dbf... > Transferring '2_12_872079635.dbf.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [12332 KB/s] - done. Compressing 2_13_872079635.dbf... > Transferring '2_13_872079635.dbf.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [15808 KB/s] - done. Compressing 2_14_872079635.dbf... > Transferring '2_14_872079635.dbf.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [299 KB/s] - done. Compressing 2_15_872079635.dbf... > Transferring '2_15_872079635.dbf.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [481 KB/s] - done. Shutting down regular Database TSTRAC1... Regular Database TSTRAC1 shutdown successfully. Copying subsequent new archives for TSTRAC1 to dbvrlin305... Copying redo logs ... this may take a while... Compressing X.dbvisit.2.TSTRAC1.redo_1.log... > Transferring 'X.dbvisit.2.TSTRAC1.redo_1.log.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [67174 KB/s] - done. Compressing X.dbvisit.2.TSTRAC1.redo_2.log... > Transferring 'X.dbvisit.2.TSTRAC1.redo_2.log.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [86041 KB/s] - done. Compressing X.dbvisit.2.TSTRAC1.redo_3.log... > Transferring 'X.dbvisit.2.TSTRAC1.redo_3.log.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [72281 KB/s] - done. Compressing X.dbvisit.2.TSTRAC1.redo_4.log... > Transferring 'X.dbvisit.2.TSTRAC1.redo_4.log.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [87212 KB/s] - done. Waiting for Checkpoint 5 on dbvrlin305... Checkpoint 5 completed. Key found on dbvrlin305 Backing up current control files for TSTRAC1 dbvrlin301... Database TSTRAC1 on dbvrlin301 is already down. No action taken. Starting Regular Database TSTRAC1... Regular Database TSTRAC1 started nomount. Control file backed up as /usr/dbvisit/standby/tmp/GS/TSTRAC1/X.dbvisit.2.TSTRAC1.controlfile. Shutting down standby Database TSTRAC1... Standby Database TSTRAC1 shutdown successfully. Waiting for Checkpoint 6 on dbvrlin305... Checkpoint 6 completed. Key found on dbvrlin305 Waiting for Checkpoint 7 on dbvrlin305... Checkpoint 7 completed. Key found on dbvrlin305 Waiting for Checkpoint 8 on dbvrlin305... Checkpoint 8 completed. Key found on dbvrlin305 Waiting for Checkpoint 9 on dbvrlin305... Checkpoint 9 completed. Key found on dbvrlin305 Waiting for Checkpoint 10 on dbvrlin305... Checkpoint 10 completed. Key found on dbvrlin305 Waiting for Checkpoint 11 on dbvrlin305... Checkpoint 11 completed. Key found on dbvrlin305 Waiting for Checkpoint 12 on dbvrlin305... Checkpoint 12 completed. Key found on dbvrlin305 Database TSTRAC1 on dbvrlin301 is already down. No action taken. Starting Regular Database TSTRAC1... Regular Database TSTRAC1 started nomount. STANDBY control file(s) restored from /usr/dbvisit/standby/tmp/GS/TSTRAC1/X.dbvisit.2.TSTRAC1.standbycontrolfile. Shutting down standby Database TSTRAC1... Standby Database TSTRAC1 shutdown successfully. Starting Standby Database TSTRAC1... Standby Database TSTRAC1 started . Waiting for Checkpoint 13 on dbvrlin305... Checkpoint 13 completed. Key found on dbvrlin305 Waiting for Checkpoint 14 on dbvrlin305... Checkpoint 14 completed. Key found on dbvrlin305 File /usr/dbvisit/standby/conf/dbv_TSTRAC1.env copied to /usr/dbvisit/standby/conf/dbv_TSTRAC1.env.201503031440. Dbvisit Database configuration (DDC) file /usr/dbvisit/standby/conf/dbv_TSTRAC1.env has been updated and variables have been reversed between primary and standby server. SOURCE=dbvrlin305 DESTINATION=dbvrlin301. Waiting for Checkpoint 15 on dbvrlin305... Checkpoint 15 completed. Key found on dbvrlin305 Uncompressing 1_19_872079635.dbf.gz... Uncompressing 1_20_872079635.dbf.gz... Uncompressing 1_21_872079635.dbf.gz... Uncompressing 1_22_872079635.dbf.gz... Uncompressing 1_23_872079635.dbf.gz... Uncompressing 1_24_872079635.dbf.gz... Uncompressing 1_25_872079635.dbf.gz... Uncompressing 2_11_872079635.dbf.gz... Uncompressing 2_12_872079635.dbf.gz... Uncompressing 2_13_872079635.dbf.gz... Uncompressing 2_14_872079635.dbf.gz... Uncompressing 2_15_872079635.dbf.gz... Waiting for Checkpoint 16 on dbvrlin305... Checkpoint 16 completed. Key found on dbvrlin305 Standby file +DATA/tstrac/onlinelog/group_1.281.872079635 renamed to +DATA in database TSTRAC1. Standby file +FRA/tstrac/onlinelog/group_1.463.872079635 renamed to +FRA in database TSTRAC1. Standby file +DATA/tstrac/onlinelog/group_2.287.872079637 renamed to +DATA in database TSTRAC1. Standby file +FRA/tstrac/onlinelog/group_2.458.872079637 renamed to +FRA in database TSTRAC1. Standby file +DATA/tstrac/onlinelog/group_3.294.872079921 renamed to +DATA in database TSTRAC1. Standby file +FRA/tstrac/onlinelog/group_3.843.872079921 renamed to +FRA in database TSTRAC1. Standby file +DATA/tstrac/onlinelog/group_4.261.872079921 renamed to +DATA in database TSTRAC1. Standby file +FRA/tstrac/onlinelog/group_4.927.872079921 renamed to +FRA in database TSTRAC1. Waiting for Checkpoint 17 on dbvrlin305... Checkpoint 17 completed. Key found on dbvrlin305 Waiting for Checkpoint 18 on dbvrlin305... Checkpoint 18 completed. Key found on dbvrlin305 Graceful switchover completed. This database (TSTRAC1) is now a standby database. To keep this new standby database in synch, reschedule Dbvisit as per normal: dbvisit TSTRAC1 ============================================================= dbv_oraStartStop ended on dbvrlin301: Tue Mar 3 14:49:17 2015 =============================================================	oracle@dbvrlin305[/usr/dbvisit/standby]: ./dbv_oraStartStop switchover TSTRAC1 2 ============================================================= Dbvisit Standby Database Technology (7.0.36.13688) (pid 18000) dbv_oraStartStop started on dbvrlin305: Tue Mar 3 14:40:07 2015 () ============================================================= ============================================================= Graceful Switchover starting on Standby Database TSTRAC. Timestamp: 201503031440. Database TSTRAC will be shutdown and restarted! Ensure Dbvisit is no longer scheduled. Please enter unique key to begin graceful switchover for database TSTRAC. The same key must be entered on both primary and standby server. Please start command: dbv_oraStartStop switchover TSTRAC on dbvrlin301 if not already started. Batch key will be used: 2 Key 2 entered. Contacting dbvrlin301 to ensure the same unique key is entered for TSTRAC. Waiting for Key 1 on dbvrlin301... Checkpoint 1 completed. Key found on dbvrlin301 Waiting for Checkpoint 2 on dbvrlin301... Checkpoint 2 completed. Key found on dbvrlin301 Waiting for Checkpoint 3 on dbvrlin301... Checkpoint 3 completed. Key found on dbvrlin301 Tempfiles dropped. Waiting for Checkpoint 4 on dbvrlin301... Checkpoint 4 completed. Key found on dbvrlin301 Shutting down standby Database TSTRAC... Standby Database TSTRAC shutdown successfully. Waiting for Checkpoint 5 on dbvrlin301... Checkpoint 5 completed. Key found on dbvrlin301 Uncompressing 1_19_872079635.arc.gz... Uncompressing 1_20_872079635.arc.gz... Uncompressing 1_21_872079635.arc.gz... Uncompressing 1_22_872079635.arc.gz... Uncompressing 1_23_872079635.arc.gz... Uncompressing 1_24_872079635.arc.gz... Uncompressing 1_25_872079635.arc.gz... Rename 2_11_872079635.arc to 2_11_872079635.arc.201503031440... Uncompressing 2_11_872079635.arc.gz... Uncompressing 2_12_872079635.arc.gz... Uncompressing 2_13_872079635.arc.gz... Uncompressing 2_14_872079635.arc.gz... Uncompressing 2_15_872079635.arc.gz... Uncompressing X.dbvisit.2.TSTRAC1.redo_1.log.gz... Uncompressing X.dbvisit.2.TSTRAC1.redo_2.log.gz... Uncompressing X.dbvisit.2.TSTRAC1.redo_3.log.gz... Uncompressing X.dbvisit.2.TSTRAC1.redo_4.log.gz... Waiting for Checkpoint 6 on dbvrlin301... Checkpoint 6 completed. Key found on dbvrlin301 Waiting for Checkpoint 7 on dbvrlin301... Checkpoint 7 completed. Key found on dbvrlin301 Backing up current control files for TSTRAC dbvrlin305... Database TSTRAC on dbvrlin305 is already down. No action taken. Starting Standby Database TSTRAC... Standby Database TSTRAC started nomount. Control file backed up as /usr/dbvisit/standby/tmp/GS/TSTRAC1/X.dbvisit.2.TSTRAC1.stdby_ctl_bck. Shutting down standby Database TSTRAC... Standby Database TSTRAC shutdown successfully. Waiting for Checkpoint 8 on dbvrlin301... Checkpoint 8 completed. Key found on dbvrlin301 Starting Standby Database TSTRAC... Standby Database TSTRAC started nomount. PRIMARY control file(s) restored from /usr/dbvisit/standby/tmp/GS/TSTRAC1/X.dbvisit.2.TSTRAC1.controlfile. Mount Database TSTRAC... Database TSTRAC mounted. Waiting for Checkpoint 9 on dbvrlin301... Checkpoint 9 completed. Key found on dbvrlin301 Recovering new primary database... Catalog archivelog '/u01/app/oracle/dbvisit_archdest/TSTRAC/1_19_872079635.arc'... Catalog archivelog '/u01/app/oracle/dbvisit_archdest/TSTRAC/1_20_872079635.arc'... Catalog archivelog '/u01/app/oracle/dbvisit_archdest/TSTRAC/1_21_872079635.arc'... Catalog archivelog '/u01/app/oracle/dbvisit_archdest/TSTRAC/1_22_872079635.arc'... Catalog archivelog '/u01/app/oracle/dbvisit_archdest/TSTRAC/1_23_872079635.arc'... Catalog archivelog '/u01/app/oracle/dbvisit_archdest/TSTRAC/1_24_872079635.arc'... Catalog archivelog '/u01/app/oracle/dbvisit_archdest/TSTRAC/1_25_872079635.arc'... Catalog archivelog '/u01/app/oracle/dbvisit_archdest/TSTRAC/2_11_872079635.arc'... Catalog archivelog '/u01/app/oracle/dbvisit_archdest/TSTRAC/2_12_872079635.arc'... Catalog archivelog '/u01/app/oracle/dbvisit_archdest/TSTRAC/2_13_872079635.arc'... Catalog archivelog '/u01/app/oracle/dbvisit_archdest/TSTRAC/2_14_872079635.arc'... Catalog archivelog '/u01/app/oracle/dbvisit_archdest/TSTRAC/2_15_872079635.arc'... Completed. Waiting for Checkpoint 10 on dbvrlin301... Checkpoint 10 completed. Key found on dbvrlin301 Archive log all for Database TSTRAC... Completed. Tempfiles dropped. Waiting for Checkpoint 11 on dbvrlin301... Checkpoint 11 completed. Key found on dbvrlin301 Open Database TSTRAC... Database TSTRAC opened. Waiting for Checkpoint 12 on dbvrlin301... Checkpoint 12 completed. Key found on dbvrlin301 Waiting for Checkpoint 13 on dbvrlin301... Checkpoint 13 completed. Key found on dbvrlin301 Waiting for Checkpoint 14 on dbvrlin301... Checkpoint 14 completed. Key found on dbvrlin301 File /usr/dbvisit/standby/conf/dbv_TSTRAC1.env copied to /usr/dbvisit/standby/conf/dbv_TSTRAC1.env.201503031440. Dbvisit Database configuration (DDC) file /usr/dbvisit/standby/conf/dbv_TSTRAC1.env has been updated and variables have been reversed between primary and standby server. SOURCE=dbvrlin305 DESTINATION=dbvrlin301. Waiting for Checkpoint 15 on dbvrlin301... Checkpoint 15 completed. Key found on dbvrlin301 Tempfiles created. Compressing 1_19_872079635.arc... Compressing 1_20_872079635.arc... Compressing 1_21_872079635.arc... Compressing 1_22_872079635.arc... Compressing 1_23_872079635.arc... Compressing 1_24_872079635.arc... Compressing 1_25_872079635.arc... Compressing 2_11_872079635.arc... Compressing 2_12_872079635.arc... Compressing 2_13_872079635.arc... Compressing 2_14_872079635.arc... Compressing 2_15_872079635.arc... FORCE LOGGING is left turned off in the primary database TSTRAC on dbvrlin305. Waiting for Checkpoint 16 on dbvrlin301... Checkpoint 16 completed. Key found on dbvrlin301 Waiting for Checkpoint 17 on dbvrlin301... Checkpoint 17 completed. Key found on dbvrlin301 Waiting for Checkpoint 18 on dbvrlin301... Checkpoint 18 completed. Key found on dbvrlin301 Graceful switchover completed. This database (TSTRAC) is now the primary database. You may consider re-creating a password file to make sure all sysdba passwords are preserved. To keep the standby database on dbvrlin301 in synch with this primary database, reschedule Dbvisit as per normal: dbvisit TSTRAC1 ============================================================= dbv_oraStartStop ended on dbvrlin305: Tue Mar 3 14:49:19 2015 =============================================================
See column to the right -->>	Step 3 - Send logs from primary to standby (run dbvisit on the original standby server dbvrlin305 - which is now the primary following the switch)
See column to the right →>	oracle@dbvrlin305[/usr/dbvisit/standby]: ./dbvisit TSTRAC1 ============================================================= Dbvisit Standby Database Technology (7.0.36.13688) (pid 19080) dbvisit started on dbvrlin305: Tue Mar 3 14:53:49 2015 () ============================================================= >>> Obtaining information from standby database (RUN_INSPECT=Y)... >>> Checking Dbvisit Standby for configurational differences between dbvrlin305 and dbvrlin301... No configurational differences found between dbvrlin305 and dbvrlin301. >>> Log file(s) for TSTRAC will be transferred from dbvrlin305 to dbvrlin301... > Transferring '1_26_872079635.dbf.gz' to server dbvrlin301:7890 Progress: 0%...20%...40%...60%...80%...100% [17511 KB/s] - done. > Transferring '1_27_872079635.dbf.gz' to server dbvrlin301:7890 Progress: 0%...20%...40%...60%...80%...100% [3659 KB/s] - done. > Transferring '1_28_872079635.dbf.gz' to server dbvrlin301:7890 Progress: 0%...20%...40%...60%...80%...100% [463 KB/s] - done. 3 archive log transfers to dbvrlin301 for TSTRAC completed. Last sequence was 28 thread 1. No Mail sent as SEND_MAIL_FLAG = N >>> Log file(s) for TSTRAC will be transferred from dbvrlin305 to dbvrlin301... > Transferring '2_16_872079635.dbf.gz' to server dbvrlin301:7890 Progress: 0%...20%...40%...60%...80%...100% [539 KB/s] - done. > Transferring '2_17_872079635.dbf.gz' to server dbvrlin301:7890 Progress: 0%...20%...40%...60%...80%...100% [452 KB/s] - done. 2 archive log transfers to dbvrlin301 for TSTRAC completed. Last sequence was 17 thread 2. No Mail sent as SEND_MAIL_FLAG = N >>> Dbvisit Archive Management Module (AMM) Config: number of archives to keep = 0 Config: number of days to keep archives = 3 Config: archive backup count = 0 Config: diskspace full threshold = 80% Current disk percent full (/dbvacfs/dbvisit_archdest/TSTRAC) = 66% Current disk percent full (+FRA) = 1% Current disk percent full (FRA) = 5% Number of archive logs deleted = 0 ============================================================= dbvisit ended on dbvrlin305: Tue Mar 3 14:54:39 2015 =============================================================
Step 4 - Run Dbvisit on the new standby (original primary RAC node 1) to apply the logs shipped in step 3
oracle@dbvrlin301[/usr/dbvisit/standby]: ./dbvisit TSTRAC1 ============================================================= Dbvisit Standby Database Technology (7.0.36.13688) (pid 28268) dbvisit started on dbvrlin301: Tue Mar 3 14:54:44 2015 () ============================================================= >>> Log file(s) for TSTRAC1 from dbvrlin305 will be applied to dbvrlin301 201503031454 - Log seq 26 thread 1 applied to standby database TSTRAC1. 201503031454 - Log seq 27 thread 1 applied to standby database TSTRAC1. 201503031454 - Log seq 16 thread 2 applied to standby database TSTRAC1. 201503031454 - Log seq 17 thread 2 applied to standby database TSTRAC1. 201503031454 - Log seq 27 thread 1 applied to standby database TSTRAC1. 201503031454 - Log seq 28 thread 1 applied to standby database TSTRAC1. 201503031454 - Log seq 17 thread 2 applied to standby database TSTRAC1. No Mail sent as SEND_MAIL_FLAG_DR = N >>> Dbvisit Archive Management Module (AMM) Config: number of archives to keep = 0 Config: number of days to keep archives = 3 Config: diskspace full threshold = 80% Processing /u01/app/oracle/dbvisit_archdest/TSTRAC... Archive log dir: /u01/app/oracle/dbvisit_archdest/TSTRAC Total number of archive files : 6 Number of archive logs deleted = 0 Current Disk percent full : 50% ============================================================= dbvisit ended on dbvrlin301: Tue Mar 3 14:54:56 2015 =============================================================
See column to the right -->>	Step 5 - Before you switch back to the original configuration, run the log gap report - need to be 1 or less.
See column to the right -->>	oracle@dbvrlin305[/usr/dbvisit/standby]: ./dbvisit -i TSTRAC1 ============================================================= Dbvisit Standby Database Technology (7.0.36.13688) (pid 19587) dbvisit started on dbvrlin305: Tue Mar 3 15:03:36 2015 () ============================================================= Dbvisit Standby log gap report for TSTRAC at 201503031503 (thread 1): ------------------------------------------------------------- Standby database on dbvrlin301 is at sequence: 28. Primary database on dbvrlin305 is at log sequence: 29. Primary database on dbvrlin305 is at archived log sequence: 28. Dbvisit Standby last transfer log sequence: 28. Dbvisit Standby last transfer at: 201503031453. Archive log gap for TSTRAC: 0. Transfer log gap for TSTRAC: 0. Standby database time lag (HH:MI:SS): 00:14:46. No Mail sent as SEND_MAIL_FLAG = N Dbvisit Standby log gap report for TSTRAC at 201503031503 (thread 2): ------------------------------------------------------------- Standby database on dbvrlin301 is at sequence: 17. Primary database on dbvrlin305 is at log sequence: 18. Primary database on dbvrlin305 is at archived log sequence: 17. Dbvisit Standby last transfer log sequence: 17. Dbvisit Standby last transfer at: 201503031453. Archive log gap for TSTRAC: 0. Transfer log gap for TSTRAC: 0. Standby database time lag (HH:MI:SS): 00:14:52. No Mail sent as SEND_MAIL_FLAG = N ============================================================= dbvisit ended on dbvrlin305: Tue Mar 3 15:03:49 2015 =============================================================
Step 6 - Start the Graceful Switchover Back (From RAC Node 1 - where standby is running)	Step 6 - Start the Graceful Switchover Back (From Primary Node)
oracle@dbvrlin301[/usr/dbvisit/standby]: ./dbv_oraStartStop switchover TSTRAC1 3 ============================================================= Dbvisit Standby Database Technology (7.0.36.13688) (pid 29001) dbv_oraStartStop started on dbvrlin301: Tue Mar 3 15:04:00 2015 () ============================================================= ============================================================= Graceful Switchover starting on Standby Database TSTRAC1. Timestamp: 201503031503. Database TSTRAC1 will be shutdown and restarted! Ensure Dbvisit is no longer scheduled. Please enter unique key to begin graceful switchover for database TSTRAC1. The same key must be entered on both primary and standby server. Please start command: dbv_oraStartStop switchover TSTRAC1 on dbvrlin305 if not already started. Batch key will be used: 3 Key 3 entered. Contacting dbvrlin305 to ensure the same unique key is entered for TSTRAC1. Waiting for Key 1 on dbvrlin305... Checkpoint 1 completed. Key found on dbvrlin305 Waiting for Checkpoint 2 on dbvrlin305... Checkpoint 2 completed. Key found on dbvrlin305 Init parameter CLUSTER_DATABASE parameter is set to TRUE. To proceed with switchover CLUSTER_DATABASE will be set to FALSE. Database uses spfile +DATA/tstrac/spfiletstrac.ora. CLUSTER_DATABASE will be set to FALSE in spfile. Waiting for Checkpoint 3 on dbvrlin305... Checkpoint 3 completed. Key found on dbvrlin305 Tempfiles dropped. Waiting for Checkpoint 4 on dbvrlin305... Checkpoint 4 completed. Key found on dbvrlin305 Shutting down standby Database TSTRAC1... Standby Database TSTRAC1 shutdown successfully. Waiting for Checkpoint 5 on dbvrlin305... Checkpoint 5 completed. Key found on dbvrlin305 Uncompressing 1_29_872079635.arc.gz... Uncompressing 1_30_872079635.arc.gz... Uncompressing X.dbvisit.3.TSTRAC1.redo_1.log.gz... Uncompressing X.dbvisit.3.TSTRAC1.redo_2.log.gz... Uncompressing X.dbvisit.3.TSTRAC1.redo_3.log.gz... Uncompressing X.dbvisit.3.TSTRAC1.redo_4.log.gz... Waiting for Checkpoint 6 on dbvrlin305... Checkpoint 6 completed. Key found on dbvrlin305 Waiting for Checkpoint 7 on dbvrlin305... Checkpoint 7 completed. Key found on dbvrlin305 Backing up current control files for TSTRAC1 dbvrlin301... Database TSTRAC1 on dbvrlin301 is already down. No action taken. Starting Standby Database TSTRAC1... Standby Database TSTRAC1 started nomount. Control file backed up as /usr/dbvisit/standby/tmp/GS/TSTRAC1/X.dbvisit.3.TSTRAC1.stdby_ctl_bck. Shutting down standby Database TSTRAC1... Standby Database TSTRAC1 shutdown successfully. Waiting for Checkpoint 8 on dbvrlin305... Checkpoint 8 completed. Key found on dbvrlin305 Starting Standby Database TSTRAC1... Standby Database TSTRAC1 started nomount. PRIMARY control file(s) restored from /usr/dbvisit/standby/tmp/GS/TSTRAC1/X.dbvisit.3.TSTRAC1.controlfile. Mount Database TSTRAC1... Database TSTRAC1 mounted. Waiting for Checkpoint 9 on dbvrlin305... Checkpoint 9 completed. Key found on dbvrlin305 Recovering new primary database... Catalog archivelog '/u01/app/oracle/dbvisit_archdest/TSTRAC/1_29_872079635.arc'... Catalog archivelog '/u01/app/oracle/dbvisit_archdest/TSTRAC/1_30_872079635.arc'... Completed. Waiting for Checkpoint 10 on dbvrlin305... Checkpoint 10 completed. Key found on dbvrlin305 Archive log all for Database TSTRAC1... Completed. Tempfiles dropped. Waiting for Checkpoint 11 on dbvrlin305... Checkpoint 11 completed. Key found on dbvrlin305 Open Database TSTRAC1... Database TSTRAC1 opened. Waiting for Checkpoint 12 on dbvrlin305... Checkpoint 12 completed. Key found on dbvrlin305 Waiting for Checkpoint 13 on dbvrlin305... Checkpoint 13 completed. Key found on dbvrlin305 Waiting for Checkpoint 14 on dbvrlin305... Checkpoint 14 completed. Key found on dbvrlin305 File /usr/dbvisit/standby/conf/dbv_TSTRAC1.env copied to /usr/dbvisit/standby/conf/dbv_TSTRAC1.env.201503031503. Dbvisit Database configuration (DDC) file /usr/dbvisit/standby/conf/dbv_TSTRAC1.env has been updated and variables have been reversed between primary and standby server. SOURCE=dbvrlin301 DESTINATION=dbvrlin305. Waiting for Checkpoint 15 on dbvrlin305... Checkpoint 15 completed. Key found on dbvrlin305 Tempfiles created. Compressing 1_29_872079635.arc... Compressing 1_30_872079635.arc... FORCE LOGGING is left turned off in the primary database TSTRAC1 on dbvrlin301. Waiting for Checkpoint 16 on dbvrlin305... Checkpoint 16 completed. Key found on dbvrlin305 Waiting for Checkpoint 17 on dbvrlin305... Checkpoint 17 completed. Key found on dbvrlin305 Waiting for Checkpoint 18 on dbvrlin305... Checkpoint 18 completed. Key found on dbvrlin305 Graceful switchover completed. This database (TSTRAC1) is now the primary database. You may consider re-creating a password file to make sure all sysdba passwords are preserved. To keep the standby database on dbvrlin305 in synch with this primary database, reschedule Dbvisit as per normal: dbvisit TSTRAC1 Init parameter CLUSTER_DATABASE parameter is set to FALSE. To change single instance database to RAC database set: SQL>alter system set cluster_database = true scope = spfile; SQL>alter system set cluster_database_instances = x scope = spfile; and restart all instances. ============================================================= dbv_oraStartStop ended on dbvrlin301: Tue Mar 3 15:11:42 2015 ============================================================= IMPORTANT: Please note the last few lines in the Graceful Switchover on the RAC node 1: ... Init parameter CLUSTER_DATABASE parameter is set to FALSE. To change single instance database to RAC database set: SQL>alter system set cluster_database = true scope = spfile; SQL>alter system set cluster_database_instances = x scope = spfile; and restart all instances. ....	oracle@dbvrlin305[/usr/dbvisit/standby]: ./dbv_oraStartStop switchover TSTRAC1 3 ============================================================= Dbvisit Standby Database Technology (7.0.36.13688) (pid 19687) dbv_oraStartStop started on dbvrlin305: Tue Mar 3 15:03:54 2015 () ============================================================= ============================================================= Graceful Switchover starting on Primary Database TSTRAC. Timestamp: 201503031503. >>> RAC Database TSTRAC will be shutdown and restarted as single instance <<< Ensure Dbvisit is no longer scheduled. >>> Graceful Switchover will attempt to reset Oracle parameters db_file_name_convert and log_file_name_convert to default values (null strings) in the spfile <<< Obtaining archive log gap.... Contacting Standby Database TSTRAC on dbvrlin301... Next standby sequence required for recovery (29) for thread 1. Archive Log Gap for thread 1 is: 0. This is correct to continue. Contacting Standby Database TSTRAC on dbvrlin301... Next standby sequence required for recovery (18) for thread 2. Archive Log Gap for thread 2 is: 0. This is correct to continue. Please enter unique key to begin graceful switchover for database TSTRAC. The same key must be entered on both primary and standby server. Please start command: dbv_oraStartStop switchover TSTRAC on dbvrlin301 if not already started. Batch key will be used: 3 Key 3 entered. Contacting dbvrlin301 to ensure the same unique key is entered for TSTRAC. Waiting for Key 1 on dbvrlin301... Checkpoint 1 completed. Key found on dbvrlin301 Waiting for Checkpoint 2 on dbvrlin301... Checkpoint 2 completed. Key found on dbvrlin301 Waiting for Checkpoint 3 on dbvrlin301... Checkpoint 3 completed. Key found on dbvrlin301 Shutting down regular Database TSTRAC... Regular Database TSTRAC shutdown successfully. Starting Regular Database TSTRAC... Regular Database TSTRAC started restrict. Performing Oracle Checkpoint. Waiting 3 seconds for log switch completion... Creating standby control file as '/usr/dbvisit/standby/tmp/GS/TSTRAC1/X.dbvisit.3.TSTRAC1.standbycontrolfile'... Waiting for Checkpoint 4 on dbvrlin301... Checkpoint 4 completed. Key found on dbvrlin301 Copying new archives for TSTRAC to dbvrlin301... Compressing 1_29_872079635.dbf... > Transferring '1_29_872079635.dbf.gz' to server dbvrlin301:7890 Progress: 0%...20%...40%...60%...80%...100% [22059 KB/s] - done. Compressing 1_30_872079635.dbf... > Transferring '1_30_872079635.dbf.gz' to server dbvrlin301:7890 Progress: 0%...20%...40%...60%...80%...100% [16751 KB/s] - done. Shutting down regular Database TSTRAC... Regular Database TSTRAC shutdown successfully. Copying subsequent new archives for TSTRAC to dbvrlin301... Copying redo logs ... this may take a while... Compressing X.dbvisit.3.TSTRAC1.redo_1.log... > Transferring 'X.dbvisit.3.TSTRAC1.redo_1.log.gz' to server dbvrlin301:7890 Progress: 0%...20%...40%...60%...80%...100% [44522 KB/s] - done. Compressing X.dbvisit.3.TSTRAC1.redo_2.log... > Transferring 'X.dbvisit.3.TSTRAC1.redo_2.log.gz' to server dbvrlin301:7890 Progress: 0%...20%...40%...60%...80%...100% [34096 KB/s] - done. Compressing X.dbvisit.3.TSTRAC1.redo_3.log... > Transferring 'X.dbvisit.3.TSTRAC1.redo_3.log.gz' to server dbvrlin301:7890 Progress: 0%...20%...40%...60%...80%...100% [40545 KB/s] - done. Compressing X.dbvisit.3.TSTRAC1.redo_4.log... > Transferring 'X.dbvisit.3.TSTRAC1.redo_4.log.gz' to server dbvrlin301:7890 Progress: 0%...20%...40%...60%...80%...100% [52167 KB/s] - done. Waiting for Checkpoint 5 on dbvrlin301... Checkpoint 5 completed. Key found on dbvrlin301 Backing up current control files for TSTRAC dbvrlin305... Database TSTRAC on dbvrlin305 is already down. No action taken. Starting Regular Database TSTRAC... Regular Database TSTRAC started nomount. Control file backed up as /usr/dbvisit/standby/tmp/GS/TSTRAC1/X.dbvisit.3.TSTRAC1.controlfile. Shutting down standby Database TSTRAC... Standby Database TSTRAC shutdown successfully. Waiting for Checkpoint 6 on dbvrlin301... Checkpoint 6 completed. Key found on dbvrlin301 Waiting for Checkpoint 7 on dbvrlin301... Checkpoint 7 completed. Key found on dbvrlin301 Waiting for Checkpoint 8 on dbvrlin301... Checkpoint 8 completed. Key found on dbvrlin301 Waiting for Checkpoint 9 on dbvrlin301... Checkpoint 9 completed. Key found on dbvrlin301 Waiting for Checkpoint 10 on dbvrlin301... Checkpoint 10 completed. Key found on dbvrlin301 Waiting for Checkpoint 11 on dbvrlin301... Checkpoint 11 completed. Key found on dbvrlin301 Waiting for Checkpoint 12 on dbvrlin301... Checkpoint 12 completed. Key found on dbvrlin301 Database TSTRAC on dbvrlin305 is already down. No action taken. Starting Regular Database TSTRAC... Regular Database TSTRAC started nomount. STANDBY control file(s) restored from /usr/dbvisit/standby/tmp/GS/TSTRAC1/X.dbvisit.3.TSTRAC1.standbycontrolfile. Shutting down standby Database TSTRAC... Standby Database TSTRAC shutdown successfully. Starting Standby Database TSTRAC... Standby Database TSTRAC started . Waiting for Checkpoint 13 on dbvrlin301... Checkpoint 13 completed. Key found on dbvrlin301 Waiting for Checkpoint 14 on dbvrlin301... Checkpoint 14 completed. Key found on dbvrlin301 File /usr/dbvisit/standby/conf/dbv_TSTRAC1.env copied to /usr/dbvisit/standby/conf/dbv_TSTRAC1.env.201503031503. Dbvisit Database configuration (DDC) file /usr/dbvisit/standby/conf/dbv_TSTRAC1.env has been updated and variables have been reversed between primary and standby server. SOURCE=dbvrlin301 DESTINATION=dbvrlin305. Waiting for Checkpoint 15 on dbvrlin301... Checkpoint 15 completed. Key found on dbvrlin301 Uncompressing 1_29_872079635.dbf.gz... Uncompressing 1_30_872079635.dbf.gz... Waiting for Checkpoint 16 on dbvrlin301... Checkpoint 16 completed. Key found on dbvrlin301 Standby file +DATA/tstrac/onlinelog/group_1.269.873319723 renamed to +DATA in database TSTRAC. Standby file +FRA/tstrac/onlinelog/group_1.289.873319725 renamed to +FRA in database TSTRAC. Standby file +DATA/tstrac/onlinelog/group_2.260.873319729 renamed to +DATA in database TSTRAC. Standby file +FRA/tstrac/onlinelog/group_2.278.873319729 renamed to +FRA in database TSTRAC. Standby file +DATA/tstrac/onlinelog/group_3.257.873319731 renamed to +DATA in database TSTRAC. Standby file +FRA/tstrac/onlinelog/group_3.259.873319731 renamed to +FRA in database TSTRAC. Standby file +DATA/tstrac/onlinelog/group_4.261.873319731 renamed to +DATA in database TSTRAC. Standby file +FRA/tstrac/onlinelog/group_4.302.873319733 renamed to +FRA in database TSTRAC. Waiting for Checkpoint 17 on dbvrlin301... Checkpoint 17 completed. Key found on dbvrlin301 Waiting for Checkpoint 18 on dbvrlin301... Checkpoint 18 completed. Key found on dbvrlin301 Graceful switchover completed. This database (TSTRAC) is now a standby database. To keep this new standby database in synch, reschedule Dbvisit as per normal: dbvisit TSTRAC1 ============================================================= dbv_oraStartStop ended on dbvrlin305: Tue Mar 3 15:11:41 2015 =============================================================
Step 7 - Re-enable the RAC database (Done from Node 1). It is important that if you do this all RAC nodes are available.
oracle@dbvrlin301[/usr/dbvisit/standby]: . oraenv ORACLE_SID = [TSTRAC1] ? TSTRAC1 The Oracle base remains unchanged with value /u01/app/oracle oracle@dbvrlin301[/usr/dbvisit/standby]: sqlplus /nolog SQL*Plus: Release 11.2.0.4.0 Production on Tue Mar 3 15:39:27 2015 Copyright (c) 1982, 2013, Oracle. All rights reserved. 15:39:27 SQL> connect / as sysdba Connected. 15:39:30 SQL> alter system set cluster_database = true scope = spfile; System altered. 15:39:37 SQL> alter system set cluster_database_instances=2 scope=spfile; System altered. 15:39:48 SQL> shutdown immediate; Database closed. Database dismounted. ORACLE instance shut down. 15:40:12 SQL> exit Disconnected from Oracle Database 11g Release 11.2.0.4.0 - 64bit Production With the Real Application Clusters and Automatic Storage Management options oracle@dbvrlin301[/usr/dbvisit/standby]: srvctl start database -d TSTRAC oracle@dbvrlin301[/usr/dbvisit/standby]: srvctl status database -d TSTRAC Instance TSTRAC1 is running on node dbvrlin301 Instance TSTRAC2 is running on node dbvrlin302
Step 8 - Send logs from both Primary Nodes to Standby
From node 1: oracle@dbvrlin301[/usr/dbvisit/standby]: ./dbvisit TSTRAC1 ============================================================= Dbvisit Standby Database Technology (7.0.36.13688) (pid 32485) dbvisit started on dbvrlin301: Tue Mar 3 15:43:06 2015 () ============================================================= >>> Obtaining information from standby database (RUN_INSPECT=Y)... >>> Checking Dbvisit Standby for configurational differences between dbvrlin301 and dbvrlin305... No configurational differences found between dbvrlin301 and dbvrlin305. >>> Log file(s) for TSTRAC1 will be transferred from dbvrlin301 to dbvrlin305... > Transferring '1_31_872079635.dbf.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [18242 KB/s] - done. > Transferring '1_32_872079635.dbf.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [4531 KB/s] - done. > Transferring '1_33_872079635.dbf.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [5324 KB/s] - done. > Transferring '1_34_872079635.dbf.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [3890 KB/s] - done. > Transferring '1_35_872079635.dbf.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [3582 KB/s] - done. > Transferring '1_36_872079635.dbf.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [3465 KB/s] - done. > Transferring '1_37_872079635.dbf.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [2621 KB/s] - done. > Transferring '1_38_872079635.dbf.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [4537 KB/s] - done. 8 archive log transfers to dbvrlin305 for TSTRAC1 completed. Last sequence was 38 thread 1. No Mail sent as SEND_MAIL_FLAG = N >>> Dbvisit Archive Management Module (AMM) Config: number of archives to keep = 0 Config: number of days to keep archives = 3 Config: archive backup count = 0 Config: diskspace full threshold = 80% Current disk percent full (/dbvacfs/dbvisit_archdest/TSTRAC) = 2% Current disk percent full (+FRA) = 4% Current disk percent full (FRA) = 5% Number of archive logs deleted = 0 ============================================================= dbvisit ended on dbvrlin301: Tue Mar 3 15:43:59 2015 ============================================================= From node 2: oracle@dbvrlin302[/usr/dbvisit/standby]: ./dbvisit TSTRAC2 ============================================================= Dbvisit Standby Database Technology (7.0.36.13688) (pid 27104) dbvisit started on dbvrlin302: Tue Mar 3 15:43:12 2015 () ============================================================= >>> Obtaining information from standby database (RUN_INSPECT=Y)... >>> Checking Dbvisit Standby for configurational differences between dbvrlin302 and dbvrlin305... No configurational differences found between dbvrlin302 and dbvrlin305. >>> Log file(s) for TSTRAC2 will be transferred from dbvrlin302 to dbvrlin305... > Transferring '2_18_872079635.dbf.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [337 KB/s] - done. > Transferring '2_19_872079635.dbf.gz' to server dbvrlin305:7890 Progress: 0%...20%...40%...60%...80%...100% [443 KB/s] - done. 2 archive log transfers to dbvrlin305 for TSTRAC2 completed. Last sequence was 19 thread 2. No Mail sent as SEND_MAIL_FLAG = N >>> Dbvisit Archive Management Module (AMM) Config: number of archives to keep = 0 Config: number of days to keep archives = 3 Config: archive backup count = 0 Config: diskspace full threshold = 80% Current disk percent full (/dbvacfs/dbvisit_archdest/TSTRAC) = 2% Current disk percent full (+FRA) = 4% Current disk percent full (FRA) = 5% Number of archive logs deleted = 0 ============================================================= dbvisit ended on dbvrlin302: Tue Mar 3 15:44:02 2015 =============================================================
See column to the right -->>	Step 9 - Apply logs to standby
See column to the right -->>	oracle@dbvrlin305[/usr/dbvisit/standby]: ./dbvisit TSTRAC1 ============================================================= Dbvisit Standby Database Technology (7.0.36.13688) (pid 20993) dbvisit started on dbvrlin305: Tue Mar 3 15:44:19 2015 () ============================================================= >>> Log file(s) for TSTRAC from dbvrlin301 will be applied to dbvrlin305 201503031544 - Log seq 31 thread 1 applied to standby database TSTRAC. 201503031544 - Log seq 32 thread 1 applied to standby database TSTRAC. 201503031544 - Log seq 33 thread 1 applied to standby database TSTRAC. 201503031544 - Log seq 34 thread 1 applied to standby database TSTRAC. 201503031544 - Log seq 35 thread 1 applied to standby database TSTRAC. 201503031544 - Log seq 36 thread 1 applied to standby database TSTRAC. 201503031544 - Log seq 37 thread 1 applied to standby database TSTRAC. 201503031544 - Log seq 18 thread 2 applied to standby database TSTRAC. 201503031544 - Log seq 19 thread 2 applied to standby database TSTRAC. 201503031544 - Log seq 37 thread 1 applied to standby database TSTRAC. 201503031544 - Log seq 38 thread 1 applied to standby database TSTRAC. 201503031544 - Log seq 19 thread 2 applied to standby database TSTRAC. No Mail sent as SEND_MAIL_FLAG_DR = N >>> Dbvisit Archive Management Module (AMM) Config: number of archives to keep = 0 Config: number of days to keep archives = 3 Config: diskspace full threshold = 80% Processing /u01/app/oracle/dbvisit_archdest/TSTRAC... Archive log dir: /u01/app/oracle/dbvisit_archdest/TSTRAC Total number of archive files : 35 Number of archive logs deleted = 0 Current Disk percent full : 40% ============================================================= dbvisit ended on dbvrlin305: Tue Mar 3 15:44:36 2015 =============================================================