1. Introduction

When disaster strikes and the primary database is no longer available the standby database must be activated to become the new primary database to continue operation. This is also called failover to the standby database.

The steps to activate the standby database are:

Stop the scheduling of Dbvisit Standby.
Change the network configuration (or DNS) so that users will connect to the standby database (or server) instead of the primary database (or server).
Activate and open the standby database for normal operation as per instructions below. Note that as soon as the standby database is activated and becomes the new primary database that is open read/write. The link to the original primary database is lost and it is no longer possible to apply new logs to the original standby database.

To activate and open the standby database for continued operation in the event of a disaster the following procedure discussed in this section must be used.

2. Activate (Failover) - Using the Command Line Interface

The command line options available to activate the standby database are:

dbvctl -d <ddc> -o activate [--force]

This command is used to Activate the standby database, during which process it becomes the new primary database. Important, this operation MUST be done with caution as the process is NOT reversible. This action is performed in the event that the primary database is no longer available. Dbvisit Standby will prompt before activating the standby database. If the --force an option is provided then Dbvisit Standby will no longer prompt before activating the standby database (this option can be used in batch mode).

Note: On the event that APPLY_DELAY_LAG_MINUTES (i.e. APPLY_DELAY_LAG_MINUTES = 180) is set in the DDC, and standby is activated in batch mode, it will most likely fail or cancelled. If APPLY_DELAY_LAG_MINUTES is set in the DDC, the correct way of activating the standby is via interactive mode and not using the option --force.

Before you activate the standby database, running a quick test by ensuring the standby database is in a consistent state (datafile headers and controlfile are in sync) and ready to be activated. This can be done by opening the standby database read-only.

This is done by running the following command: "./dbvctl -d DEV -o read"

Example:

[oracle@dbv2 standby]$ ./dbvctl -d DEV -o read
=============================================================
Dbvisit Standby Database Technology (9.0.0_1271_g4b00fb49) (pid 2555)
dbvctl started on dbv2: Tue May  7 11:36:30 2019
=============================================================


>>> Running pre-checks please wait... done
Open Standby instance DEV in READ ONLY mode...
Standby instance DEV on dbv2 opened in READ ONLY mode.
Log files cannot be applied to Database while in READ ONLY mode.
Database tempfile(s) may need to be added to this database.

=============================================================
dbvctl ended on dbv2: Tue May  7 11:36:31 2019
=============================================================

As we can see in the example above, the standby database can be opened read-only without any errors.

If you did get any errors, example stating file 1 needs more recovery, you will need to apply more archive logs before you will be able to successfully activate (open the standby database read/write).

Now that we know we can open the standby read-only, let's start it back into recovery mode (mount state), by running: "./dbvctl -d DEV -o restart"

Example:

[oracle@dbv2 standby]$ ./dbvctl -d DEV -o restart
=============================================================
Dbvisit Standby Database Technology (9.0.0_1271_g4b00fb49) (pid 2574)
dbvctl started on dbv2: Tue May  7 11:37:16 2019
=============================================================

Stopping instance DEV...
Standby Database DEV shutdown successfully on dbv2.
Starting instance DEV...
Standby Database DEV started on dbv2.

=============================================================
dbvctl ended on dbv2: Tue May  7 11:37:24 2019
=============================================================

Now that we know the standby database can be opened read-only and that it is in a consistent state that it can be opened, we can execute the "Activation"

The example below shows the process is executed to activate the standby database for DDC file DEV:

[oracle@dbv2 /usr/dbvisit/standby]$ ./dbvctl -d DEV -o activate
=============================================================
Dbvisit Standby Database Technology (8.0.04.18169) (pid 3420)
dbvctl started on dbv2: Sun Nov 13 10:52:38 2016
=============================================================

Activating means this database will become a Primary Database.
It will no longer be a Standby Database for DEV on dbv1.
Activation cannot be reversed.
=>Activate Standby Database on dbv2? <Yes/No> [No]: Yes
Are you sure? <Yes/No> [No]: Yes
>>> Activating now...

File /usr/dbvisit/standby/conf/dbv_DEV.env copied to /usr/dbvisit/standby/conf/dbv_DEV.env.201611131052.

>>> Activation complete. Please ensure a backup of this Database is made
>>> Old archives from before the activation should be removed to avoid mix-up between new
    and old archive logs


>>> If the Dbvisit Standby process is to be reversed, then database on dbv1 will need to be
    rebuilt as a Standby Database

=============================================================
dbvctl ended on dbv2: Sun Nov 13 10:53:35 2016
=============================================================

Next Steps:

As soon as possible, back up your new production database.

At this point, the former standby database is now your production database.

This task, while not required, is a recommended safety measure because you cannot recover changes made after activation without a backup.

3. Disaster Recovery Actions - Using the Central Console

In case of disaster strikes, you will want to activate (failover) to the standby database. During this process, a resetlogs is performed and the standby database is opened read/write.

The standby database is then converted to a primary database and cannot be used as a standby database anymore.

In version 9 of Dbvisit Standby, a number of new options have been introduced in this section. First select "Disaster Recovery Actions" from the main menu [1]

Once you select a DDC on which the operation is to be performed, there are 3 options are now provided as listed below:

Activate Now
1. Select this option if you want to activate the standby database now - this is usually in the case when the disaster happened and you need to activate (failover) to the standby database.
Perform DR Test
1. It is always recommended that sites perform DR testing, activating a standby database and making use if a disaster did happen everything work as expected. This option will allow you to perform a backup of the standby database prior to activating it.
2. Two options will be available:
  1. Create a backup to a local disk using RMAN Backupsets
  2. Create a backup to a local disk using RMAN Image Copies
    1. This option can be useful to quickly switch back to your image copy instead of running a restore process.
      
      Use Filesystem/ACFS locations to take backups and image copies of the database.The current release does not support taking backups and image copies in ASM diskgroups
Standby Read Only Test
1. This option allows you to quickly test if your standby database can be opened read-only. This is an excellent test to see if the standby database is in a consistent state. If a standby database is not consistent, you will not be able to open it read-only or activate it. In this case, you will have to send and apply more logs before you will be able to activate or open the standby read-only.

The screen capture below shows what the Perform DR Test screen looks like:

Select "Perform DR Test"
If you are running this the first time you will need to select "RUN DR TEST" - if you already performed this task, you will be given the option to "REINSTATE" if you have backups (backupset or image copies) available
Confirm if you want to create a backup of the standby database prior to the test - NOTE: if you select NO, you will not have backups unless you create them manually
Select the backup method type - Image Copy or Backupset
Specify the format - this is making use of the RMAN format options - for more detail please see the Oracle RMAN documentation
Specify the location for the backup - this location must have at least enough space to hold a full copy of the standby database.
You can specify the process to stop between the backup and activation stages
Submit to continue

If you have selected the option to create an Image copy backup, once done with the first step of the image copy creation you will get a completed task indicating the image backup was created and validated.

You will then be presented with the option to resume and continue with the activation. Once that is complete you will get the task that indicates the completion of the activation/failover process.

Once the above is complete, you will be able to perform you tests on the newly activate standby.

When you are ready to re-instantiate the standby again you can navigate to the Disaster Recovery Actions page and select the DDC.

You will then select the Perform DR Test and you will get the option REINSTATE, which will allow you to get to the next screen that allows you to continue restoring from the backup taken in the previous step.

...

You will get two options if using Image Copy
- Switch to the image copy
  - This will be fast as we will switch to the image copy instead of restoring. You will need to be aware that in this scenario the database will be switched to the copy.
- Restore from the image copy
  - This method will perform a full restore from the image copy - this process is not as fast but will restore the database into the original location.

Once this is selected, click on Submit to start the operation. As mentioned the switching to the image copy is faster.

Once the re-instate is complete, for example, if you have picked to switch to copy the following task details, will be displayed.

4. Disaster Recovery Actions - Using CLI

The DR test and the Read-Only test can also be performed using the command line. The below example will show how it is done. dbvctl -o -h will display the help on the commands which can be used .

[oracle@dbv2 standby]$ ./dbvctl -o -h

    ... 
	dbvctl -d <ddc> -o ro_test
    dbvctl -d <ddc> -o activate [--force] [--noprompt]
    dbvctl -d <ddc> -o dr_test [--backup --backup_type image|backupset --backup_location bck_dir] [--noprompt]
    dbvctl -d <ddc> -o reinstate [--switch] [--noprompt]
    ....
    ...

4.1 Perform a Read-Only test

This option will help in determining if the database is standby is in a consistent state. This would open the database in read-only mode and the restart the database to mount mode so that the archivelogs can be applied.

[oracle@dbv2 standby]$ ./dbvctl -d DEV -o ro_test
=============================================================
Dbvisit Standby Database Technology (9.0.00_0_g28d7024c) (pid 1245)
dbvctl started on dbv2: Fri May 31 11:53:44 2019
=============================================================


>>> Running pre-checks please wait... done
Open Standby database DEV in READ ONLY mode...
Standby database DEV on dbv2 opened in READ ONLY mode.
Log files cannot be applied to Database while in READ ONLY mode.
Database tempfile(s) may need to be added to this database.
Stopping database DEV...
Standby Database DEV shutdown successfully on dbv2.
Starting database DEV...
Standby Database DEV started on dbv2.

=============================================================
dbvctl ended on dbv2: Fri May 31 11:53:54 2019
=============================================================

4.2 Perform a DR Test and Re-instate

Use Filesystem/ACFS locations to take backups and image copies of the database. The current release does not support taking backups and image copies in ASM diskgroups

[oracle@dbv2 standby]$ ./dbvctl -d DEV -o dr_test
=============================================================
Dbvisit Standby Database Technology (9.0.00_0_g28d7024c) (pid 1383)
dbvctl started on dbv2: Fri May 31 12:07:47 2019
=============================================================


>>> Running pre-checks please wait... done

>>> No backup of the standby database DEV is available


-------------------------------------------------------------------------------

=>Do you want to backup the standby database prior to activation? [yes]: yes
Your input: 1

Is this correct? <Yes/No> [Yes]:

-------------------------------------------------------------------------------

=>Specify backup type: 1 for image copy, 2 for backupset [1]: 2
Your input: 2

Is this correct? <Yes/No> [Yes]:

-------------------------------------------------------------------------------

=>Specify backup location: []: /home/oracle/temp
Your input: /home/oracle/temp

Is this correct? <Yes/No> [Yes]:

-------------------------------------------------------------------------------

=>Specify backup format: [%U.bkp]:
Your input: %U.bkp

Is this correct? <Yes/No> [Yes]:

-------------------------------------------------------------------------------

=>Do you want to be prompted to confirm activation on successful completion of the backup? [yes]:
Your input: 1

Is this correct? <Yes/No> [Yes]:

>>> Standby database size is 1.39GB. Backing up database might take some time.

>>> Backing up standby database... done

>>> Validating backup of the standby database... done

-------------------------------------------------------------------------------
Standby Backup summary
-------------------------------------------------------------------------------
tag                           DBV_DEV_1
location                      /home/oracle/temp
type                          backupset
db_unique_name                DEV
server                        dbv2
resetlogs_id                  1004372670
resetlogs_time                2019-03-31:16:24:30
created_on                    2019-05-31 12:09:00
status                        VALID

Files:
DBV_DEV_1_03u2u0tk_1_1.bkp
DBV_DEV_1_04u2u0ud_1_1.bkp
DBV_DEV_1_CONTROL.CTL

-------------------------------------------------------------------------------

-------------------------------------------------------------------------------

=>Do you want to proceed with activating the standby database? [no]: yes
Your input: 1

Is this correct? <Yes/No> [Yes]:
>>> Activating now...


>>> Activation complete. Please ensure a backup of this Database is made
>>> Old archives from before the activation should be removed to avoid mix-up between new
    and old archive logs


>>> Process complete

=============================================================
dbvctl ended on dbv2: Fri May 31 12:09:29 2019
=============================================================

Status of the database

[oracle@dbv2 standby]$ ./dbvctl -d DEV -o status
=============================================================
Dbvisit Standby Database Technology (9.0.00_0_g28d7024c) (pid 1670)
dbvctl started on dbv2: Fri May 31 12:11:29 2019
=============================================================

Database Instance DEV on dbv2 status: Regular database open in read write mode

=============================================================
dbvctl ended on dbv2: Fri May 31 12:11:29 2019
=============================================================

Now the next step is to reinstate the database back to the standby database

[oracle@dbv2 standby]$ ./dbvctl -d DEV -o reinstate
=============================================================
Dbvisit Standby Database Technology (9.0.00_0_g28d7024c) (pid 1689)
dbvctl started on dbv2: Fri May 31 12:12:48 2019
=============================================================


>>> Running pre-checks please wait...
>>> Validating backup of the standby database... done
done

-------------------------------------------------------------------------------
Standby Backup summary
-------------------------------------------------------------------------------
tag                           DBV_DEV_1
location                      /home/oracle/temp
type                          backupset
db_unique_name                DEV
server                        dbv2
resetlogs_id                  1004372670
resetlogs_time                2019-03-31:16:24:30
created_on                    2019-05-31 12:09:00
status                        VALID

Files:
DBV_DEV_1_03u2u0tk_1_1.bkp
DBV_DEV_1_04u2u0ud_1_1.bkp
DBV_DEV_1_CONTROL.CTL

-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
>>> Reinstating standby will result in permanent removal of current database files.

=>Do you want to view a list of files to be removed? [yes]: yes
Your input: 1

Is this correct? <Yes/No> [Yes]:
REINSTATE will remove the following files:
datafile:
/u01/app/oracle/oradata/DEV/system01.dbf
/u01/app/oracle/oradata/DEV/sysaux01.dbf
/u01/app/oracle/oradata/DEV/undotbs01.dbf
/u01/app/oracle/oradata/DEV/users01.dbf
tempfile:
/u01/app/oracle/oradata/DEV/temp01.dbf
controlfile:
/u01/app/oracle/oradata/DEV/control01.ctl
/u01/app/oracle/fast_recovery_area/DEV/control02.ctl
redolog:
/u01/app/oracle/oradata/DEV/redo01.log
/u01/app/oracle/oradata/DEV/redo02.log
/u01/app/oracle/oradata/DEV/redo03.log
other:

-------------------------------------------------------------------------------

=>Proceed with reinstating? [no]: yes
Your input: 1

Is this correct? <Yes/No> [Yes]:

>>> Shutting down the database... done

>>> Removing database files... done

>>> Restoring standby control file... done

>>> Starting standby database... done

>>> Restoring standby database... done
Standby database reinstated.

=============================================================
dbvctl ended on dbv2: Fri May 31 12:13:59 2019
=============================================================