/
Automatic Failover - Dbvisit Observer

Automatic Failover - Dbvisit Observer

Dec 28, 2020

Purpose

The purpose of this page is to provide detailed information regarding the Dbvisit Observer, which is a new tool/component that is a part of Dbvisit Standby. It works together with and in addition to, the other major Dbvisit Standby components, which are dbvctl (Dbvisit Standby CLI), dbvnet (Networking communication), dbvagent (Inter-communication Agent) and dbvserver (Web-based Central Console).

The main functions of Dbvisit Observer are:

  1. Provide remote monitoring of existing DDCs, and inform the DBA of problems in close to real-time, and

  2. Automatically perform a Failover of the DDC if previously-specified conditions are met.

Introduction

Most will be familiar with the Dbvisit Standby architecture as shown in the diagram below. It can be summarized as follows:

  1. Dbvisit Standby has 4 key components:

    • Dbvserver - also known as the Central Console or GUI.

    • Dbvagent - the agent that runs on each host allowing secure connections from the Central Console.

    • Standby Core - the heart of Dbvisit Standby from where core functions can be executed - this must be installed on the database server.

    • Dbvnet - the network component that allows secure communication for the Dbvisit Standby components between primary and standby.





As you can see above, in most cases you will have a default configuration, where there is a Primary Site (at the top of the diagram) with a Standby Site (bottom of diagram), where the standby database will be created and kept up to date (by shipping archive logs from primary and applying them to the standby database) via the use of Dbvisit Standby. The primary and standby site can make use of ASM, OMF or even Oracle RAC configurations.

It is important to understand the above architecture before we discuss the new Dbvisit Observer feature that allows Automatic Failover capability.



Dbvisit Observer Architecture Diagram

First, lets visualize this new component from a high level:

As you can see, the Observer is a completely stand-alone component that can be used to keep an eye on your configurations.

Some important points:

  • The Observer can be installed both completely on its own as well as together with any other Dbvisit component, e.g. you could have the Observer and the Central Console running on the same host.

  • Once installed, the Observer should be always running, even if there are no defined DDCs for it to monitor, or if all monitoring is switched off.

  • Observer & Agent: The Observer initiates all communication with dbvagent, i.e. dbvagent does not push information to the Observer.

  • Observer & Central Console: The Central Console initiates all communication with the Observer, i.e. the Observer does not push information to the Central Console.

Video Overview of Dbvisit Observer

This video covers all the basics of how the Observer works, how to install and use it. Please watch it before continuing to read this document:





System Requirements

The following Operating Systems are currently supported:

  • Oracle Linux 6 and above

  • Red Hat Linux 6 and above

  • Windows Server 2008 and above (64bit)

Please note: This does not mean you cannot monitor a Dbvisit Standby configuration that is running on Solaris or AIX. Only the dbvobserver component must run on the supported OSs above, but the Database servers themselves can be on any OS supported by Dbvisit Standby.

We do not recommend installing the Observer on your Database servers, but rather on a separate system - we recommend installation on the same system where you install the Central Console (Dbvserver).

For example, you can install the dbvserver and dbvobserver components on a Oracle Linux or Red Hat Linux 7 system (Virtual Machine or even Docker - just make sure you have persistent storage for the configuration folders), and have this system monitor and manage any other Dbvisit Standby version 10 supported configuration.





Getting Started

In this section, we will take you through a basic setup on a new system to explain how this works.

Example - Installing the environment

The environment we will use in this configuration consists out of 2 systems:

  • dbvel71 - Primary Database Server running an 18c database called orcl

  • dbvel72 - Standby Database Server that will be used as the Standby Database - Oracle 18c database software is installed and matches the Primary exactly.

  • dbvel72 - Central Console and Observer system.  This is where the Dbvserver (GUI) will be installed as well as the new Dbvisit Observer component.

Step 1:  Installing the Dbvisit Standby Software on Primary and Standby. 

This step should be familiar to most, if not please refer to the Install Dbvisit Standby section in the User guide

Step 2:  Installing Dbvisit Observer

The next step is to install the Dbvisit Observer onto the 2nd Node dbvel72. This system will also run the Central Console (GUI) and the Dbvisit Observer component.

[oracle@dbvel72 ~]$ pwd /home/oracle [oracle@dbvel72 ~]$ cd 10.0/ [oracle@dbvel72 10.0]$ ls dbvisit dbvisit-standby-10.0.0RC_24_g94ba1d85-el6.tar dbvisit-standby-10.0.0RC_24_g94ba1d85-el6.zip INSTALL.txt README.txt [oracle@dbvel72 10.0]$ cd dbvisit [oracle@dbvel72 dbvisit]$ cd installer/ [oracle@dbvel72 installer]$ ls doc install-dbvisit [oracle@dbvel72 installer]$ pwd /home/oracle/10.0/dbvisit/installer [oracle@dbvel72 installer]$ ./install-dbvisit ----------------------------------------------------------- Welcome to the Dbvisit software installer. ----------------------------------------------------------- It is recommended to make a backup of our current Dbvisit software location (Dbvisit Base location) for rollback purposes. Installer Directory /home/oracle/10.0/dbvisit >>> Please specify the Dbvisit installation directory (Dbvisit Base). The various Dbvisit products and components - such as Dbvisit Standby, Dbvisit Dbvnet will be installed in the appropriate subdirectories of this path. Enter a custom value or press ENTER to accept default [/usr/dbvisit]: > DBVISIT_BASE = /usr/dbvisit ----------------------------------------------------------- Component Installer Version Installed Version ----------------------------------------------------------- standby 10.0.0RC_24_g94ba1d85 10.0.0RC_24_g94ba1d85 dbvnet 10.0.0RC_24_g94ba1d85 (dbvnet v1.5.0-301-g3896e54)10.0.0RC_24_g94ba1d85 (dbvnet v1.5.0-301-g3896e54) dbvagent 10.0.0RC_24_g94ba1d85 10.0.0RC_24_g94ba1d85 dbvserver 10.0.0RC_24_g94ba1d85 10.0.0RC_24_g94ba1d85 observer 10.0.0RC_24_g94ba1d85 not installed ----------------------------------------------------------- What action would you like to perform? 1 - Install component(s) 2 - Uninstall component(s) 3 - Exit Your choice: 1 Choose component(s): 1 - Core Components (Dbvisit Standby Cli, Dbvnet, Dbvagent) 2 - Dbvisit Standby Core (Command Line Interface) 3 - Dbvnet (Dbvisit Network Communication) 4 - Dbvagent (Dbvisit Agent) 5 - Dbvserver (Dbvisit Central Console) - Not available on Solaris/AIX 6 - Dbvisit Observer (Automatic Failover Option) - Not available on Solaris/AIX Press ENTER to exit Installer Your choice: 6 ----------------------------------------------------------- Summary of the Dbvisit OBSERVER configuration ----------------------------------------------------------- DBVISIT_BASE /usr/dbvisit Press ENTER to continue ----------------------------------------------------------- About to install Dbvisit OBSERVER ----------------------------------------------------------- Component observer installed. ----------------------------------------------------------- Component Installer Version Installed Version ----------------------------------------------------------- standby 10.0.0RC_24_g94ba1d85 10.0.0RC_24_g94ba1d85 dbvnet 10.0.0RC_24_g94ba1d85 (dbvnet v1.5.0-301-g3896e54)10.0.0RC_24_g94ba1d85 (dbvnet v1.5.0-301-g3896e54) dbvagent 10.0.0RC_24_g94ba1d85 10.0.0RC_24_g94ba1d85 dbvserver 10.0.0RC_24_g94ba1d85 10.0.0RC_24_g94ba1d85 observer 10.0.0RC_24_g94ba1d85 10.0.0RC_24_g94ba1d85 ----------------------------------------------------------- What action would you like to perform? 1 - Install component(s) 2 - Uninstall component(s) 3 - Exit Your choice: 3 >>> Installation completed Install log /tmp/dbvisit_install.log.202012040925.

If you have followed the steps above you would now have the Dbvisit Standby GUI and Dbvisit Observer installed on the 2nd Node. In this example dbvel72

Step 3:  Start the Observer component

The observer component must be started as a background process in Linux. In the below example we will show the options of Observer components and how to start it.

[oracle@dbvel72 ~]$ [oracle@dbvel72 ~]$ cd /usr/dbvisit/observer/ [oracle@dbvel72 observer]$ ls -lrt total 11596 drwxr-xr-x. 2 oracle oinstall 6 Dec 4 09:26 log drwxr-xr-x. 2 oracle oinstall 6 Dec 4 09:26 doc drwxr-xr-x. 2 oracle oinstall 84 Dec 4 09:26 conf -rwxr-xr-x. 1 oracle oinstall 11872364 Dec 4 09:26 observersvc [oracle@dbvel72 observer]$ tree . . ├── conf │   ├── observer.conf │   ├── sample_userscript.bat │   └── sample_userscript.sh ├── doc ├── log └── observersvc [oracle@dbvel72 observer]$ ./observersvc -h dbvobserver Usage: ./observersvc [flags] -V report version and exit -d action service action. -ddc id DDC id for resetting retries with -reset -f file load config from file instead of system-wide /usr/dbvisit/observer/conf/observer.conf -log logfile logfile override. Overrides the file and path of the log. Set to "-" to output to stderr. This parameter has highest priority over settings and defaults. -ls list configs in the settings file -p change the password for /api/reset call -reset reset retries count for DDC -v verbose messages -version report full version header and exit -vv very verbose messages



[oracle@dbvel72 observer]$ nohup ./observersvc -f ./conf/observer.conf & [1] 5310 [oracle@dbvel72 observer]$ nohup: ignoring input and appending output to ‘nohup.out’ [oracle@dbvel72 observer]$ ps -ef |grep observer |grep -v grep oracle 5310 4544 0 09:35 pts/0 00:00:00 ./observersvc -f ./conf/observer.conf [oracle@dbvel72 observer]$ ./observersvc -ls -----------> No DDC is configured to be monitored by Observer Directory of Observer Config File: /usr/dbvisit/observer/conf/observer.conf Failed Failed Failed Poll ID Name Enabled Primary Standby MAX Every



Step 4:  Adding the Dbvisit Observer to the GUI

The next step once you have a standby database configured is to add the Observer to the Dbvisit Standby console.

This is done by navigating to the "Manage Configurations" menu option. From the Configuration page, click on "New" next to the "Add a Dbvisit Observer" section at the top as shown below:



There are three values required to add the observer to the Central Console:

  • The hostname or IP where the observer is configured

  • The port number on which the observer process is listening (Default is port 10853)

  • The observer passphrase - (default is admin900)



  • Fill in the values as per below 

1- the hostname - in this case, dbvel72

2- the port - which is the default 10853

3- the passphrase - admin900 (the default all lower case)

4- click on Save

Once the above is complete you will have the observer now listed in the Configuration Section as can be seen below:



IMPORTANT: Do not modify the observer configuration file via the command line (text editor) - it must be managed via the GUI (Central Console).

If you are only using the CLI and NOT using the Central Console (Advanced users only) then you should not add the Observer to the Central Console.



You have the observer now configured, and added to the Dbvisit Standby Console. Remember: you can only have one Dbvisit Observer process per Central Console configuration.



Step 5:  Adding a specific DDC to the Observer for monitoring

The next step is to enable the Dbvisit Observer to monitor a particular DDC.

This is done for each specific DDC and to enable this you can click on the "Monitor" icon under the Dbvisit Observer section - as highlighted in the section below:

Once you click on Monitor the following dialogue will be displayed:

This dialog will show you the basic settings you can configure:

1- The Poll Interval 

This is in seconds and indicates how often the Dbvisit Observer should monitor the particular configuration.

NOTE: It is very important that this poll interval is always greater then your expected network delay/timeout. For example, if a connection to your Primary or Standby host fails, how long does it take to timeout and report an error? If this is 60 seconds (a common default network timeout setting), this poll interval should be set to at least 61 seconds.

Setting the poll interval to a value less than the expected network delay/timeout can result in various incorrect error states being reported by the Observer.

If you do not want to modify your network timeout settings, but still want to set the Observer poll interval to be less than the maximum expected values for these, you can do so by use of the advanced "remote_timeout" parameter - please refer to the Advanced Configuration Parameters section below.

2- Retries 

The retry option is the number of times the Dbvisit Observer will monitor the particular DDC if a Failure is detected, Example, in the above the Observer will monitor the specific DDC configuration every 2 minutes (120 seconds), and if any errors are detected it will retry 5 times. After 5 errors (failures of the rules, more on this below), the "Operation mode" action will be executed.

3- Operation Mode

The operation mode is a very important setting - this is where you can select between two options:

"Manual Mode" - This is the default and puts the Dbvisit observer in what is called a "Dry Run" or Observe Only mode, meaning if a failure of the rules is triggered, Alerts will begin appearing in the GUI and Notifications will be sent out (configured under Advanced Settings) but the standby database is will NOT be activated. So no actual Automatic Failover is performed. It is highly recommended that this option be used for thorough testing in your configuration before you enable the "Failover Mode" - which if enabled will perform activation of the standby if the rule checks fail.

"Failover Mode" - This is the "real mode" of the Observer, where Automatic Failover (Activation) of the standby database will be performed if the rule checks fail the maximum specified number of times (as set in the Retries setting). For example, if the primary database goes down, and stays down for more than 10 minutes in this scenario (5x retries of 120 seconds each), the Failover will be triggered and the standby database will be activated. Note that Notifications will also be sent if enabled under the Advanced Settings section.

4- Log File Notification

By default as soon as you enable a DDC to be monitored by Dbvisit Observer, a log file will be created for any notifications specific to that configuration This log file is created on the node where the observer is running, which in this example is dbvlin03 and the location is: /usr/dbvisit/observer/log as we installed the Dbvisit Observer into the default DBVISIT_BASE location which is /usr/dbvisit. This setting cannot be disabled or modified. Note that this log will only contain DDC-specific entries, the same entries that would be sent out for any of the Notifications specified for the DDC in Advanced Settings. The full Observer log, including non-DDC-specific events, can be viewed in the Observer installation directory under /log/observersvc.log.

If you now click on SAVE SETTINGS, the default rule ("observer", specified on the Advanced Settings page, more on this below) will be applied, and monitoring of the DDC will begin.

The particular DDC - in this case, orcl - is now being monitored. The monitoring can be stopped at any time by selecting the "Stop Monitoring" option. You will also notice the Operating Mode which in this case is "Manual Mode" is displayed.  This means that this configuration is only being monitored (Auto Failover if off), and if the connectivity checks fail, only notifications will be sent and the monitoring will be stopped. No Automatic Failover will happen, as in Manual Mode, the Dbvisit Observer is only, funnily enough, observing.

[oracle@dbvel72 observer]$ ./observersvc -ls Directory of Observer Config File: /usr/dbvisit/observer/conf/observer.conf Failed Failed Failed Poll ID Name Enabled Primary Standby MAX Every 1 "orcl" true 0 0 5 2m0s



There are also the settings as shown above in point [3], which can be used to configure the monitoring for this DDC in more detail, such as enabling email notifications or changing the ruleset.

Step 6: Advanced Configuration

Once you click to edit settings for a particular DDC, you can navigate to the "Advanced Settings" section, from where you can configure the following:

  1. Email and Slack Notifications

  2. User Scripts (this can be a user shell script that is located in the DBVISIT_BASE/standby folder on both the primary and standby)

  3. Selecting the Rule Priority - default is "Observer", which means connectivity and database status checks only

We will now cover each of these in detail.

Windows Installation



Step1: 

The observer installation is similar to the installation of the other dbvisit components in windows. Double click on the Installation executable and when you click Next you will see the options to install the Dbvisit components. You can see the option of the Observer as well  [ 1].

Once the installation steps are completed, the observer service is started [2 ], you can check the service status from windows services [3 ].



Step2: Configuration

The configuration is exactly the same as the Linux configuration and its done from the central console. Go to Manage Configurations page and click on New

  1. Enter the host in which Observer was installed.

  2. The default port for Observer is 10853

  3. The password is admin900 

  4. Save the configuration

  1. Click on Monitor to configure the Observer and to start monitoring the Standby database.

The default configuration has poll interval as 120 and retries as 5 with Manual mode (this does not activate the standby)