2.  The Dbvnet Command Line Options

2.1.  Obtain quick help options

To show the Dbvnet full help from the command line you can specify the -h or --help command line option.  

  -h [ --help ]         This message

2.2.  Start or Stopping Dbvnet

Dbvnet runs as a background process.  To start and stop dbvnet use the "-d" flag with either start, stop or status keywords.  

-d start|stop|status|killpid
    Dbvnet daemon process control:

    Arguments:
       start  : this will start the dbvnet daemon process on Unix;
       stop   : the stop command will terminate the dbvnet daemon process;
       status : the status command will provide the state of the dbvnet daemon process;
       killpid: removes the pid-file of the dbvnet daemon process.

Example:

./dbvnet -d start
./dbvnet -d stop
./dbvnet -d status

2.3.  The Dbvnet Configuration File

The default configuration file for Dbvnet is located in the "DBVISIT_BASE/dbvnet/conf" directory. It is possible to specify a custom configuration file if required using the -c flag.  This may be useful in advanced DR architectures to connect to different Dbvnet servers using different configurations.

  -c [ --config ] arg   [-c config_file]
                        It is optional to specify configuration file if one is not present in the default location.
                        The default configuration file is located in DBVISIT_BASE/dbvnet/conf sub directory
                        The default configuration file is called dbvnet.conf
                        Example: -c /usr/dbvisit/dbvnet/conf/dbvnet.conf

Example:

./dbvnet -d start -c /usr/dbvisit/dbvnet/conf/dbvnet-custom.conf

2.4.  The Dbvnet Port

The default port used by Dbvnet is 7890.  It is possible to change this or to run different dbvnet configurations on the same server.  This will require you to either specify different configuration files that have different specified settings, or you can specify the required options on the command line using the command line options.  To specify the Dbvnet port from the command line, use the "-p" flag followed by the port number.  

In this case, if the port being specified is used with the start command, this will be the port to listen on.  If you are performing a remote execution or copy of a file, the port specified will be the one used to connect to the remote server.

  -p [ --port ] arg     [-p remote_port]
                        This is optional as the remote server port can be specified in the Dbvnet configuration file which is the
                        recommended approach. This parameter can be set to overwrite the configuration file

Example:

./dbvnet -d start -p 7788

2.5.  Executing a Remote Command

Dbvnet can perform two key functions:

• The execution of a remote command 
• Copy of a file from current to the remote server

If you configured the [local] and [remote] sections in the dbvnet.conf file, you would have listed the local server (also referred to as the client) and the remote system (also referred to as the remote dbvnet server). You can execute remote commands by using the "-e" flag.  

The argument following the "-e" flag must be enclosed in double quotes (to cater for spaces or arguments that might be part of the remote command to be executed).

  -e [ --command ] arg  [-e remote_command]
                        This is one of the key options.  To execute a remote command, you can specify the command with the -e flag.
                        It is important that you specify the full path of the command to execute.  Example if you want to execute
                        "uname -a" command on the remote server, you have to specify the full path to the uname executable.
                        Example:  -e "/bin/uname -a"

Example:

oracle@kiwi81[/usr/dbvisit/dbvnet]: hostname
kiwi81.oraclekiwi.co.nz
oracle@kiwi81[/usr/dbvisit/dbvnet]: ./dbvnet -e "uname -n"
kiwi91.oraclekiwi.co.nz
oracle@kiwi81[/usr/dbvisit/dbvnet]:

2.6.  Remote Copy of Files

As mentioned above, Dbvnet has two core functions, to execute remote commands and to allow files to be copied between the primary and standby servers.

This file copy is done by making use of two flags "-f" and "-o".  The first flag "-f" specifies the local file to be copied to the remote server.  The "-o" flag is used to specify the file location and name on the remote server.

  -f [ --file ] arg     [-f file]
                        This is optional as the file to be transferred
  -o [ --output ] arg   [-o file]
                        This is optional as the file name on server side to be transferred

Example:

oracle@dbvlin503[/usr/dbvisit/8.0/dbvnet]: ./dbvnet -f /tmp/dbv_prod_sync_0ur26di8_1_1.rman -o /tmp/testfile

oracle@dbvlin503[/usr/dbvisit/8.0/dbvnet]: ./dbvnet -e "uname -n; ls -al /tmp/test*"
dbvlin504.dbvisit.co.nz
-rw-r--r-- 1 oracle oinstall 10231808 Apr  6 15:30 /tmp/testfile

3.  The Dbvnet Configuration

In this section, we will cover the configuration file and the parameters you should be familiar with.

IMPORTANT: The only parameters that should require change is:

passphrase=
listener_address=
listener_port=
remote_address=
remote_port=

• The passphrase is used to establish a secure encrypted connection between the source and target

• The listener address is the local address on which the dbvnet server will listen 

• The listener port is by default 7890 and is the TCP port dbvnet will be listening on for incoming connections 

• The remote address is optional and can be the same as the local - but using the remote (target) systems address (hostname) here is recommended

• The remote port is the remote systems dbvnet port

Other parameters should not be modified unless instructed by Dbvisit Support.

Example Configuration File:

[general]
listener_address=dbv1
listener_port=7890
passphrase=kiwi123
remote_address=dbv1
remote_port=7890
cert=conf/cert.pem
prikey=conf/prikey.pem
ca=conf/ca.pem
certkey=
debug=3

4.  Start and Stop Dbvnet (Server)

Dbvnet can be stopped and started using the dbvnet executable located in the dbvnet subdirectory (example - /usr/dbvisit/dbvnet) when using Linux (Unix) based systems and when using Windows-based systems using the Windows Services to stop and start the "Dbvnet" service.

4.1.  Linux (UNIX)

Below is an example of stopping and starting the Dbvnet background processes on Linux (Unix).

oracle@dbvlin503[/usr/dbvisit/dbvnet]: ./dbvnet -d start

oracle@dbvlin503[/usr/dbvisit/dbvnet]: ps -ef|grep dbvnet |grep -v grep
oracle    6631     1  0 14:42 ?        00:00:00 ./dbvnet -d start
 
oracle@dbvlin503[/usr/dbvisit/dbvnet]: ./dbvnet -d stop
Dbvnetd stopping pid: 6631

Once the Dbvnet background processes are started,  you will notice 1 process running (in earlier versions of Dbvnet there used to be more than one process running by default)

4.2. Windows

Starting and stopping Dbvnet on Windows-based systems is done via the Windows Services.  

Below is an example showing the Windows service details for the Dbvnet Service.  From here you can stop/start dbvnet.

It is recommended that this process be left with "Startup Type" as "Automatic"

5.  Testing Dbvnet Communication

Once you have Dbvisit Standby installed on all the servers, it is important to make sure that the communication between the Primary and Standby servers are working.  

5.1.  Test 1: Performing a Basic Status Check

The first check is to run a remote command.  This can be done by using the "-e" flag when running dbvnet.

It is important to note that before you can run this command, you should have the client and server configuration settings in the dbvnet.conf file.

oracle@dbvlin503[/usr/dbvisit/9.0/dbvnet]: hostname
dbvlin503.dbvisit.co.nz

oracle@dbvlin503[/usr/dbvisit/9.0/dbvnet]: ./dbvnet -e "uname -n"
dbvlin504.dbvisit.co.nz

From the above test, you can see that Dbvnet was able to successfully establish a connection between server dbvlin101 and dbvlin102.

5.2.  Test 2: Copy a file between primary and standby

Example:

oracle@dbvlin503[/usr/dbvisit/9.0/dbvnet]: ./dbvnet -f /tmp/dbv_prod_sync_0ur26di8_1_1.rman -o /tmp/testfile
 
If the -v flag is used:

oracle@dbvlin503[/usr/dbvisit/9.0/dbvnet]: ./dbvnet -f /tmp/dbv_prod_sync_0ur26di8_1_1.rman -o /tmp/testfile -v
/tmp/dbv_prod_sync_0ur26di8_1_1.rman -> /tmp/testfile File transmitted successfully
 
oracle@dbvlin503[/usr/dbvisit/9.0/dbvnet]: ./dbvnet -e "uname -n; ls -al /tmp/test*"
dbvlin504.dbvisit.co.nz
-rw-r--r-- 1 oracle oinstall 10231808 Apr  6 15:30 /tmp/testfile
 

5.3.  Test 3:  Test using "system readiness" function

When installing Dbvisit Standby, it is important that once the installation on the Primary and the Standby servers are complete and all required processes are started (Dbvnet, Dbvagent and Dbvserver) that we then test the network connectivity between the primary and standby.
This process can help confirm that network communication is working before you continue creating DDC files or if upgrading - performing the DDC file upgrade.
The command to test connectivity is: dbvctl -f system_readiness
A series of questions will be asked, for example, the remote server name. We recommend you run this command on the primary as well as the standby to ensure network connectivity is working prior to continuing with creating or upgrading DDC files.

Example:
Below is an example using the "system readiness" function:

[oracle@dbv1 /usr/dbvisit/standby]$ ./dbvctl -f system_readiness

Please supply the following information to complete the test.
Default values are in [].

Enter Dbvisit Standby location on local server: [/usr/dbvisit]:
Your input: /usr/dbvisit

Is this correct? <Yes/No> [Yes]:

Enter the name of the remote server: []: dbv2
Your input: dbv2

Is this correct? <Yes/No> [Yes]:

Enter Dbvisit Standby location on remote server: [/usr/dbvisit]:
Your input: /usr/dbvisit

Is this correct? <Yes/No> [Yes]:

Enter the name of a file to transfer relative to local install directory
/usr/dbvisit: [standby/doc/README.txt]:
Your input: standby/doc/README.txt

Is this correct? <Yes/No> [Yes]:

Choose copy method:
1)   /usr/dbvisit/dbvnet/dbvnet
2)   /usr/bin/scp
Please enter choice [1] : 1

Is this correct? <Yes/No> [Yes]:

Enter port for method /usr/dbvisit/dbvnet/dbvnet: [7890]:
Your input: 7890

Is this correct? <Yes/No> [Yes]:
-------------------------------------------------------------
Testing the network connection between local server and remote server dbv2.
-------------------------------------------------------------
Settings
========
Remote server                                          =dbv2
Dbvisit Standby location on local server               =/usr/dbvisit
Dbvisit Standby location on remote server              =/usr/dbvisit
Test file to copy                                      =/usr/dbvisit/standby/doc/README.txt
Transfer method                                        =/usr/dbvisit/dbvnet/dbvnet
port                                                   =7890
-------------------------------------------------------------
Checking network connection by copying file to remote server dbv2...
-------------------------------------------------------------
Trace file /usr/dbvisit/standby/trace/230_dbvctl_system_readiness_201611172115.trc

File copied successfully. Network connection between local and dbv2 correctly
configured.

[oracle@dbv1 /usr/dbvisit/standby]$

6.  The Dbvnet Passphrase

It is important that the same password is used on both the primary and standby dbvnet configurations.  

7.  Troubleshooting Dbvnet 

The default port used for Dbvnet is 7890, it is important to make sure this port is open on both primary and standby servers. Ensure there is no firewall in between these systems blocking this port.

To review if Dbvnet is running and listening on port 7890 the "netstat" utility can be used.  This option is available on both Linux (UNIX) based systems as well as Windows. 

7.1.  Reviewing the Dbvnet Port - Linux

The netstat and lsof commands can be used to identify if Dbvnet is running and/or what is using port 7890.

The examples below show how you can use these two commands to identify if port 7890 is in use and what is listening/using this port:

[root@dbvlin101 ~]# netstat -a|grep 7890
tcp 0 0 *:7890 *:* LISTEN

From the above, we can see that something is listening on port 7890.  This is a good indication of if Dbvnet is already running.  If you see the above but Dbvnet is not running, that indicates something else might be using port 7890.

To further investigate what is using port 7890, you can make use of the "lsof" command.  This command must be executed as the "root" user.  Using the lsof command with the parameter "-i" with the port number 7890 we can see what application is using this port.  From the command below we can see that Dbvnet is indeed the process listening on this port.

root@dbvlin503[/root]: lsof -i :7890
COMMAND  PID   USER   FD   TYPE  DEVICE SIZE NODE NAME
dbvnet  6631 oracle    8u  IPv4 1998193       TCP dbvlin503.dbvisit.co.nz:7890 (LISTEN)

Another test you can perform to make sure there is no firewall blocking the two systems is the "nc" (netcat) utility and the well known "telnet" utility.

Example 1 using netcat "nc":  nc -vtz <remote_ip_address> <dbvnet_port

Example testing if you can connect to remote ip (172.16.1.154) on the dbvnet port 7890:
 
root@dbvlin503[/root]: nc -vtz 172.16.1.154 7890
Connection to 172.16.1.154 7890 port [tcp/*] succeeded!

Example 2 using telnet:  telnet <remote_ip_address> <dbvnet_port>

Note - the key line is line 3 ->  "Connected to..."

root@dbvlin503[/root]: telnet 172.16.1.154 7890
Trying 172.16.1.154...
Connected to dbvlin504.dbvisit.co.nz (172.16.1.154).
Escape character is '^]'.

7.2.  Reviewing the Dbvnet Port - Windows

Using the netstat and tasklist commands you can find out if Dbvnet is running, or identify what is using port 7890.

In the example below, Dbvnet is running on both Primary and Standby using port 7890.  The commands netstat and task list are used to show that Dbvnet is listening on port 7890.

C:\Program Files (x86)\Dbvisit\Standby>netstat -ano|findstr 7890
  TCP    0.0.0.0:7890           0.0.0.0:0              LISTENING       2396
  TCP    172.16.1.92:7890       172.16.1.91:61319      ESTABLISHED     2396

From the above, we can see that something is listening on port 7890 and that a connection was established to this port.

Using the process ID shown in the last column (2396 in this example) we can use the tasklist command to identify the process using port 7890

C:\Program Files (x86)\Dbvisit\Standby>tasklist /svc /FI "PID eq 2396"
Image Name                     PID Services
========================= ======== ============================================
dbvnet.exe                    2396 Dbvnet

7.3.  Testing Network Bandwidth 

This section will cover the steps using the "iperf" utility to test network bandwidth and connectivity between the primary and standby servers.

The iperf utility can be downloaded from here - https://iperf.fr/.  For more detail on the usage of this utility, please see the website.

The use of this utility is highlighted in the documentation here as it can help you troubleshoot network connectivity and bandwidth problems.

Start iPerf on the one system as a server - use the -s switch - using a specific port with the -p switch.  Example to run a test on port 7890 (make sure dbvnet is stopped).

The command on Server A would be:  iperf3.exe -s -i 1 -p 7891

Example on Server A:

C:\iperf311>iperf3.exe -s -i 1 -p 7891
warning: this system does not seem to support IPv6 - trying IPv4
-----------------------------------------------------------
Server listening on 7891
-----------------------------------------------------------

It will wait here and appear to hang.  Now start on the second server (remote server) a client connection (using the -c switch) and transfer example 100MB of data (using the -n <size> switch).  Make sure you specify the same port as you did on server A which is set with the -p <port> switch.

The command to be executed on server B to transfer data to server A is:    iperf3 -n 100M -i 1 -c serverA -p 7891

As soon as you start the command on the "client" server (which is server B in this case), you will see output appearing on both Server A and Server B.  Once complete, you can review and analyze the output.

There are a number of options available in the iperf utility.  It might be good to also test with larger files, example 1024M and monitor.

Example Server B Output (Client):

C:\iperf311>iperf3  -n 100M  -i 1 -c serverA -p 7891
Connecting to host serverA, port 7891
[  4] local 172.16.1.80 port 3840 connected to 172.16.1.90 port 7891
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  9.78 MBytes  82.1 Mbits/sec
[  4]   1.00-2.00   sec  3.63 MBytes  30.4 Mbits/sec
[  4]   2.00-3.00   sec  15.6 MBytes   131 Mbits/sec
[  4]   3.00-4.00   sec  7.81 MBytes  65.5 Mbits/sec
[  4]   4.00-5.00   sec  3.38 MBytes  28.4 Mbits/sec
[  4]   5.00-6.00   sec  11.6 MBytes  97.0 Mbits/sec
[  4]   6.00-7.00   sec  5.41 MBytes  45.4 Mbits/sec
[  4]   7.00-8.00   sec  2.77 MBytes  23.2 Mbits/sec
[  4]   8.00-9.00   sec  12.2 MBytes   102 Mbits/sec
[  4]   9.00-10.00  sec  21.5 MBytes   180 Mbits/sec
[  4]  10.00-10.89  sec  6.46 MBytes  60.8 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.89  sec   100 MBytes  77.1 Mbits/sec                  sender
[  4]   0.00-10.89  sec   100 MBytes  77.0 Mbits/sec                  receiver

iperf Done.

Example Server A Output:

C:\iperf311>iperf3.exe -s -i 1 -p 7891
warning: this system does not seem to support IPv6 - trying IPv4
-----------------------------------------------------------
Server listening on 7891
-----------------------------------------------------------
Accepted connection from 172.16.1.80, port 3839
[  5] local 172.16.1.90 port 7891 connected to 172.16.1.80 port 3840
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  9.60 MBytes  80.5 Mbits/sec
[  5]   1.00-2.00   sec  3.81 MBytes  32.0 Mbits/sec
[  5]   2.00-3.00   sec  15.6 MBytes   131 Mbits/sec
[  5]   3.00-4.00   sec  7.75 MBytes  65.0 Mbits/sec
[  5]   4.00-5.00   sec  3.45 MBytes  28.9 Mbits/sec
[  5]   5.00-6.00   sec  5.11 MBytes  42.8 Mbits/sec
[  5]   6.00-7.00   sec  10.4 MBytes  87.2 Mbits/sec
[  5]   7.00-8.00   sec  3.51 MBytes  29.4 Mbits/sec
[  5]   8.00-9.00   sec  12.9 MBytes   108 Mbits/sec
[  5]   9.00-10.00  sec  21.3 MBytes   179 Mbits/sec
[  5]  10.00-10.99  sec  6.60 MBytes  56.2 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.99  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.99  sec   100 MBytes  76.3 Mbits/sec                  receiver
-----------------------------------------------------------
Server listening on 7891
-----------------------------------------------------------

The output provided by the iperf utility can help you establish the bandwidth of your network link during a specific port. 

This can be useful to test when you have an environment where packet shaping is happening on certain ports.

8.  Using XCOPY

This option is only available from 7.0.38.  However, using Dbvnet is recommended.

On Windows (Recommended 2008 R2 and above) you can enable the use of "xcopy" for the shipping of archive logs.

This option is only used for the transfer of archive logs from the primary to standby during normal operation.

All other operations will still use Dbvnet.  

To enable this option, set the following values in the primary DDC file where DEST_SHARE. This is the share location on the standby server where the archive logs are copied to (same location as the Dbvisit ARCHDEST).

For example: If the ARCHDEST is e:\dbvisit_archdest\testdb, and the share location is created as "dbvisit_testdb" and points at this directory, you will use \\server_name\dbvisit_testdb as the parameter.

If for instance, DEST_SHARE location is different from the ARCHDEST location, ARCHDEST needs to be adjusted in the DDC file to be pointing to the DEST_SHARE location. This way, applying of logs on the standby will not fail.

Note: It is important to make sure the user running Dbvisit Standby has access to this share on the standby server.

 
CP = C:\Windows\system32\xcopy.exe
DEST_SHARE = \\dbvwin108\dbvisit_testdb
 

If a Graceful Switchover is performed, this value must be updated in the new primary.

9.  Using SSH (UNIX Only)

When using Dbvisit Standby version 9 on Unix based systems, you can also configure the use of SSH instead of using Dbvnet.

On Windows-based systems, Dbvnet is the only option when using Dbvisit Standby version 9.

The key requirement for using SSH is to ensure SSH user equivalence between the primary and standby servers.

IMPORTANT:   If you have Oracle RAC configured:

If you already have Oracle RAC configured, you will already have SSH equivalence between the RAC nodes - this is one of the pre-requisites when using Oracle RAC.  In this case, you do not have to re-generate any SSH keys on the RAC nodes as they would already have them configured.  The requirement is to make sure you can ssh between the primary and standby servers without passwords.  So, in this case, you then have to extend the ssh equivalence not just between the RAC nodes, but also between the primary and standby (where both the primary and standby could potentially be RAC).

If you need to create new keys and configure SSH equivalence between the nodes, the following can be used as a guideline to do this:

If using Oracle Linux 5 and above or RHEL 5 and above you can easily make use of the following commands:

To Generate the required keys:

ssh-keygen -t dsa

To Update the remote server's authorized keys file:

ssh-copy-id -i $HOME/.ssh/id_dsa.pub oracle@remote_server_name

For example, if you have two servers Primary Node and Standby Node and are using the "oracle" Unix account you can follow these steps to enable SSH User Equivalence:

On primary node as the oracle Unix account:

• Run the ssh-keygen command and press enter, accepting null values when asked for a pass-phrase (do not supply a pass-phrase). The required DSA keys will be created in the $HOME/.ssh/ directory

  ssh-keygen -t dsa

  
  

• Run the ssh-copy-id command to update the local and remote authorized_keys files:

  ssh-copy-id -i $HOME/.ssh/id_dsa.pub oracle@PrimaryNode ssh-copy-id -i $HOME/.ssh/id_dsa.pub oracle@StandbyNode

  
  

On StandbyNode as the oracle Unix account:

• Run the ssh-keygen command and press enter. Accept null values when asked for a pass-phrase (do not supply a pass-phrase). The required DSA keys will be created in $HOME/.ssh/ directory

  ssh-keygen -t dsa

  
  

• Run the ssh-copy-id command to update the local and remote authorized_keys files:

  ssh-copy-id -i $HOME/.ssh/id_dsa.pub oracle@PrimaryNode ssh-copy-id -i $HOME/.ssh/id_dsa.pub oracle@StandbyNode

  
  

Following the above steps, you should now be able to SSH between the primary and standby servers without being asked for a password. For example, running the following you should not be asked for any passwords and should just echo back the date from the remote server:

On PrimaryNode:

 ssh PrimaryNode "date"

On Standby Node:

 ssh StandbyNode "date"

If you do not have the option to use the ssh-copy-id command you can update the authorized_keys file manually. Detailed steps to perform this are explained in the Dbvisit Standby v7 User Guide here:

https://dbvisit.atlassian.net/wiki/display/UGDS7/Appendix+E%3A+Configure+Secure+Shell+%28SSH%29+Equivalence

9.1.  Dbvisit Standby Required Changes to Enable the Use of SSH

Once you have SSH user equivalence configured, you can now update the following values in the Dbvisit Standby DDC file on the primary server:

NETPORT=22
CP = /usr/bin/scp
RSH =/usr/bin/ssh

9.2.  If using SSH Banner

If you are using an SSH banner, this can effect the use of SSH for Dbvisit Standby. 

You need to review the banner size (lines) and you can specify:  SSH_SKIP_OUTPUT_LINES in the Dbvisit Standby DDC file to tell Dbvisit Standby to ignore these lines.

Example if the SSH Banner looks like this one below, we can see 3 lines are used:

#########################
# This is server X
#########################

To tell Dbvisit Standby to ignore this banner, set the following value in the DDC file:   

 SSH_SKIP_OUTPUT_LINES=3

9.3.  Summary

• Set the DBVNET_PORT to a null (empty) value.
• Ensure that the SSH_PORT is specified (default is 22).
• Set the CP variable to the full path of the scp command. In most cases this will be /usr/bin/scp.
• Set the RSH variable to the full path of the ssh command. In most cases this will be /usr/bin/ssh.

You should now be able to run Dbvisit Standby as normal and it will make use of SSH instead of Dbvnet