Dbvisit Standby Networking (Dbvnet and SSH)
Introduction
Dbvisit Standby’s new Network Layer is one of the exciting new features introduced in Dbvisit Standby version 7. Dbvnet’s functionality removes the dependency earlier Dbvisit Standby versions had on SSH for providing network communication between the primary and standby nodes. By making use of Dbvnet no additional SSH software is required to be installed, configured or maintained on Windows. When using UNIX based systems, Dbvisit Standby can now be configured to make use of the new Dbvnet (default option) or if required, SSH is still fully supported by Dbvisit Standby and can be configured and used on Unix based systems only. Password-less SSH authentication must be enabled between the primary and standby nodes if using Dbvisit Standby with SSH as communication layer.
Using Dbvnet
- Dbvnet is started as a background process and is run independent from the already known GUI (Dbvserver) background processes.
- Dbvnet provides the following key features:
- Independence of SSH-based client/server network communication
- Secure network communication (128bit or 256bit Encryption)
128bit Encryption is enabled by default
This setting can be controlled by the setting of DBVNET_ENCRYPT parameter in the Dbvisit Database Configuration (DDC) file for a particular database (located in DBVISIT_BASE/standby/conf) - Compression of network traffic between primary and standby nodes
Compression is disabled by default and is not recommended unless you have an extremely low bandwidth network.
By enabling network compression more CPU will be required to perform the compress/uncompress of the data and on systems already CPU bound this is not recommended. It is recommended to rather use the COMPRESS and UNCOMPRESS parameters in the Dbvisit Database Configuration (DDC) file for the particular database.
Dbvnet compression can be enabled by setting the DBVNET_COMPRESS parameter in the Dbvisit Database Configuration (DDC) file for a particular database (located in DBVISIT_BASE/standby/conf)
- Secure network communication (128bit or 256bit Encryption)
Dbvnet is located in the DBVISIT_BASE directory under the dbvnet sub-directory. Example Dbvisit Standby directory structure below:
dbvisit <-- DBVISIT_BASE |-- dbvnet <-- Dbvnet Home/Sub directory | |-- conf <-- Dbvnet Configuration Directory | | `-- init.d | |-- doc | |-- log <-- Dbvnet Log Files | `-- tmp |-- dbvserver <-- Dbvserver (GUI) Home/Sub directory | |-- conf | | `-- init.d | |-- doc | |-- log | `-- tmp `-- standby |-- conf <------ New location of Dbvisit Standby Configuration (DDC) file |-- doc |-- log |-- pid |-- tmp `-- trace <------ Default location for Dbvisit Standby Trace files
Stop and Start Dbvnet
Dbvnet can be stopped and started using the dbvnetd executable located in the dbvnet sub directory (example - /usr/dbvisit/dbvnet) when using Linux (Unix) based systems and when using Windows based systems using the Windows Services to stop and start the "Dbvnet" service.
Linux (UNIX)
Below is an example of stopping and starting the Dbvnet background processes on Linux (Unix).
Once the Dbvnet background processes are started, you will notice 5 processes running.
Windows
Starting and stopping Dbvnet on Windows based systems is done via the Windows Services.
Below is an example showing the Windows service details for the Dbvnet Service. From here you can stop/start dbvnet.
It is recommended that this process be left with "Startup Type" as "Automatic"
Testing Dbvnet Communication
Once you have Dbvisit Standby installed on all the servers, it is important to make sure that the communication between the Primary and Standby servers are working. This can be done by making use of the Dbvnet Test utility provided with the Dbvisit Standby installation - dbvnet-test, which is located under the dbvnet sub directory. When using the default path this utility will be in /usr/dbvisit/dbvnet/dbvnet-test
To test communication between the primary and standby servers you can run the following recommended test commands:
- To perform a basic status check which confirms that the primary and standby server can communicate using Dbvnet:
- dbvnet-test [-s|--status] SERVER[:PORT]
- dbvnet-test [-s|--status] SERVER[:PORT]
- To perform a more detail check, testing network connectivity, you can run the following:
- dbvnet-test [-f|--fulltest] TMPDIR SERVER:PORT
In the example below, dbvlin101 is the primary server and dbvlin102 is the standby server. Dbvisit Standby was installed using default values and no additional configuration has been performed at this stage. Dbvnet and Dbvserver is running on both nodes, but only Dbvnet is required and used for these tests:
Test 1: Basic Status Check
From the above test you can see that Dbvnet was able to successfully establish a connection between server dbvlin101 and dbvlin102.
Test 2: Full Test - 10MB file is copied (and verified) between primary and standby
Note: When using the dbvnet-test -f option, please specify the temp directory that exists on both systems. This is where the temp 10MB file will be copied to test connectivity. Example usage: ./dbvnet-test -f TEMPDIR SERVERNAME:7890
This temp directory is required from Dbvisit Standby version 7.0.10.
Example:
From the above test you can see that Dbvnet was able to successfully copy a temporary 10MB file between the two servers.
Note that the speed (transfer rate) is informational only and not an exact indication of the network speed.
Dbvnet Configuration Options
Dbvnet does not have a large number of configuration options and should work without any configuration changes in most environments.
Do not adjust any of the settings in the dbvnet.conf file located in DBVISIT_BASE/dbvnet/conf directory, unless instructed by Dbvisit Support.
The only parameter that can be adjusted manually in this file is the PORT you would like Dbvnet to use. By default the port number is 7890.
To adjust this port, stop dbvnet and adjust the parameter "bind_port = 7890" in the dbvnet.conf file followed by a restart of dbvnet. Note that if you do change this port number that you will need to adjust this on both the Primary and Standby servers and that Dbvnet must be restarted.
There are three Dbvisit Database Configuration (DDC - located in DBVISIT_BASE/standby/conf/) file parameters that can be adjusted (default values shown below):
- DBVNET_PORT=7890
- DBVNET_ENCRYPT=Y
- DBVNET_COMPRESS=N
Possible values for the DBVNET_ENCRYPT:
- DBVNET_ENCRYPT = Y - Meaning "Yes", enable the default 128bit encryption
- DBVNET_ENCRYPT = N - Meaning "No", disable encryption
- DBVNET_ENCRYPT = 256 - to enable 256bit encryption (Note this will slow down network transfer speeds as higher computing power (CPU) will be required to perform the encryption)
Possible values for the DBVNET_COMPRESS:
- DBVNET_COMPRESS = Y - Meaning "Yes", enable compression (Only recommended for environments with extremely low bandwidth between the primary and standby)
- DBVNET_COMPRESS = N - Meaning "No", disable compression
Reset Dbvnet Default Password
It is important that the same password be used on both the primary and standby dbvnet configurations.
If for some reason these passwords - stored in encrypted form in the DDC file, is not in sync, you can follow these steps to update the password:
- Stop Dbvnet on both Primary and Standby
- Generate a new encrypted password using the following command: "./dbvnetd -encryptpw YOUR_NEW_PASSWORD"
- Example:
oracle@dbvrlin303[/usr/dbvisit/dbvnet]: ./dbvnetd -encryptpw admin1 ENCRYPT53616c7465645f5f624381a94bb55307e9d7c6b253261044
- Not update the dbvnet configuration file dbvnet.conf located in DBVISIT_BASE/dbvnet/conf
- Update the "password" parameter to match the new password.
- Example, in this case change the entry from:
password = ENCRYPT53616c7465645f5fd0486bd36f60c368a307dd7576e414f1
to this (the one generated with the dbvnet -encrypt command)
password = ENCRYPT53616c7465645f5f624381a94bb55307e9d7c6b253261044
- Add the same password value in both the primary and standby server dbvnet.conf file and then restart Dbvnet.
- Once Dbvnet is restarted on both primary and standby, run the dbvnet-test tool to ensure the primary and standby servers can communicate.
Troubleshooting Dbvnet
The default port used for Dbvnet is 7890 and it is important to make sure this port is open on both primary and standby servers and that there is no firewall in between these systems blocking this port.
Connectivity can be tested using the "dbvnet-test" tool provided and located in the DBVISIT_BASE/dbvnet directory. To test communication between the primary and standby servers you can run the following recommended test commands:
- To perform a basic status check which confirms that the primary and standby server can communicate using Dbvnet:
- dbvnet-test [-s|--status] SERVER[:PORT]
- dbvnet-test [-s|--status] SERVER[:PORT]
- To perform a more detail check, testing network connectivity, you can run the following:
- dbvnet-test [-f|--fulltest] SERVER:PORT
Examples
oracle@dbvlin101[/usr/dbvisit/dbvnet]: ./dbvnet-test -s dbvlin102:7890
>>> Determining Dbvnet status on server dbvlin102:7890...
Dbvnet on server dbvlin102:7890 is running.
oracle@dbvlin101[/usr/dbvisit/dbvnet]: ./dbvnet-test -f /tmp dbvlin102:7890
>>> Determining Dbvnet status on server dbvlin102:7890...
Dbvnet on server dbvlin102:7890 is running.
>>> Running file transfer round trip on server dbvlin102:7890...
Creating file '/tmp/dbvnet-test.tmp' containing 10 MB of data... - done.
> Transferring 'dbvnet-test.tmp' to server dbvlin102:7890
Progress: 0%...20%...40%...60%...80%...100% [10109 KB/s] - done.
> Retrieving file "dbvnet-test.tmp" from server dbvlin102:7890
Progress: 0%...20%...40%...60%...80%...100% [9669 KB/s] - done.
Comparing checksums: all three checksums (local/remote/local) are
identical. File transfer round trip finished successfully.
Reviewing the Dbvnet Port
To review if Dbvnet is running and listening on port 7890 the "netstat" utility can be used. This option is available on both Linux (UNIX) based systems as well as Windows.
On Linux
The netstat and lsof commands can be used to identify if Dbvnet is running and/or what is using port 7890.
The examples below shows how you can use these two commands to identify if port 7890 is in use and what is listening/using this port:
[root@dbvlin101 ~]# netstat -a|grep 7890 tcp 0 0 *:7890 *:* LISTEN
From the above we can see that something is listening on port 7890. This is a good indication if Dbvnet is already running. If Dbvnet is not running and you see the above that indicates something else might be using the port 7890.
To further investigate what is using port 7890 you can make use of the "lsof" command. This command must be executed as the "root" user. Using the lsof command with the parameter "-i" with the port number 7890 we can see what application is using this port. From the command below we can see that Dbvnet is indeed the process listening on this port.
[root@dbvlin101 ~]# /usr/sbin/lsof -i :7890 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME ./dbvnetd 19199 oracle 8u IPv4 6680151 0t0 TCP *:7890 (LISTEN) ./dbvnetd 19201 oracle 8u IPv4 6680151 0t0 TCP *:7890 (LISTEN) ./dbvnetd 19202 oracle 8u IPv4 6680151 0t0 TCP *:7890 (LISTEN) ./dbvnetd 19203 oracle 8u IPv4 6680151 0t0 TCP *:7890 (LISTEN) ./dbvnetd 19204 oracle 8u IPv4 6680151 0t0 TCP *:7890 (LISTEN)
Another tests you can perform to make sure there are no firewall blocking between the two systems are using the "nc" (netcat) utility and the well known "telnet" utility.
Example 1 using netcat "nc": nc -vtz <remote_ip_address> <dbvnet_port
Example testing if you can connect to remote ip (172.16.1.162) on the dbvnet port 7890: nc -vtz 172.16.1.162 7890 Connection to 172.16.1.162 7890 port [tcp/*] succeeded!
Example 2 using telnet: telnet <remote_ip_address> <dbvnet_port>
Note - the key line is line 3 -> "Connected to..."
telnet 172.16.1.162 7890 Trying 172.16.1.162... Connected to 172.16.1.162 (172.16.1.162). Escape character is '^]'. Connection closed by foreign host.
On Windows
Using the netstat and tasklist commands you can find out if Dbvnet is running, or identify what is using port 7890.
In the example below Dbvnet is running on both Primary and Standby using port 7890. The commands netstat and tasklist is used to show that Dbvnet is listening on port 7890.
C:\Program Files (x86)\Dbvisit\Standby>netstat -ano|findstr 7890 TCP 0.0.0.0:7890 0.0.0.0:0 LISTENING 2396 TCP 172.16.1.92:7890 172.16.1.91:61319 ESTABLISHED 2396
From the above we can see that something is listening on port 7890 and that a connection was established to this port.
Using the process id shown in the last column (2396 in this example) we can use the tasklist command to identify the process using port 7890
C:\Program Files (x86)\Dbvisit\Standby>tasklist /svc /FI "PID eq 2396" Image Name PID Services ========================= ======== ============================================ dbvnetd.exe 2396 Dbvnet
IO::Socket Error
If you run the dbvnet-test tool and receive IO::Socket error with error code 20, it is an indication that Dbvnet is not running on the remote server or that the port 7890 is block between the servers.
Testing Network Bandwidth Between Primary and Standby Server
This section will cover the steps using the "iperf" utility to test network bandwidth and connectivity between the primary and standby servers.
The iperf utility can be downloaded from here - https://iperf.fr/ . For more detail on the usage of this utility, please see its web site.
The use of this utility is just highlighted in the documentation here as it can help you troubleshoot network connectivity and bandwidth problems.
Start iPerf on the one system as a server - use the -s switch - using a specific port with the -p switch. Example to run a test on port 7890 (make sure dbvnet is stopped).
The command on Server A, would be: iperf3.exe -s -i 1 -p 7891
Example on Server A:
C:\iperf311>iperf3.exe -s -i 1 -p 7891 warning: this system does not seem to support IPv6 - trying IPv4 ----------------------------------------------------------- Server listening on 7891 -----------------------------------------------------------
It will wait here and appear to hang. Now start on the second server (remote server) a client connection (using the -c switch) and transfer example 100MB of data (using the -n <size> switch). Make sure you specify the same port as you did on server A which is set with the -p <port> switch.
The command to be executed on server B to transfer data to server A is: iperf3 -n 100M -i 1 -c serverA -p 7891
As soon as you start the command on the "client" server (which is server B in this case), you will see output appearing on both Server A and Server B. Once complete, you can review and analyze the output.
There are a number of options available in the iperf utility. It might be good to also test with larger files, example 1024M and monitor.
Example Server B Output (Client):
C:\iperf311>iperf3 -n 100M -i 1 -c serverA -p 7891 Connecting to host serverA, port 7891 [ 4] local 172.16.1.80 port 3840 connected to 172.16.1.90 port 7891 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 9.78 MBytes 82.1 Mbits/sec [ 4] 1.00-2.00 sec 3.63 MBytes 30.4 Mbits/sec [ 4] 2.00-3.00 sec 15.6 MBytes 131 Mbits/sec [ 4] 3.00-4.00 sec 7.81 MBytes 65.5 Mbits/sec [ 4] 4.00-5.00 sec 3.38 MBytes 28.4 Mbits/sec [ 4] 5.00-6.00 sec 11.6 MBytes 97.0 Mbits/sec [ 4] 6.00-7.00 sec 5.41 MBytes 45.4 Mbits/sec [ 4] 7.00-8.00 sec 2.77 MBytes 23.2 Mbits/sec [ 4] 8.00-9.00 sec 12.2 MBytes 102 Mbits/sec [ 4] 9.00-10.00 sec 21.5 MBytes 180 Mbits/sec [ 4] 10.00-10.89 sec 6.46 MBytes 60.8 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.89 sec 100 MBytes 77.1 Mbits/sec sender [ 4] 0.00-10.89 sec 100 MBytes 77.0 Mbits/sec receiver iperf Done.
Example Server A Output:
C:\iperf311>iperf3.exe -s -i 1 -p 7891 warning: this system does not seem to support IPv6 - trying IPv4 ----------------------------------------------------------- Server listening on 7891 ----------------------------------------------------------- Accepted connection from 172.16.1.80, port 3839 [ 5] local 172.16.1.90 port 7891 connected to 172.16.1.80 port 3840 [ ID] Interval Transfer Bandwidth [ 5] 0.00-1.00 sec 9.60 MBytes 80.5 Mbits/sec [ 5] 1.00-2.00 sec 3.81 MBytes 32.0 Mbits/sec [ 5] 2.00-3.00 sec 15.6 MBytes 131 Mbits/sec [ 5] 3.00-4.00 sec 7.75 MBytes 65.0 Mbits/sec [ 5] 4.00-5.00 sec 3.45 MBytes 28.9 Mbits/sec [ 5] 5.00-6.00 sec 5.11 MBytes 42.8 Mbits/sec [ 5] 6.00-7.00 sec 10.4 MBytes 87.2 Mbits/sec [ 5] 7.00-8.00 sec 3.51 MBytes 29.4 Mbits/sec [ 5] 8.00-9.00 sec 12.9 MBytes 108 Mbits/sec [ 5] 9.00-10.00 sec 21.3 MBytes 179 Mbits/sec [ 5] 10.00-10.99 sec 6.60 MBytes 56.2 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 5] 0.00-10.99 sec 0.00 Bytes 0.00 bits/sec sender [ 5] 0.00-10.99 sec 100 MBytes 76.3 Mbits/sec receiver ----------------------------------------------------------- Server listening on 7891 -----------------------------------------------------------
The output provided by the iperf utility can help you establish the bandwidth of your network link during a specific port.
This can be useful to test when you have an environment where packet shaping is happening on certain ports.
Using XCOPY
This option is only available from 7.0.38. Using Dbvnet is however recommended.
On Windows (Recommended 2008 R2 and above) you can enable the use of "xcopy" for the shipping of archive logs.
This option is only used for the transfer of archive logs from the primary to standby during normal operation.
All other operations will still use Dbvnet.
To enable this option set the following values in the primary DDC file where DEST_SHARE is the share location on the standby server where the archive logs is copied to (same location as the Dbvisit ARCHDEST)
Example if the ARCHDEST is e:\dbvisit_archdest\testdb and the share is created as "dbvisit_testdb" pointing at this directory, you will use \\server_name\dbvisit_testdb as the parameter.
If for instance, DEST_SHARE location is different from the ARCHDEST location, ARCHDEST needs to be adjusted in the DDC file to be pointing to the DEST_SHARE location. This way, applying of logs on the standby will not fail.
Note: It is important to make sure the user running Dbvisit Standby does have access to this share on the standby server.
CP = C:\Windows\system32\xcopy.exe DEST_SHARE = \\dbvwin108\dbvisit_testdb
If a Graceful Switchover is performed, this value must be updated in the new primary.
Auto Startup on Linux
Dbvnet can be auto started on Linux during a system restart/start using the startup script.
An example startup script that can be placed in /etc/init.d/ can be found in DBVISIT_BASE/dbvnet/conf/init.d/
Once the script is placed in the /etc/init.d/ folder, it should be owned by the "root" user with execute permission. You can modify the "chkconfig: 345 95 5" line if required and then add the startup script using "chkconfig --add dbvnetd" command.
To review the startup options run "chkconfig --list dbvnetd" to view the status of the current start levels.
An example of this script is below:
#!/bin/bash
#
# /etc/rc.d/init.d/dbvnetd
#
# Starts the dbvnetd daemon
#
# chkconfig: 345 95 5
# description: Dbvisit network infrastructure daemon. \
# Copyright (c) 2012 by Dbvisit Software Ltd. \
# All rights reserved.
# processname: dbvnetd
# Source function library.
. /etc/init.d/functions
DBVISIT=/usr/dbvisit
DBVNETD=$DBVISIT/dbvnet/dbvnetd
PIDFILE=$DBVISIT/dbvnet/tmp/dbvnetd.pid
PATH=/usr/bin:/bin:/usr/sbin:/sbin
RETVAL=0
prog="dbvnetd"
start() {
# Check if already running
echo -n $"Starting $prog: "
su -l oracle -c "$DBVNETD"
RETVAL=$?
echo
return $RETVAL
}
stop() {
echo -n $"Stopping $prog: "
if [ -f "$PIDFILE" ]; then
killproc $DBVNETD
fi
RETVAL=$?
echo
return $RETVAL
}
restart() {
stop
start
}
case "$1" in
start)
start
;;
stop)
stop
;;
restart)
restart
;;
*)
echo $"Usage: $0 {start|stop|restart}"
exit 1
esac
exit $?
exit $RETVAL
Using SSH (UNIX Only)
When using Dbvisit Standby version 7 on Unix based systems, you can also configure the use of SSH instead of using Dbvnet.
On Windows based systems Dbvnet is the only option when using Dbvisit Standby version 7.
The key requirement for using SSH is to ensure SSH user equivalence between the primary and standby servers.
IMPORTANT: If you have Oracle RAC configured:
If you already have Oracle RAC configured, you will already have SSH equivalence between the RAC nodes - this is one of the pre-requisites when using Oracle RAC. In this case you do not have to re-generate any SSH keys on the RAC nodes as they would already have them configured. The requirement is to make sure you can ssh between the primary and standby servers without passwords. So in this case you then have to extend the ssh equivalence not just between the RAC nodes, but also between the primary and standby (where both the primary and standby could potentially be RAC).
If you need to create new keys and configure SSH equivalence between the nodes, the following can be used as guideline to do this:
If using Oracle Linux 5 and above or RHEL 5 and above you can easily make use of the following commands:
To Generate the required keys:
ssh-keygen -t dsa
To Update the remote server's authorized keys file:
ssh-copy-id -i $HOME/.ssh/id_dsa.pub oracle@remote_server_name
For example, if you have two servers PrimaryNode and StandbyNode and are using the "oracle" Unix account you can follow these steps to enable SSH User Equivalence:
On PrimaryNode as the oracle Unix account:
Run the
ssh-keygencommand and press enter, accepting null values when asked for a pass-phrase (do not supply a pass-phrase). The required DSA keys will be created in the$HOME/.ssh/directoryssh-keygen -t dsa
Run the
ssh-copy-idcommand to update the local and remote authorized_keys files:ssh-copy-id -i $HOME/.ssh/id_dsa.pub oracle@PrimaryNode ssh-copy-id -i $HOME/.ssh/id_dsa.pub oracle@StandbyNode
On StandbyNode as the oracle Unix account:
Run the
ssh-keygencommand and press enter. Accept null values when asked for a pass-phrase (do not supply a pass-phrase). The required DSA keys will be created in$HOME/.ssh/directoryssh-keygen -t dsa
Run the
ssh-copy-idcommand to update the local and remoteauthorized_keysfiles:ssh-copy-id -i $HOME/.ssh/id_dsa.pub oracle@PrimaryNode ssh-copy-id -i $HOME/.ssh/id_dsa.pub oracle@StandbyNode
Following the above steps, you should now be able to SSH between the primary and standby servers without being asked for a password. For example, running the following you should not be asked for any passwords and should just echo back the date from the remote server:
On PrimaryNode:
ssh PrimaryNode "date"
On StandbyNode:
ssh StandbyNode "date"
If you do not have the option to use the ssh-copy-id command you can update the authorized_keys file manually. Detailed steps to perform this are explained in the Dbvisit Standby v7 User Guide here:
Dbvisit Standby Required Changes to Enable the Use of SSH
Once you have SSH user equivalence configured, you can now update the following values in the Dbvisit Standby DDC file on the primary server:
DBVNET_PORT= SSH_PORT = 22 CP = /usr/bin/scp RSH = /usr/bin/ssh
If using SSH Banner
If you are using an SSH banner, this can effect the use of SSH for Dbvisit Standby.
You need to review the banner size (lines) and you can specify: SSH_SKIP_OUTPUT_LINES in the Dbvisit Standby DDC file to tell Dbvisit Standby to ignore these lines.
Example if the SSH Banner looks like this one below, we can see 3 lines are used:
######################### # This is server X #########################
To tell Dbvisit Standby to ignore this banner, set the following value in the DDC file:
SSH_SKIP_OUTPUT_LINES=3
Summary
- Set the
DBVNET_PORTto a null (empty) value. - Ensure that the
SSH_PORTis specified (default is 22). - Set the CP variable to the full path of the
scpcommand. In most cases this will be/usr/bin/scp. - Set the RSH variable to the full path of the ssh command. In most cases this will be
/usr/bin/ssh.
You should now be able to run Dbvisit Standby as normal and it will make use of SSH instead of Dbvnet.