APPLY (or MINE) cannot connect - connectivity issues

The MINE and APPLY processes have to communicate with each other for the replication to be successful. If this is not the case, then replication will be stopped. 

In the console this can sometimes be seen with the following error message:

MINE IS running. Currently at plog 1071 and SCN 9832904 (06/10/2016 23:10:19).
Could not connect to APPLY process. Process not started or connection refused.

This section describes how to trouble shoot connectivity issues between the MINE and APPLY processes.

Solution:

1. Was the Mine process started on the Mine server?

2. Was the Apply process started on the Apply server?

3. Check if the mine process is able to connect through the network to the APPLY process.

dbvrep> engine apply send engine status 
-1: Connection refused

This indicates that there is a network issue. Check to ensure the firewall is not blocking the Dbvisit Replicate ports.

4. Run a health check command:

dbvrep> healthcheck 
ERROR: Could not connect to APPLY, it is down or unreachable.
(Connection refused)
Connectivity test for APPLY: FAILED
ERROR: Could not connect to APPLY, it is down or unreachable. (Connection refused)
Connectivity test for MINE: PASSED
2 processes checked.

Find out which ports are being used:

dbvrep> show interface
MINE1.MINE_LISTEN_INTERFACE = 0.0.0.0:7903
*.MINE_REMOTE_INTERFACE = localhost:7890
APPLY.APPLY_LISTEN_INTERFACE = 0.0.0.0:7902
APPLY.APPLY_REMOTE_INTERFACE = xxx:7902
MINE.MINE_LISTEN_INTERFACE = 0.0.0.0:7901
APPLY1.APPLY_REMOTE_INTERFACE = xxx:7904
*.APPLY_REMOTE_INTERFACE = localhost:7891
APPLY1.APPLY_LISTEN_INTERFACE = 0.0.0.0:7904
*.FETCHER_REMOTE_INTERFACE = localhost:7889
*.MINE_LISTEN_INTERFACE = localhost:7890
MINE.MINE_REMOTE_INTERFACE = xxx:7901
*.APPLY_LISTEN_INTERFACE = localhost:7891
*.FETCHER_LISTEN_INTERFACE = localhost:7889
MINE1.MINE_REMOTE_INTERFACE = xxx.cr:7903

Please ensure all ports listed above are open on the specific servers. 

5. Try to ping the unreachable process server; try the ping from the MINE machine, the machine console (if it is different from MINE machine) and the APPLY machine itself. Check that the IP addresses reported by the ping are the same. (It definitely should not be 127.0.0.1 unless the replication is completely running on one machine.)

6. If the processes does not seem to see each other, try running "telnet machineport" on each of the machines to check connectivity to each other machine. The connection should not be refused. Use exactly same names as in your DDC file (use copy-paste if possible). This checks various DNS problems, typos in names/IPs, closed firewall ports etc.

Use nz -z <host> <port> command:

nc -z dbvisitbox 7901
Connection to dbvisitbox 7901 port [tcp/tnos-sp] succeeded!

Or use telnet:

telnet dbvisitbox 7901

If the message says: "Escape character is" then the connection is successful.

7. Ensure there is enough free memory for the dbvrep process to start. If the dbvrep process suddenly dies, it may mean that there is not enough free memory on the server and the OS will terminate the dbvrep process without warning. This can be verified by running "top" in Linux/Unix while the dbvrep APPLY or MINE process is started.

8. Check that the processes are running. On LInux/Unix check with "ps -ef | grep dbvrep". On Windows, check the services control panel. There should be a MINE process running on the source server and an APPLY process running on the target server. For 2-way replication there will be a MINE and APPLY1 process running on the source server and an APPLY and MINE1 process running on the target server.

9. Check the Mine log is being updated. On the Mine server, the Mine log is located in the "log" directory and the file name is similar to dbvrep_MINE_*.log. This log will be updated every few seconds. Check for error messages in the log if the file is not being updated. 

10. Check the Apply log is being updated. On the Apply server, the Apply log is located in the "log" directory and the file name is similar to dbvrep_APPLY_*.log. This log will be updated every few seconds. Check for error messages in the log if the file is not being updated. 

11. Run further internal engine commands. These commands can be run to check the connectivity

dbvrep> engine apply send engine id 
dbvrep> engine apply send engine status
dbvrep> engine apply send list progress