Troubleshooting
- Anton Els (Deactivated)
- Vijayganesh Sivaprakasam
- Deepika Kumar (Deactivated)
1. Introduction
In this section, we will cover specific topics around troubleshooting Dbvisit Standby version 8.
2. Creating a support package
One of the most important steps you should perform when logging a support call is to generate a support package.
This section will cover the process in more detail and show you how easy it is to create a support package.
2.1. Using the Command Line Interface (dbvctl)
When running Dbvisit Standby processes, the Process ID or "pid" is displayed in the header. In the example below, we can see the (pid 13750).
============================================================= Dbvisit Standby Database Technology (8.0.04.18184) (pid 13750) dbvctl started on dbvlin103: Thu Nov 17 16:54:02 2016 ============================================================= ... ...
Now that we have the process id, we can create a support package with the command: dbvctl -d <ddc> -f support_package [-a pid=<pid>]
Unless you are aware of the specific PID process that is having a problem use this command: dbvctl -d <ddc> -f support_package. This would create a generic support package, collecting database, Dbvisit Standby configuration details required by the support team.
Example:
In the example below a support, the package is created for process id 13750.
oracle@dbvlin103[/usr/dbvisit8/standby]: ./dbvctl -d CDB1 -f support_package -a pid=13750 Support package /usr/dbvisit8/standby/support/13750_201611171757.tar.gz created. oracle@dbvlin103[/usr/dbvisit8/standby]: ls -al ./support/13750_201611171757.tar.gz -rw-r--r--. 1 oracle oinstall 505391 Nov 17 17:58 ./support/13750_201611171757.tar.gz
Once the support package is created in the DBVISIT_BASE/standby/support subfolder, you can upload this to the Dbvisit Support team to assist.
2.2. Using the Central Console
This section will show you how to create a support package using the Central Console.
There are two methods for creating a support package:
- On completion or failure of a task
- At the completion of each Active task or if any error is detected in the Central Console the option to create and then download the support package will be presented at the top of the task output.
- Using the Create Support Package option from the Active Task List menu (bottom bar)
- Using this option you can create a support package for a specific server (primary or standby) for a specific DDC file, or you can create a support package for a specific PID for a DDC file.
- The steps to do this is described using the images below:
Step 1: Click on "Create Support Package" (1), then select the Dbvisit Standby Configuration (DDC) (2) - see image below:
Step 2: Select the specific Host (on which the Support Package will be created) for the specific DDC (2) - see image below:
Step 3: If you want to create a support package for a specific PID (4 - below) specify the pid, otherwise, click on submit to create a generic support package.
Step 4: Download the support package (3) below. Once downloaded upload the support package to the support ticket if you have one logged with the Dbvisit Support team.
3. General Troubleshooting
3.1. Error opening file for writing
During an upgrade on a Windows-based system (if you are installing version 8.0.06 and above) you need to make sure that all Dbvisit Standby Schedules and Services are stopped prior to starting the installer.
If you get an error "Error opening file for writing" while running the Windows installer, you have not stopped the Dbvisit services, which means the dbvsmgr.exe application is in use and cannot be updated.
Stop the installer, stop the Dbvisit services and restart the installer.
3.2. Troubleshooting - "specify valid username and password"
When installing Dbvisit Standby version 8, it is important to make sure the user that is running the installer is in the Local Administrator group.
The user that is used to Run the dbvisit services - username and password are asked during the installer - must be in the following groups:
- Local Administrators
- ORA_DBA
- USERS
This user must also have the "Logon as a service" permission.
For more detail on how to do this please see the example below:
4. Graceful Switchover
This section will contain notes with regards to Graceful Switchover and possible errors and their fixes.
4.1. GS Fail due to NON-OMF redo logs and OMF redo logs expected
The following error is shown when performing a Graceful Switchover (GS) between two Windows-based Databases. This is the second Graceful Switchover, meaning a switch was performed between primary and standby, where the primary database was using ASM based storage and standby not (filesystem based). The second switch over is when we want to move from the filesystem based primary database to the ASM standby.
Error message:
Dbvisit Standby terminated... Error Code: 1 Remote execution error on KIWI301. ========================Remote Output: KIWI301======================== Dbvisit Standby terminated... Error Code: 2147 Dbvisit Standby cannot proceed: Cannot create standby redo log C:\APP\ORACLE\ORADATA\DEV\ONLINELOG\GROUP_1.272.933526145 for primary log C:\APP\ORACLE\ORADATA\DEV\ONLINELOG\GROUP_1.272.933526145 as ASM file. Cannot create standby redo log C:\APP\ORACLE\ORADATA\DEV\ONLINELOG\GROUP_1.276.933526145 for primary log C:\APP\ORACLE\ORADATA\DEV\ONLINELOG\GROUP_1.276.933526145 as ASM file. Cannot create standby redo log C:\APP\ORACLE\ORADATA\DEV\ONLINELOG\GROUP_2.268.933526147 for primary log C:\APP\ORACLE\ORADATA\DEV\ONLINELOG\GROUP_2.268.933526147 as ASM file. Cannot create standby redo log C:\APP\ORACLE\ORADATA\DEV\ONLINELOG\GROUP_2.269.933526147 for primary log C:\APP\ORACLE\ORADATA\DEV\ONLINELOG\GROUP_2.269.933526147 as ASM file. Cannot create standby redo log C:\APP\ORACLE\ORADATA\DEV\ONLINELOG\GROUP_3.265.933526151 for primary log C:\APP\ORACLE\ORADATA\DEV\ONLINELOG\GROUP_3.265.933526151 as ASM file. Cannot create standby redo log C:\APP\ORACLE\ORADATA\DEV\ONLINELOG\GROUP_3.266.933526151 for primary log C:\APP\ORACLE\ORADATA\DEV\ONLINELOG\GROUP_3.266.933526151 as ASM file. Review the following standby database parameters: db_create_file_dest = +DATA db_create_online_log_dest_1 = db_create_online_log_dest_2 = db_create_online_log_dest_3 = db_create_online_log_dest_4 = db_create_online_log_dest_5 = db_recovery_file_dest = +DATA log_file_name_convert = Tracefile from server: KIWI301 (PID:2796) 2796_dbvctl_f_gs_get_info_standby_DEV_201701251733.trc
In this example, the primary database is running on Windows and using ASM storage with Oracle Managed Files configured (OMF)
The standby database was created on a Windows system using Filesystem Based storage.
But during the Standby Creation instead of having proper OMF configuration, the convert parameters where set.
Standby Server Details: Using ASM (this was the original primary)
SQL> select * from v$logfile;
GROUP# STATUS TYPE MEMBER IS_ CON_ID
---------- ------- ------- -------------------------------------------------- --- ----------
1 ONLINE +DATA YES 0
1 ONLINE +DATA YES 0
2 ONLINE +DATA YES 0
2 ONLINE +DATA YES 0
3 ONLINE +DATA YES 0
3 ONLINE +DATA YES 0
6 rows selected.
SQL> show parameter convert
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_file_name_convert string
log_file_name_convert string
pdb_file_name_convert string
SQL> show parameter create
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
create_bitmap_area_size integer 8388608
create_stored_outlines string
db_create_file_dest string +DATA
db_create_online_log_dest_1 string
db_create_online_log_dest_2 string
db_create_online_log_dest_3 string
db_create_online_log_dest_4 string
db_create_online_log_dest_5 string
SQL> show parameter recovery
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest string +DATA
db_recovery_file_dest_size big integer 6420M
recovery_parallelism integer 0
Primary System Details: Using Filesystem (this was the original standby)
SQL> show parameter create NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ create_bitmap_area_size integer 8388608 create_stored_outlines string db_create_file_dest string c:\app\oracle\oradata db_create_online_log_dest_1 string db_create_online_log_dest_2 string db_create_online_log_dest_3 string db_create_online_log_dest_4 string db_create_online_log_dest_5 string SQL> show parameter recovery NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ db_recovery_file_dest string c:\app\oracle\oradata db_recovery_file_dest_size big integer 6420M recovery_parallelism integer 0 SQL> show parameter convert NAME TYPE VALUE ------------------------------------ ----------- ------------------------------ db_file_name_convert string +DATA, c:\app\oracle\oradata log_file_name_convert string +DATA, c:\app\oracle\oradata pdb_file_name_convert string
One way to show that these files are not OMF files are to add a new one, example below - you will see how the new OMF file names are different:
SQL> alter database add logfile group 4 size 50M;
Database altered.
SQL> select * from v$logfile;
GROUP# STATUS TYPE MEMBER IS_ CON_ID
---------- ------- ------- -------------------------------------------------------------------------------- --- ----------
1 ONLINE C:\APP\ORACLE\ORADATA\DEV\ONLINELOG\GROUP_1.272.933526145 NO 0
1 ONLINE C:\APP\ORACLE\ORADATA\DEV\ONLINELOG\GROUP_1.276.933526145 NO 0
2 ONLINE C:\APP\ORACLE\ORADATA\DEV\ONLINELOG\GROUP_2.268.933526147 NO 0
2 ONLINE C:\APP\ORACLE\ORADATA\DEV\ONLINELOG\GROUP_2.269.933526147 NO 0
3 ONLINE C:\APP\ORACLE\ORADATA\DEV\ONLINELOG\GROUP_3.265.933526151 NO 0
3 ONLINE C:\APP\ORACLE\ORADATA\DEV\ONLINELOG\GROUP_3.266.933526151 NO 0
4 ONLINE C:\APP\ORACLE\ORADATA\DEV\ONLINELOG\O1_MF_4_D8LODWLK_.LOG NO 0
4 ONLINE C:\APP\ORACLE\ORADATA\DEV\ONLINELOG\O1_MF_4_D8LODWTC_.LOG YES 0
8 rows selected.
SQL>
The Solution:
The solution in this scenario is to do the following:
- From the Primary which is currently using Filesystem Based Storage, remove (drop) redo log groups 1,2 and 3 which is not true OMF files, then recreate them as OMF redo (same as group 4 above)
- Remove the db_file_name_convert and log_file_name_convert parameters
- Run a number of log switches
- Send logs to the standby
- Apply logs
- Run Log Gap Report
- If the report shows 0 archive log gap, re-start the Graceful Switchover.