Using Data Pump

Owned by Copy Space for Confluence
Jan 22, 2017

In two way replication, Dbvisit Replicate can not prepare tables for replication on the target system as they do not exist there. Thus we need to create empty tables on target. If there are a few tables to be replicated then we recommend creating those tables manually. When there are several tables to be replicated then you can use auto generated data pump script of Dbvisit Replicate with a small modification.    

The steps for this are:

  1. Complete setup wizard
  2. Run the *-all.sh script which will generate APPLY.sh file with datapump script.
  3. Modify APPLY.sh file and add "CONTENT=METADATA_ONLY" and run on target system (this only creates the tables - DDL and does not load the data).
  4. Run the *-all.sh script again to prepare the tables
  5. Start the replication (i.e. follow the steps listed in the Nextsteps.txt file). 
    Running the APPLY.sh script takes place within this step. This time you don't modify the APPLY.sh script and so it actually loads the data from the source to the target database. Note that APPLY1.sh script for loading data from target to source is also generated, however it should not be run, as the data is already in the first database.


In step 5 it has been stated that the APPLY1.sh script should not be run. This statement, however, is only true if you are configuring a simple 2-way replication. For the complex scenarios (some data does not exist in the second database, but some data exists in the second database but does not exist in the first one) the APPLY1.sh might be necessary. In such a case the data instantiation must be considered carefully. If not sure, please contact the Dbvisit Support.

Example of using datapump for creating starting point

This example shows creation of starting point for two-way replication for avi schema on database reptest1 and reptest2.


1. In setup wizard, specify two way replication in "Step 2 - Replication pairs" and "Step 3 - Replicated tables".

Step 2 - Replication pairs
========================================
The second step is to set source and targets for each replication pair. This is usually just choosing the first database as
source and the second one as target, but many more configurations are possible.
Let's configure the replication pair, selecting source and target.
Following databases are described:
1: reptest1 (Oracle) 
2: reptest2 (Oracle) 
Select source database: [1] 
Select target database: [2] 
Will be DDL replication enabled? (If YES, the script will grant more privileges to the Dbvisit Replicate users and enable database-wide supplemental logging): [yes] 
Use fetcher to offload the mining to a different server? (yes/no) [no] 
Would you like to encrypt the data across the network (yes or no) [no]
Would you like to compress the data across the network (yes or no) [no]
How long do you want to set the network timeouts.
Recommended range between 60-300 seconds [60]
Lock and copy the data initially one-by-one or at a single SCN? (one-by-one/single-scn/ddl-only/resetlogs) [single-scn] 
What data copy script to create? (dp_networklink/dp_exp/exp/ddl_file/ddl_run/none) [dp_networklink]
Following replication pairs are now configured:
1: reptest1 (Oracle) ==> reptest2 (Oracle), DDL: yes, fetcher: no, process suffix: (no suffix), 
compression: no, encryption: no, network timeout: 60, prepare type:
single-scn, data load: dp_networklink
Enter number of replication pair to modify it, or "add", or "done": [done] add 
Let's configure the replication pair, selecting source and target.
Following databases are described:
1: reptest1 (Oracle) 
2: reptest2 (Oracle) 
Select source database: [1] 2 
Select target database: [2] 1 
Will be DDL replication enabled? (If YES, the script will grant more privileges to the Dbvisit Replicate users and enable database-wide supplemental logging): [yes] 
Use fetcher to offload the mining to a different server? (yes/no) [no] 
Would you like to encrypt the data across the network (yes or no) [no]
Would you like to compress the data across the network (yes or no) [no]
How long do you want to set the network timeouts.
Recommended range between 60-300 seconds [60] 
Lock and copy the data initially one-by-one or at a single SCN? (one-by-one/single-scn/ddl-only/resetlogs) [single-scn] 
What data copy script to create? (dp_networklink/dp_exp/exp/ddl_file/ddl_run/none) [dp_networklink]
Following replication pairs are now configured:
1: reptest1 (Oracle) ==> reptest2 (Oracle), DDL: yes, fetcher: no, process suffix: (no suffix), compression: no, encryption: no,
network timeout: 60, prepare type: single-scn, data load: dp_networklink
2: reptest2 (Oracle) ==> reptest1 (Oracle), DDL: yes, fetcher: no, process suffix: 1, 
compression: no, encryption: no, network timeout: 60, prepare type: single-scn,
data load: dp_networklink
Enter number of replication pair to modify it, or "add", or "done": [done]
Step 3 - Replicated tables
========================================
The third step is to choose the schemas and tables to be replicated. If the databases are reachable, the tables are checked for
existence, datatype support, etc., schemas are queried for tables. Note that all messages are merely hints/warnings and may be
ignored if issues are rectified before the scripts are actually executed.
Note the following assumptions are made in this wizard - which can be modified by editing the resulting script:
1. All replicated tables have a primary key defined.
2. All columns of the tables and all tables of the specified schemas are replicated, and if DDL support is enabled, whenever a
new column/table is added, this should be replicate as well.
3. If an apply conflict arises, the default option is to try again repeatedly, until a different option is given or the
underlying issue is resolved. 
4. If an apply conflict arises, no more data will be replicated until the issue is resolved or ignored.
You can also specify some advanced options:
1. Rename schemas or tables.
2. Specify filtering conditions.
3. (Tables only) Configure Change Data Capture; this does not maintain a copy of the source table, but logs all operations as
separate entries. This is useful for ETL or as an audit trail. This usually requires adding of new columns (timestamps, old/new
values etc.) to the target table. Dbvisit Replicate can create the target table with the additional columns during setup by
selecting "ddl_run" in the data copy section of Step 2.
Following tables are defined for replication pairs:
1: reptest1 (Oracle) ==> reptest2 (Oracle), DDL: yes, suffix: (no suffix), prepare: single-scn
 No tables defined.
2: reptest2 (Oracle) ==> reptest1 (Oracle), DDL: yes, suffix: 1, prepare: single-scn
 No tables defined.
Enter number of replication pair to modify it, or "done": [1]
Please enter list of all individual tables to be replicated. Enter schema name(s) only to replicate all tables in that schema.
Use comma or space to delimit the entries.
Enter the tables and schemas: [] avi 
Selected schemas: AVI
Add more tables or schemas? (YES/NO) [NO]
To replicate changes of PL/SQL objects in schema(s), please enter the schemas to be replicated. Note that specifying any entry
will cause additional privileges to be granted to dbvrep. Enter through a comma or space-delimited list.
Enter the list of schemas (PL/SQL): [] 
Specify rename name or filter condition for any of the specified schemas (YES/NO): [no]
Following tables are defined for replication pairs:
1: reptest1 (Oracle) ==> reptest2 (Oracle), DDL: yes, suffix: (no suffix), prepare: single-scn
 AVI(tables)
2: reptest2 (Oracle) ==> reptest1 (Oracle), DDL: yes, suffix: 1, prepare: single-scn
 No tables defined.
Enter number of replication pair to modify it, or "done": [2]
Please enter list of all individual tables to be replicated. Enter schema name(s) only to replicate all tables in that schema.
Use comma or space to delimit the entries.
Enter the tables and schemas: [] avi 
Selected schemas: AVI
Add more tables or schemas? (YES/NO) [NO]
To replicate changes of PL/SQL objects in schema(s), please enter the schemas to be replicated. Note that specifying any entry
will cause additional privileges to be granted to dbvrep. Enter through a comma or space-delimited list.
Enter the list of schemas (PL/SQL): [] 
Specify rename name or filter condition for any of the specified schemas (YES/NO): [no]
Following tables are defined for replication pairs:
1: reptest1 (Oracle) ==> reptest2 (Oracle), DDL: yes, suffix: (no suffix), prepare: single-scn
 AVI(tables)
2: reptest2 (Oracle) ==> reptest1 (Oracle), DDL: yes, suffix: 1, prepare: single-scn
 AVI(tables)
Enter number of replication pair to modify it, or "done": [done]


2. After completing setup wizard, run *-all.sh that will generate APPLY.sh file with datapump script. Modify the script,add "CONTENT=METADATA_ONLY" and run on target system to create empty tables.

impdp SYSTEM/password@reptest2 table_exists_action=TRUNCATE network_link=reptest1 directory=DATA_PUMP_DIR flashback_scn=7792693 tables=AVI.AVI_OBJECTS,AVI.AVI_TEST2 CONTENT=METADATA_ONLY logfile=AVI_AVI_OBJECTS.log JOB_NAME=DP_reptest1_0001


3. Now , On source machine, run *-all.sh script again.This time it will successfully prepare tables on target system for two-way replication because tables do exist on target machine.

oracle@dbvldemo101[/home/oracle/reptest1]: ./reptest1-all.sh 
Setting up Dbvisit Replicate configuration
Configure database reptest1...
Configure database reptest2...
Object grants for database reptest1...
Object grants for database reptest2...
Setting up the configuration
Initializing......done
WARN-1850: No DDC DB available, dictionary table does not exist.
DDC loaded from database (0 variables).
Dbvisit Replicate version 2.x.x.x.
Copyright (C) Dbvisit Software Limited. All rights reserved.
DDC file /home/oracle/reptest1/reptest1-onetime.ddc loaded.
MINE: Cannot determine Dbvisit Replicate dictionary version. (no dictionary exists)
APPLY: Cannot determine Dbvisit Replicate dictionary version. (no dictionary exists)
dbvrep> set ON_WARNING SKIP
Variable ON_WARNING set to SKIP for process *.
dbvrep> set ON_ERROR EXIT
Variable ON_ERROR set to EXIT for process *.
dbvrep> ENGINE SETUP MINE DROP DICTIONARY
0 dictionary objects dropped.
dbvrep> ENGINE SETUP MINE CREATE DICTIONARY
dbvrep> ENGINE SETUP MINE LOAD DICTIONARY
Supplemental logging on database set.
Loading dictionary table DBRSCOL$
Loading dictionary table DBRSOBJ$
Loading dictionary table DBRSTAB$
Loading dictionary table DBRSUSER$
Loading dictionary table DBRSV_$DATABASE
dbvrep> ENGINE SETUP APPLY DROP DICTIONARY
0 dictionary objects dropped.
dbvrep> ENGINE SETUP APPLY CREATE DICTIONARY
dbvrep> ENGINE SETUP APPLY LOAD DICTIONARY
dbvrep> ENGINE PREPARE_DP SETUP CLEAR
dbvrep> ENGINE SETUP PAIR MINE AND APPLY
ID of mine proces is 88F20AEA-DC6B-11E1-A1D3-7A9D111BFFE8. If not using DDC in database, set MINE_UNIQUE_ID to this value.
1 applier SCN set.
dbvrep> SET APPLY.INSTANTIATE_SCN NOW
Variable INSTANTIATE_SCN set to NOW for process APPLY.
dbvrep> ENGINE SUPPLEMENTAL LOGGING SCHEMA AVI ENABLE PRIMARY KEY
dbvrep> ENGINE SWITCH_REDOLOG
dbvrep> #single-scn instantiation: lock all tables and schemas
dbvrep> ENGINE LOCK SCHEMAS AVI
Locking all schemas.
Lock done.
dbvrep> #single-scn instantiation: unlock all tables and schemas, but keep the SCN
dbvrep> ENGINE LOCK RELEASE LOCKS
dbvrep> 
dbvrep> #prepare the tables (we use OFFLINE as neither MINE not APPLY is running; with OFFLINE we won't wait on network timeout)
dbvrep> PREPARE OFFLINE SCHEMA AVI
Table AVI.AVI_OBJECTS instantiated at SCN 7792693
Table AVI.AVI_TEST2 instantiated at SCN 7792693
dbvrep> #single-scn instantiation: unlock all tables and schemas, forget the SCN (so it does not affect any further PREPARE
statements)
dbvrep> ENGINE LOCK CLEAR SCN
dbvrep> ENGINE SWITCH_REDOLOG
dbvrep> #prepare script for instantiation
dbvrep> ENGINE PREPARE_DP WRITE DP_NETWORKLINK DIRECTORY DATA_PUMP_DIR FILE /home/oracle/reptest1/APPLY.sh DBLINK reptest1 USERID
SYSTEM/password@reptest2
Created Data Pump script /home/oracle/reptest1/APPLY.sh, using network import.
dbvrep> 
dbvrep> # Configuring non-default processes
dbvrep> choose process MINE1
Process type MINE set to: MINE1.
dbvrep> choose process APPLY1
Process type APPLY set to: APPLY1.
dbvrep> ENGINE SETUP MINE LOAD DICTIONARY
Supplemental logging on database set.
Loading dictionary table DBRSCOL$
Loading dictionary table DBRSOBJ$
Loading dictionary table DBRSTAB$
Loading dictionary table DBRSUSER$
Loading dictionary table DBRSV_$DATABASE
dbvrep> ENGINE SETUP APPLY LOAD DICTIONARY
dbvrep> ENGINE PREPARE_DP SETUP CLEAR
dbvrep> ENGINE SETUP PAIR MINE AND APPLY
ID of mine proces is 95164E3A-DC6B-11E1-A1D3-7A9D111BFFE8. If not using DDC in database, set MINE_UNIQUE_ID to this value.
1 applier SCN set.
dbvrep> SET APPLY1.INSTANTIATE_SCN NOW
Variable INSTANTIATE_SCN set to NOW for process APPLY1.
dbvrep> ENGINE SUPPLEMENTAL LOGGING SCHEMA AVI ENABLE PRIMARY KEY
dbvrep> ENGINE SWITCH_REDOLOG
dbvrep> #single-scn instantiation: lock all tables and schemas
dbvrep> ENGINE LOCK SCHEMAS AVI
Locking all schemas.
Lock done.
dbvrep> #single-scn instantiation: unlock all tables and schemas, but keep the SCN
dbvrep> ENGINE LOCK RELEASE LOCKS
dbvrep> 
dbvrep> #prepare the tables (we use OFFLINE as neither MINE not APPLY is running; with OFFLINE we won't wait on network timeout)
dbvrep> PREPARE OFFLINE SCHEMA AVI
Table AVI.AVI_OBJECTS instantiated at SCN 7793482
Table AVI.AVI_TEST2 instantiated at SCN 7793482
dbvrep> #single-scn instantiation: unlock all tables and schemas, forget the SCN (so it does not affect any further PREPARE
statements)
dbvrep> ENGINE LOCK CLEAR SCN
dbvrep> ENGINE SWITCH_REDOLOG
dbvrep> #prepare script for instantiation
dbvrep> ENGINE PREPARE_DP WRITE DP_NETWORKLINK DIRECTORY DATA_PUMP_DIR FILE /home/oracle/reptest1/APPLY1.sh DBLINK reptest2
USERID SYSTEM/password@reptest1
Created Data Pump script /home/oracle/reptest1/APPLY1.sh, using network import.
dbvrep> create ddcdb from ddcfile
DDC loaded into database (224 variables).
dbvrep> set ON_WARNING SKIP
Variable ON_WARNING set to SKIP for process *.
dbvrep> set ON_ERROR SKIP
Variable ON_ERROR set to SKIP for process *.
OK-0: Completed successfully.
These steps are required after the reptest1-all.sh script runs:
1) Create the necessary directory(ies) on the servers:
dbvldemo102: /home/oracle/reptest1
2) Copy the DDC files to the server(s) where the processes will run:
dbvldemo102: /home/oracle/reptest1/reptest1-APPLY.ddc
dbvldemo101: /home/oracle/reptest1/reptest1-APPLY1.ddc
dbvldemo101: /home/oracle/reptest1/reptest1-MINE.ddc
dbvldemo102: /home/oracle/reptest1/reptest1-MINE1.ddc
3) Review that path to dbvrep executable is correct in the run scripts:
/home/oracle/reptest1/reptest1-run-dbvldemo101.sh
/home/oracle/reptest1/reptest1-run-dbvldemo102.sh
4) Copy the run script to the server(s) where the processes will run:
/home/oracle/reptest1/reptest1-run-dbvldemo101.sh
/home/oracle/reptest1/reptest1-run-dbvldemo102.sh
5) Ensure firewall is open for listen interfaces 0.0.0.0:7902, 0.0.0.0:7904, 0.0.0.0:7901, 0.0.0.0:7903 used by the processes.
6) Make sure the data on apply are in sync as of time when setup was run.
Scripts for Data Pump/export/DDL were created as requested:
/home/oracle/reptest1/APPLY.sh
/home/oracle/reptest1/APPLY1.sh
Create referenced database links (if any) before running the scripts.
7) Start the replication processes on all servers:
/home/oracle/reptest1/reptest1-run-dbvldemo101.sh
/home/oracle/reptest1/reptest1-run-dbvldemo102.sh
8) Start the console to monitor the progress:
chmod 0750 /home/oracle/reptest1/start-console.sh
/home/oracle/reptest1/start-console.sh
The above list is stored in /home/oracle/reptest1/Nextsteps.txt.


4. Copy required scripts to target system explained in Nextsteps.txt, run APPLY.sh script to copy the initial data and start replication.  Note if data pump is used you may have to add TABLE_EXISTS_ACTION=APPEND to the impdp command in the APPLY.sh file