Target Hadoop (HDFS)

Owned by Copy Space for Confluence
Last updated: Jul 04, 2017 by Miroslav Sivon (Deactivated)
18 min read

In all replications, the source database must be an Oracle database. 

NOTE

  • Two-way replication with Hadoop (HFDS) is not supported
  • Replicating DDL is not currently supported in this release
  • Make sure Hadoop is running. (ps -ef | grep had)
  • Make sure that you know the location of the below directories, ie. hadoop/hdfs, hadoop/common & hadoop/common/lib & have added to your CLASSPATH, i.e.
		cd ~/hadoop/share/hadoop/hdfs/
		export CLASSPATH=`for f in $PWD/*.jar; do CLASSPATH=$CLASSPATH:$f; done; echo $CLASSPATH;`
		cd ~/hadoop/share/hadoop/common/
		export CLASSPATH=`for f in $PWD/*.jar; do CLASSPATH=$CLASSPATH:$f; done; echo $CLASSPATH;`
		cd ~/hadoop/share/hadoop/common/lib
		export CLASSPATH=`for f in $PWD/*.jar; do CLASSPATH=$CLASSPATH:$f; done; echo $CLASSPATH;`
		export CLASSPATH=$CLASSPATH:~/hadoop/etc/hadoop

Setup Wizard Example

This example shows an Oracle-to-Hadoop one-way replication using the following information.

Server/Database
Name
Source databaseORCL (Oracle)
Source serversource (Linux)
Target databasen/a
Target servertarget (Linux)
TNS alias for source database   ORCL

 

Throughout the wizard, defaults are shown in [brackets]. Just press enter to accept them. The replication name in this example is "reptest".

 

First of all start the Dbvisit Replicate command line console.

 

[oracle@source replicate]$ ./dbvrep
Initializing......done
Dbvisit Replicate version 2.8
Copyright (C) Dbvisit Software Limited.  All rights reserved.
No DDC file loaded.
Run "setup wizard" to start the configuration wizard or try "help" to see all commands available.
dbvrep>

Then start the setup wizard and follow the instructions. Most of the defaults can be chosen.

dbvrep> setup wizard

This wizard configures Dbvisit Replicate.
The setup wizard creates configuration scripts, which need to be run after the wizard ends. No changes to the databases are made before that.
The progress is saved every time a list of databases, replications, etc. is shown. It will be re-read if wizard is restarted and the same DDC name and script path
is selected.
Run the wizard now? [Yes]
Accept end-user license agreement? (View/Yes/No) [View] Yes
Before starting the actual configuration, some basic information is needed. The DDC name and script path determines where all files created by the wizard go (and
where to reread them if wizard is rerun) and the license key determines which options are available for this configuration.
(DDC_NAME) - Please enter a name for this replication: [] reptest
(LICENSE_KEY) - Please enter your license key: [(trial)]
Which Replicate edition do you want to trial (LTD/XTD/MAX): [MAX]
(SETUP_SCRIPT_PATH) - Please enter a directory for location of configuration scripts on this machine: [/u01/app/oracle/reptest] /home/oracle/reptest
Network configuration files were detected on this system in these locations:
/u01/app/oracle/product/11.2.0/xe/network/admin
(TNS_ADMIN) - Please enter TNS configuration directory for this machine: [/u01/app/oracle/product/11.2.0/xe/network/admin]

 

Specify databases participating in the replication in the first step of setup wizard.

Step 1 - Describe databases
========================================
The first step is to describe databases used in the replication. There are usually two of them (source and target).
Store SYSDBA and DBA passwords? Passwords only required during setup and initialization? (Yes/No) [Yes]
Let's configure the database, describing its type, connectivity, user names etc.
What type of database is this? (Oracle/MySQL/SQL Server/Oracle AWS RDS/CSV/Hadoop): [Oracle]
Please enter database TNS alias: [] ORCL
Please enter SYSDBA user name: [SYS]
Please enter password for this user: [change_on_install]
Please enter user with DBA role: [SYSTEM]
Please enter password for this user: [manager]
Connecting to database ORCL as SYSTEM to query list of tablespaces and to detect ASM (by looking whether any redo logs or archived logs are stored in ASM).
Enter the Dbvisit Replicate owner (this user will be created by this script): [dbvrep]
Please enter password for this user: [dbvpasswd]
Permanent tablespaces detected on the database: REPOE_DATA, USERS.
Please enter default permanent tablespace for this user: [REPOE_DATA] USERS
Temporary tablespaces detected on the database: TEMP.
Please enter default temporary tablespace for this user: [TEMP]
Following databases are now configured:
1: Oracle ORCL, SYS/***, SYSTEM/***, dbvrep/***, USERS/TEMP, dbvrep/, ASM:No, TZ: +00:00
Enter the number of the database to modify it, or "add", or "done": [add]                                                        
Let's configure the database, describing its type, connectivity, user names etc.
What type of database is this? (Oracle/MySQL/SQL Server/Oracle AWS RDS/CSV/Hadoop): [Oracle] Hadoop                              
File-based output selected. Questions regarding file locations and format will be asked in Step 4 - apply process(es).
Dbvisit Replicate needs to store its metadata in a database. Please configure it now. You can also use the source database, but
use different DBVREP user in such case.
What type of database to use for the repository? (Oracle/MySQL/SQL Server/Oracle AWS RDS): [Oracle]                              
Please enter database TNS alias: [] ORCL                                                                                        
Please enter SYSDBA user name: [SYS]                                                                                             
Please enter password for this user: [change_on_install] ********
Please enter user with DBA role: [SYSTEM]                                                                                        
Please enter password for this user: [manager] ********
Connecting to database SDB11 as SYSTEM to query list of tablespaces and to detect ASM (by looking whether any redo logs or
archived logs are stored in ASM).
Enter the Dbvisit Replicate owner (this user will be created by this script): [dbvrep] dbvrep2                                   
Please enter password for this user: [dbvpasswd] 
Permanent tablespaces detected on the database: DATA, USERS.
Please enter default permanent tablespace for this user: [DATA]                                                                  
Temporary tablespaces detected on the database: TEMP.
Please enter default temporary tablespace for this user: [TEMP]                                                                  

Following databases are now configured:
1: Oracle ORCL, SYS/***, SYSTEM/***, dbvrep/***, DATA/TEMP, dbvrep/, ASM:No, TZ: +00:00
2: Hadoop(Oracle) ORCL, SYS/***, SYSTEM/***, dbvrep2/***, DATA/TEMP, dbvrep2/, ASM:No, TZ: +00:00

Enter the number of the database to modify it, or "add", or "done": [done]

 

In the second step of setup wizard,we configure replication pair by specifying source and target databases. Only Oracle database can be the source here.

Step 2 - Replication pairs
========================================
The second step is to set source and targets for each replication pair.
Let's configure the replication pair, selecting source and target.
Following databases are described:
1: ORCL (Oracle)  
2: ORCL (Hadoop-Oracle) (cannot be source, is not Oracle) 
Select source database: [1]                                                                                                      
Select target database: [2]                                                                                                      
Will DDL replication be enabled? (Yes/No) [No]                                                                                  
Use fetcher to offload the mining to a different server? (Yes/No) [No]                                                           
Should where clauses (and change auditing) include all columns, not just changed and PK? (Yes/No) [No]                           
Would you like to encrypt the data across the network? (Yes/No) [No]                                                             
Would you like to compress the data across the network? (Yes/No) [No]                                                            
How long do you want to set the network timeouts. Recommended range between 60-300 seconds [60]                                  
Lock and copy the data initially one-by-one or at a single SCN?
one-by-one : Lock tables one by one and capture SCN
single-scn : One SCN for all tables
ddl-only   : Only DDL script for target objects
resetlogs  : Use SCN from last resetlogs operation (standby activation, rman incomplete recovery)
no-lock    : Do not lock tables. Captures previous SCN of oldest active transaction. Requires pre-requisite running of
pre-all.sh script   (one-by-one/single-scn/ddl-only/resetlogs/no-lock) [single-scn] 
What data instantiation script to create?
dp_networklink : Data Pump with network link. No export datafile is created (APPLY.sh)
dp_exp         : Data Pump with export datafile. Creates an export datafile (APPLY.sh)
exp            : Pre-datapump exp/imp with export datafile (APPLY.sh)
ddl_file       : DDL file created (APPLY.sql)
ddl_run        : DDL is automatically executed on target
load           : All replicated data is created and loaded automatically 
none                                                                                                                             (dp_networklink/dp_exp/exp/ddl_file/ddl_run/load/none) [none] 
Following replication pairs are now configured:
1: ORCL (Oracle) ==> ORCL (Hadoop-Oracle), DDL: No, fetcher: No, process suffix: (no suffix), compression: No, encryption:
No, network timeout: 60, prepare type: single-scn, data load: none
Enter number of replication pair to modify it, or "add", or "done": [done]

 

We will now choose the tables that we want to replicate.

Step 3 - Replicated tables
========================================
The third step is to choose the schemas and tables to be replicated. If the databases are reachable, the tables are checked for existence, datatype support, etc.,
schemas are queried for tables. Note that all messages are merely hints/warnings and may be ignored if issues are rectified before the scripts are actually
executed.
Following tables are defined for replication pairs:
1: ORCL (Oracle) ==> ORCL (Hadoop-Oracle), DDL: No, suffix: (no suffix), prepare: single-scn
  No tables defined.
Enter number of replication pair to modify it, or "done": [1]
Please enter list of all individual tables to be replicated. Enter schema name(s) only to replicate all tables in that schema. Use comma or space to delimit the
entries.
Enter the tables and schemas: [] HR.EMPLOYEES HR.DEPARTMENTS
Selected tables: HR.EMPLOYEES, HR.DEPARTMENTS
Add more tables or schemas? (Yes/No) [No]

Filter the data to be replicated? If yes, use single condition for ALL DML operations or use CUSTOM conditions for each operation? (No/ALL/CUSTOM) [No]
Configure change auditing? (No/Yes) [No]
Following tables are defined for replication pairs:
1: ORCL (Oracle) ==> ORCL (Hadoop-Oracle), DDL: NO, suffix: (no suffix), prepare: single-scn
  HR.TEST1 => HR_TARGET.TEST1_TARGET, HR.TEST2 => TEST2_TARGET.TEST2
Enter number of replication pair to modify it, or "done": [done]
To replicate changes of PL/SQL objects in schema(s), please enter the schemas
to be replicated. Note that specifying any entry will cause additional
privileges to be granted to dbvrep. Enter through a comma or space-delimited
list.
Enter the list of schemas (PL/SQL): []  
 
File-based output implies Change Data Capture.

Configure table HR.EMPLOYEES: 
Capture DELETE operations? (Yes/No) [Yes] 
Capture UPDATE operations - old values? (Yes/No) [Yes]                          
Capture UPDATE operations - new values? (Yes/No) [Yes]                          
Capture INSERT operations? (Yes/No) [Yes]                                       
The columns at the target table can be called the same as on the source table,
or they can be prefixed to indicate whether they contain old or new values. As
both new and old values for update are specified, at least one prefix has to be
defined. 
Prefix for columns with OLD values: [] OLD_
Prefix for columns with NEW values: [] NEW_  
Dbvisit Replicate can include additional information into the target table;
this can be used to identify type of change, when it occurred, who initiated
the change etc.
Add basic additional information about the changes (SCN, time, operation type)? (Yes/No) [Yes]  
Add more transactional information (transaction id, commit time)? (Yes/No) [No] Yes
Add auditing columns (login user, machine, OS user...)? (Yes/No) [No]                                  
Setup wizard chose following default names for the columns; you can use them or choose own names. Note
that empty answer confirms the proposed default; use "-" (minus) to remove the column from the audit.
OPERATION: Operation code (U/I/D) (opcol)
SCN: SCN at source (scn)
DATE_CHANGE: Date and time of the change (timestamp_change)
DATE_COMMIT: Date and time of transaction commit (timestamp_commit)
TRANSACTION_ID: Transaction ID (mandatory if timestamp_commit is used; please define an index on
transaction ID in such case) (xidcol)
Accept these settings? (Yes/No) [Yes] 
Configure table HR.DEPARTMENTS:                                                                           
Configure table in a different way? Use COPY to copy settings from previous table? (Yes/COPY) [COPY] 

Following tables are defined for replication pairs:
1: ORCL (Oracle) ==> ORCL (Hadoop-Oracle), DDL: Yes, suffix: (no suffix), prepare: single-scn
  HR.EMPLOYEES, HR.DEPARTMENTS
Enter number of replication pair to modify it, or "done": [done]

 

In the fourth and the final step of setup wizard, we can configure MINE and APPLY process parameters.

Step 4 - Process configuration
========================================
The fourth step is to configure the replication processes for each replication.
Following processes are defined:
1: MINE on ORCL
  Not configured.
2: APPLY on ORCL
  Not configured.
Enter number of process to modify it, or "done": [1]
Fully qualified name of the server for the process (usually co-located with the database, unless mine is offloaded using fetcher): [source]
Server type (Windows/Linux/Unix): [Linux]
Enable email notifications about problems? (Yes/No) [No]
Enable SNMP traps/notifications about problems? (Yes/No) [No]
Directory with DDC file and default where to create log files etc. (recommended: same as global setting, if possible)? [/home/oracle/reptest]
Following settings were pre-filled with defaults or your reloaded settings:
----------------------------------------
[MINE_REMOTE_INTERFACE]: Network remote interface: source:7901
[MINE_DATABASE]: Database TNS: ORCL
[TNS_ADMIN]: tnsnames.ora path: /u01/app/oracle/product/11.2.0/xe/network/admin
[MINE_PLOG]: Filemask for generated plogs: /home/oracle/reptest/mine/%S.%E (%S is sequence, %T thread, %F original filename (stripped extension), %P process type,
%N process name, %E default extension)
[LOG_FILE]: General log file: /home/oracle/reptest1/log/dbvrep_%N_%D.%E
[LOG_FILE_TRACE]: Error traces: /home/oracle/reptest1/log/trace/dbvrep_%N_%D_%I_%U.%E
Checking that these settings are valid...
Do you want to change any of the settings? [No]
Following processes are defined:
1: MINE on ORCL
  Host: source, SMTP: No, SNMP: No
2: APPLY on ORCL
  Not configured.
Enter number of process to modify it, or "done": [2]
Fully qualified name of the server for the process (usually co-located with the database, unless mine is offloaded using fetcher): [target]
Server type (Windows/Linux/Unix): [] Linux
Enable email notifications about problems? (Yes/No) [No]
Enable SNMP traps/notifications about problems? (Yes/No) [No]
Directory with DDC file and default where to create log files etc. (recommended: same as global setting, if possible)? [/home/oracle/reptest]
Following settings were pre-filled with defaults or your reloaded settings:
----------------------------------------
[APPLY_REMOTE_INTERFACE]: Network remote interface: target:7902
[APPLY_DATABASE]: Database MySQL connection string: database=dbvrep;host=target
[TNS_ADMIN]: tnsnames.ora path: /u01/app/oracle/product/11.2.0/xe/network/admin
[APPLY_SCHEMA]: Dbvisit Replicate database (schema): dbvrep
[APPLY_STAGING_DIR]: Directory for received plogs: /home/oracle/reptest/apply
[LOG_FILE]: General log file: /home/oracle/reptest1/log/dbvrep_%N_%D.%E
[LOG_FILE_TRACE]: Error traces: /home/oracle/reptest1/log/trace/dbvrep_%N_%D_%I_%U.%E
Checking that these settings are valid...
Do you want to change any of the settings? [No]
Following processes are defined:
1: MINE on ORCL
  Host: source, SMTP: No, SNMP: No
2: APPLY on ORCL
  Host: target, SMTP: No, SNMP: No
Enter number of process to modify it, or "done": [done]
Created file /home/oracle/reptest/reptest-APPLY.ddc.
Created file /home/oracle/reptest/reptest-MINE.ddc.
Created file /home/oracle/reptest/config/reptest-setup.dbvrep.
Created file /home/oracle/reptest/config/reptest-dbsetup_ttorcl_src_dbvrep.sql.
Created file /home/oracle/reptest/config/reptest-dbsetup_database_dbvrep_host_target_root.sql.
Created file /home/oracle/reptest/config/reptest-grants_ttorcl_src_dbvrep.sql.
Created file /home/oracle/reptest/config/reptest-grants_database_dbvrep_host_target_root.sql.
Created file /home/oracle/reptest/config/reptest-onetime.ddc.
Created file /home/oracle/reptest/start-console.sh.
Created file /home/oracle/reptest/reptest-run-source.sh.
Created file /home/oracle/reptest/scripts/reptest-source-start-MINE.sh.
Created file /home/oracle/reptest/scripts/reptest-source-stop-MINE.sh.
Created file /home/oracle/reptest/scripts/reptest-source-dbvrep-MINE.sh.
Created file /home/oracle/reptest/scripts/systemd-dbvrep-MINE_reptest1.service.
Created file /home/oracle/reptest/scripts/upstart-dbvrep-MINE_reptest1.conf.
Created file /home/oracle/reptest/reptest-run-target.sh.
Created file /home/oracle/reptest/scripts/reptest-target-start-APPLY.sh.
Created file /home/oracle/reptest/scripts/reptest-target-stop-APPLY.sh.
Created file /home/oracle/reptest/scripts/reptest-target-dbvrep-APPLY.sh.
Created file /home/oracle/reptest/scripts/systemd-dbvrep-APPLY_reptest1.service.
Created file /home/oracle/reptest/scripts/upstart-dbvrep-APPLY_reptest1.conf.
Created file /home/oracle/reptest/Nextsteps.txt.
Created file /home/oracle/reptest/reptest-all.sh.
===================================================================================================================================================================
Dbvisit Replicate wizard completed
Script /home/oracle/reptest/reptest-all.sh created. This runs all the above created scripts. Please exit out of dbvrep, review and run script as current user to
setup and start Dbvisit Replicate.
===================================================================================================================================================================
Optionally, the script can be invoked now by this wizard.
Run this script now? (Yes/No) [No]

 

Exit out of Dbvisit Replicate and start the script that has been generated by the setup wizard. These scripts configure the source and target databases for replication and create the dbvrep schema in Oracle and the dbvrep2 schema that in this case is also in the Oracle database.  Out of all scripts, the only script to be executed is <replication-name>-all.sh (in this case the ORCL-all.sh script) as this actually executes all the other scripts in the appropriate order. 

[oracle@source reptest1]$ ./reptest1-all.sh
Setting up Dbvisit Replicate configuration
Configure database ORCL...
This check fails if the DBID is not the expected one...

Ok, check passed.
Setting up the configuration
Initializing......done
DDC loaded from database (0 variables).
Dbvisit Replicate version 2.8
Copyright (C) Dbvisit Software Limited.  All rights reserved.
DDC file /home/oracle/reptest1/config/reptest1-onetime.ddc loaded.
MINE: Cannot determine Dbvisit Replicate dictionary version. (no
dictionary exists)
APPLY: Cannot determine Dbvisit Replicate dictionary version. (no
dictionary exists)
dbvrep> #clear the no-DDC-DB-available warning
dbvrep> process clear previous warnings
dbvrep> set ON_WARNING SKIP
Variable ON_WARNING set to SKIP for process *.
dbvrep> set ON_ERROR EXIT
Variable ON_ERROR set to EXIT for process *.
dbvrep>
dbvrep> # Configuring default processes
dbvrep> choose process MINE
Process type MINE set to: MINE.
dbvrep> choose process APPLY
Process type APPLY set to: APPLY.
dbvrep> PROCESS SWITCH_REDOLOG
Redo log switch requested.dbvrep> PROCESS SETUP MINE DROP DICTIONARY
0 dictionary objects dropped.
dbvrep> PROCESS SETUP MINE CREATE DICTIONARY
dbvrep> PROCESS SETUP MINE LOAD DICTIONARY
Oldest active transaction SCN: 2238258 (no active transaction)
Supplemental logging on database set.
Loading dictionary table DBRSCCOL$
Loading dictionary table DBRSCDEF$
Loading dictionary table DBRSCOL$
Loading dictionary table DBRSIND$
Loading dictionary table DBRSOBJ$
Loading dictionary table DBRSTAB$
Loading dictionary table DBRSUSER$
Loading dictionary table DBRSV_$DATABASE
dbvrep> PROCESS SETUP APPLY DROP DICTIONARY
0 dictionary objects dropped.
dbvrep> PROCESS SETUP APPLY CREATE DICTIONARY
dbvrep> PROCESS SETUP APPLY LOAD DICTIONARY
dbvrep> PROCESS SETUP PAIR MINE AND APPLY
Applier SCN set (start=2238267, current=2238267).
dbvrep> SET APPLY.INSTANTIATE_SCN NOW
Variable INSTANTIATE_SCN set to NOW for process APPLY.
dbvrep> SET MINE._PREPARE_SUPLOG_TYPE PK
Variable _PREPARE_SUPLOG_TYPE set to PK for process MINE.
dbvrep> EXCLUDE CREATE TABLE %.DBMS_TABCOMP_TEMP_UNCMP #Ignore tables
created by Compression Advisor
Exclude rule created.
dbvrep> EXCLUDE CREATE TABLE %.DBMS_TABCOMP_TEMP_CMP #Ignore tables
created by Compression Advisor
Exclude rule created.
dbvrep> EXCLUDE CREATE TABLE %.SCHEDULER$_% #Ignore tables created by
Oracle scheduler (also used by schema/full expdp/impdp)
Exclude rule created.
dbvrep> EXCLUDE CREATE TABLE %.CMP1$% #Ignore tables created by
Compression Advisor since 11.2.0.4
Exclude rule created.
dbvrep> EXCLUDE CREATE TABLE %.CMP2$% #Ignore tables created by
Compression Advisor since 11.2.0.4
Exclude rule created.
dbvrep> EXCLUDE CREATE TABLE %.CMP3$% #Ignore tables created by
Compression Advisor since 11.2.0.4
Exclude rule created.
dbvrep> EXCLUDE CREATE TABLE %.CMP4$% #Ignore tables created by
Compression Advisor since 11.2.0.4
Exclude rule created.
dbvrep> PROCESS SUPPLEMENTAL LOGGING TABLE "HR"."EMPLOYEES" ENABLE PRIMARY
KEY
dbvrep> PROCESS SUPPLEMENTAL LOGGING TABLE "HR"."DEPARTMENTS" ENABLE PRIMARY
KEY
dbvrep> PROCESS SWITCH_REDOLOG
Redo log switch requested.dbvrep> PROCESS WAIT_SCN_FLIP
Waited 3 seconds until scn_to_timestamp changed.
dbvrep> #single-scn instantiation: lock all tables and schemas
dbvrep> PROCESS LOCK TABLES "HR"."EMPLOYEES" "HR"."DEPARTMENTS"
Locking all tables.
Lock done.
dbvrep> #single-scn instantiation: unlock all tables and schemas, but
keep the SCN
dbvrep> PROCESS LOCK RELEASE LOCKS
dbvrep>
dbvrep> #prepare the tables (we use OFFLINE as neither MINE nor APPLY
is running; with OFFLINE we won't wait on network timeout)
dbvrep> PREPARE OFFLINE TABLE "HR"."EMPLOYEES"
Table HR.EMPLOYEES instantiated at SCN 2239997
dbvrep> PREPARE OFFLINE TABLE "HR"."DEPARTMENTS"
Table HR.DEPARTMENTS instantiated at SCN 2239997
dbvrep> #single-scn instantiation: unlock all tables and schemas,
forget the SCN (so it does not affect any further PREPARE statements)
dbvrep> PROCESS LOCK CLEAR SCN
dbvrep> PROCESS SWITCH_REDOLOG
Redo log switch requested.dbvrep> #prepare script for instantiation
dbvrep> create ddcdb from ddcfile
DDC loaded into database (391 variables).
dbvrep> set ON_WARNING SKIP
Variable ON_WARNING set to SKIP for process *.
dbvrep> set ON_ERROR SKIP
Variable ON_ERROR set to SKIP for process *.
OK-0: Completed.
These steps are required after the reptest1-all.sh script runs:
1) Create the necessary directory(ies) on the servers:
target: /home/oracle/reptest1
2) Copy the DDC files to the server(s) where the processes will run:
target: /home/oracle/reptest1/reptest1-APPLY.ddc
source: /home/oracle/reptest1/reptest1-MINE.ddc
3) Review that path to dbvrep executable is correct in the run scripts:
/home/oracle/reptest1/reptest1-run-source.sh
/home/oracle/reptest1/reptest1-run-target.sh
4) Copy the run script to the server(s) where the processes will run:
source: /home/oracle/reptest1/reptest1-run-source.sh
target: /home/oracle/reptest1/reptest1-run-target.sh
5) Ensure firewall is open for listen interfaces 0.0.0.0:7902, 0.0.0.0:7901 used by the processes.
6) Ensure firewall is open for remote access to mysql (usually port 3306) on target.
7) Make sure the data on apply are in sync as of time when setup was run.
8) Start the replication processes on all servers:
source: /home/oracle/reptest1/reptest1-run-source.sh
target: /home/oracle/reptest1/reptest1-run-target.sh
9) Start the console to monitor the progress:
/home/oracle/reptest1/start-console.sh
The above list is stored in /home/oracle/reptest1/Nextsteps.txt.

Starting the MINE and APPLY process requires their respective DDC config files and run scripts that actually starts these processes in background. ( On Windows system, MINE and APPLY act as services). These steps are outlined in Nextsteps.txt, but in the above example we copy reptest1-APPLY.ddc and reptest1-run-target.sh to the server (target) and then start MINE and APPLY on source and target, respectively. 

Starting MINE on source

[oracle@source reptest1]$ ./reptest1-run-source.sh
Initializing......done
DDC loaded from database (391 variables).
Dbvisit Replicate version 2.8
Copyright (C) Dbvisit Software Limited.  All rights reserved.
DDC file /home/oracle/reptest1/reptest1-MINE.ddc loaded.
Starting process MINE...Created directory /home/oracle/reptest1/ddc_backup
Created directory /home/oracle/reptest1/log/trace/
Created directory /home/oracle/reptest1/mine/
started
[oracle@source reptest1]$

 

NOTE

Before you start an APPLY process check where DBVisit replicate will store Oracle's data on Hadoop.. During step `Step 4 - Process Configuration` you can manage location by variables APPLY_FILEOUT_STAGING_DIR and APPLY_FILEOUT_TARGET_DIR. In the output of step for you'll see recapitulation:

[APPLY_FILEOUT_STAGING_DIR]: Staging HDFS directory for output files: /replicate/Hadoop/out_stage/stage-%O.%A-%I_%C_%X.%E
[APPLY_FILEOUT_TARGET_DIR]: Target HDFS directory for output files: /replicate/Hadoop/out_target/%O.%A-%I_%C_%X.%E

Ensure you have created /replicate directory in the Hadoop too.

If you hit error: FATAL-8001: Internal error: load of libhdfs.so failed: Inappropriate ioctl for device at expression handle at apply_file.c:961

Add libjvm* library to $LD_LIBRARY_PATH variable.


Starting APPLY on target

[oracle@target reptest1]$ ./reptest1-run-target.sh
Initializing......done
DDC loaded from database (392 variables).
Dbvisit Replicate version 2.8
Copyright (C) Dbvisit Software Limited.  All rights reserved.
DDC file /home/oracle/reptest1/reptest1-APPLY.ddc loaded.
Starting process APPLY...Created directory /home/oracle/reptest1/ddc_backup
Created directory /home/oracle/reptest1/log/
Created directory /home/oracle/reptest1/log/trace/
Created directory /home/oracle/reptest1/apply
started
[oracle@target reptest1]$

 

Dbvisit Replicate has now been configured and started in the background. To monitor replication progress start the console on source by executing start-console.sh script.

[oracle@source reptest1]$ ./start-console.sh
Initializing......done
DDC loaded from database (395 variables).
Dbvisit Replicate version 2.8
Copyright (C) Dbvisit Software Limited.  All rights reserved.
\ Dbvisit Replicate 2.8 (MAX edition) - Evaluation License expires in 30 days
MINE IS running. Currently at plog 229 and SCN 2223644 (11/18/2015 15:13:12).
APPLY IS running. Currently at plog 229 and SCN 2223626 (11/18/2015 15:13:08).

 

When starting the MINE and APPLY processes for the first time they enter the INITIALIZATION state. In this phase Dbvisit Replicate replicates the contents of its internal dictionary and repository from the source to the target schema (in this case dbvrep2). To monitor the progress of initialization, the LIST PROGRESS ALL command can be used.

Replication proper does not commence until the initialization phase has been completed. This is completed when the console show both MINE and APPLY are running, as per the following:

MINE IS running. Currently at plog 229 and SCN 2224596 (11/18/2015 15:16:47).
APPLY IS running. Currently at plog 229 and SCN 2224582 (11/18/2015 15:16:45).

.