Oracle to MySQL Replication Setup Example 01

[oracle@source replicate]$ ./dbvrep
Initializing......done
Dbvisit Replicate version 2.8
Copyright (C) Dbvisit Software Limited.  All rights reserved.
No DDC file loaded.
Run "setup wizard" to start the configuration wizard or try "help" to see all commands available.
dbvrep>

dbvrep> setup wizard

This wizard configures Dbvisit Replicate.
The setup wizard creates configuration scripts, which need to be run after the wizard ends. No changes to the databases are made before that.
The progress is saved every time a list of databases, replications, etc. is shown. It will be re-read if wizard is restarted and the same DDC name and script path
is selected.
Run the wizard now? [Yes]
Accept end-user license agreement? (View/Yes/No) [View] Yes
Before starting the actual configuration, some basic information is needed. The DDC name and script path determines where all files created by the wizard go (and
where to reread them if wizard is rerun) and the license key determines which options are available for this configuration.
(DDC_NAME) - Please enter a name for this replication: [] reptest1
(LICENSE_KEY) - Please enter your license key: [(trial)]
Which Replicate edition do you want to trial (LTD/XTD/MAX): [MAX]
(SETUP_SCRIPT_PATH) - Please enter a directory for location of configuration scripts on this machine: [/u01/app/oracle/reptest1] /home/oracle/reptest1
Network configuration files were detected on this system in these locations:
/u01/app/oracle/product/11.2.0/xe/network/admin
(TNS_ADMIN) - Please enter TNS configuration directory for this machine: [/u01/app/oracle/product/11.2.0/xe/network/admin]

Step 1 - Describe databases
========================================
The first step is to describe databases used in the replication. There are usually two of them (source and target).
Store SYSDBA and DBA passwords? Passwords only required during setup and initialization? (Yes/No) [Yes]
Let's configure the database, describing its type, connectivity, user names etc.
What type of database is this? (Oracle/MySQL/SQL Server/Oracle AWS RDS/CSV/Hadoop): [Oracle]
Please enter database TNS alias: [] ttorcl_src
Please enter SYSDBA user name: [SYS]
Please enter password for this user: [change_on_install]
Please enter user with DBA role: [SYSTEM]
Please enter password for this user: [manager]
Connecting to database ttorcl_src as SYSTEM to query list of tablespaces and to detect ASM (by looking whether any redo logs or archived logs are stored in ASM).
Enter the Dbvisit Replicate owner (this user will be created by this script): [dbvrep]
Please enter password for this user: [dbvpasswd]
Permanent tablespaces detected on the database: REPOE_DATA, USERS.
Please enter default permanent tablespace for this user: [REPOE_DATA] USERS
Temporary tablespaces detected on the database: TEMP.
Please enter default temporary tablespace for this user: [TEMP]
Following databases are now configured:
1: Oracle ttorcl_src, SYS/***, SYSTEM/***, dbvrep/***, USERS/TEMP, dbvrep/, ASM:No, TZ: +00:00
Enter the number of the database to modify it, or "add", or "done": [add]
Let's configure the database, describing its type, connectivity, user names etc.
What type of database is this? (Oracle/MySQL/SQL Server/Oracle AWS RDS/CSV/Hadoop): [Oracle] MySQL
Please enter database hostname: [localhost] target
Please enter user name of an administrator: [root]
Please enter password for this user: [] ******
Enter the user to log into apply database: [root]
Please enter password for this user: [password] ******
Enter the database (schema) to use for Dbvisit Replicate internal data (will be created in the script): [dbvrep]
Following databases are now configured:
1: Oracle ttorcl_src, SYS/***, SYSTEM/***, dbvrep/***, USERS/TEMP, dbvrep/, ASM:No, TZ: +00:00
2: MySQL host=target, root/***, root/***, root/***, /, dbvrep/, ASM:n/a, TZ:
Enter the number of the database to modify it, or "add", or "done": [done]

Step 2 - Replication pairs
========================================
The second step is to set source and targets for each replication pair.
Let's configure the replication pair, selecting source and target.
Following databases are described:
1: ttorcl_src (Oracle)
2: host=target (MySQL) (cannot be source, is not Oracle)
Select source database: [1]
Select target database: [2]
Will limited DDL replication be enabled? (Yes/No) [Yes] No
Use fetcher to offload the mining to a different server? (Yes/No) [No]
Should where clauses (and change auditing) include all columns, not just changed and PK? (Yes/No) [No]
Would you like to encrypt the data across the network? (Yes/No) [No]
Would you like to compress the data across the network? (Yes/No) [No]
How long do you want to set the network timeouts. Recommended range between 60-300 seconds [60]
Lock and copy the data initially one-by-one or at a single SCN?
one-by-one : Lock tables one by one and capture SCN
single-scn : One SCN for all tables
ddl-only   : Only DDL script for target objects
resetlogs  : Use SCN from last resetlogs operation (standby activation, rman incomplete recovery)
no-lock    : Do not lock tables. Captures previous SCN of oldest active transaction. Requires pre-requisite running of pre-all.sh script                             
(one-by-one/single-scn/ddl-only/resetlogs/no-lock) [single-scn]
What data instantiation script to create?
ddl_file       : DDL file created (APPLY.sql)
ddl_run        : DDL is automatically executed on target
load           : All replicated data is created and loaded automatically
none                                                                                                                                                                 
(ddl_file/ddl_run/load/none) [ddl_file] none
Following replication pairs are now configured:
1: ttorcl_src (Oracle) ==> host=target (MySQL), DDL: NO, fetcher: No, process suffix: (no suffix), compression: No, encryption: No, network timeout: 60, prepare
type: single-scn, data load: none
Enter number of replication pair to modify it, or "add", or "done": [done]

Step 3 - Replicated tables
========================================
The third step is to choose the schemas and tables to be replicated. If the databases are reachable, the tables are checked for existence, datatype support, etc.,
schemas are queried for tables. Note that all messages are merely hints/warnings and may be ignored if issues are rectified before the scripts are actually
executed.
Following tables are defined for replication pairs:
1: ttorcl_src (Oracle) ==> host=target (MySQL), DDL: NO, suffix: (no suffix), prepare: single-scn
  No tables defined.
Enter number of replication pair to modify it, or "done": [1]
Please enter list of all individual tables to be replicated. Enter schema name(s) only to replicate all tables in that schema. Use comma or space to delimit the
entries.
Enter the tables and schemas: [] HR.TEST1, HR.TEST2
Selected tables: HR.TEST1,HR.TEST2
Add more tables or schemas? (Yes/No) [No]
You can also specify some advanced options:
1. Exclude some tables from schema-level replication
2. Rename schemas or tables.
3. Specify filtering conditions.
4. (Tables only) Configure Change Data Capture; this does not maintain a copy of the source table, but logs all operations as separate entries. This is useful for
ETL or as an audit trail. This usually requires adding of new columns (timestamps, old/new values etc.) to the target table.
Specify rename name, filter condition or audit for any of the specified tables? (Yes/No) [No] Yes
Rename SCHEMA name for HR.TEST1 (empty means no rename): [] HR_TARGET
Rename TABLE name for HR.TEST1 (empty means no rename): [] TEST1_TARGET
Filter the data to be replicated? If yes, use single condition for ALL DML operations or use CUSTOM conditions for each operation? (No/ALL/CUSTOM) [No]
Configure change auditing? (No/Yes) [No]
Rename SCHEMA name for HR.TEST2 (empty means no rename): [] TEST2_TARGET
Rename TABLE name for HR.TEST2 (empty means no rename): []
Filter the data to be replicated? If yes, use single condition for ALL DML operations or use CUSTOM conditions for each operation? (No/ALL/CUSTOM) [No]
Configure change auditing? (No/Yes) [No]
Following tables are defined for replication pairs:
1: ttorcl_src (Oracle) ==> host=target (MySQL), DDL: NO, suffix: (no suffix), prepare: single-scn
  HR.TEST1 => HR_TARGET.TEST1_TARGET, HR.TEST2 => TEST2_TARGET.TEST2
Enter number of replication pair to modify it, or "done": [done]

Step 4 - Process configuration
========================================
The fourth step is to configure the replication processes for each replication.
Following processes are defined:
1: MINE on ttorcl_src
  Not configured.
2: APPLY on host=target
  Not configured.
Enter number of process to modify it, or "done": [1]
Fully qualified name of the server for the process (usually co-located with the database, unless mine is offloaded using fetcher): [source]
Server type (Windows/Linux/Unix): [Linux]
Enable email notifications about problems? (Yes/No) [No]
Enable SNMP traps/notifications about problems? (Yes/No) [No]
Directory with DDC file and default where to create log files etc. (recommended: same as global setting, if possible)? [/home/oracle/reptest1]
Following settings were pre-filled with defaults or your reloaded settings:
----------------------------------------
[MINE_REMOTE_INTERFACE]: Network remote interface: source:7901
[MINE_DATABASE]: Database TNS: ttorcl_src
[TNS_ADMIN]: tnsnames.ora path: /u01/app/oracle/product/11.2.0/xe/network/admin
[MINE_PLOG]: Filemask for generated plogs: /home/oracle/reptest1/mine/%S.%E (%S is sequence, %T thread, %F original filename (stripped extension), %P process type,
%N process name, %E default extension)
[LOG_FILE]: General log file: /home/oracle/reptest1/log/dbvrep_%N_%D.%E
[LOG_FILE_TRACE]: Error traces: /home/oracle/reptest1/log/trace/dbvrep_%N_%D_%I_%U.%E
Checking that these settings are valid...
Do you want to change any of the settings? [No]
Following processes are defined:
1: MINE on ttorcl_src
  Host: source, SMTP: No, SNMP: No
2: APPLY on host=target
  Not configured.
Enter number of process to modify it, or "done": [2]
Fully qualified name of the server for the process (usually co-located with the database, unless mine is offloaded using fetcher): [target]
Server type (Windows/Linux/Unix): [] Linux
Enable email notifications about problems? (Yes/No) [No]
Enable SNMP traps/notifications about problems? (Yes/No) [No]
Directory with DDC file and default where to create log files etc. (recommended: same as global setting, if possible)? [/home/oracle/reptest1]
Following settings were pre-filled with defaults or your reloaded settings:
----------------------------------------
[APPLY_REMOTE_INTERFACE]: Network remote interface: target:7902
[APPLY_DATABASE]: Database MySQL connection string: database=dbvrep;host=target
[TNS_ADMIN]: tnsnames.ora path: /u01/app/oracle/product/11.2.0/xe/network/admin
[APPLY_SCHEMA]: Dbvisit Replicate database (schema): dbvrep
[APPLY_STAGING_DIR]: Directory for received plogs: /home/oracle/reptest1/apply
[LOG_FILE]: General log file: /home/oracle/reptest1/log/dbvrep_%N_%D.%E
[LOG_FILE_TRACE]: Error traces: /home/oracle/reptest1/log/trace/dbvrep_%N_%D_%I_%U.%E
Checking that these settings are valid...
Do you want to change any of the settings? [No]
Following processes are defined:
1: MINE on ttorcl_src
  Host: source, SMTP: No, SNMP: No
2: APPLY on host=target
  Host: target, SMTP: No, SNMP: No
Enter number of process to modify it, or "done": [done]
Created file /home/oracle/reptest1/reptest1-APPLY.ddc.
Created file /home/oracle/reptest1/reptest1-MINE.ddc.
Created file /home/oracle/reptest1/config/reptest1-setup.dbvrep.
Created file /home/oracle/reptest1/config/reptest1-dbsetup_ttorcl_src_dbvrep.sql.
Created file /home/oracle/reptest1/config/reptest1-dbsetup_database_dbvrep_host_target_root.sql.
Created file /home/oracle/reptest1/config/reptest1-grants_ttorcl_src_dbvrep.sql.
Created file /home/oracle/reptest1/config/reptest1-grants_database_dbvrep_host_target_root.sql.
Created file /home/oracle/reptest1/config/reptest1-onetime.ddc.
Created file /home/oracle/reptest1/start-console.sh.
Created file /home/oracle/reptest1/reptest1-run-source.sh.
Created file /home/oracle/reptest1/scripts/reptest1-source-start-MINE.sh.
Created file /home/oracle/reptest1/scripts/reptest1-source-stop-MINE.sh.
Created file /home/oracle/reptest1/scripts/reptest1-source-dbvrep-MINE.sh.
Created file /home/oracle/reptest1/scripts/systemd-dbvrep-MINE_reptest1.service.
Created file /home/oracle/reptest1/scripts/upstart-dbvrep-MINE_reptest1.conf.
Created file /home/oracle/reptest1/reptest1-run-target.sh.
Created file /home/oracle/reptest1/scripts/reptest1-target-start-APPLY.sh.
Created file /home/oracle/reptest1/scripts/reptest1-target-stop-APPLY.sh.
Created file /home/oracle/reptest1/scripts/reptest1-target-dbvrep-APPLY.sh.
Created file /home/oracle/reptest1/scripts/systemd-dbvrep-APPLY_reptest1.service.
Created file /home/oracle/reptest1/scripts/upstart-dbvrep-APPLY_reptest1.conf.
Created file /home/oracle/reptest1/Nextsteps.txt.
Created file /home/oracle/reptest1/reptest1-all.sh.
===================================================================================================================================================================
Dbvisit Replicate wizard completed
Script /home/oracle/reptest1/reptest1-all.sh created. This runs all the above created scripts. Please exit out of dbvrep, review and run script as current user to
setup and start Dbvisit Replicate.
===================================================================================================================================================================
Optionally, the script can be invoked now by this wizard.
Run this script now? (Yes/No) [No]

[oracle@source reptest1]$ ./reptest1-all.sh
Setting up Dbvisit Replicate configuration
Configure database ttorcl_src...
This check fails if the DBID is not the expected one...

Ok, check passed.
Configure database database=dbvrep
Warning: Using a password on the command line interface can be insecure.
Object grants for database ttorcl_src...
Object grants for database database=dbvrep
Warning: Using a password on the command line interface can be insecure.
Setting up the configuration
Initializing......done
DDC loaded from database (0 variables).
Dbvisit Replicate version 2.8
Copyright (C) Dbvisit Software Limited.  All rights reserved.
DDC file /home/oracle/reptest1/config/reptest1-onetime.ddc loaded.
MINE: Cannot determine Dbvisit Replicate dictionary version. (no
dictionary exists)
APPLY: Cannot determine Dbvisit Replicate dictionary version. (no
dictionary exists)
dbvrep> #clear the no-DDC-DB-available warning
dbvrep> process clear previous warnings
dbvrep> set ON_WARNING SKIP
Variable ON_WARNING set to SKIP for process *.
dbvrep> set ON_ERROR EXIT
Variable ON_ERROR set to EXIT for process *.
dbvrep>
dbvrep> # Configuring default processes
dbvrep> choose process MINE
Process type MINE set to: MINE.
dbvrep> choose process APPLY
Process type APPLY set to: APPLY.
dbvrep> PROCESS SWITCH_REDOLOG
Redo log switch requested.dbvrep> PROCESS SETUP MINE DROP DICTIONARY
0 dictionary objects dropped.
dbvrep> PROCESS SETUP MINE CREATE DICTIONARY
dbvrep> PROCESS SETUP MINE LOAD DICTIONARY
Oldest active transaction SCN: 2238258 (no active transaction)
Supplemental logging on database set.
Loading dictionary table DBRSCCOL$
Loading dictionary table DBRSCDEF$
Loading dictionary table DBRSCOL$
Loading dictionary table DBRSIND$
Loading dictionary table DBRSOBJ$
Loading dictionary table DBRSTAB$
Loading dictionary table DBRSUSER$
Loading dictionary table DBRSV_$DATABASE
dbvrep> PROCESS SETUP APPLY DROP DICTIONARY
0 dictionary objects dropped.
dbvrep> PROCESS SETUP APPLY CREATE DICTIONARY
dbvrep> PROCESS SETUP APPLY LOAD DICTIONARY
dbvrep> PROCESS SETUP PAIR MINE AND APPLY
Applier SCN set (start=2238267, current=2238267).
dbvrep> SET APPLY.INSTANTIATE_SCN NOW
Variable INSTANTIATE_SCN set to NOW for process APPLY.
dbvrep> SET MINE._PREPARE_SUPLOG_TYPE PK
Variable _PREPARE_SUPLOG_TYPE set to PK for process MINE.
dbvrep> EXCLUDE CREATE TABLE %.DBMS_TABCOMP_TEMP_UNCMP #Ignore tables
created by Compression Advisor
Exclude rule created.
dbvrep> EXCLUDE CREATE TABLE %.DBMS_TABCOMP_TEMP_CMP #Ignore tables
created by Compression Advisor
Exclude rule created.
dbvrep> EXCLUDE CREATE TABLE %.SCHEDULER$_% #Ignore tables created by
Oracle scheduler (also used by schema/full expdp/impdp)
Exclude rule created.
dbvrep> EXCLUDE CREATE TABLE %.CMP1$% #Ignore tables created by
Compression Advisor since 11.2.0.4
Exclude rule created.
dbvrep> EXCLUDE CREATE TABLE %.CMP2$% #Ignore tables created by
Compression Advisor since 11.2.0.4
Exclude rule created.
dbvrep> EXCLUDE CREATE TABLE %.CMP3$% #Ignore tables created by
Compression Advisor since 11.2.0.4
Exclude rule created.
dbvrep> EXCLUDE CREATE TABLE %.CMP4$% #Ignore tables created by
Compression Advisor since 11.2.0.4
Exclude rule created.
dbvrep> PROCESS SUPPLEMENTAL LOGGING TABLE "HR"."TEST1" ENABLE PRIMARY
KEY
dbvrep> PROCESS SUPPLEMENTAL LOGGING TABLE "HR"."TEST2" ENABLE PRIMARY
KEY
dbvrep> PROCESS SWITCH_REDOLOG
Redo log switch requested.dbvrep> PROCESS WAIT_SCN_FLIP
Waited 3 seconds until scn_to_timestamp changed.
dbvrep> #single-scn instantiation: lock all tables and schemas
dbvrep> PROCESS LOCK TABLES "HR"."TEST1" "HR"."TEST2"
Locking all tables.
Lock done.
dbvrep> #single-scn instantiation: unlock all tables and schemas, but
keep the SCN
dbvrep> PROCESS LOCK RELEASE LOCKS
dbvrep>
dbvrep> #prepare the tables (we use OFFLINE as neither MINE nor APPLY
is running; with OFFLINE we won't wait on network timeout)
dbvrep> PREPARE OFFLINE TABLE "HR"."TEST1" NODDL RENAME TO
"HR_TARGET"."TEST1_TARGET"
Table HR.TEST1 instantiated at SCN 2239997
dbvrep> PREPARE OFFLINE TABLE "HR"."TEST2" NODDL RENAME TO
"TEST2_TARGET"."TEST2"
Table HR.TEST2 instantiated at SCN 2239997
dbvrep> #single-scn instantiation: unlock all tables and schemas,
forget the SCN (so it does not affect any further PREPARE statements)
dbvrep> PROCESS LOCK CLEAR SCN
dbvrep> PROCESS SWITCH_REDOLOG
Redo log switch requested.dbvrep> #prepare script for instantiation
dbvrep> create ddcdb from ddcfile
DDC loaded into database (391 variables).
dbvrep> set ON_WARNING SKIP
Variable ON_WARNING set to SKIP for process *.
dbvrep> set ON_ERROR SKIP
Variable ON_ERROR set to SKIP for process *.
OK-0: Completed, but some operation(s) raised warnings and/or errors.
WARN-1850: No DDC DB available, dictionary table does not exist.
WARN-9247: Column HR.TEST1.TEST_FLOAT is of type FLOAT on mine, but
HR_TARGET.TEST1_TARGET.TEST_FLOAT is DOUBLE on apply.
WARN-9247: Column HR.TEST1.TEST_LONG is of type LONG on mine, but
HR_TARGET.TEST1_TARGET.TEST_LONG is TEXT on apply.
WARN-9247: Column HR.TEST1.TEST_NCHAR is of type NCHAR on mine, but
HR_TARGET.TEST1_TARGET.TEST_NCHAR is VARCHAR on apply.
WARN-9247: Column HR.TEST1.TEST_NVARCHAR2 is of type NVARCHAR2 on mine,
but HR_TARGET.TEST1_TARGET.TEST_NVARCHAR2 is VARCHAR on apply.
WARN-9247: Column HR.TEST1.TEST_RAW is of type RAW on mine, but
HR_TARGET.TEST1_TARGET.TEST_RAW is VARBINARY on apply.
WARN-9083: Table TEST2_TARGET.TEST2 not found at apply.
WARN-1645: No columns found for TEST2_TARGET.TEST2, check privileges
for Dbvisit Replicate dictionary user at apply or create the target
table if it does not exist - before you start the apply.
WARN-9246: Column HR.TEST2.TEST_LONG_RAW is not in table
TEST2_TARGET.TEST2 on apply.
WARN-9246: Column HR.TEST2.TEST_NUMBER is not in table
TEST2_TARGET.TEST2 on apply.
These steps are required after the reptest1-all.sh script runs:
1) Create the necessary directory(ies) on the servers:
target: /home/oracle/reptest1
2) Copy the DDC files to the server(s) where the processes will run:
target: /home/oracle/reptest1/reptest1-APPLY.ddc
source: /home/oracle/reptest1/reptest1-MINE.ddc
3) Review that path to dbvrep executable is correct in the run scripts:
/home/oracle/reptest1/reptest1-run-source.sh
/home/oracle/reptest1/reptest1-run-target.sh
4) Copy the run script to the server(s) where the processes will run:
source: /home/oracle/reptest1/reptest1-run-source.sh
target: /home/oracle/reptest1/reptest1-run-target.sh
5) Ensure firewall is open for listen interfaces 0.0.0.0:7902, 0.0.0.0:7901 used by the processes.
6) Ensure firewall is open for remote access to mysql (usually port 3306) on target.
7) Make sure the data on apply are in sync as of time when setup was run.
8) Start the replication processes on all servers:
source: /home/oracle/reptest1/reptest1-run-source.sh
target: /home/oracle/reptest1/reptest1-run-target.sh
9) Start the console to monitor the progress:
/home/oracle/reptest1/start-console.sh
The above list is stored in /home/oracle/reptest1/Nextsteps.txt.

[oracle@source reptest1]$ ./reptest1-run-source.sh
Initializing......done
DDC loaded from database (391 variables).
Dbvisit Replicate version 2.8
Copyright (C) Dbvisit Software Limited.  All rights reserved.
DDC file /home/oracle/reptest1/reptest1-MINE.ddc loaded.
Starting process MINE...Created directory /home/oracle/reptest1/ddc_backup
Created directory /home/oracle/reptest1/log/trace/
Created directory /home/oracle/reptest1/mine/
started
[oracle@source reptest1]$

[oracle@target reptest1]$ ./reptest1-run-target.sh
Initializing......done
DDC loaded from database (392 variables).
Dbvisit Replicate version 2.8
Copyright (C) Dbvisit Software Limited.  All rights reserved.
DDC file /home/oracle/reptest1/reptest1-APPLY.ddc loaded.
Starting process APPLY...Created directory /home/oracle/reptest1/ddc_backup
Created directory /home/oracle/reptest1/log/
Created directory /home/oracle/reptest1/log/trace/
Created directory /home/oracle/reptest1/apply
started
[oracle@target reptest1]$

[oracle@source reptest1]$ ./start-console.sh
Initializing......done
DDC loaded from database (395 variables).
Dbvisit Replicate version 2.8
Copyright (C) Dbvisit Software Limited.  All rights reserved.
\ Dbvisit Replicate 2.8 (MAX edition) - Evaluation License expires in 30 days
MINE IS running. Currently at plog 229 and SCN 2223644 (11/18/2015 15:13:12).
APPLY IS running. Currently at plog 229 and SCN 2223626 (11/18/2015 15:13:08).

MINE IS running. Currently at plog 229 and SCN 2224596 (11/18/2015 15:16:47).
APPLY IS running. Currently at plog 229 and SCN 2224582 (11/18/2015 15:16:45).