Introduction
This guide will provide you with an overview on getting started with Dbvisit Replicate and Amazon EC2 The scenario used in this guide can be illustrated as:
...
This document will provide you with an overview on the steps required to implement the above solution.
Source (MINE) Server
Dbvisit Replicate will be installed on the database server, from where the mine process will be configured.
...
- Server Name
- Server IP Address (Make sure that you have an internal and external IP address. Talk to your admin about firewalls.)
- Oracle (Unix User) password
- Source Database Name
- Source TNS Alias (Make sure it is located in local tnsnames.ora) also have the full path ready
- SYS password
- SYSTEM password
- List of Schemas or Objects you want to replicate
- Sufficient disk space to store Dbvisit Replicate Mine PLOGS (default location /home/oracle/<DDC>/)
For a list of system requirements:
System Requirements
Target (APPLY) Server
Assumption is that Oracle database is installed, up and running and has a sys/system user. Also a tablespace will need to be created. Note that in this guide the the Apply server is an Oracle Linux EC2 Instance in AWS. Note that the example initial export dump and import (instantiation) of the source data into the target will be performed from the source database server using Oracle Datapump over a database link. 'Regular' export/import may also be done. Before starting with the Dbvisit Replicate configuration it is recommended to make sure you have the following information ready with regards to the Apply server and process:
...
Ensure you have configured Security Group to allow the Replicate ports (7901-7904, 1521):
Dbvisit Replicate Installation/Configuration
Installing latest Dbvisit Replicate on the source on premise machine and the target EC2 instance:
...
Please note that the main configuration (Setup Wizard) will be executed from the source (Mine) server which is also the same server hosting the source data we want to replicate to EC2.
Starting the Dbvisit Replicate Configuration
Execute “dbvrep” as the Oracle Unix account on the source server
...
| Code Block | ||
|---|---|---|
| ||
Run "setup wizard" to start the configuration wizard or try "help" to see all commands available. dbvrep> setup wizard This wizard configures Dbvisit Replicate. The setup wizard creates configuration scripts, which need to be run after the wizard ends. No changes to the databases are made before that. The progress is saved every time a list of databases, replications, etc. is shown. It will be re-read if wizard is restarted and the same DDC name and script path is selected. Run the wizard now? [yes] Accept end-user license agreement? (view/yes/no) [view] yes Before starting the actual configuration, some basic information is needed. The DDC name and script path determines where all files created by the wizard go (and where to reread them if wizard is rerun) and the license key determines which options are available for this configuration. (DDC_NAME) - Please enter a name for this replication (suggestion: use the name of the source database): [] TEST-EC2 (LICENSE_KEY) - Please enter your license key: [(trial)] Which Replicate edition do you want to trial (LTD/XTD/MAX):[MAX] (SETUP_SCRIPT_PATH) - Please enter a directory for location of configuration scripts on this machine: [/home/oracle/TEST-EC2] /home/oracle/TEST Network configuration files were detected on this system in these locations: /u01/app/oracle/product/11.2.0/db_1/network/admin /u01/app/oracle/product/10.2.0/db_1/network/admin /u01/app/oracle/product/11.2.0/db_1/network/admin (TNS_ADMIN) - Please enter TNS configuration directory for this machine: [/u01/app/oracle/product/11.2.0/db_1/network/admin] Step 1 - Describe databases ======================================== The first step is to describe databases used in the replication. There are usually two of them (source and target). Store SYSDBA and DBA passwords? Passwords only required during setup and initialization? (yes/no) [yes] Let's configure the database, describing it's type, connectivity, user names etc. What type of database is this? (Oracle/MySQL/SQL Server/Oracle AWS RDS/CSV/Hadoop): [Oracle] Please enter database TNS alias: [] SOURCE_DB Please enter SYSDBA user name: [SYS] Please enter password for this user: [change_on_install] ****** Please enter user with DBA role: [SYSTEM] Please enter password for this user: [manager] ****** Connecting to database SOURCE_DB as SYSTEM to query list of tablespaces and to detect ASM (by looking whether any redo logs or archived logs are stored in ASM). Enter the Dbvisit Replicate owner (this user will be created by this script): [dbvrep] Please enter password for this user: [dbvpasswd] Permanent tablespaces detected on the database: USERS. Please enter default permanent tablespace for this user:[USERS] Temporary tablespaces detected on the database: TEMP. Please enter default temporary tablespace for this user:[TEMP] Following databases are now configured: 1: Oracle DZ11G, SYS/***, SYSTEM/***, dbvrep/***, USERS/TEMP, dbvrep/, ASM:NO, TZ: +00:00 Enter the number of the database to modify it, or "add", or "done": [add] Let's configure the database, describing it's type, connectivity, user names etc. What type of database is this? (Oracle/MySQL/SQL Server/Oracle AWS RDS/CSV/Hadoop): [Oracle] Please enter database TNS alias: [] EC2_TARGET Please enter SYSDBA user name: [SYS] Please enter password for this user: [change_on_install] ******* Please enter user with DBA role: [SYSTEM] Please enter password for this user: [manager] ********** Connecting to database EC2_TARGET as SYSTEM to query list of tablespaces and to detect ASM (by looking whether any redo logs or archived logs are stored in ASM). Enter the Dbvisit Replicate owner (this user will be created by this script): [dbvrep] Please enter password for this user: [dbvpasswd] Permanent tablespaces detected on the database: USERS. Please enter default permanent tablespace for this user: [USERS] Temporary tablespaces detected on the database: TEMP. Please enter default temporary tablespace for this user: [TEMP] Following databases are now configured: 1: Oracle SOURCE_DB, SYS/***, SYSTEM/***, dbvrep/***, USERS/TEMP, dbvrep/, ASM:NO, TZ: +00:00 2: Oracle EC2_TARGET, SYS/***, SYSTEM/***, dbvrep/***, USERS/TEMP, dbvrep/***, ASM:NO, TZ: +00:00 Enter the number of the database to modify it, or "add", or "done": [done] Step 2 - Replication pairs ======================================== The second step is to set source and targets for each replication pair. Let's configure the replication pair, selecting source and target. Following databases are described: 1: SOURCE_DB (Oracle) 2: EC2_TARGET (Oracle) Select source database: [1] Select target database: [2] Will be DDL replication enabled? [yes] Use fetcher to offload the mining to a different server? (yes/no) [no] Would you like to encrypt the data across the network (yes or no) [no] Would you like to compress the data across the network (yes or no) [no] How long do you want to set the network timeouts. Recommended range between 60-300 seconds [60] Lock and copy the data initially one-by-one or at a single SCN? one-by-one : Lock tables one by one and capture SCN single-scn : One SCN for all tables ddl-only : Only DDL script for target objects resetlogs : Use SCN from last resetlogs operation (standby activation, rman incomplete recovery) (one-by-one/single-scn/ddl-only/resetlogs) [single-scn] What data instantiation script to create? dp_networklink : Data Pump with network link. No export datafile is created (APPLY.sh) dp_exp : Data Pump with export datafile. Creates an export datafile (APPLY.sh) exp : Pre-datapump exp/imp with export datafile (APPLY.sh) ddl_file : DDL file created (APPLY.sql) ddl_run : DDL is automatically executed on target load : All replicated data is created and loaded automatically none (dp_networklink/dp_exp/exp/ddl_file/ddl_run/load/none) [dp_networklink] Following replication pairs are now configured: 1: SOURCE_DB (Oracle) ==> EC2_TARGET (Oracle), DDL: yes, fetcher: no, process suffix: (no suffix), compression: no, encryption: no, network timeout: 60, prepare type: single-scn, data load: dp_networklink Enter number of replication pair to modify it, or "add", or "done": [done] Step 3 - Replicated tables ======================================== The third step is to choose the schemas and tables to be replicated. If the databases are reachable, the tables are checked for existence, datatype support, etc., schemas are queried for tables. Note that all messages are merely hints/warnings and may be ignored if issues are rectified before the scripts are actually executed. Following tables are defined for replication pairs: 1: SOURCE_DB (Oracle) ==> EC2_TARGET (Oracle), DDL: yes, suffix: (no suffix), prepare: single-scn No tables defined. Enter number of replication pair to modify it, or "done": [1] Please enter list of all individual tables to be replicated. Enter schema name(s) only to replicate all tables in that schema. Use comma or space to delimit the entries. Enter the tables and schemas: [] SCOTT Add more tables or schemas? (YES/NO) [NO] NO To replicate changes of PL/SQL objects in schema(s), please enter the schemas to be replicated. Note that specifying any entry will cause additional privileges to be granted to dbvrep. Enter through a comma or space-delimited list. Enter the list of schemas (PL/SQL): [] You can also specify some advanced options: 1. Rename schemas or tables. 2. Specify filtering conditions. 3. (Tables only) Configure Change Data Capture; this does not maintain a copy of the source table, but logs all operations as separate entries. This is useful for ETL or as an audit trail. This usually requires adding of new columns (timestamps, old/new values etc.) to the target table. Specify rename name or filter condition for any of the specified schemas (YES/NO): [no] Following tables are defined for replication pairs: 1: SOURCE_DB (Oracle) ==> EC2_TARGET (Oracle), DDL: yes, suffix: (no suffix), prepare: single-scn SCOTT(tables) Enter number of replication pair to modify it, or "done": [done] Step 4 - Process configuration ======================================== The fourth step is to configure the replication processes for each replication. Following processes are defined: 1: MINE on SOURCE_DB Not configured. 2: APPLY on EC2_TARGET Not configured. Enter number of process to modify it, or "done": [1] Fully qualified name of the server for the process (usually co-located with the database, unless mine is offloaded using fetcher):[sourceserver.com] Server type (Windows/Linux/Unix): [Linux] Enable email notifications about problems (yes/no)? [no] Enable SNMP traps/notifications about problems (yes/no)?[no] Directory with DDC file and default where to create log files etc. (recommended: same as global setting, if possible)? [/home/oracle/TEST] Following settings were pre-filled with defaults or your reloaded settings: ---------------------------------------- [MINE_REMOTE_INTERFACE]: Network remote interface:sourceserver.com:7901 [MINE_DATABASE]: Database TNS: SOURCE_DB [TNS_ADMIN] tnsnames.ora path: /u01/app/oracle/product/11.2.0/db_1/network/admin [MINE_PLOG]: Filemask for generated plogs: /home/oracle/TEST/mine/%S.%E (%S is sequence, %T thread, %F original filename (stripped extension), %P process type, %N process name, %E default extension) [LOG_FILE]: General log file: /home/oracle/TEST/log/dbvrep_%N_%D.%E [LOG_FILE_TRACE]: Error traces: /home/oracle/TEST/log/trace/dbvrep_%N_%D_%I_%U.%E Checking that these settings are valid... Do you want change any of the settings? [no] Following processes are defined: 1: MINE on SOURCE_DB Host: sourceserver.com, SMTP: no, SNMP: no 2: APPLY on EC2_TARGET Not configured. Enter number of process to modify it, or "done": [2] Fully qualified name of the server for the process (usually co-located with the database, unless mine is offloaded using fetcher):[ip-10-47-141-210] ec2-54-82-3-111.compute-1.amazonaws.com Server type (Windows/Linux/Unix): [Linux] Enable email notifications about problems (yes/no)? [no] Enable SNMP traps/notifications about problems (yes/no)?[no] Directory with DDC file and default where to create log files etc. (recommended: same as global setting, if possible)? [/home/oracle/TEST] Following settings were pre-filled with defaults or your reloaded settings: ---------------------------------------- [APPLY_REMOTE_INTERFACE]: Network remote interface: ec2-54-82-3-111.compute-1.amazonaws.com:7902 [APPLY_DATABASE]: Database TNS: EC2_TARGET [TNS_ADMIN]: tnsnames.ora path: /u01/app/oracle/product/11.2.0/db_1/network/admin [APPLY_STAGING_DIR]: Directory for received plogs: /home/oracle/TEST/apply [LOG_FILE]: General log file: /home/oracle/TEST/log/dbvrep_%N_%D.%E [LOG_FILE_TRACE]: Error traces: /home/oracle/TEST/log/trace/dbvrep_%N_%D_%I_%U.%E Checking that these settings are valid... Do you want change any of the settings? [no] no Following processes are defined: 1: MINE on SOURCE_DB Host: sourceserver.com, SMTP: no, SNMP: no 2: APPLY on EC2_TARGET Host: ec2-54-82-3-111.compute-1.amazonaws.com, SMTP: no, SNMP: no Enter number of process to modify it, or "done": [done] Created file /home/oracle/TEST/TEST-EC2-APPLY.ddc. Created file /home/oracle/TEST/TEST-EC2-MINE.ddc. Created file /home/oracle/TEST/config/TEST-EC2-setup.dbvrep. Created file /home/oracle/TEST/config/TEST-EC2-dbsetup_SOURCE_DB.sql. Created file /home/oracle/TEST/config/TEST-EC2-dbsetup_EC2_TARGET.sql. Created file /home/oracle/TEST/config/TEST-EC2-grants_SOURCE_DB.sql. Created file /home/oracle/TEST/config/TEST-EC2-grants_EC2_TARGET.sql. Created file /home/oracle/TEST/config/TEST-EC2-onetime.ddc. Created file /home/oracle/TEST/start-console.sh. Created file /home/oracle/TEST/TEST-EC2-run-sourceserver.com.sh. Created file /home/oracle/TEST/TEST-EC2-run-ec2-54-82-3-111.compute-1.amazonaws.com.sh. Created file /home/oracle/TEST/Nextsteps.txt. Created file /home/oracle/TEST/TEST-EC2-all.sh. ======================================================================================================================================================================================================== Dbvisit Replicate wizard completed Script /home/oracle/TEST/TEST-EC2-all.sh created. This runs all the above created scripts. Please exit out of dbvrep, review and run script as current user to setup and start Dbvisit Replicate. ======================================================================================================================================================================================================== Optionally, the script can be invoked now by this wizard. Run this script now? (yes/no) [NO] dbvrep> |
Review APPLY DDC file
Ensure you have the correct TNS_ADMIN details in the APPLY ddc file. In this test example environment Oracle Express is used on the APPLY server. The location of the TNS_ADMIN variable in the TEST-EC2-APPLY.ddc is:
...
| Code Block | ||
|---|---|---|
| ||
memory_set TNS_ADMIN /u01/app/oracle/product/11.2.0/xe/network/admin |
Execute TEST-EC2-all.sh script on sourceserver.com
Run the TEST-EC2-all.sh script to initialize Dbvisit Replicate and load the dbvrep repository in both the source and target. It is important that the steps above be executed before running this step.
...
| Code Block | ||
|---|---|---|
| ||
[oracle@ ]$ scp /home/oracle/TEST/TEST-EC2-run-ec2-54-82-3-111.compute-1.amazonaws.com.sh ec2-54-82-3-111.compute-1.amazonaws.com:/home/oracle/TEST/ [oracle@ ]$ scp /home/oracle/TEST/TEST-EC2-APPLY.ddc ec2-54-82-3-111.compute-1.amazonaws.com:/home/oracle/TEST/ |
Instantiate Data (Import data into EC2)
The first step is to create the required database link. This link is required to pull the data from the source database into the target database. . For this database link the SYSTEM account is used.
...
| Code Block | ||
|---|---|---|
| ||
SQL> create public database link XX connect to system identified by manager using '(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = SOURCESERVER.COM)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = SOURCE_DB)))';
Database link created.
SQL> select * from v$instance@SOURCE_DB;
INSTANCE_NUMBER INSTANCE_NAME
--------------- ----------------
HOST_NAME
----------------------------------------------------------------
VERSION STARTUP_T STATUS PAR THREAD# ARCHIVE LOG_SWITCH_WAIT
----------------- --------- ------------ --- ---------- ------- ---------------
LOGINS SHU DATABASE_STATUS INSTANCE_ROLE ACTIVE_ST BLO EDITION
---------- --- ----------------- ------------------ --------- --- -------
1 SOURCE_DB
sourceserver.com
11.2.0.2.0 30-SEP-13 OPEN NO 1 STARTED
ALLOWED NO ACTIVE PRIMARY_INSTANCE NORMAL NO XE
|
Start the import process
To import the data into the EC2 database you will need to use Datapump using the database links.
...
| Code Block | ||
|---|---|---|
| ||
[oracle\]$ cat APPLY.sh impdp system/xxxx@EC2_TARGET table_exists_action=TRUNCATE network_link=XX directory=DATA_PUMP_DIR flashback_scn=1229827 tables=SCOTT.EMP SCOTT.DEPT SCOTT.SALGRADE SCOTT.LOADTEST JOB_NAME=DP_EC2-TEST_0001 exclude=statistics [oracle@]$ ./APPLY.sh Import: Release 11.2.0.2.0 - Production on Mon Sep 30 15:45:45 2013 Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved. Connected to: Oracle Database 11g Release 11.2.0.2.0 - 64bit Production Starting "SYSTEM"."DP_C_0001": SYSTEM/********@EC2_TARGET table_exists_action=TRUNCATE network_link=XX directory=DATA_PUMP_DIR flashback_scn=1229827 tables=SCOTT.EMP,SCOTT.DEPT, SCOTT.SALGRADE, SCOTT.LOADTEST JOB_NAME=DP__0001 exclude=statistics Estimate in progress using BLOCKS method... Processing object type TABLE_EXPORT/TABLE/TABLE_DATA Total estimation using BLOCKS method: 102.6 MB Processing object type TABLE_EXPORT/TABLE/TABLE . . imported "SCOTT"."EMP" 14 rows . . imported "SCOTT"."DEPT" 4 rows . . imported "SCOTT"."SALGRADE" 5 rows . . imported "SCOTT"."LOADTEST" 100 rows Processing object type TABLE_EXPORT/TABLE/GRANT/OWNER_GRANT/OBJECT_GRANT Processing object type TABLE_EXPORT/TABLE/INDEX/INDEX Processing object type TABLE_EXPORT/TABLE/CONSTRAINT/CONSTRAINT Job "DBVADMIN"."DP__0001" successfully completed at 02:46:32 |
Start the Replication Processes
Mine Server - sourceserver.com
| Code Block | ||
|---|---|---|
| ||
[oracle@]$ ./TEST-EC2.sourceserver.com.sh Initializing......done [oracle@ ]$ DDC loaded from database (266 variables). Dbvisit Replicate version 2.6 Copyright (C) Dbvisit Software Limited. All rights reserved. DDC file /home/oracle/TEST/TEST=EC2-MINE.ddc loaded. Starting process MINE...Created directory /home/oracle/TEST/ddc_backup Created directory /home/oracle/TEST/log/ Created directory /home/oracle/TEST/log/trace/ Created directory /home/oracle/TEST/mine/ started |
Apply Server - EC2:
| Code Block | ||
|---|---|---|
| ||
[oracle@]$ ./TEST-EC2-run-ec2-54-82-3-111.compute-1.amazonaws.com.sh Initializing......done DDC loaded from database (266 variables). Dbvisit Replicate version 2.6 Copyright (C) Dbvisit Software Limited. All rights reserved. DDC file /home/oracle/TEST/TEST-EC2-APPLY.ddc loaded. Starting process APPLY...Created directory /home/oracle/TEST/ddc_backup Created directory /home/oracle/TEST/log/ Created directory /home/oracle/TEST/log/trace/ Created directory /home/oracle/TEST/apply started |
...
--------------------------------------------------------------------------------------------------------------------------------------------
Testing the replication
Insert some rows into the source tables.
...