Dbvisit Replicate from on-premise to Amazon RDS (Oracle)

Owned by Chris Lawless (Deactivated)
Last updated: Jun 14, 2014

Introduction

This guide will provide you with an overview on getting started with  Dbvisit Replicate and Amazon RDS.  The scenario used in this guide can be illustrated as:

The rest of the document will provide you with an overview on the steps required to implement the above solution.  In this example the On-Site database is also located in Amazon and is running on an EC2 instance.

Source (MINE) Server

Dbvisit Replicate will be installed on the database server, from where the mine process will be configured.  It is possible to run the apply process on the same server, but using an interim EC2 instance for the apply server is recommended.

 

Before starting the Dbvisit Replicate configuration, make sure you have the following details of the Source Database server where the mine process will be configured.  (Note that in this guide it is assumed that the source server is Unix based):

  • Server Name
  • Server IP Address
  • root (Unix User) Password
  • oracle (Unix User) Password
  • Source Database Name
  • Source TNS Alias (Make sure it is located in local tnsnames.ora)
  • Target Database Name (Amazon RDS Instance)
  • Target TNS Alias (Make sure it is located in local tnsnames.ora)
  • SYS (Oracle Super User) Password
  • SYSTEM (Oracle System User) Password
  • DBVREP (Dbvisit Repository) Password                       
  • List of Schemas or Objects you want to replicate
  • Sufficient disk space to store Dbvisit Replicate Mine PLOGS (default location /home/oracle/<DDC>/)

Target (APPLY) Server

In this example we recommend the use of an Amazon EC2 instance, however it is possible to configure the Apply process to also run from the same server as the Mine process.  Note that in this guide it is assumed that the Apply server is an Oracle Linux EC2 Instance in AWS.  This is done to offload the Apply processing to a different server than the one running the Mine.  Note that the initial export dump and import (instantiation) of the source data into the target will be performed from the source database server using Oracle Datapump over a database link.  Before starting with the Dbvisit Replicate configuration it is recommended to make sure you have the following information ready with regards to the Apply server and process:

  • EC2 - Server Name
  • Server IP Address (This example will use an Elastic IP)
  • root (Unix User) Password
  • oracle (Unix User) Password
    • Even though this system will not run an Oracle Database, we will make use of an Oracle Unix account as well as the Oracle Instance Client to establish connections between the source and target)
  • Source Database Name
  • Source TNS Alias (Make sure it is located in local tnsnames.ora)
  • Target Database Name (Amazon RDS Instance)
  • Target TNS Alias (Make sure it is located in local tnsnames.ora)
  • SYS (Oracle Super User) Password for Source
  • SYSTEM (Oracle System User) Password for Source
  • A DBA username and password in the Amazon RDS Instance
    • Note that in AWS RDS you do not have access to the SYS or SYSTEM user accounts and passwords.  You will need to make sure you create a DBA user account with the DBA role granted.  This is the user that Dbvisit Replicate will use during initial setup and configuration.
  • DBVREP (Dbvisit Repository) Password
    • Dbvisit Replicate has a repository in both source and target using the “dbvrep” user name by default.  This account is created during Dbvisit Replicate installation and configuration.       
  • Sufficient disk space to store Dbvisit Replicate Apply PLOGS (default location /home/oracle/<DDC>/)

The Amazon RDS (Target) Database

The first step is to create an Amazon Oracle RDS database environment.  Once the environment is created, make sure that you have created an additional DBA account, example “dbvdba” with the DBA role assigned to it.  This account is only used during installation and configuration.

The following information should be ready before you start the Dbvisit Replicate installation and configuration:

  • Amazon Oracle RDS instance created and configured
  • Tablespaces created as required, with sufficient space to hold replicated data
    • Dbvisit Replicate repository is small and can be stored in the default USERS tablespace.  Or if required can be stored in a different tablespace.
    • Make sure that you pre-create all the required tablespaces before you start the Dbvisit Replicate configuration.  Dbvisit Replicate will not create tablespaces and datafiles; this should be done as pre-requisite.
  • DBA User Account and Password of DBA user in RDS database
  • TNS Alias details required (tnsnames.ora entries to RDS instance should be available and added to the Mine and Apply server’s tnsnames.ora files and tested to ensure connections to the DBA user account (dbvdba) can be established.

 

TNS Entry in tnsnames.ora

 

RDSTGT =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = awsdb.cu1f3524nvj.us-east-1.rds.amazonaws.com)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = RDSTGT)
    )
  )

 

Creating Amazon EC2 Instances 

In this example we created an AWS EC2  that  will be configured to perform the APPLY related tasks inside the AWS environment.  In an ideal configuration this instance should be in the same region and availability zone as the RDS database environment.  Also ensure there is sufficient resources on this system.


EC2 Instance

This environment will only run an Oracle Instant Client installation with Dbvisit Replicate installed.

Environment Summary:

  • EC2 instance running Oracle Linux 6.3
  • Install Dbvisit Replicate 2.6 from RPM.
  • Oracle client
    • Oracle client or instant client can be used, but in this example, the Oracle Express installation was used.
    • Configure the TNS_ADMIN environment:
      • /u01/app/oracle/product/11.2.0/xe/network/admin
      • create tnsnames.ora file with details for the source (SOURCEDB) database and the target RDS database called “ RDSTGT”

 

SOURCEDB =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = dvrds01)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = SOURCEDB)
    )
  )
RDSTGT  =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = awsdb.cu1fdi2sdn7nvj.us-east-1.rds.amazonaws.com)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = RDSTGT)
    )
  )

 

Ensure the firewall on both environments (IPTABLES) as well as the AWS security groups is configures to allow communication between these systems.


Ensure you have configured Security Group to allow the Replicate ports (7901-7904, 1521).

 

Dbvisit Replicate Installation/Configuration

Installing latest Dbvisit Replicate on both dbvrds01 (source) and dbvrds02 (apply):

 

root@akl:[/root] # rpm -ivh dbvisit_replicate-2.7.08-1.x86_64.rpm
Preparing...                ########################################### [100%]
   1:dbvisit_replicate      ########################################### [100%]

 

Please note that the main configuration (Setup Wizard) will be executed from the source (Mine) server which is also the same server hosing the source data we want to replicate to RDS.

Starting the Dbvisit Replicate Configuration

Execute “dbvrep” as the Oracle Unix account from the dbvrds01 (source) server.

 

[oracle@dbvrds01 ~]$ dbvrep
Initializing......done
Dbvisit Replicate version 2.7
Copyright (C) Dbvisit Software Limited.  All rights reserved.
No DDC file loaded.
Run "setup wizard" to start the configuration wizard or try "help" to see all commands available.

 

Once the command line interface (dbvrep) started, type in “setup wizard” to start the configuration process (below is an example):

 

dbvrep> setup wizard
This wizard configures Dbvisit Replicate.

The setup wizard creates configuration
scripts, which need to be run after the wizard ends. No changes to the
databases are made before that.

The progress is saved every time a list of databases, replications, etc. is shown. It will be re-read if
wizard is restarted and the same DDC name and script path is selected.

Run the wizard now? [yes]

Accept end-user license agreement? (view/yes/no) [view] yes


Before starting the actual configuration, some basic information is needed. The DDC name and script path
determines where all files created by the wizard go (and where to reread them
if wizard is rerun) and the license key determines which options are available for this configuration.

(DDC_NAME) - Please enter a name for this replication (suggestion: use the name of the source database): [] TEST_RDS

(LICENSE_KEY) - Please enter your license key: [(trial)]

Which Replicate edition do you want to trial (LTD/XTD/MAX): [MAX]

(SETUP_SCRIPT_PATH) - Please enter a directory for location of configuration scripts on this machine: [/home/oracle/TEST_RDS]

Network configuration files were detected on this system in these locations:
/u01/app/oracle/product/11.2.0/db_1/network/admin
/u01/app/oracle/product/10.2.0/db_1/network/admin
/u01/app/oracle/product/11.2.0/db_1/network/admin
(TNS_ADMIN) - Please enter TNS configuration directory for this machine:
[/u01/app/oracle/product/11.2.0/db_1/network/admin]


Step 1 - Describe databases
========================================

The first step is to describe databases used in the replication. There are usually two of them (source and target).

Store SYSDBA and DBA passwords? Passwords only required during setup and initialization? (yes/no) [yes]

Let's configure the database, describing it's type, connectivity, user names etc.

What type of database is this?
(Oracle/MySQL/SQL Server/Oracle AWS RDS/CSV/Hadoop): [Oracle]

Please enter database TNS alias: [] SOURCEDB

Please enter SYSDBA user name: [SYS]

Please enter password for this user: [change_on_install] ******

Please enter user with DBA role: [SYSTEM]

Please enter password for this user: [manager] ******

Connecting to database
SOURCEDB as SYSTEM to query list of tablespaces and to detect ASM (by looking
whether any redo logs or archived logs are stored in ASM).

Enter the Dbvisit Replicate owner (this user will be created by this script): [dbvrep]

Please enter password for this user: [dbvpasswd]

Permanent tablespaces detected on the database: USERS.

Please enter default permanent tablespace for this user: [USERS]

Temporary tablespaces detected on the database: TEMP.

Please enter default temporary tablespace for this user: [TEMP]


Following databases are now configured:
1: Oracle SOURCEDB, SYS/***, SYSTEM/***, dbvrep/***, USERS/TEMP, dbvrep/, ASM:NO, TZ: +00:00

Enter the number of the database to modify it, or "add", or "done": [add]

Let's configure the database, describing it's type, connectivity, user names etc.

What type of database is this? (Oracle/MySQL/SQL Server/Oracle AWS RDS/CSV/Hadoop): [Oracle] Oracle AWS RDS
Please enter database TNS alias: [] RDSTGT

Specify AWS RDS instance DBA user Account [] rds_admin

Please enter password for this user: [] ***********

Connecting to database RDSTGT as rds_admin to query list of tablespaces and to detect ASM (by looking whether
any redo logs or archived logs are stored in ASM).

Enter the Dbvisit Replicate owner (this user will be created by this script): [dbvrep]
Please enter password for this user: [dbvpasswd]
Permanent tablespaces detected on the database: RDSADMIN, USERS.
Please enter default permanent tablespace for this user: [RDSADMIN] USERS
Temporary tablespaces detected on the database: TEMP.
Please enter default temporary tablespace for this user: [TEMP]

Following databases are now configured:

1: Oracle SOURCEDB, SYS/***,
SYSTEM/***, dbvrep/***, USERS/TEMP, dbvrep/, ASM:NO, TZ: +00:00
2: Oracle AWS RDS RDSTGT, rds_admin/***, rds_admin/***, dbvrep/***, USERS/TEMP, dbvrep/, ASM:NO, TZ: +00:00

Enter the number of the database to modify it, or "add", or "done": [done]

Step 2 - Replication pairs
========================================
The second step is to set source and targets for each replication pair.

Let's configure the replication pair, selecting source and target.
Following databases are described:
1: SOURCEDB (Oracle)
2: RDSTGT (Oracle AWS RDS)
Select source database: [1]
Select target database: [2]

Will be DDL replication enabled? [yes]

Use fetcher to offload the mining to a different server? (yes/no) [no]

Would you like to encrypt the data across the network (yes or no) [no]

Would you like to compress the data across the network (yes or no) [no]

How long do you want to set the network timeouts. Recommended range between 60-300 seconds [60]

Lock and copy the data initially one-by-one or at a single SCN?
one-by-one : Lock tables one by one and capture SCN
single-scn : One SCN for all tables
ddl-only   : Only DDL script for target objects
resetlogs  : Use SCN from last resetlogs operation (standby activation, rman incomplete recovery)  
no-lock    : Do not lock tables. Captures previous SCN of oldest active transaction. Requires pre-requisite running of pre-all.sh script                                                      
(one-by-one/single-scn/ddl-only/resetlogs/no-lock) [single-scn]

What data instantiation script to create?

dp_networklink : Data Pump with network link. No export datafile is created (APPLY.sh)
dp_exp         : Data Pump with export datafile. Creates an export datafile (APPLY.sh)
exp            : Pre-datapump exp/imp with export datafile (APPLY.sh)
ddl_file       : DDL file created (APPLY.sql)
ddl_run        : DDL is automatically executed on target
load           : All replicated data is created and loaded automatically
none                                                                                                                                                                                      
(dp_networklink/dp_exp/exp/ddl_file/ddl_run/load/none) [dp_networklink]


Following replication pairs are now configured:
1: SOURCEDB (Oracle) ==> RDSTGT (Oracle AWS RDS), DDL: yes, fetcher: no, process suffix: (no suffix),
compression: no, encryption: no, network timeout: 60, prepare type: single-scn, dataload: dp_networklink

Enter number of replication pair to modify it, or "add", or "done": [done]
Following replication pairs are now configured:


Step 3 - Replicated tables
========================================
The third step is to choose the schemas and tables to be replicated. If the databases are reachable, the
tables are checked for existence, datatype support, etc., schemas are queried
for tables. Note that all messages are merely hints/warnings and may be ignored if issues are rectified
before the scripts are actually executed.

Following tables are definedfor replication pairs:
1: SOURCEDB (Oracle) ==> RDSTGT (Oracle AWS RDS), DDL: yes, suffix: (no suffix), prepare: single-scn
  No tables defined.
Enter number of replication pair to modify it, or "done": [1]

Please enter list of all individual tables to be replicated. Enter schema name(s) only to replicate all
tables in that schema. Use comma or space to delimit the entries.

Enter the tables and schemas:[] SCOTT
Selected schemas: SCOTT
Add more tables or schemas? (YES/NO) [NO]


To replicate changes of PL/SQL objects in schema(s), please enter the schemas to be replicated. Note
that specifying any entry will cause additional privileges to be granted to dbvrep.
Enter through a comma or space-delimited list.
Enter the list of schemas (PL/SQL): []
You can also specify some advanced options:

1. Rename schemas or tables.
2. Specify filtering conditions.
3. (Tables only) Configure
Change Data Capture; this does not maintain a copy of the source table, but logs all operations as separate entries. This is useful for ETL or as an audittrail. This usually requires adding of new columns (timestamps, old/new values etc.) to the target table.

Specify rename name or filter condition for any of the specified schemas (YES/NO): [no]

Following tables are defined for replication pairs:

1: SOURCEDB (Oracle) ==> RDSTGT (Oracle AWS RDS), DDL: yes, suffix: (no suffix), prepare: single-scn
  SCOTT(tables)
Enter number of replication pair to modify it, or "done": [done]


Step 4 - Process configuration
========================================
The fourth step is to configure the replication processes for each replication. 
Following processes are defined:
1: MINE on SOURCEDB
  Not configured.
2: APPLY on RDSTGT
  Not configured.
Enter number of process to modify it, or "done": [1]

Fully qualified name of the server for the process (usually co-located with the database, unless mine is
offloaded using fetcher): [dbvlin210.dbvisit.co.nz]
Server type (Windows/Linux/Unix): [Linux]
Enable email notifications about problems (yes/no)? [no]
Enable SNMP traps/notifications about problems (yes/no)? [no]
Directory with DDC file and default where to create log files etc. (recommended: same as global setting, if
possible)? [/home/oracle/TEST_RDS]

Following settings were pre-filled with defaults or your reloaded settings:
[MINE_REMOTE_INTERFACE]: Network remote interface: dbvlin210.dbvisit.co.nz:7901
[MINE_DATABASE]: Database TNS: SOURCEDB
[TNS_ADMIN]: tnsnames.ora path: /u01/app/oracle/product/11.2.0/db_1/network/admin
[MINE_PLOG]: Filemask for generated plogs: /home/oracle/TEST_RDS/mine/%S.%E (%S is sequence, %T thread, %F original filename (stripped extension), %P process type, %N process name, %E default extension)
[LOG_FILE]: General log file: /home/oracle/TEST_RDS/log/dbvrep_%N_%D.%E
LOG_FILE_TRACE]: Error traces: /home/oracle/TEST_RDS/log/trace/dbvrep_%N_%D_%I_%U.%E
 
Checking that these settings are valid...
Do you want change any of the settings? [no]


Following processes are defined:
1: MINE on SOURCEDB
  Host: dbvlin210.dbvisit.co.nz, SMTP: no, SNMP: no
2: APPLY on RDSTGT
  Not configured.
Enter number of process to modify it, or "done": [2]

 
Fully qualified name of the server for the process (usually co-located with the database, unless mine is offloaded using fetcher): [ip-10-139-39-29] ec2-54-82-47-176.compute-1.amazonaws.com

Server type (Windows/Linux/Unix): [Linux]
Enable email notifications about problems (yes/no)? [no]
Enable SNMP traps/notifications about problems (yes/no)? [no]
Directory with DDC file and default where to create log files etc. (recommended: same as global setting, if
possible)? [/home/oracle/TEST_RDS]

Following settings were pre-filled with defaults or your reloaded settings:
----------------------------------------
[APPLY_REMOTE_INTERFACE]: Network remote interface: ec2-54-82-47-176.compute-1.amazonaws.com:7902
[APPLY_DATABASE]: Database TNS: RDSTGT
[TNS_ADMIN]: tnsnames.ora path: /u01/app/oracle/product/11.2.0/db_1/network/admin
[APPLY_STAGING_DIR]: Directory for received plogs: /home/oracle/TEST_RDS/apply
[LOG_FILE]: General log file: /home/oracle/TEST_RDS/log/dbvrep_%N_%D.%E
[LOG_FILE_TRACE]: Error traces: /home/oracle/TEST_RDS/log/trace/dbvrep_%N_%D_%I_%U.%E

Checking that these settings are valid...
Do you want change any of the settings? [no]
Following processes are defined:
1: MINE on SOURCEDB
  Host: dbvlin210.dbvisit.co.nz, SMTP: no, SNMP: no
2: APPLY on RDSTGT
  Host: ec2-54-82-47-176.compute-1.amazonaws.com, SMTP: no, SNMP: no

Enter number of process to modify it, or "done": [done]
Do you want change any of the settings? [no]


Following processes are defined:
1: MINE on SOURCEDB
  Host: dbvlin210.dbvisit.co.nz, SMTP: no, SNMP: no
2: APPLY on RDSTGT
  Host: ec2-54-82-47-176.compute-1.amazonaws.com, SMTP: no, SNMP: no

Enter number of process to modify it, or "done": [done]
Created file /home/oracle/TEST_RDS/TEST_RDS-APPLY.ddc.
Created file /home/oracle/TEST_RDS/TEST_RDS-MINE.ddc.
Created file /home/oracle/TEST_RDS/config/TEST_RDS-setup.dbvrep.
Created file /home/oracle/TEST_RDS/config/TEST_RDS-dbsetup_SOURCEDB.sql.
Created file /home/oracle/TEST_RDS/config/TEST_RDS-dbsetup_RDSTGT.sql.
Created file /home/oracle/TEST_RDS/config/TEST_RDS-grants_SOURCEDB.sql.
Created file /home/oracle/TEST_RDS/config/TEST_RDS-grants_RDSTGT.sql.
Created file /home/oracle/TEST_RDS/config/TEST_RDS-onetime.ddc.
Created file /home/oracle/TEST_RDS/start-console.sh.
Created file /home/oracle/TEST_RDS/TEST_RDS-run-dbvlin210.dbvisit.co.nz.sh.
Created file /home/oracle/TEST_RDS/TEST_RDS-run-ec2-54-82-47-176.compute-1.amazonaws.com.sh.
Created file /home/oracle/TEST_RDS/Nextsteps.txt.
Created file /home/oracle/TEST_RDS/TEST_RDS-all.sh.


=========================================================================================================================================================================================
Dbvisit Replicate wizard completed
Script /home/oracle/TEST_RDS/TEST_RDS-all.sh created. This runs all the above created scripts. Please exit out of dbvrep, review and run script as current user to
setup and start DbvisitReplicate.
=========================================================================================================================================================================================
Optionally, the script can be invoked now by this wizard.
Run this script now? (yes/no) [NO]
dbvrep> exit

 

Review APPLY DDC file

Ensure you have the correct TNS_ADMIN details in the APPLY ddc file.  In this test example environment Oracle Express is used on the APPLY server.  The location of the TNS_ADMIN variable in the TEST_RDS-APPLY.ddc is:

 

memory_set TNS_ADMIN /u01/app/oracle/product/11.2.0/db_1/network/admin

 

Execute prod-all.sh script on dbvrdsrep01

Run the TEST_RDS-all.sh script to initialize Dbvisit Replicate and load the dbvrep repository in both the source and target.  It is important that the steps above be executed before running this step.

 

[oracle@dbvrds01 TEST_RDS]$ 
$ ./TEST_RDS-all.sh

Setting up Dbvisit Replicate configuration

Configure database SOURCEDB...

This check fails if the DBID is not the expected one...

Ok, check passed.

Configure database RDSTGT...

This check fails if the DBID is not the expected one...

Ok, check passed.

Object grants for database SOURCEDB...

Object grants for database RDSTGT...

Setting up the configuration

Initializing......done

WARN-1850: No DDC DB available, dictionary table does not exist.
DDC loaded from database (0 variables).

Dbvisit Replicate version 2.7
Copyright (C) Dbvisit Software Limited.  All rights reserved.

DDC file /home/oracle/TEST_RDS/config/TEST_RDS-onetime.ddc loaded.

MINE: Cannot determine Dbvisit Replicate dictionary version. (no dictionary exists)

APPLY: Cannot determine Dbvisit Replicate dictionary version. (no dictionary exists)

dbvrep> #clear the no-DDC-DB-available warning

dbvrep> engine clear previous warnings

dbvrep> set ON_WARNING SKIP

Variable ON_WARNING set to SKIP for process *.

dbvrep> set ON_ERROR EXIT

Variable ON_ERROR set to EXIT for process *.

dbvrep> ENGINE SWITCH_REDOLOG
dbvrep> ENGINE SETUP MINE DROP DICTIONARY

0 dictionary objects dropped.

dbvrep> ENGINE SETUP MINE CREATE DICTIONARY
dbvrep> ENGINE SETUP MINE LOAD DICTIONARY
Supplemental logging on database set.

Loading dictionary table DBRSCOL$
Loading dictionary table DBRSIND$
Loading dictionary table DBRSOBJ$
Loading dictionary table DBRSTAB$
Loading dictionary table DBRSUSER$
Loading dictionary table DBRSV_$DATABASE
ID of mine process is
2E363D28-AAD3-11E3-B759-C37E51CB262D. If not using
DDC in database, set MINE_UNIQUE_ID to this value.

dbvrep> ENGINE SETUP APPLY DROP DICTIONARY
0 dictionary objects dropped.

dbvrep> ENGINE SETUP APPLY CREATE DICTIONARY
dbvrep> ENGINE SETUP APPLY LOAD DICTIONARY
dbvrep> ENGINE PREPARE_DP SETUP CLEAR
dbvrep> ENGINE SETUP PAIR MINE AND APPLY
1 applier SCN set.


dbvrep> SET APPLY.INSTANTIATE_SCN NOW

Variable INSTANTIATE_SCN set to NOW for process APPLY.

dbvrep> ENGINE SUPPLEMENTAL LOGGING SCHEMA SCOTT ENABLE PRIMARY KEY
dbvrep> ENGINE SWITCH_REDOLOG
dbvrep> ENGINE WAIT_SCN_FLIP
Waited 2 seconds until scn_to_timestamp changed.
dbvrep> #single-scn instantiation: lock all tables and schemas

dbvrep> ENGINE LOCK SCHEMAS SCOTT
Locking all schemas.
Lock done.
dbvrep> #single-scn instantiation: unlock all tables and schemas, but
keep the SCN
dbvrep> ENGINE LOCK RELEASE LOCKS
dbvrep>

dbvrep> #prepare the tables (we use OFFLINE as neither MINE not APPLY is running; with OFFLINE we
won't wait on network timeout)

dbvrep> PREPARE OFFLINE SCHEMA SCOTT
Table SCOTT.BONUS instantiated at SCN 1034905
Table SCOTT.DEPT instantiated at SCN 1034905
Table SCOTT.EMP instantiated at SCN 1034905
Table SCOTT.LOADTEST instantiated at SCN 1034905
Table SCOTT.SALGRADE instantiated at SCN 1034905
dbvrep> #single-scn instantiation: unlock all tables and schemas,
forget the SCN (so it does not affect any further PREPARE statements)

dbvrep> ENGINE LOCK CLEAR SCN
dbvrep> ENGINE SWITCH_REDOLOG

dbvrep> #prepare script for instantiation

dbvrep> ENGINE PREPARE_DP WRITE DP_NETWORKLINK DIRECTORY DATA_PUMP_DIR

FILE /home/oracle/TEST_RDS/APPLY.sh DBLINK SOURCEDB USERID
rds_admin/dbvisit2014@RDSTGT

Created Data Pump script /home/oracle/TEST_RDS/APPLY.sh, using network import.

dbvrep> create ddcdb from ddcfile
DDC loaded into database (319 variables).

dbvrep> set ON_WARNING SKIP

Variable ON_WARNING set to SKIP for process *.

dbvrep> set ON_ERROR SKIP

Variable ON_ERROR set to SKIP for process *.
OK-0: Completed successfully.


These steps are required after the TEST_RDS-all.sh
script runs:


1) Create the necessary directory(ies) on the servers:

ec2-54-82-47-176.compute-1.amazonaws.com: /home/oracle/TEST_RDS

2) Copy the DDC files to the server(s) where the processes will run:

ec2-54-82-47-176.compute-1.amazonaws.com: /home/oracle/TEST_RDS/TEST_RDS-APPLY.ddc
dbvlin210.dbvisit.co.nz: /home/oracle/TEST_RDS/TEST_RDS-MINE.ddc

3) Review that path to dbvrep executable is correct in the run scripts:
/home/oracle/TEST_RDS/TEST_RDS-run-dbvlin210.dbvisit.co.nz.sh
/home/oracle/TEST_RDS/TEST_RDS-run-ec2-54-82-47-176.compute-1.amazonaws.com.sh

4) Copy the run script to the server(s) where the processes will run:
dbvlin210.dbvisit.co.nz: /home/oracle/TEST_RDS/TEST_RDS-run-dbvlin210.dbvisit.co.nz.sh
ec2-54-82-47-176.compute-1.amazonaws.com: /home/oracle/TEST_RDS/TEST_RDS-run-ec2-54-82-47-176.compute-1.amazonaws.com.sh

5) Ensure firewall is open for listen interfaces 0.0.0.0:7902, 0.0.0.0:7901 used by the processes.

6) Make sure the data on apply are in sync as of time when setup was run.
Scripts for Data Pump/export/DDL were created as requested:
/home/oracle/TEST_RDS/APPLY.sh
Create referenced database links (if any) before running the scripts.

7) Start the replication processes on all servers:
dbvlin210.dbvisit.co.nz: /home/oracle/TEST_RDS/TEST_RDS-run-dbvlin210.dbvisit.co.nz.sh
ec2-54-82-47-176.compute-1.amazonaws.com: /home/oracle/TEST_RDS/TEST_RDS-run-ec2-54-82-47-176.compute-1.amazonaws.com.sh

8) Start the console to monitor the progress:
/home/oracle/TEST_RDS/start-console.sh


The above list is stored in /home/oracle/TEST_RDS/Nextsteps.txt.

 

 

Once the above scripts are created, copy the TEST_RDS--run-ec2-54-82-47-176.compute-1.amazonaws.com.sh and TEST_RDS-APPLY.ddc to the EC2 server, example:

 

$ scp -i /home/oracle/aws/dbv-lab-keypair.pem /home/oracle/TEST_RDS/TEST_RDS-APPLY.ddc 
root@ec2-54-82-47-176.compute-1.amazonaws.com:/home/oracle/TEST_RDS/TEST_RDS-APPLY.ddc

$ scp -i /home/oracle/aws/dbv-lab-keypair.pem /home/oracle/TEST_RDS/TEST_RDS-run-ec2-54-82-47-176.compute-1.amazonaws.com.sh 
root@ec2-54-82-47-176.compute-1.amazonaws.com: /home/oracle/TEST_RDS/TEST_RDS-run-ec2-54-82-47-176.compute-1.amazonaws.com.sh

 

Instantiate Data (Import data into AWSDB)

 

 

 

 

Start the import process

 

To import the data into the RDS database you will need to use Datapump using the database links.

 

The import script was created as part of the setup wizard process.  You will have a file called APPLY.sh located in the /home/oracle/TEST_RDS directory.  Edit this file and add the option “exclude=statistics” at the end.  This will ensure you do not get any errors at the end of the import process.  Once you have updated the file and as per the above instructions you have your database links in place you can execute the script from the dbvrdsrep01 server.

 

[oracle@dbvrds01 ]$ cat APPLY.sh

impdp dbvadmin/xxxx@RDS_TGT table_exists_action=TRUNCATE network_link=XX directory=DATA_PUMP_DIR flashback_scn=1034905 tables=SCOTT. 
Table = SCOTT.BONUS, SCOTT.DEPT,SCOTT.EMP,SCOTT.LOADTEST,SCOTT.SALGRADE 
[oracle@dbvrds01 ]$ ./APPLY.sh

Import: Release 11.2.0.2.0 - Production on 

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

Connected to: Oracle Database 11g Release 11.2.0.2.0 - 64bit Production

Processing object type TABLE_EXPORT/TABLE/TABLE_DATA
Total estimation using BLOCKS method: 102.6 MB
Processing object type TABLE_EXPORT/TABLE/TABLE
. . imported "SCOTT"."BONUS"                         1200672 rows
. . imported "SCOTT"."DEPT"                            354141 rows
. . imported "SCOTT"."EMP"                               401688 rows
. . imported "SCOTT"."LOADTEST"                          900441 rows
. . imported "OE"."SALGRADE"                                114403 rows

Processing object type TABLE_EXPORT/TABLE/GRANT/OWNER_GRANT/OBJECT_GRANT
Processing object type TABLE_EXPORT/TABLE/INDEX/INDEX
Processing object type TABLE_EXPORT/TABLE/CONSTRAINT/CONSTRAINT
Job "DBVADMIN"."DP_RDSPOC_0001" successfully completed at 02:46:32

 

Start the Replication Processes

 

Mine Server (SOURCE) 

 

 

[oracle@dbvrds01]$ ./home/oracle/TEST_RDS/TEST_RDS-run-dbvlin210.dbvisit.co.nz.sh
Initializing......done
[oracle@dbvrds01 ]$ DDC loaded from database (266 variables).
Dbvisit Replicate version 2.7
Copyright (C) Dbvisit Software Limited.  All rights reserved.
DDC file /home/oracle/TEST_RDS/TEST_RDS-MINE.ddc loaded.
Starting process MINE...Created directory /home/oracle/TEST_RDS/ddc_backup
Created directory /home/oracle/TEST_RDS/log/
Created directory /home/oracle/TEST_RDS/log/trace/
Created directory /home/oracle/TEST_RDS/mine/
started

 

Apply (EC2) Server :

 

[oracle@ ]$ ./home/oracle/TEST_RDS/TEST_RDS-run-ec2-54-82-47-176.compute-1.amazonaws.com.sh
Initializing......done
DDC loaded from database (266 variables).
Dbvisit Replicate version 2.6.x
Copyright (C) Dbvisit Software Limited.  All rights reserved.
DDC file /home/oracle/TEST_RDS/TEST_RDS-APPLY.ddc loaded.
Starting process APPLY...Created directory /home/oracle/TEST_RDS/ddc_backup
Created directory /home/oracle/TEST_RDS/log/
Created directory /home/oracle/TEST_RDS/log/trace/
Created directory /home/oracle/TEST_RDS/apply
started

 

Starting the Dbvisit Replicate console

 

The console can be started from the Mine server (source) using the “start-console.sh” script

 

[oracle@ ]$ ./start-console.sh

 

 Note that the mine and apply process does take some time to initialize.

Once both Mine and Apply is initialized.

Testing the replication

 

While the above is running you can open the Dbvisit Replicate console and you can see the progress of the replication.  Insert some rows into the source tables and then look in the target to see them appear there.