Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 5 Next »

Introduction

This guide will provide you with an overview on getting started with  Dbvisit Replicate and Amazon EC2 The scenario used in this guide can be illustrated as:

FIX this slide.... 


The rest of the document will provide you with an overview on the steps required to implement the above solution.  

Source (MINE) Server

Dbvisit Replicate will be installed on the database server, from where the mine process will be configured.  

 

Before starting the Dbvisit Replicate configuration, make sure you have the following details of the Source Database server where the mine process will be configured.  (Note that in this guide it is assumed that the source server is Unix based):

  • Server Name
  • Server IP Address
  • root (Unix User) Password
  • oracle (Unix User) Password
  • Source Database Name
  • Source TNS Alias (Make sure it is located in local tnsnames.ora)
  • Target TNS Alias (Make sure it is located in local tnsnames.ora)
  • SYS (Oracle Super User) Password
  • SYSTEM (Oracle System User) Password
  • DBVREP (Dbvisit Repository) Password                       
  • List of Schemas or Objects you want to replicate
  • DBVREP (Dbvisit Repository) Password
    • Dbvisit Replicate has a repository in both source and target using the “dbvrep” user name by default.  This account is created during Dbvisit Replicate installation and configuration.       
  • Sufficient disk space to store Dbvisit Replicate Mine PLOGS (default location /home/oracle/<DDC>/)

Target (APPLY) Server

In this example we recommend the use of an Amazon EC2 instance, however it is possible to configure the Apply process to also run from the same server as the Mine process.  Note that in this guide it is assumed that the Apply server is an Oracle Linux EC2 Instance in AWS.  This is done to offload the Apply processing to a different server than the one running the Mine.  Note that the initial export dump and import (instantiation) of the source data into the target will be performed from the source database server using Oracle Datapump over a database link.  Before starting with the Dbvisit Replicate configuration it is recommended to make sure you have the following information ready with regards to the Apply server and process:

  • EC2 - Server Name
  • Server IP Address (This example will use an Elastic IP)   ******* Do we need to say this?*****
  • root (Unix User) Password  ********
  • oracle (Unix User) Password  **********
    • Even though this system will not run an Oracle Database, we will make use of an Oracle Unix account as well as the Oracle Instance Client to establish connections between the source and target)
  • Target Database Name (Amazon AWS EC2 instance)
  • Target TNS Alias (Make sure it is located in local tnsnames.ora)
  • A DBA username and password in the EC2 instance
  • DBVREP (Dbvisit Repository) Password
    • Dbvisit Replicate has a repository in both source and target using the “dbvrep” user name by default.  This account is created during Dbvisit Replicate installation and configuration.       
  • Sufficient disk space to store Dbvisit Replicate Apply PLOGS (default location /home/oracle/<DDC>/)

 

 

 

AWSDB =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = awsdb.cu1fmi2n7nvj.us-east-1.rds.amazonaws.com)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = awsdb)
    )
  )

 

Testing Connection

 


sqlplus /nolog


SQL*Plus: Release 10.2.0.4.0 - Production on Mon Sep 30 12:11:14 2013


Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.


SQL> connect dbvadmin@awsdb
Enter password:
Connected.
SQL> select * from v$tablespace;


       TS# NAME                       INC BIG FLA ENC
---------- ------------------------------ --- --- --- ---
        0 SYSTEM                      YES YES YES
        1 SYSAUX                      YES YES YES
        2 UNDO_T1                     YES YES YES
        4 USERS                       YES YES YES
        5 RDSADMIN                    YES YES YES
        3 TEMP                        NO  YES YES


 

Preparing the Amazon EC2 (Target) Database

This step is required to prepare the RDS database for the data to be loaded from the Source database.  In this example we will load the Swingbench OE schema from the PROD database into the EC2 database DBVRDS into the tablespace DATA.  The first step is to create the DATA tablespace.  In the example below we added a 5G datafile.

 

SQL> select * from v$tablespace;

       TS# NAME                       INC BIG FLA ENC
---------- --------------------------- --- --- --- ---
        0 SYSTEM                      YES YES YES
        1 SYSAUX                      YES YES YES
        2 UNDO_T1                     YES YES YES
        4 USERS                       YES YES YES
        5 RDSADMIN                    YES YES YES
        3 TEMP                        NO  YES YES

6 rows selected.

SQL> create tablespace DATA datafile size 5G autoextend on next 200M maxsize 10G;
Tablespace created.

SQL> select tablespace_name, file_name, bytes/1024/1024 from dba_data_files order by 1;

TABLESPACE_NAME  FILE_NAME                                          BYTES/1024/1024
---------------- --------------------------------------------------------- ---------------
DATA          /rdsdbdata/db/AWSDB_A/datafile/o1_mf_data_94kfndhh_.dbf          5120
RDSADMIN      /rdsdbdata/db/AWSDB_A/datafile/o1_mf_rdsadmin_8r2stzcm_.dbf      1
SYSAUX        /rdsdbdata/db/AWSDB_A/datafile/o1_mf_sysaux_8r2sb6jm_.dbf        100
SYSTEM        /rdsdbdata/db/AWSDB_A/datafile/o1_mf_system_8r2sb398_.dbf        300
UNDO_T1       /rdsdbdata/db/AWSDB_A/datafile/o1_mf_undo_t1_8r2sb91t_.dbf              200
USERS         /rdsdbdata/db/AWSDB_A/datafile/o1_mf_users_8r2sb9sl_.dbf         100

6 rows selected.

 

Pre-Create Swingbench OE Schema in EC2 (Target)

In this test example we will be using Swingbench to generate transactions in the OE (Order Entry) schema on the source database. 

The next step is to create the OE schema in the Amazon EC2 Database, only the user and permissions, no table structures are pre-created as this will be done as part of the data instantiation.

 

create user oe identified by oe default tablespace data temporary tablespace temp;
  GRANT QUERY REWRITE TO "OE";
  GRANT CREATE MATERIALIZED VIEW TO "OE";
  GRANT CREATE VIEW TO "OE";
  GRANT UNLIMITED TABLESPACE TO "OE";
  GRANT ALTER SESSION TO "OE";
  GRANT "RESOURCE" TO "OE";
  GRANT create session to oe;

 

 

 

The OE schema in the source (XE) database consist out of the following:

 

SQL> select owner, segment_name, segment_type, bytes from dba_segments where owner='OE';

OWNER         SEGMENT_NAME                SEGMENT_TYPE                    BYTES
-------------- --------------------------- ------------------------------ ----------
OE            CUSTOMERS                   TABLE                         27262976
OE            INVENTORIES                 TABLE                         19922944
OE            LOGON                       TABLE                         3145728
OE            ORDERENTRY_METADATA         TABLE                           65536
OE            ORDERS                      TABLE                         23068672
OE            ORDER_ITEMS                 TABLE                         33554432
OE            PRODUCT_DESCRIPTIONS        TABLE                         327680
OE            PRODUCT_INFORMATION         TABLE                         262144
OE            WAREHOUSES                  TABLE                           65536
OE            CUSTOMERS_PK                INDEX                         6291456
OE            LOGON_PK                    INDEX                         2097152
OE            ORDERS_PK                   INDEX                         7340032
OE            PRODUCT_DESCRIPTIONS_PK      INDEX                          65536
OE            PRODUCT_INFORMATION_PK             INDEX                           65536
OE            WAREHOUSES_PK               INDEX                           65536

15 rows selected.


SQL> select owner, object_type, count(1) from dba_objects where owner='OE' group by owner, object_type;

OWNER                        OBJECT_TYPE         COUNT(1)
---------------------------- -------------------  ----------
OE                           INDEX                     6
OE                           SEQUENCE                  8
OE                           TABLE                     9
OE                           VIEW                      2
OE                           PACKAGE BODY              1
OE                           PACKAGE                   1

6 rows selected.

 

Creating Amazon EC2 Instances (Mine/Apply)

In this example we created one AWS EC2 instances to show the concept of loading data into EC2 using Replicate.  Depending on the requirements the Source database could be on-premise or in the AWS Cloud.  If this example I created the “dbvrds01” ****** to simulate the Source Database.  The second instance “dbvrds02” will be configured to perform the APPLY related tasks inside the AWS environment. 

Note this is an example, where the source database is Oracle Express with the Swingbench Order Entry environment loaded to simulate transaction load.


Instance 1: dbvrds01   SOURCE

Environment Summary:

  • Oracle Standard ENTERPRISE? 11.2.0.2 (XE)
  • Database Called “XE” created
  • Database in archivelog mode
  • Dbvisit Replicate 2.6.0.1 installed.
  • Swingbench installed into /home/oracle/swingbench
  • Java JDK installed
  • Swingbench Generate Load shell script created /home/oracle/generate_load


Instance 2: dbvrds02  TARGET

This environment will only run an Oracle Instant Client installation with Dbvisit Replicate installed.

Environment Summary:

  • EC2 instance running Oracle Linux 6.3
  • Install Dbvisit Replicate 2.6.02 from RPM.
  • Oracle client
    • Oracle client or instant client can be used, but in this example, the Oracle Express installation was used.
    • Configure the TNS_ADMIN environment:
      • /u01/app/oracle/product/11.2.0/xe/network/admin
      • create tnsnames.ora file with details for the XE (source) database and the target EC2 instance.

 

SOURCE_DB =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = dbvrds01)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = XE)
    )
  )


EC2_TARGET =
  (DESCRIPTION =
    (ADDRESS = (PROTOCOL = TCP)(HOST = awsdb.cu1fmi2n7nvj.us-east-1.rds.amazonaws.com)(PORT = 1521))
    (CONNECT_DATA =
      (SERVER = DEDICATED)
      (SERVICE_NAME = awsdb)
    )
  )

 

Ensure the firewall on both environments (IPTABLES) as well as the AWS security groups is configures to allow communication between these systems.  Example:

root@dbvrds01:[/root] # cd /etc/sysconfig


root@dbvrds01:[/etc/sysconfig] # cat iptables
# Firewall configuration written by system-config-securitylevel
# Manual customization of this file is not recommended.
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A FORWARD -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 1521 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 7901 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 7902 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 7903 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 7904 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 8081 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 8080 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 9090 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 8443 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
COMMIT


Ensure you have configured Security Group to allow the Replicate ports (7901-7904, 1521):

 

Dbvisit Replicate Installation/Configuration

Installing latest Dbvisit Replicate on both dbvrds01 (source) and dbvrds02 (apply):

 

root@akl:[/root] # rpm -ivh dbvisit_replicate-2.5.08-1.x86_64.rpm
Preparing...                ########################################### [100%]
   1:dbvisit_replicate      ########################################### [100%]

 

Please note that the main configuration (Setup Wizard) will be executed from the source (Mine) server which is also the same server hosing the source data we want to replicate to RDS.

Starting the Dbvisit Replicate Configuration

Execute “dbvrep” as the Oracle Unix account from the dbvrds01 (source) server.

 

[oracle@dbvrds01 ~]$ dbvrep
Initializing......done
Dbvisit Replicate version 2.5.08.3341
Copyright (C) Dbvisit Software Limited.  All rights reserved.
No DDC file loaded.
Run "setup wizard" to start the configuration wizard or try "help" to see all commands available.

 

Once the command line interface (dbvrep) started, type in “setup wizard” to start the configuration process (below is an example):

 

dbvrep> setup wizard
This wizard configures Dbvisit Replicate to start a replication process.


The setup wizard creates configuration scripts, which need to be run after the wizard ends. No changes to the databases are made before that.


The progress is saved every time a list of databases, replications, etc. is shown. It will be re-read if wizard is restarted and the same DDC name and
script path is selected.
Run the wizard now? [yes]
Accept end-user license agreement? (view/yes/no) [view] yes


Before starting the actual configuration, some basic information is needed. The DDC name and script path determines where all files created by the wizard
go (and where to reread them if wizard is rerun) and the license key determines which options are available for this configuration.
(DDC_NAME) - Please enter a name for this replication (suggestion: use the name of the source database): [] RDSPOC
(LICENSE_KEY) - Please enter your license key (or just enter "(trial)"): [(trial)]
Which Replicate edition do you want to trial (LTD/XTD/MAX): [MAX]
(SETUP_SCRIPT_PATH) - Please enter a directory for location of configuration scripts on this machine: [/home/oracle/RDSPOC]


Network configuration files were detected on this system in these locations:
/u01/app/oracle/product/11.2.0/xe/network/admin
(TNS_ADMIN) - Please enter TNS configuration directory for this machine: [/u01/app/oracle/product/11.2.0/xe/network/admin]


Step 1 - Describe databases
========================================
The first step is to describe databases used in the replication. There are usually two of them (source and target); however, there can be just one (when
source and target is the same) or more than two (one-to-many or other complex configurations.)
Store SYSDBA and DBA passwords? Passwords only required during setup and initialization? (yes/no) [yes]
Let's configure the database, describing it's type, connectivity, user names etc.
What type of database is this? (Oracle/MySQL/SQL Server): [Oracle]
Please enter database TNS alias: [] XE
Please enter SYSDBA user name: [SYS]
Please enter password for this user: [change_on_install] <<- Test system is using default passwords
Please enter user with DBA role: [SYSTEM]
Please enter password for this user: [manager]


Connecting to database XE as SYSTEM to query list of tablespaces and to detect ASM (by looking whether any redo logs or archived logs are stored in ASM).
Enter the Dbvisit Replicate owner and apply user (this user will be created by this script): [dbvrep]
Please enter password for this user: [dbvpasswd]


Permanent tablespaces detected on the database: DATA, USERS.
Please enter default permanent tablespace for this user: [DATA] USERS


Temporary tablespaces detected on the database: TEMP.
Please enter default temporary tablespace for this user: [TEMP]


Following databases are now configured:
1: Oracle XE, SYS/***, SYSTEM/***, dbvrep/***, USERS/TEMP, dbvrep/, ASM:NO, TZ: +00:00
Enter number of database to modify it, or "add", or "done": [add]
Let's configure the database, describing it's type, connectivity, user names etc.
What type of database is this? (Oracle/MySQL/SQL Server): [Oracle]
Please enter database TNS alias: [] AWSDB
Please enter SYSDBA user name: [SYS] dbvadmin  <<- Specify AWS RDS instance DBA user Account
Please enter password for this user: [change_on_install] ********
Please enter user with DBA role: [SYSTEM] dbvadmin <<- Specify AWS RDS instance DBA user Account
Please enter password for this user: [manager] ********


Connecting to database AWSDB as dbvadmin to query list of tablespaces and to detect ASM (by looking whether any redo logs or archived logs are stored in
ASM).
Enter the Dbvisit Replicate owner and apply user (this user will be created by this script): [dbvrep]
Please enter password for this user: [dbvpasswd]


Permanent tablespaces detected on the database: DATA, RDSADMIN, USERS.
Please enter default permanent tablespace for this user: [DATA] USERS


Temporary tablespaces detected on the database: TEMP.
Please enter default temporary tablespace for this user: [TEMP]


Following databases are now configured:
1: Oracle XE, SYS/***, SYSTEM/***, dbvrep/***, USERS/TEMP, dbvrep/, ASM:NO, TZ: +00:00
2: Oracle AWSDB, dbvadmin/***, dbvadmin/***, dbvrep/***, USERS/TEMP, dbvrep/, ASM:NO, TZ: +00:00
Enter number of database to modify it, or "add", or "done": [done]


Step 2 - Replication pairs
========================================
The second step is to set source and targets for each replication pair. This is usually just choosing the first database as source and the second one as
target, but many more configurations are possible.
Let's configure the replication pair, selecting source and target.
Following databases are described:
1: XE (Oracle)
2: AWSDB (Oracle)
Select source database: [1]
Select target database: [2]
Will be DDL replication enabled? (If YES, the script will grant more privileges to the Dbvisit Replicate users and enable database-wide supplemental        logging): [yes]
Use fetcher to offload the mining to a different server? (yes/no) [no] NO
(NETWORK_QUALITY) - Please specify your network type (LAN or WAN). Autoconfigures timeouts, use of compression etc. [LAN] WAN
Lock and copy the data initially one-by-one or at a single SCN? (one-by-one/single-scn/ddl-only/resetlogs) [single-scn] single-scn
What data copy script to create? (dp_networklink/dp_exp/exp/ddl_file/ddl_run/load/none) [dp_networklink] dp_networklink


Following replication pairs are now configured:
1: XE (Oracle) ==> AWSDB (Oracle), DDL: yes, fetcher: NO, process suffix: (no suffix), network: WAN, prepare type: single-scn, data load: dp_networklink
Enter number of replication pair to modify it, or "add", or "done": [done]


Step 3 - Replicated tables
========================================
The third step is to choose the schemas and tables to be replicated. If the databases are reachable, the tables are checked for existence, datatype
support, etc., schemas are queried for tables. Note that all messages are merely hints/warnings and may be ignored if issues are rectified before the
scripts are actually executed.


Note the following assumptions are made in this wizard – which can be modified by editing the resulting script:
1. All replicated tables have a primary key defined.
2. All columns of the tables and all tables of the specified schemas are replicated, and if DDL support is enabled, whenever a new column/table is added,
this should be replicate as well.
3. If an apply conflict arises, the default option is to try again repeatedly, until a different option is given or the underlying issue is resolved.
4. If an apply conflict arises, no more data will be replicated until the issue is resolved or ignored.


You can also specify some advanced options:
1. Rename schemas or tables.
2. Specify filtering conditions.
3. (Tables only) Configure Change Data Capture; this does not maintain a copy of the source table, but logs all operations as separate entries. This is
useful for ETL or as an audit trail. This usually requires adding of new columns (timestamps, old/new values etc.) to the target table. Dbvisit Replicate
can create the target table with the additional columns during setup by selecting "ddl_run" in the data copy section of Step 2.


Following tables are defined for replication pairs:
1: XE (Oracle) ==> AWSDB (Oracle), DDL: yes, suffix: (no suffix), prepare: single-scn
  No tables defined.
Enter number of replication pair to modify it, or "done": [1]


Please enter list of all individual tables to be replicated. Enter schema name(s) only to replicate all tables in that schema. Use comma or space to
delimit the entries.
Enter the tables and schemas: [] OE
Selected schemas: OE
Add more tables or schemas? (YES/NO) [NO] NO


To replicate changes of PL/SQL objects in schema(s), please enter the schemas to be replicated. Note that specifying any entry will cause additional
privileges to be granted to dbvrep. Enter through a comma or space-delimited list.
Enter the list of schemas (PL/SQL): []
Specify rename name or filter condition for any of the specified schemas (YES/NO): [no]


Following tables are defined for replication pairs:
1: XE (Oracle) ==> AWSDB (Oracle), DDL: yes, suffix: (no suffix), prepare: single-scn
  OE(tables)
Enter number of replication pair to modify it, or "done": [done]


Step 4 - Process configuration
========================================
The fourth step is to configure the replication processes for each replication. Although most options have reasonable defaults, manual input is required.


Following processes are defined:
1: MINE on XE
  Not configured.
2: APPLY on AWSDB
  Not configured.
Enter number of process to modify it, or "done": [1]
Fully qualified name of the server for the process (usually co-located with the database, unless mine is offloaded using fetcher): [dbvrds01]
Server type (Windows/Linux/Unix): [Linux]
Enable email notifications about problems (yes/no)? [YES] NO
Enable SNMP traps/notifications about problems (yes/no)? [NO] NO
Directory with DDC file and default where to create log files etc. (recommended: same as global setting, if possible)? [/home/oracle/RDSPOC]


Following settings were pre-filled with defaults or your reloaded settings:
----------------------------------------
[MINE_REMOTE_INTERFACE]: Network remote interface: dbvrds01:7901
[MINE_DATABASE]: Database TNS: XE
[TNS_ADMIN]: tnsnames.ora path: /u01/app/oracle/product/11.2.0/xe/network/admin
[MINE_USER]: Dbvisit Replicate database username: dbvrep
[MINE_PASSWORD]: Dbvisit Replicate database password: *********
[MINE_PLOG]: Filemask for generated plogs: /home/oracle/RDSPOC/mine/%S.%E (%S is sequence, %T thread, %F original filename (stripped extension), %P process
type, %N process name, %E default extension)
[LOG_FILE]: General log file: /home/oracle/RDSPOC/log/dbvrep_%N_%D.%E
[LOG_FILE_TRACE]: Error traces: /home/oracle/RDSPOC/log/trace/dbvrep_%N_%D_%I_%U.%E


Checking that these settings are valid...
Do you want change any of the settings? [no]


Following processes are defined:
1: MINE on XE
  Host: dbvrds01, SMTP: NO, SNMP: NO
2: APPLY on AWSDB
  Not configured.
Enter number of process to modify it, or "done": [2]
<usually co-located with the database, unless mine is offloaded using fetcher): [ip-10-143-181-189] dbvrds02
Server type (Windows/Linux/Unix): [Linux]
Enable email notifications about problems (yes/no)? [YES] NO
Enable SNMP traps/notifications about problems (yes/no)? [NO] NO
Directory with DDC file and default where to create log files etc. (recommended: same as global setting, if possible)? [/home/oracle/RDSPOC]


Following settings were pre-filled with defaults or your reloaded settings:
----------------------------------------
[APPLY_REMOTE_INTERFACE]: Network remote interface: dbvrds02:7902
[APPLY_DATABASE]: Database TNS: AWSDB
[TNS_ADMIN]: tnsnames.ora path: /u01/app/oracle/product/11.2.0/xe/network/admin
[APPLY_USER]: Dbvisit Replicate database username: dbvrep
[APPLY_PASSWORD]: Dbvisit Replicate database password: *********
[APPLY_STAGING_DIR]: Directory for received plogs: /home/oracle/RDSPOC/apply
[LOG_FILE]: General log file: /home/oracle/RDSPOC/log/dbvrep_%N_%D.%E
[LOG_FILE_TRACE]: Error traces: /home/oracle/RDSPOC/log/trace/dbvrep_%N_%D_%I_%U.%E


Checking that these settings are valid...
Do you want change any of the settings? [no]


Following processes are defined:
1: MINE on XE
  Host: dbvrds01, SMTP: NO, SNMP: NO
2: APPLY on AWSDB
  Host: dbvrds02, SMTP: NO, SNMP: NO
Enter number of process to modify it, or "done": [done]
Created file /home/oracle/RDSPOC/RDSPOC-APPLY.ddc.
Created file /home/oracle/RDSPOC/RDSPOC-MINE.ddc.
Created file /home/oracle/RDSPOC/config/RDSPOC-setup.dbvrep.
Created file /home/oracle/RDSPOC/config/RDSPOC-dbsetup_XE.sql.
Created file /home/oracle/RDSPOC/config/RDSPOC-dbsetup_AWSDB.sql.
Created file /home/oracle/RDSPOC/config/RDSPOC-grants_XE.sql.
Created file /home/oracle/RDSPOC/config/RDSPOC-grants_AWSDB.sql.
Created file /home/oracle/RDSPOC/config/RDSPOC-onetime.ddc.
Created file /home/oracle/RDSPOC/start-console.sh.
Created file /home/oracle/RDSPOC/RDSPOC-run-dbvrds01.sh.
Created file /home/oracle/RDSPOC/RDSPOC-run-dbvrds02.sh.
Created file /home/oracle/RDSPOC/Nextsteps.txt.
Created file /home/oracle/RDSPOC/RDSPOC-all.sh.
==========================================================================================================================================================


Dbvisit Replicate wizard completed


Script /home/oracle/RDSPOC/RDSPOC-all.sh created. This runs all the above created scripts. Please exit out of dbvrep, review and run script as current user
to setup and start Dbvisit Replicate.
==========================================================================================================================================================
Optionally, the script can be invoked now by this wizard.
Run this script now? (yes/no) [NO] NO
dbvrep> exit

 

Required steps due to lack of SYS/SYSTEM account access in RDS

Before you run the “RDSPOC-all.sh” script you need to make a small change.  Update this line:

 

sqlplus -S "dbvadmin/odb10g11@AWSDB as sysdba" @/home/oracle/RDSPOC/config/RDSPOC-dbsetup_AWSDB.sql

 

TO the one below (remove the “as sysdba”)

 

sqlplus -S "dbvadmin/odb10g11@AWSDB" @/home/oracle/RDSPOC/config/RDSPOC-dbsetup_AWSDB.sql

 

The second change is to the RDSPOC-dbsetup_awsdb.sql script.  This change is required as we do not have access as the SYS or SYSTEM user to create the dbvrep user or grants.  Note the “grant dba to dbvrep” at the end of the script.  This is to make sure that dbvrep have sufficient permission to perform required tasks.

 

set feedback off
whenever sqlerror exit failure
-- alter database add supplemental log data;     <<----- comment out this line


prompt This check fails if the DBID is not the expected one...
set heading off
select to_number(decode(dbid, 2623380325, null, 'NotTheDatabaseDBIDWeExpected'))
from v$database;
prompt Ok, check passed.


begin execute immediate 'drop user dbvrep cascade'; exception when others then if sqlcode = -1918 then null; else raise; end if; end;
/
create user dbvrep identified by dbvpasswd default tablespace USERS temporary tablespace TEMP;
-- grants because this is a target database


alter user dbvrep quota unlimited on USERS;
grant create session to dbvrep;
grant create table to dbvrep;
grant create view to dbvrep;
grant create sequence to dbvrep;
grant create procedure to dbvrep;
grant execute on dbms_flashback to dbvrep;
-- grant select on v_$database to dbvrep;   ß comment out this line
-- grant select on v_$mystat to dbvrep;    ß comment out this line
-- grant select on v_$session to dbvrep;   ß comment out this line
-- grant select on gv_$session to dbvrep;  ß comment out this line
-- grant select on v_$lock to dbvrep;   ß comment out this line
-- grant select on gv_$lock to dbvrep;    ß comment out this line
-- grant select on sys.user$ to dbvrep;    ß comment out this line
-- grant alter system to dbvrep;
grant alter session to dbvrep;
-- This is for 10.2.0.5 trigger-fire-once
begin execute immediate 'begin dbms_streams_auth.grant_admin_privilege(''dbvrep''); end; '; exception when others then null; end;
/
-- This will fail on versions prior to 11.2.0.2
begin execute immediate 'begin dbms_goldengate_auth.grant_admin_privilege(''dbvrep''); end; '; exception when others then null; end;
/
-- this is needed for 9.2 only:
-- grant alter database to dbvrep;   ß comment out this line


-- grants because this is a target database with DDL replication enabled


grant select any table to dbvrep;
grant update any table to dbvrep;
grant insert any table to dbvrep;
grant delete any table to dbvrep;
grant create any table to dbvrep;
grant alter any table to dbvrep;
grant drop any table to dbvrep;
grant create any index to dbvrep;
grant alter any index to dbvrep;
grant drop any index to dbvrep;
grant create any view to dbvrep;
grant drop any view to dbvrep;


-- ADD THIS FOR AMAZON RDS
-- This is required as we do not have SYS user access to grant specific permissions to dbvrep
--
grant dba to dbvrep;  <<-- IMPORTANT - ADD This line


exit

 

Review APPLY DDC file

Ensure you have the correct TNS_ADMIN details in the APPLY ddc file.  In this test example environment Oracle Express is used on the APPLY server.  The location of the TNS_ADMIN variable in the RDSPOC-APPLY.ddc is:

 

memory_set TNS_ADMIN /u01/app/oracle/product/11.2.0/xe/network/admin

 

Execute prod-all.sh script on dbvrdsrep01

Run the RDSPOC-all.sh script to initialize Dbvisit Replicate and load the dbvrep repository in both the source and target.  It is important that the steps above be executed before running this step.

 

[oracle@dbvrds01 RDSPOC]$ ./RDSPOC-all.sh
Setting up Dbvisit Replicate configuration
Configure database XE...
This check fails if the DBID is not the expected one...




Ok, check passed.
Configure database AWSDB...
This check fails if the DBID is not the expected one...




Ok, check passed.
Object grants for database XE...
Object grants for database AWSDB...
Setting up the configuration
Initializing......done
WARN-1850: No DDC DB available, dictionary table does not exist.
DDC loaded from database (0 variables).
Dbvisit Replicate version 2.5.08.3341
Copyright (C) Dbvisit Software Limited.  All rights reserved.
DDC file /home/oracle/RDSPOC/config/RDSPOC-onetime.ddc loaded.
MINE: Cannot determine Dbvisit Replicate dictionary version. (no dictionary exists)
APPLY: Cannot determine Dbvisit Replicate dictionary version. (no dictionary exists)
dbvrep> set ON_WARNING SKIP
Variable ON_WARNING set to SKIP for process *.
dbvrep> set ON_ERROR EXIT
Variable ON_ERROR set to EXIT for process *.
dbvrep> ENGINE SWITCH_REDOLOG
dbvrep> ENGINE SETUP MINE DROP DICTIONARY
0 dictionary objects dropped.
dbvrep> ENGINE SETUP MINE CREATE DICTIONARY
dbvrep> ENGINE SETUP MINE LOAD DICTIONARY
Supplemental logging on database set.
Loading dictionary table DBRSCOL$
Loading dictionary table DBRSIND$
Loading dictionary table DBRSOBJ$
Loading dictionary table DBRSTAB$
Loading dictionary table DBRSUSER$
Loading dictionary table DBRSV_$DATABASE
ID of mine proces is 5B57FC60-2978-11E3-A0AD-58D8E5C48113. If not using DDC in database, set MINE_UNIQUE_ID to this value.
dbvrep> ENGINE SETUP APPLY DROP DICTIONARY
0 dictionary objects dropped.
dbvrep> ENGINE SETUP APPLY CREATE DICTIONARY
dbvrep> ENGINE SETUP APPLY LOAD DICTIONARY
dbvrep> ENGINE PREPARE_DP SETUP CLEAR
dbvrep> ENGINE SETUP PAIR MINE AND APPLY
1 applier SCN set.
dbvrep> SET APPLY.INSTANTIATE_SCN NOW
Variable INSTANTIATE_SCN set to NOW for process APPLY.
dbvrep> ENGINE SUPPLEMENTAL LOGGING SCHEMA OE ENABLE PRIMARY KEY
dbvrep> ENGINE SWITCH_REDOLOG
dbvrep> ENGINE WAIT_SCN_FLIP
Waited 2 seconds until scn_to_timestamp changed.
dbvrep> #single-scn instantiation: lock all tables and schemas
dbvrep> ENGINE LOCK SCHEMAS OE
Locking all schemas.
Lock done.
dbvrep> #single-scn instantiation: unlock all tables and schemas, but keep the SCN
dbvrep> ENGINE LOCK RELEASE LOCKS
dbvrep>
dbvrep> #prepare the tables (we use OFFLINE as neither MINE not APPLY is running; with OFFLINE we won't wait on network timeout)
dbvrep> PREPARE OFFLINE SCHEMA OE
Table OE.CUSTOMERS instantiated at SCN 1229827
Table OE.INVENTORIES instantiated at SCN 1229827
Table OE.LOGON instantiated at SCN 1229827
Table OE.ORDERENTRY_METADATA instantiated at SCN 1229827
Table OE.ORDERS instantiated at SCN 1229827
Table OE.ORDER_ITEMS instantiated at SCN 1229827
Table OE.PRODUCT_DESCRIPTIONS instantiated at SCN 1229827
Table OE.PRODUCT_INFORMATION instantiated at SCN 1229827
Table OE.WAREHOUSES instantiated at SCN 1229827
dbvrep> #single-scn instantiation: unlock all tables and schemas, forget the SCN (so it does not affect any further PREPARE statements)
dbvrep> ENGINE LOCK CLEAR SCN
dbvrep> ENGINE SWITCH_REDOLOG
dbvrep> #prepare script for instantiation
dbvrep> ENGINE PREPARE_DP WRITE DP_NETWORKLINK DIRECTORY DATA_PUMP_DIR FILE /home/oracle/RDSPOC/APPLY.sh DBLINK XE USERID dbvadmin/xxxx@AWSDB
Created Data Pump script /home/oracle/RDSPOC/APPLY.sh, using network import.
dbvrep> create ddcdb from ddcfile
DDC loaded into database (266 variables).
dbvrep> set ON_WARNING SKIP
Variable ON_WARNING set to SKIP for process *.
dbvrep> set ON_ERROR SKIP
Variable ON_ERROR set to SKIP for process *.
OK-0: Completed, but some operation(s) raised warnings and/or errors.
These steps are required after the RDSPOC-all.sh script runs:


1) Create the necessary directory(ies) on the servers:
dbvrds02: /home/oracle/RDSPOC


2) Copy the DDC files to the server(s) where the processes will run:
dbvrds02: /home/oracle/RDSPOC/RDSPOC-APPLY.ddc
dbvrds01: /home/oracle/RDSPOC/RDSPOC-MINE.ddc


3) Review that path to dbvrep executable is correct in the run scripts:
/home/oracle/RDSPOC/RDSPOC-run-dbvrds01.sh
/home/oracle/RDSPOC/RDSPOC-run-dbvrds02.sh


4) Copy the run script to the server(s) where the processes will run:
dbvrds01: /home/oracle/RDSPOC/RDSPOC-run-dbvrds01.sh
dbvrds02: /home/oracle/RDSPOC/RDSPOC-run-dbvrds02.sh


5) Ensure firewall is open for listen interfaces 0.0.0.0:7902, 0.0.0.0:7901 used by the processes.


6) Make sure the data on apply are in sync as of time when setup was run.
Scripts for Data Pump/export/DDL were created as requested:
/home/oracle/RDSPOC/APPLY.sh
Create referenced database links (if any) before running the scripts.


7) Start the replication processes on all servers:
dbvrds01: /home/oracle/RDSPOC/RDSPOC-run-dbvrds01.sh
dbvrds02: /home/oracle/RDSPOC/RDSPOC-run-dbvrds02.sh


8) Start the console to monitor the progress:
/home/oracle/RDSPOC/start-console.sh


The above list is stored in /home/oracle/RDSPOC/Nextsteps.txt.

 

 

Once the above scripts are created, copy the RDSPOC-rundbvrds02.sh and RDSPOC-APPLY.ddc to the dbvrds02 server, example:

 

[oracle@dbvrds01 RDSPOC]$ scp /home/oracle/RDSPOC/RDSPOC-run-dbvrds02.sh dbvrds02:/home/oracle/RDSPOC/


[oracle@dbvrds01 RDSPOC]$ scp /home/oracle/RDSPOC/RDSPOC-APPLY.ddc dbvrds02:/home/oracle/RDSPOC/

 

Instantiate Data (Import data into AWSDB)

 

The first step is to create the required database link.  This link is required to pull the data from the AWSDB database into the XE database.  The full network connection description is used for the connection detail, as we do not have access to the local system to modify the tnsnames.ora.  For this database link the SYSTEM account is used.

 

SQL> create public database link XE connect to system identified by manager using '(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = 10.85.90.146)(PORT = 1521)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = XE)))';

Database link created.


SQL> select * from v$instance@xe;


INSTANCE_NUMBER INSTANCE_NAME
--------------- ----------------
HOST_NAME
----------------------------------------------------------------
VERSION         STARTUP_T STATUS    PAR   THREAD# ARCHIVE LOG_SWITCH_WAIT
----------------- --------- ------------ --- ---------- ------- ---------------
LOGINS    SHU DATABASE_STATUS       INSTANCE_ROLE     ACTIVE_ST BLO EDITION
---------- --- ----------------- ------------------ --------- --- -------
             1 XE
dbvrds01
11.2.0.2.0      30-SEP-13 OPEN      NO          1 STARTED
ALLOWED    NO  ACTIVE        PRIMARY_INSTANCE   NORMAL    NO  XE

 

 

Start the import process

 

To import the data into the RDS database you will need to use Datapump using the database links.

 

The import script was created as part of the setup wizard process.  You will have a file called APPLY.sh located in the /home/oracle/RDSPOC directory.  Edit this file and add the option “exclude=statistics” at the end.  This will ensure you do not get any errors at the end of the import process.  Once you have updated the file and as per the above instructions you have your database links in place you can execute the script from the dbvrdsrep01 server.

 

[oracle@dbvrds01 RDSPOC]$ cat APPLY.sh

impdp dbvadmin/xxxx@AWSDB table_exists_action=TRUNCATE network_link=XE directory=DATA_PUMP_DIR flashback_scn=1229827 tables=OE.CUSTOMERS,OE.INVENTORIES,OE.LOGON,OE.ORDERENTRY_METADATA,OE.ORDERS,OE.ORDER_ITEMS,OE.PRODUCT_DESCRIPTIONS,OE.PRODUCT_INFORMATION,OE.WAREHOUSES   logfile=OE_WAREHOUSES.log JOB_NAME=DP_RDSPOC_0001 exclude=statistics


[oracle@dbvrds01 RDSPOC]$ ./APPLY.sh

Import: Release 11.2.0.2.0 - Production on Mon Sep 30 15:45:45 2013

Copyright (c) 1982, 2009, Oracle and/or its affiliates.  All rights reserved.

Connected to: Oracle Database 11g Release 11.2.0.2.0 - 64bit Production
Starting "DBVADMIN"."DP_RDSPOC_0001":  dbvadmin/********@AWSDB table_exists_action=TRUNCATE network_link=XE directory=DATA_PUMP_DIR flashback_scn=1229827 tables=OE.CUSTOMERS,OE.INVENTORIES,OE.LOGON,OE.ORDERENTRY_METADATA,OE.ORDERS,OE.ORDER_ITEMS,OE.PRODUCT_DESCRIPTIONS,OE.PRODUCT_INFORMATION,OE.WAREHOUSES logfile=OE_WAREHOUSES.log JOB_NAME=DP_RDSPOC_0001 exclude=statistics
Estimate in progress using BLOCKS method...
Processing object type TABLE_EXPORT/TABLE/TABLE_DATA
Total estimation using BLOCKS method: 102.6 MB
Processing object type TABLE_EXPORT/TABLE/TABLE
. . imported "OE"."ORDER_ITEMS"                         1200672 rows
. . imported "OE"."CUSTOMERS"                            354141 rows
. . imported "OE"."ORDERS"                               401688 rows
. . imported "OE"."INVENTORIES"                          900441 rows
. . imported "OE"."LOGON"                                114403 rows
. . imported "OE"."PRODUCT_DESCRIPTIONS"                   1000 rows
. . imported "OE"."PRODUCT_INFORMATION"                    1000 rows
. . imported "OE"."ORDERENTRY_METADATA"                       4 rows
. . imported "OE"."WAREHOUSES"                             1000 rows
Processing object type TABLE_EXPORT/TABLE/GRANT/OWNER_GRANT/OBJECT_GRANT
Processing object type TABLE_EXPORT/TABLE/INDEX/INDEX
Processing object type TABLE_EXPORT/TABLE/CONSTRAINT/CONSTRAINT
Job "DBVADMIN"."DP_RDSPOC_0001" successfully completed at 02:46:32

 

Start the Replication Processes

 

Mine Server - dbvrds01

 

 

[oracle@dbvrds01 RDSPOC]$ ./RDSPOC-run-dbvrds01.sh
Initializing......done
[oracle@dbvrds01 RDSPOC]$ DDC loaded from database (266 variables).
Dbvisit Replicate version 2.5.08.3341
Copyright (C) Dbvisit Software Limited.  All rights reserved.
DDC file /home/oracle/RDSPOC/RDSPOC-MINE.ddc loaded.
Starting process MINE...Created directory /home/oracle/RDSPOC/ddc_backup
Created directory /home/oracle/RDSPOC/log/
Created directory /home/oracle/RDSPOC/log/trace/
Created directory /home/oracle/RDSPOC/mine/
started

 

Apply (Intermediate) Server - dbvrds02:

 

[oracle@dbvrds02 RDSPOC]$ ./RDSPOC-run-dbvrds02.sh
Initializing......done
DDC loaded from database (266 variables).
Dbvisit Replicate version 2.5.08.3341
Copyright (C) Dbvisit Software Limited.  All rights reserved.
DDC file /home/oracle/RDSPOC/RDSPOC-APPLY.ddc loaded.
Starting process APPLY...Created directory /home/oracle/RDSPOC/ddc_backup
Created directory /home/oracle/RDSPOC/log/
Created directory /home/oracle/RDSPOC/log/trace/
Created directory /home/oracle/RDSPOC/apply
started

 

Starting the Dbvisit Replicate console

 

The console can be started from the Mine server (source) using the “start-console.sh” script

 

[oracle@dbvrds01 RDSPOC]$ ./start-console.sh

 

Example output can be seen below.  Note that the mine and apply process does take some time to initialize.

Once both Mine and Apply is initialized:

Testing the replication

 

The next step is to generate some load on the XE database using swingbench.  This can be done using the generate_load script.  To see the status of the replicate you can open a second terminal and open the replicate console with “start-console.sh”.    Example:

While the above is running you can open the Dbvisit Replicate console and you can see the progress of the replication:

 

 

 

  • No labels