Using Amazon Key Management Service (KMS) with Oracle Database replication

Owned by Clive Otten (Unlicensed)
Jan 20, 2020
8 min read

As more organisations make the move to Cloud based applications and operations, questions around security are coming to the fore, and vendors are responding by making this an integral component of their offering. Such is the case with Amazon Web Services (AWS) who have, over the past couple of years, introduced a number of features aimed at simplifying the task of storing cloud-based information in encrypted form - and chief amongst these is the Key Management Service (KMS). In short the Key Management Service (KMS) is a managed service that makes it easy to create and control encryption keys used to encrypt data across a number of the AWS offerings. It is available with Elastic Block Store (EBS) and the Relational Database Service (RDS); these are the services which are relevant when working with the Dbvisit products on the AWS platform, and so we will focus on these in this post. However, the complete current list of AWS services supporting KMS is as follows:

This is not the place to go into all the inner workings and capabilities of KMS, so we recommend you take a look at the product page on the AWS website to understand the service concepts and details in depth. What is so brilliant about KMS is that encryption is basically handled transparently for you, meaning that no further manual intervention is required in accessing encrypted volumes, as AWS handles this all “behind the scenes”. It sounds simple enough, but like anything new, the array of options and configurations can seem overwhelming at first. What I found, as with other AWS offerings, is that there is a well thought out, logical order and flow – making it simple to navigate your way around exactly what you need, leaving the rest aside. In this post we will only focus on those aspects that relate to the Dbvisit products specifically.

Do Dbvisit Replicate and Dbvisit Standby work with KMS?

Both Dbvisit Standby and Dbvisit Replicate are supported on AWS, whether in cloud based or hybrid (on-premise to AWS) configurations, and have been for a number of years. In fact we actually make use of this platform for our product Test Drive configurations, which are accessible on our website. However, when it comes to KMS the specific point of interest was whether this recently added service - and the techniques by which it provides encryption of data - would present challenges to our applications and affect their operations. We have had a number of questions to this end from prospective hosting partners and customers in recent months, and this is what we want to answer in this post. But first, a quick recap… Working with either of the Dbvisit products on AWS requires the creation of at least one EC2 instance - with the OS type and Oracle edition/version of your preference. These can be sourced from community/vendor AMIs for the sake of convenience, or built yourself. An on-premise to AWS configuration (hybrid) for Dbvisit Replicate then has the following form:

EC2

Although the running processes have different names the shape is also essentially the same for a hybrid Dbvisit Standby configuration. And for both applications, when running entirely in the cloud, at least 2 independent EC2 instances will be needed for the source/primary and secondary/target servers. For your information, specific and detailed instructions for setting up Dbvisit Standby and Dbvisit Replicate on AWS can be found in the following documentation: https://blog.dbvisit.com/oracle-disaster-recovery-in-the-cloud-part-1/ https://dbvisit.atlassian.net/wiki/pages/viewpage.action?pageId=60784823 https://dbvisit.atlassian.net/wiki/pages/viewpage.action?pageId=60784826 The point to take note of here is that EC2 instances require a persistence layer for their operations (think of this as “disk” volumes that are attached to a machine) and Elastic Block Store (EBS) is perhaps the most common option used for this. EBS comes in 2 varieties - faster, more expensive SSD - and regular HDD - and KMS is supported on both. As a side note, AWS also offers slower, cheaper storage offerings (i.e.: Glacier) but this is generally too slow to provide adequate performance for a replication system, being better suited to long term archival and storage. So KMS comes into view as an option when working with EBS as the persistence layer for EC2 instances. KMS can enable encryption when selected for EBS, providing the following types of data encryption, as the documentation states:

  • Data at rest inside the volume

  • All data moving between the volume and the instance

  • All snapshots created from the volume

  • The encryption occurs on the servers that host EC2 instances, providing encryption of data-in-transit from EC2 instances to EBS storage.

RDS

RDS is Amazon’s managed Relational Database Service, which enables you to spin up databases of various flavours and configurations, including Oracle. A lot of the groundwork of managing these databases is covered off for you by RDS (ie: backups). But the downside is that you don’t get full SYS level access to the database itself, or root access to the underlying server it is provisioned on - so that is a limitation. This restriction means it cannot be used in conjunction with Dbvisit Standby (as we require access to read and apply the archive logs with our application), but it can be employed as a target when working with Dbvisit Replicate via an EC2 hub instance

  • as depicted in the following:

The key to register for our discussion here is that these RDS instances actually run on top of storage volumes (different grades of SSD - again EBS behind the scenes), which are automatically provisioned and attached when the RDS instance is created. So in terms of making use of the KMS security options, this is as simple as selecting the checkbox to activate encryption when configuring a new RDS instance. What KMS offers on RDS is similar to that for EC2, which is encryption of the data at rest, and in transit between storage and the instance and, as the documentation states: Data that is encrypted at rest includes the underlying storage for a DB instance, its automated backups, Read Replicas, and snapshots.

So how do we get started with KMS?

First of all you need to log into the AWS Management Console and go to the “Identity and Access Management” section under Security and Identity:

From here select the Encryption Keys option in the menu on the left hand side. There is a default key automatically generated for EBS (and RDS if you use it) - but you will see that I have created a new one called “kms-test” in this case:

Creating a new key (via the Create Key button) is a simple 4 step process (as below) which is outlined in detail in the documentation. It includes provision for authorising IAM users and roles who can administer the master keys, and denoting those who can also make use of it to encrypt and decrypt data - which can also include External Accounts.

One thing to be aware of is that KMS keys are region-specific and cannot be shared across regions. But apart from this, there is a great deal of flexibility in the way they can be used and managed - including features such as the automated rotation of keys.

Using KMS with EC2 instances:

To make use of the KMS key we have created, we need to create EC2 instances which will employ this at the EBS level. Note that there is currently no direct way to encrypt an existing unencrypted volume, or to remove encryption from an encrypted volume. It is, however, possible to migrate data between encrypted and unencrypted volumes, and you can apply encryption while copying the encrypted snapshot of an unencrypted volume. However, since the announcement in December 2015 it has been possible to “…create Amazon Machine Images (AMIs) that make use of encrypted EBS boot volumes and use the AMIs to launch EC2 instances.” Perhaps the easiest way to do this is to create an encrypted boot volume by using the AMI ”Copy Image" function - which enables you to specify a Master Key for encryption during the process. If you don’t have an existing AMI then you can create one from a running instance - or get one from the community/vendor catalog. To access this copy option, under the AMIs menu of the EC2 interface right click on an existing AMI and select “Copy AMI” to bring up the following dialog box. The key step here is to select the Encryption box to encrypt the target EBS snapshot, and then select the Master Key to use from the dropdown below this. It is as simple as that! 

Note that Amazon EBS encryption is not available on all instance types, so if you select one that is not currently supported you will receive an error message to this effect. You can find the complete listing of those currently supported in the documentation. Once an AMI has been generated it can then be used to create new EC2 instances, using the Launch Instance options under the Instances menu of the EC2 console window. And when the instance(s) are up and running you can check the underlying EBS details from the following menu in the EC2 console window:

Below you can see that, for the instances I provisioned when testing out Dbvisit Replicate on AWS, the attached EBS volumes are indeed encrypted, with the custom key which I generated earlier and assigned: 

Using KMS with RDS instances:

In terms of adding KMS as an option to RDS the process is even simpler! All you need to do is select encryption as an option when creating an RDS instance. So, from the RDS console in AWS select the “Launch a DB Instance” to create a new Instance:

Then select the database flavour you want to create:

For my test I have gone with Oracle SE, with general purpose SSD storage, and a instance class which I know is supported for KMS - m3.large:

The critical next step is, in the Advanced Settings configuration screen, to select the “Enable Encryption” option, as demonstrated below. This brings up additional drop down boxes allowing you to set the KMS Master Key for the region in your account:

Once this is done, and you have launched the DB instance, then the summary screen will show something like the following, in which we can clearly see that encryption has been enabled, in the Encryption Details section on the right hand side:

Running Dbvisit Standby and Dbvisit Replicate with KMS on AWS:

Sure enough, KMS lives up to its promise of providing transparent data encryption at rest, and in transfer, between storage and instances. In running both Dbvisit Standby and Dbvisit Replicate our tests showed that the applications were unaffected, and unimpeded in any way when this encryption facility was enabled. Dbvisit Standby In the following screenshots, taken from our AWS test environment we can see archive logs being sent from the primary to the standby server, and then applied on the latter:

And a Log Gap report confirms the healthy, functional status of the configuration:

Dbvisit Replicate:

In this test (one of many performed) we replicated data from the HR schema on a local Oracle database to the HR schema on an Oracle RDS instance - pushing a reasonable amount of test data through. As can be seen from the Dbvisit Replicate console, the mining and applying of changes has been processed successfully, bringing the target data set up to date with that on the source: 

And we can also query this replicated data from the RDS instance using regular SQL tools, without issue, as per the following: 

Conclusion:

The good news is that both Dbvisit Standby and Dbvisit Replicate support KMS on AWS - so you can go ahead and make use of this when working with our applications on this platform. We hope that this information will give you confidence, and a starting point, to begin experimenting with this security offering on AWS yourself. Good luck - and if you have any questions or comments then please drop us a line.