Exporting database snapshots to the Data Platform Landing Zone

This section covers the technical overview of uploading data into the data platform from a db instance in AWS. For step by step instructions on how to do this, refer to exporting db snapshot to the DataPlatform Landing zone guide.

The terraform module db_snapshot_to_s3 will provision the following resources: exporting-snapshot-to-landing-zone

After the deployment these resources will export the data from any instance listed in this environment variable by executing the following steps:

  1. An RDS event subscription is provisioned, which attaches to the defined instance and pushes events to an SNS topic
  2. When a snapshot of the defined instance is created a message is sent to an SQS queue
  3. The sent SQS message invokes the export lambda by passing it the RDS instance id
  4. The export lambda retrieves the last snapshot created for the received instance id and starts an RDS snapshot export task
  5. Once the export task is started the lambda queues an SQS message with the export task identifier and the RDS export storage bucket name
  6. The queued message invoked the copier lambda
  7. The copier lambda checks the state of the export task and once the export is completed, it transfers the exported data into an S3 bucket in the data platform account