Skip to main content

Integrating AWS S3

Capabilities​

DataGrail's AWS S3 integration provides the following capabilities:

ProductCapability
Request Manager
Foundation ConnectionsCloud Storage

Before You Start​

To successfully configure this integration, please ensure you have sufficient privileges:

  • DataGrail User Role: Super Admin, Connections Manager
  • AWS S3 User Role: Admin

Setup Cloud Storage​

DataGrail uses two buckets to support your privacy request storage, one for the data retrieved during privacy requests, and another for logging.

Create The Bucket​

  1. Log in to AWS, and navigate to S3.
  2. Select Create Bucket.
  3. Enter a Bucket Name. We recommend using: yourcompanyname-datagrail-reports
  4. Enter an AWS Region. There are no restrictions on region.
  5. Take note of the bucket name and region. You will need this for later.

Configure The Bucket​

The following are highly recommended configuration options. They ensure that you're both logging access to any objects produced during an access or deletion request and ensures that data is also encrypted at rest.

  1. Confirm that the Block All Public Access is checked.
  2. Under Default Encryption, select either:
    1. Server-side encryption with Amazon S3 managed keys (SSE-S3)
    2. Server-side encryption with AWS Key Management Service keys (SSE-KMS)
      Configuring the KMS Key
      1. Select or create a KMS key that works for your security posture. However, per Amazon's recommendation, use symmetric encryption on the KMS Key.
      2. Add the following permissions in the Key Policy: kms:GenerateDataKey, kms:Encrypt and kms:Decrypt
      3. The Key Policy should look like the following. The Principal should be set to arn:aws:iam::338780525468:user/dg_app for Access Key Connections. For AssumeRole Connections, the Principal should be the ARN of the role.
      {
          "Sid": "Allow use of the key",
          "Effect": "Allow",
          "Principal": {
              "AWS": [
                  "<SUBSTITUTEME>"
              ]
          },
          "Action": [
              "kms:Encrypt",
              "kms:Decrypt",
              "kms:GenerateDataKey"
          ],
          "Resource": "<The ARN of the KMS Key>"
      }

      It's important that the Resource is properly set to restrict access to only the AWS KMS Key.

  3. Select Create Bucket.

Create a Bucket For Logging​

  1. Follow the steps in Create The Bucket and Configure The Bucket to create another bucket named yourcompanyname-datagrail-logs. This bucket will be used to store server access logs for your bucket.
  2. Select the original bucket (yourcompanyname-datagrail-reports) and click Properties from the top menu.
  3. Select Edit under Server Access Logging.
  4. Enable Server Access Logging and select the yourcompanyname-datagrail-logs bucket you created earlier as the Target Bucket.
  5. Select Save Changes.
  6. Scroll down to AWS CloudTrail data events and select Configure in CloudTrail.
  7. After being redirected, select Create Trail and enter a trail name. We suggest: datagrail-reports-events
  8. Take note of the name of the new bucket that will be created under Trail log bucket and folder.
  9. Enter an AWS KMS alias. We suggest: datagrail-reports-key
  10. Select Next at the bottom of the page.
  11. Deselect Management events and select Data events.
  12. Select Switch to basic event selectors.
  13. Set Data event type to S3. Uncheck the Read and Write options under All current and future buckets.
  14. Under Individual bucket selection, enter the name of the original bucket (yourcompanyname-datagrail-reports). Ensure Read and Write are checked.
  15. Click Next, Review, and then select Create Trail.

Create The IAM Bucket Policy​

  1. Navigate to IAM within AWS and select Policies from the left menu.
  2. Select Create Policy.
  3. Switch to the JSON policy editor.
  4. Enter the Recommended IAM Policy.
Recommended IAM Policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowListBucketIfSpecificPrefixIsIncludedInRequest",
            "Action": [
                "s3:ListBucket"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::SUBSTITUTEME"
            ]
        },
        {
            "Sid": "AllowUserToReadWriteObjectDataInFolder",
            "Action": [
                "s3:GetObject",
                "s3:DeleteObject",
                "s3:PutObject"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::SUBSTITUTEME/*"
            ]
        }
    ]
}

Ensure you substitute the ARN of your bucket for SUBSTITUTEME.

Connect to DataGrail​

DataGrail provides two methods for securely connecting AWS S3:

One of the main security benefits of using AssumeRole is the lack of key rotation requirements. Read more about IAM Users and Roles here.

Create The IAM Role​

  1. Navigate to IAM within AWS and select Roles from the left menu.
  2. Select Create Role.
  3. Choose Custom trust policy.
  4. Enter the Recommended JSON Policy, ensuring you create and substitute an External ID.
Recommended JSON Policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::338780525468:user/dg_app"
            },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "sts:ExternalId": "SUBSTITUTEME"
                }
            }
        }
    ]
}

Ensure you create and substitute an External ID for SUBSTITUTEME.
The External ID is, in effect, a password. Keep it secure. We recommend using a secure password generator with a long length (32+ alphanumeric characters, no symbols). Save the External ID for later; it will be used to set up the DataGrail connection to this role.

  1. Select Next and choose the policy created in Creating The IAM Bucket Policy.
  2. Select Next and enter a Role name.
  3. Select Create role and then ensure that this IAM Role ARN is populated on the JSON KMS Key Policy you created in Configure The Bucket.
Important

Please send the ARN of the newly created role to support@datagrail.io. Our support team will work with you to assume the necessary permissions on this role.

Connect to DataGrail​

  1. Navigate back to DataGrail.
  2. Under Bucket, enter the name of the original bucket (yourcompanyname-datagrail-reports).
  3. Under Region, enter the name of the AWS region of the original bucket (i.e. us-west-2).
  4. Select AWS Assume Role under Authentication Type.
  5. Under AssumeRole ARN, enter the ARN of the IAM Role you just created.
  6. Under AssumeRole External ID, enter the External ID used in the IAM Role you just created.
  7. Select Configure Integration.

Connect With an Access Key​

Create The IAM User​

  1. Navigate to IAM within AWS and select Users from the left menu.
  2. Select Create User.
  3. Enter a User name and ensure Provide user access to the AWS Management Console is unchecked.
  4. Select Next and then Attach policies directly.
  5. Select the policy created in Creating The IAM Bucket Policy.
  6. Select Next and then Create user.
  7. Save the Access Key ID and Secret Access Key in a secure, temporary location, then select Close.
  8. Ensure the IAM User ARN is populated on the JSON KMS Key Policy you created in Configure The Bucket.

Connect to DataGrail​

  1. Navigate back to DataGrail.
  2. Under Bucket, enter the name of the original bucket (yourcompanyname-datagrail-reports).
  3. Under Region, enter the name of the AWS region of the original bucket (i.e. us-west-2).
  4. Select AWS Access Key under Authentication Type.
  5. Under Access Key ID, enter the Access Key ID from the IAM User you just created.
  6. Under Secret Access Key, enter the Secret Access Key from the IAM User you just created.
  7. Select Configure Integration.

Next Steps​

Now that you've successfully connected the integration, check out the following resources:

Troubleshooting​

If you are unable to successfully connect the integration, review these common troubleshooting steps:

Ensure you have sent the IAM Role ARN to DataGrail Support.

For AssumeRole connections, the DataGrail Support Team needs the ARN of the IAM role you created to assume the necessary permissions.

Please send the ARN of the newly created role to support@datagrail.io.

Ensure all placeholder values in JSON policies are populated.

All JSON policies in this documentation contain multiple placeholders for credentials you have created during the setup process. Please review the JSON policies for the KMS Key and any IAM Users, Roles, or Resources.

Ensure Required Permissions Are Granted on API Keys/Resources

DataGrail checks required permissions and scopes on API keys/resources used by each integration. If all necessary permissions are not granted, new connections will fail.

Review Required Account Types and User Roles

If users do not have the necessary permissions or the minimum required user role in the connecting system, the connection to DataGrail will fail. Additionally, connections will fail if your account type does not match the one required by the integration.

See Before You Start to review these requirements.

 

Need help?
If you have any questions, please reach out to your dedicated CSM or contact us at support@datagrail.io.

Disclaimer: The information contained in this message does not constitute as legal advice. We would advise seeking professional counsel before acting on or interpreting any material.