Amazon Elastic Container Service

Overview

DataGrail recommends deploying the Request Manager Agent in Amazon Elastic Container Service (ECS). Deploying an ECS service will result in most of the details of load balancing, SSL termination, and service uptime being managed in a simple and standard manner. This toolchain provides simple and robust management of your deployment. To get an overview of ECS, check out Amazon's documentation.

Sourcing the Agent Image

The Request Manager Agent Docker image is hosted in DataGrail's Elastic Container Registry. Once access is granted, use this path for retrieving the image:

338780525468.dkr.ecr.us-west-2.amazonaws.com/datagrail-rm-agent:v0.8.8

If you would like to, you may clone this image into your own Docker repository, or use it directly from our repository upon service startup.

Network Communication Requirements

The Agent service does not support internal SSL termination, so configuring a load balancer or another form of SSL termination is required. TLS 1.2+ is required to provide secure communication between services.

Ingress will be made to the Agent over port 443 and will arrive from our VPC IP: 52.36.177.91. Inbound requests from any other source should be rejected.

The Agent will make network requests to:

• Systems you have configured
• Storage Bucket for storing results
• Secret Manager for retrieving credentials
• DataGrail at https://<customer-name>.datagrail.io

Environment Variables

Agent Configuration

The Request Manager Agent's primary configuration is sourced from an environment variable named DATAGRAIL_AGENT_CONFIG which defines the connections available for the Agent and the credentials used for authenticating with DataGrail.

Example DATAGRAIL_AGENT_CONFIG

{
  "connections": [
    {
      "name": "Metrics DB",
      "uuid": "272c0934-0a06-4b11-8ec9-7755499001a3",
      "capabilities": ["privacy/access","privacy/delete"],
      "mode": "live",
      "connector_type": "Redshift",
      "queries": {
        "access": ["call dsr('access', %(email)s)"],
        "delete": ["call dsr('delete', %(email)s)"]
      },
      "credentials_location": "arn:aws:secretsmanager:us-west-2:000123456789:secret:redshift-cvh7s2"
    }
  ],
  "customer_domain": "acme.datagrail.io",
  "datagrail_agent_credentials_location": "arn:aws:secretsmanager:us-west-2:000123456789:secret:datagrail-agent-sjc7s3",
  "datagrail_credentials_location": "arn:aws:secretsmanager:us-west-2:000123456789:secret:datagrail-bs83nh",
  "platform": {
    "credentials_manager": {
      "provider": "AWSSecretsManager"
    },
    "storage_manager": {
      "provider": "AWSS3",
      "options": {
        "bucket": "acme-datagrail-reports"
      }
    }
  }
}

Access Keys

Important

As a best practice, use temporary security credentials (such as IAM roles) instead of creating long-term credentials like access keys.

In general, we recommend giving the Agent service permissions through IAM roles if you are operating in an AWS environment. However, you may also specify account credentials as environment variables directly.

The access keys require the permissions from the Policy section.

Example Access Keys Variables

AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
AWS_REGION=us-west-2

Quick Setup Guide

The following sections contain the core steps to creating an ECS Agent service. Please note that depending on your AWS environment's pre-existing configuration, you may need to take additional steps to configure your VPC, subnets, etc. Those are not covered in this document but we are happy to provide you with any assistance we can offer.

Create Task Policy and Role

To give the ECS service permission to make API requests to AWS services, you will need to define a task IAM policy and role.

Policy

1. In the AWS console, navigate to Identity and Access Management.
2. Under Access management, select Policies and then Create policy.
3. In the Policy editor, select JSON on the right-hand side.
4. At a minimum, the Agent should be allowed to perform the s3:PutObject and secretsmanager:GetSecretValue actions on a set of defined resources. Use the below example policy and update it with the ARNs of your resources.

Example Policy

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "S3PutObject",
            "Action": "s3:PutObject",
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::<datagrail_s3_bucket_name>/*",
                "arn:aws:s3:::<datagrail_s3_bucket_name>"
            ]
        },
        {
            "Sid": "SecretsManagerGetSecretValue",
            "Action": "secretsmanager:GetSecretValue",
            "Effect": "Allow",
            "Resource": [
                "arn:aws:secretsmanager:<region>:<account_id>:secret:<datagrail_secret_name>",
                "arn:aws:secretsmanager:<region>:<account_id>:secret:<datagrail_agent_secret_name>",
                "arn:aws:secretsmanager:<region>:<account_id>:secret:<connector_1_secret_name>",
                "arn:aws:secretsmanager:<region>:<account_id>:secret:<connector_2_secret_name>",
                "<more_connector_arns...>"
            ]
        }
    ]
}

5. Once done creating the policy, click Next.
6. Create a name for the policy and optionally a description.
7. Confirm that the permissions in the Permissions defined in this policy are accurate and add any optional tags.
8. Click Create policy.

Role

1. Navigate back to the IAM home screen.
2. Under Access management, select Roles and then Create role.
3. Under Trusted entity type, select AWS service.
4. Under Use case, choose Elastic Container Service as the service, and Elastic Container Service Task as the use case and click Next.
5. In the Permissions policies section, search for the policy that you have just created and select it.
6. Under Role details, give the role a name and optionally update the description.
7. Click Create role.

Create ECS Cluster

Optional if the Agent service will be deployed in an existing cluster.

1. Navigate to Elastic Container Service, making sure you are in your desired AWS region.
2. In the left-hand menu, select Clusters and then Create cluster.
3. Under Cluster configuration, give the cluster a Name.
4. Under Infrastructure, select AWS Fargate (serverless).
5. Click Create.

Create Task Definition

To create an ECS service, you will need to define a Task Definition which contains details of the Agent container's environment, configuration, and system resources.

1. In the AWS console, navigate to Elastic Container Service, making sure to select your desired AWS region.
2. In the left-hand menu, select Task definitions, then Create new task definition, and Create new task definition with JSON.

Creating a task definition with JSON will allow you to define the container's environment variables, configuration, and resources in a single JSON object. In the below example, make sure that you update the taskRoleArn, executionRoleArn, the awslogs-region with the appropriate region, and image parameters with your own ARNs, and the value of the DATAGRAIL_AGENT_CONFIG variable with the contents created in the Environment Variables section.

Example Task Definition

{
   "taskRoleArn": "arn:aws:iam::012345678912:role/DataGrailAgentTakeRole",
   "executionRoleArn": "arn:aws:iam::012345678912:role/ecsTaskExecutionRole",
   "containerDefinitions": [
       {
           "essential": true,
           "name": "datagrail-rm-agent",
           "image": "338780525468.dkr.ecr.us-west-2.amazonaws.com/datagrail-rm-agent:v0.8.8",
           "portMappings": [
               {
                   "hostPort": 80,
                   "protocol": "tcp",
                   "containerPort": 80
               }
           ],
           "command": [
               "supervisord",
               "-n",
               "-c",
               "/etc/rm.conf"
           ],
           "cpu": 0,
           "environment": [
               {
                   "name": "DATAGRAIL_AGENT_CONFIG",
                   "value": ""
               }
           ],
           "mountPoints": [],
           "workingDirectory": "/app",
           "logConfiguration": {
               "logDriver": "awslogs",
               "options": {
                   "awslogs-group": "/ecs/datagrail-rm-agent",
                   "awslogs-region": "us-west-2",
                   "awslogs-stream-prefix": "/ecs"
               }
           },
           "healthCheck": {
               "retries": 3,
               "command": [
                   "CMD-SHELL",
                   "curl -f http://localhost/docs || exit 1"
               ],
               "timeout": 5,
               "interval": 30,
               "startPeriod": 1
           }
       }
   ],
   "family": "datagrail-rm-agent",
   "requiresCompatibilities": [
       "FARGATE"
   ],
   "runtimePlatform": {
       "operatingSystemFamily": "LINUX"
   },
   "networkMode": "awsvpc",
   "cpu": "1024",
   "memory": "2048"
}

Create Security Groups

The Agent service requires security groups for both the load balancer and service to act as a virtual firewall to control inbound and outbound traffic.

1. Navigate to EC2 and select Security Groups under Network & Security in the left-hand menu.

Load Balancer Security Group

1. Click Create security group.
2. In the Basic details section,
   1. Give the security group a Name indicating that it is for the load balancer (e.g. "datagrail-rm-agent-load-balancer-sg"). This is important to later identify the security group to use when creating the load balancer.
   2. Add a Description of the security group (e.g. "Allow HTTPS ingress to DataGrail and TCP egress to DataGrail Agent service").
   3. Select the VPC that the load balancer will be created in.
3. In the Inbound Rules section,
   1. Click Add rule.
   2. In the Type dropdown, select HTTPS.
   3. In the Source dropdown, select Custom.
      1. Add 52.36.177.91/32 (DataGrail's IP) as the only allowed CIDR block.

Important

The next step is temporary. An outbound rule will be created later to allow egress from the load balancer to the service security group after it has been created.

4. Under Outbound rules, delete the default rule.
5. Click Create security group.

Service Security Group

1. Navigate back to Security Groups and click Create security group.
2. In Basic details,
   1. Give the security group a Name, indicating that it is for the service (e.g. "datagrail-rm-agent-service-sg"). This is important to later identify the security group to use when creating the service.
   2. Add a Description (e.g. "Allow ingress to application load balancer").
   3. Select the VPC that the service will be created in.
3. In Inbound Rules,
   1. Click Add rule.
   2. In the Type dropdown, select Custom TCP.
   3. Under Port range, enter 80.
   4. In the Destination dropdown, select Custom, and select the security group created in the Load Balancer Security Group section under Security Groups.
4. Under Outbound Rule, leave the default All traffic rule.
5. Click Create security group.

Now that the service security group has been created, an outbound rule to the application load balancer security group can be added.

1. Navigate back to security groups and select the security group created in the Load Balancer Security Group.
2. Under Actions in the top-right, select Edit outbound rules.
3. Click Add rule.
4. In the Type dropdown, select Custom TCP.
5. Under Port Range, enter 80.
6. In the Destination dropdown, select Custom, and select the security group created in the Service Security Group section under Security Groups.

Create Target Group

The target group defines where the load balancer routes requests and performs health checks on the targets.

1. Navigate to EC2 and select Target Groups under Load Balancing in the left-hand menu.
2. Click Create target group.
3. In the Basic configuration section,
   1. Under Choose a target type, select IP addresses.
   2. Under Target group name, give the target group a name indicating that it is for the Agent service load balancer (e.g. "datagrail-rm-agent-target-group")
   3. Under Protocol : Port, select HTTP in the Protocol dropdown, and enter 80 in the Port field.
   4. Under IP address type, select IPv4.
   5. Under VPC, select the VPC where the Agent service will be deployed.
   6. Under Protocol version, select HTTP1.
4. In the Health checks section,
   1. Under the Health check protocol dropdown, select HTTP.
   2. Under the Health check path field, enter /docs.
5. Click Next.
6. On the Register targets, do not modify any settings and click Create target group. The Agent service will be registered during its creation.

Create Application Load Balancer

1. Navigate to EC2, and select Load Balancers under Load Balancing in the left-hand menu.
2. Click Create load balancer.
3. Under Load balancer types, select Create under Application Load Balancer.
4. Under Basic configuration,
   1. Enter a Load balancer name.
   2. Under Scheme, select Internet-facing.
   3. Under IP address type, select IPv4.
5. Under Network mapping,
   1. Select the VPC, select the VPC where the Agent service will be deployed.
   2. Under Mappings, select at least two Availability Zone and one public subnet per zone
6. Under Security groups, remove the default group and select the security group you created in the Load Balancer Security Group section.
7. Under Listeners and routing,
   1. In the Protocol dropdown, select HTTPS.
   2. In the Port field, enter 443.
   3. Set the Default action to Forward to the target group that you created in the Create Target Group section.
8. Under Security policy,
   1. In the Security category dropdown, select All security policies.
   2. In the Policy name dropdown, select ELBSecurityPolicy-TLS13-1-2-2021-06.
9. Review the Summary to make sure everything looks correct.
10. Click Create load balancer

Create ECS Service

1. Navigate to Elastic Container Service and select Clusters in the left-hand menu.
2. Select the cluster in which you will be deploying the Agent.
3. In the Services tab, select Create.
4. Under the Environment section in the Create wizard, modify the Compute configuration.
   1. Under Compute options, select Launch type.
   2. In the Launch type dropdown, select FARGATE.
   3. In the Platform version dropdown, select LATEST.
5. Under the Deployment configuration section,
   1. Under Application type, select Service.
   2. Under Task definition, select the task definition you created in the Create Task Definition section and select the latest Revision.
   3. Under Service name, give the service a name (e.g. "datagrail-rm-agent-service").
   4. Under Service type, select Replica.
   5. Under Desired tasks, enter 1.
   6. Leave all default values under Deployment options and Deployment failure detection.
6. Under the Networking section,
   1. Select the VPC that you will be deploying the Agent in.
   2. Under Subnets, select a private subnet to place the Agent in.
   3. Under Security group, select Use and existing security group.
   4. Remove the default security group and select the security group created in the Service Security Group section.
7. Under the Load balancing section,
   1. In the Load balancer type dropdown, select Application Load Balancer.
   2. The Container dropdown will prepopulate with the single container defined in the task definition.
   3. Under Application Load Balancer, select Use and existing load balancer.
   4. In the Load balancer dropdown, select the load balancer that you created in the Create Application Load Balancer section.
   5. In the Health check grace period field, enter 15.
   6. Under Listener, select Use an existing listener and select 443:HTTPS in the Listener dropdown.
   7. Under Target group, select Use an existing target group, and select the target group created in the Create Target Group section in the Target group name dropdown.
8. Select Create.

You should then see the service appear on the management page of the cluster under the Services section. Click on the service and then on the Tasks tab in the service. If tasks are not in a Running state within a few minutes, you can inspect the stopped tasks to look for any errors or failures.

Create Alias Record for Application Load Balancer in Route 53

1. Navigate to Route 53 and click on Hosted zones in the left-hand menu.
2. Select the hosted zone where you will be deploying the Agent and click Create record.
3. In the Record name field, enter the subdomain that you would like to use (e.g. "datagrail-rm-agent").
4. In the Record type dropdown, select A.
5. Toggle the Alias button ON.
6. Under Route traffic to,
   1. Select Alias to Application and Classic Load Balancer in the Choose endpoint dropdown.
   2. Select the region where the load balancer was deployed in the Choose Region dropdown.
   3. Select the load balancer in the Choose load balancer dropdown.
   4. Keep Routing policy set to Simple routing and Evaluate target health set to Yes
7. Click Create records.

Your Agent service will now be reachable at your subdomain!