Amazon Elastic Container Service

Overview

If you are hosting in AWS, DataGrail recommends using Elastic Container Service (ECS) to deploy the Request Manager Agent. This guide covers deploying the egress-only Request Manager Agent, which initiates outbound connections to DataGrail and does not require incoming traffic or load balancing. ECS simplifies the management of service uptime and networking, offering a reliable and streamlined deployment process. For more information about ECS, refer to Amazon's documentation.

Source the Agent Image

The Request Manager Agent Docker image is hosted in DataGrail's private image registry. Once you have obtained credentials from your DataGrail representative, you can pull the image using the following command:

Pulling the image

# Authenticate with the DataGrail registry
docker login contairium.datagrail.io -u $DATAGRAIL_SUBDOMAIN

# Pull the Request Manager Agent image
docker pull contairium.datagrail.io/rm-agent:$VERSION

If you would like to, you may pull the image into your own Docker repository, or use it directly from our repository upon service startup.

Docker Pull Error: No Matching Manifest

If you are attempting to docker pull the image locally and receive a "no matching manifest..." error, update the command to use the --platform option.

For example, if the original command is docker pull ${IMAGE_URI}:

• For M1 Macs, use: docker pull ${IMAGE_URI} --platform=linux/amd64
• For M2 Macs, use: docker pull ${IMAGE_URI} --platform=linux/x86_64

Quick Setup Guide

The following sections contain the core steps to creating an egress-only ECS Agent service. Since the Agent only makes outbound connections and does not receive incoming traffic, no load balancer or TLS certificate configuration is required. Please note that depending on your AWS environment's pre-existing configuration, you may need to take additional steps to configure your VPC, subnets, etc. Those are not covered in this document but we are happy to provide you with any assistance we can offer.

Task Policy and Role

To give the ECS service permission to make API requests to AWS services, you will need to define a task IAM policy and role.

Policy

1. In the AWS console, navigate to Identity and Access Management.
2. Under Access management, select Policies and then Create policy.
3. In the Policy editor, select JSON on the right-hand side.
4. At a minimum, the Agent should be allowed to perform the s3:PutObject and secretsmanager:GetSecretValue actions on a set of defined resources. Use the below example policy and update it with the ARNs of your resources.
   Example Policy

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Sid": "S3PutObject",
         "Action": "s3:PutObject",
         "Effect": "Allow",
         "Resource": [
           "arn:aws:s3:::reports-bucket/*"
         ]
       },
       {
         "Sid": "SecretsManagerGetSecretValue",
         "Action": "secretsmanager:GetSecretValue",
         "Effect": "Allow",
         "Resource": [
           "arn:aws:secretsmanager:us-west-2:012345678901:secret:datagrail.api-key"
         ]
       }
     ]
   }

5. Once done creating the policy, click Next.
6. Create a name for the policy (e.g., "rm-agent-storage-and-secrets") and optionally a description.
7. Confirm that the permissions in the Permissions defined in this policy are accurate and add any optional tags.
8. Click Create policy.

Role

1. Navigate back to the IAM home screen.
2. Under Access management, select Roles and then Create role.
3. Under Trusted entity type, select AWS service.
4. Under Use case, choose Elastic Container Service as the service, and Elastic Container Service Task as the use case and click Next.
5. In the Permissions policies section, search for the policy that you have just created and select it.
6. Under Role details, give the role a name (e.g., "rm-agent-task-role") and optionally update the description.
7. Click Create role.

ECS Cluster

Optional if the Agent service will be deployed in an existing cluster.

1. Navigate to Elastic Container Service, making sure you are in your desired AWS region.
2. In the left-hand menu, select Clusters and then Create cluster.
3. Under Cluster configuration, give the cluster a Name (e.g., "rm-agent").
4. Under Infrastructure, select AWS Fargate (serverless).
5. Click Create.

Task Definition

To create an ECS service, you will need to define a Task Definition which contains details of the Agent container's environment and system resources.

1. In the AWS console, navigate to Elastic Container Service, making sure to select your desired AWS region.
2. In the left-hand menu, select Task definitions, then Create new task definition, and Create new task definition with JSON.

Creating a task definition with JSON will allow you to define the container's environment variables, configuration, and resources in a single JSON object. In the below example, update the taskRoleArn, executionRoleArn, awslogs-region, and image URI. You'll also need to configure the environment variables according to your setup. See the Environment Variables page for detailed descriptions of each variable.

Example Task Definition

{
  "taskRoleArn": "arn:aws:iam::012345678912:role/RmAgentTaskRole", // Replace with the ARN of the Role you created in the Task Policy and Role
  "executionRoleArn": "arn:aws:iam::012345678912:role/ecsTaskExecutionRole", // Replace with the ARN of the AWS-managed ECS Task Execution Role
  "containerDefinitions": [
    {
      "essential": true,
      "name": "rm-agent",
      "image": "rm-agent:latest", // Replace with the image version you are using
      "cpu": 0,
      "environment": [
        {
          "name": "RM_CUSTOMER_DOMAIN",
          "value": "yourcompany.datagrail.io" // Replace with your DataGrail domain
        },
        {
          "name": "RM_PLATFORM_CREDENTIALS_LOCATION",
          "value": "arn:aws:secretsmanager:us-west-2:012345678901:secret:datagrail.api-key" // Replace with your Secrets Manager ARN for the platform API key
        },
        {
          "name": "RM_CREDENTIALS_MANAGER",
          "value": "{\"provider\": \"AWSSecretsManager\"}" // Update based on your credentials manager provider
        },
        {
          "name": "RM_STORAGE_MANAGER",
          "value": "{\"provider\": \"AWSS3\", \"options\": {\"bucket\": \"my-results-bucket\"}}" // Optional: configure if using cloud storage for results
        },
        {
          "name": "LOGLEVEL",
          "value": "INFO" // Optional: set logging level (DEBUG, INFO, WARNING, ERROR)
        }
      ],
      "mountPoints": [],
      "healthCheck": {
        "retries": 3,
        "command": [
          "CMD-SHELL",
          "test -f /app/healthy || exit 1"
        ],
        "timeout": 5,
        "interval": 30,
        "startPeriod": 120
      },
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-region": "us-west-2", // Replace with your AWS region
          "awslogs-group": "/ecs/rm-agent",
          "awslogs-create-group": "true",
          "awslogs-stream-prefix": "/ecs"
        }
      }
    }
  ],
  "family": "rm-agent",
  "requiresCompatibilities": ["FARGATE"],
  "runtimePlatform": {
    "operatingSystemFamily": "LINUX"
  },
  "networkMode": "awsvpc",
  "cpu": "1024",
  "memory": "2048"
}

Security Group

The Agent service only requires outbound traffic to:

• Systems you have configured
• A preconfigured S3 bucket for storing results
• A credentials manager (e.g., AWS Secrets Manager)
• DataGrail at https://<customer-name>.datagrail.io

No inbound traffic rules are required.

1. Navigate to EC2 and select Security Groups under Network & Security in the left-hand menu.
2. Click Create security group.
3. In Basic details,
   1. Give the security group a Name (e.g., "rm-agent-service-sg"). This is important to later identify the security group to use when creating the service.
   2. Add a Description (e.g., "Allow egress from RM Agent to external systems").
   3. Select the VPC that the service will be created in.
4. In Inbound Rules, do not add any rules (the Agent does not accept incoming connections).
5. Under Outbound Rule, leave the default All traffic rule.
6. Click Create security group.

ECS Service

1. Navigate to Elastic Container Service and select Clusters in the left-hand menu.
2. Select the cluster in which you will be deploying the Agent.
3. In the Services tab, select Create.
4. Under the Service details section,
   1. Under Task definition family, select the task definition you created in the Task Definition section and select the latest Revision.
   2. Under Service name, give the service a name (e.g., "rm-agent-service").
5. Under the Environment section, modify the Compute configuration.
   1. Under Compute options, select Launch type.
   2. In the Launch type dropdown, select FARGATE.
   3. In the Platform version dropdown, select LATEST.
6. Under the Deployment configuration section,
   1. Under Scheduling strategy, select Replica.
   2. Under Desired tasks, enter 1.
   3. Leave all default values under Deployment options and Deployment failure detection.
7. Under the Networking section,
   1. Select the VPC that you will be deploying the Agent in.
   2. Under Subnets, select a private subnet to place the Agent in.
   3. Under Security group, select Use an existing security group.
   4. Remove the default security group and select the security group created in the Security Group section.
   5. Under Public IP, select Turned off (the Agent only requires outbound connectivity).
8. Under the Load balancing section, leave Use load balancing unchecked (the Agent does not require incoming traffic).
9. Select Create.

You should then see the service appear on the management page of the cluster under the Services section. Click on the service and then on the Tasks tab in the service. If tasks are not in a Running state within a few minutes, you can inspect the stopped tasks to look for any errors or failures.

Your egress-only Agent service is now deployed and will initiate outbound connections to DataGrail and your configured systems!