Skip to main content

Amazon Elastic Container Service

Overview​

If you are hosting in AWS, DataGrail recommends using Elastic Container Service (ECS) to deploy the Request Manager Agent. ECS simplifies the management of load balancing, SSL termination, and service uptime, offering a reliable and streamlined deployment process. For more information about ECS, refer to Amazon's documentation.

Source the Agent Image​

The Request Manager Agent Docker image is hosted in DataGrail's private image registry. Once you have obtained credentials from your DataGrail representative, you can pull the image using the following command:

Pulling the image
# Authenticate with the DataGrail registry
docker login contairium.datagrail.io -u $DATAGRAIL_SUBDOMAIN -p $DATAGRAIL_API_KEY

# Pull the latest Request Manager Agent image (or specify a version)
docker pull contairium.datagrail.io/datagrail-rm-agent:latest

If you would like to, you may pull the image into your own Docker repository, or use it directly from our repository upon service startup.

Docker Pull Error: No Matching Manifest

If you are attempting to docker pull the image locally and receive a "no matching manifest..." error, update the command to use the --platform option.

For example, if the original command is docker pull ${IMAGE_URI}:

  • For M1 Macs, use: docker pull ${IMAGE_URI} --platform=linux/amd64
  • For M2 Macs, use: docker pull ${IMAGE_URI} --platform=linux/x86_64

Agent Configuration​

The Request Manager Agent requires a configuration environment variable named DATAGRAIL_AGENT_CONFIG to define its actions. The configuration contains metadata about the connections, credentials, and other settings required for the Agent to function properly.

For details on the configuration schema, see the DataGrailAgentConfig Schema documentation.

Example DATAGRAIL_AGENT_CONFIG
{
  "connections": [
    {
      "name": "Metrics DB",
      "uuid": "272c0934-0a06-4b11-8ec9-7755499001a3",
      "capabilities": ["privacy/access","privacy/delete"],
      "mode": "live",
      "connector_type": "Redshift",
      "queries": {
        "access": ["call dsr('access', %(email)s)"],
        "delete": ["call dsr('delete', %(email)s)"]
      },
      "credentials_location": "arn:aws:secretsmanager:us-west-2:000123456789:secret:redshift-cvh7s2"
    }
  ],
  "customer_domain": "acme.datagrail.io",
  "datagrail_agent_credentials_location": "arn:aws:secretsmanager:us-west-2:000123456789:secret:datagrail-agent-sjc7s3",
  "datagrail_credentials_location": "arn:aws:secretsmanager:us-west-2:000123456789:secret:datagrail-bs83nh",
  "platform": {
    "credentials_manager": {
      "provider": "AWSSecretsManager"
    },
    "storage_manager": {
      "provider": "AWSS3",
      "options": {
        "bucket": "acme-datagrail-reports"
      }
    }
  }
}

Define the configuration in a JSON file. Once complete, you can use the jq command-line JSON processor to convert the file to a single line string suitable for use in the DATAGRAIL_AGENT_CONFIG environment variable. Save the output of this command for use in the next steps.

Convert file to a single line string
jq -sR . datagrail-agent-config.json

Quick Setup Guide​

The following sections contain the core steps to creating an ECS Agent service. Please note that depending on your AWS environment's pre-existing configuration, you may need to take additional steps to configure your VPC, subnets, etc. Those are not covered in this document but we are happy to provide you with any assistance we can offer.

Task Policy and Role​

To give the ECS service permission to make API requests to AWS services, you will need to define a task IAM policy and role.

Policy​

  1. In the AWS console, navigate to Identity and Access Management.
  2. Under Access management, select Policies and then Create policy.
  3. In the Policy editor, select JSON on the right-hand side.
  4. At a minimum, the Agent should be allowed to perform the s3:PutObject and secretsmanager:GetSecretValue actions on a set of defined resources. Use the below example policy and update it with the ARNs of your resources.
    Example Policy
    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "S3PutObject",
          "Action": "s3:PutObject",
          "Effect": "Allow",
          "Resource": [
            "arn:aws:s3:::datagrail-reports-bucket/*" // Replace with your S3 bucket ARN
          ]
        },
        {
          "Sid": "SecretsManagerGetSecretValue",
          "Action": "secretsmanager:GetSecretValue",
          "Effect": "Allow",
          "Resource": [
            "arn:aws:secretsmanager:us-west-2:012345678901:secret:datagrail.agent-credentials",
            "arn:aws:secretsmanager:us-west-2:012345678901:secret:datagrail.credentials"
            // Include other secrets the Agent will need access to, such as connector credentials
          ]
        }
      ]
    }
  5. Once done creating the policy, click Next.
  6. Create a name for the policy and optionally a description.
  7. Confirm that the permissions in the Permissions defined in this policy are accurate and add any optional tags.
  8. Click Create policy.

Role​

  1. Navigate back to the IAM home screen.
  2. Under Access management, select Roles and then Create role.
  3. Under Trusted entity type, select AWS service.
  4. Under Use case, choose Elastic Container Service as the service, and Elastic Container Service Task as the use case and click Next.
  5. In the Permissions policies section, search for the policy that you have just created and select it.
  6. Under Role details, give the role a name and optionally update the description.
  7. Click Create role.

ECS Cluster​

Optional if the Agent service will be deployed in an existing cluster.

  1. Navigate to Elastic Container Service, making sure you are in your desired AWS region.
  2. In the left-hand menu, select Clusters and then Create cluster.
  3. Under Cluster configuration, give the cluster a Name.
  4. Under Infrastructure, select AWS Fargate (serverless).
  5. Click Create.

Task Definition​

To create an ECS service, you will need to define a Task Definition which contains details of the Agent container's environment, configuration, and system resources.

  1. In the AWS console, navigate to Elastic Container Service, making sure to select your desired AWS region.
  2. In the left-hand menu, select Task definitions, then Create new task definition, and Create new task definition with JSON.

Creating a task definition with JSON will allow you to define the container's environment variables, configuration, and resources in a single JSON object. In the below example, update the taskRoleArn, executionRoleArn, awslogs-region, image URI, and the value of the DATAGRAIL_AGENT_CONFIG variable with the contents created in the Agent COnfiguration section.

The launch command is specified in the Docker image configuration. In general, there should be no need to override the default launch command directly, however for informational purposes the command used inside the image is:

CMD ["supervisord", "-n", "-c", "/etc/rm.conf"]
Example Task Definition
{
  "taskRoleArn": "arn:aws:iam::012345678912:role/DataGrailAgentTaskRole", // Replace with the ARN of the Role you created in the Task Policy and Role
  "executionRoleArn": "arn:aws:iam::012345678912:role/ecsTaskExecutionRole", // Replace with the ARN of the AWS-managed ECS Task Execution Role
  "containerDefinitions": [
    {
      "essential": true,
      "name": "datagrail-rm-agent",
      "image": "datagrail-rm-agent:latest", // Replace with the image version you are using
      "portMappings": [
        {
          "hostPort": 8080,
          "protocol": "tcp",
          "containerPort": 8080
        }
      ],
      "command": ["supervisord", "-n", "-c", "/etc/rm.conf"],
      "cpu": 0,
      "environment": [
        {
          "name": "DATAGRAIL_AGENT_CONFIG",
          "value": "" // Replace with the JSON object from the Environment Variables section
        }
      ],
      "mountPoints": [],
      "workingDirectory": "/app",
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-region": "us-west-2", // Replace with your AWS region
          "awslogs-group": "/ecs/datagrail-rm-agent",
          "awslogs-create-group": "true",
          "awslogs-stream-prefix": "/ecs"
        }
      },
      "healthCheck": {
        "retries": 3,
        "command": [
          "CMD-SHELL",
          "curl -f http://localhost:8080/docs || exit 1"
        ],
        "timeout": 5,
        "interval": 30,
        "startPeriod": 1
      }
    }
  ],
  "family": "datagrail-rm-agent",
  "requiresCompatibilities": ["FARGATE"],
  "runtimePlatform": {
    "operatingSystemFamily": "LINUX"
  },
  "networkMode": "awsvpc",
  "cpu": "1024",
  "memory": "2048"
}

Security Groups​

Important

As of Request Manager Agent v0.9.0, the container port has changed to 8080. (If using a version older than v0.9.0, the Load Balancer and Service Security Groups will require outbound and inbound TCP rules on port 80, respectively).

Both the load balancer and service created in the following sections should have strict security groups to act as a virtual firewalls to control inbound and outbound traffic.

  1. Navigate to EC2 and select Security Groups under Network & Security in the left-hand menu.

Load Balancer Security Group​

Load balancer ingress should be allowed on port 443 from DataGrail's VPC IP of 52.36.177.91, and requests from any other source should be rejected.

  1. Click Create security group.
  2. In the Basic details section,
    1. Give the security group a Name indicating that it is for the load balancer (e.g., "datagrail-rm-agent-load-balancer-sg"). This is important to later identify the security group to use when creating the load balancer.
    2. Add a Description of the security group (e.g., "Allow HTTPS ingress to DataGrail and TCP egress to DataGrail Agent service").
    3. Select the VPC that the load balancer will be created in.
  3. In the Inbound Rules section,
    1. Click Add rule.
    2. In the Type dropdown, select HTTPS.
    3. In the Source dropdown, select Custom and add 52.36.177.91/32 (DataGrail's IP) as the only allowed CIDR block.
Important

The next step is temporary. An outbound rule will be created later to allow egress from the load balancer to the service security group after it has been created.

  1. Under Outbound rules, delete the default rule.
  2. Click Create security group.

Service Security Group​

The service should restrict ingress to the load balancer, and egress to:

  • Systems you have configured
  • A preconfigured S3 bucket for storing results
  • Secrets Manager for retrieving credentials
  • DataGrail at https://<customer-name>.datagrail.io
  1. Navigate back to Security Groups and click Create security group.
  2. In Basic details,
    1. Give the security group a Name, indicating that it is for the service (e.g., "datagrail-rm-agent-service-sg"). This is important to later identify the security group to use when creating the service.
    2. Add a Description (e.g., "Allow ingress to application load balancer").
    3. Select the VPC that the service will be created in.
  3. In Inbound Rules,
    1. Click Add rule.
    2. In the Type dropdown, select Custom TCP.
    3. Under Port range, enter 8080.
    4. In the Destination dropdown, select Custom, and select the security group created in the Load Balancer Security Group section under Security Groups.
  4. Under Outbound Rule, leave the default All traffic rule.
  5. Click Create security group.

Now that the service security group has been created, an outbound rule to the application load balancer security group can be added.

  1. Navigate back to security groups and select the security group created in the Load Balancer Security Group.
  2. Under Actions in the top-right, select Edit outbound rules.
  3. Click Add rule.
  4. In the Type dropdown, select Custom TCP.
  5. Under Port Range, enter 8080.
  6. In the Destination dropdown, select Custom, and select the security group created in the Service Security Group section under Security Groups.

Target Group​

Important

As of Request Manager Agent v0.9.0, the container port has changed to 8080. (If using a version older than v0.9.0, the Target Group port will need to be set to 80).

The target group defines where the load balancer routes requests and performs health checks on the targets.

  1. Navigate to EC2 and select Target Groups under Load Balancing in the left-hand menu.
  2. Click Create target group.
  3. In the Basic configuration section,
    1. Under Choose a target type, select IP addresses.
    2. Under Target group name, give the target group a name indicating that it is for the Agent service load balancer (e.g., "datagrail-rm-agent-target-group")
    3. Under Protocol : Port, select HTTP in the Protocol dropdown, and enter 8080 in the Port field.
    4. Under IP address type, select IPv4.
    5. Under VPC, select the VPC where the Agent service will be deployed.
    6. Under Protocol version, select HTTP/1.
  4. In the Health checks section,
    1. Under the Health check protocol dropdown, select HTTP.
    2. Under the Health check path field, enter /docs.
  5. Click Next.
  6. On the Register targets, do not modify any settings and click Create target group. The Agent service will be registered during its creation.

Application Load Balancer​

The Agent service does not support internal SSL termination, so configuring a load balancer or another form of SSL termination is required. TLS 1.2+ is required to provide secure communication between services.

  1. Navigate to EC2, and select Load Balancers under Load Balancing in the left-hand menu.
  2. Click Create load balancer.
  3. Under Load balancer types, select Create under Application Load Balancer.
  4. Under Basic configuration,
    1. Enter a Load balancer name.
    2. Under Scheme, select Internet-facing.
    3. Under IP address type, select IPv4.
  5. Under Network mapping,
    1. Select the VPC, select the VPC where the Agent service will be deployed.
    2. Under Mappings, select at least two Availability Zone and one public subnet per zone
  6. Under Security groups, remove the default group and select the security group you created in the Load Balancer Security Group section.
  7. Under Listeners and routing,
    1. In the Protocol dropdown, select HTTPS.
    2. In the Port field, enter 443.
    3. Set the Default action to Forward to the target group that you created in the Create Target Group section.
  8. Under Security policy,
    1. In the Security category dropdown, select All security policies.
    2. In the Policy name dropdown, select ELBSecurityPolicy-TLS13-1-2-2021-06.
  9. Review the Summary to make sure everything looks correct.
  10. Click Create load balancer

ECS Service​

  1. Navigate to Elastic Container Service and select Clusters in the left-hand menu.
  2. Select the cluster in which you will be deploying the Agent.
  3. In the Services tab, select Create.
  4. Under the Environment section in the Create wizard, modify the Compute configuration.
    1. Under Compute options, select Launch type.
    2. In the Launch type dropdown, select FARGATE.
    3. In the Platform version dropdown, select LATEST.
  5. Under the Deployment configuration section,
    1. Under Application type, select Service.
    2. Under Task definition, select the task definition you created in the Create Task Definition section and select the latest Revision.
    3. Under Service name, give the service a name (e.g., "datagrail-rm-agent-service").
    4. Under Service type, select Replica.
    5. Under Desired tasks, enter 1.
    6. Leave all default values under Deployment options and Deployment failure detection.
  6. Under the Networking section,
    1. Select the VPC that you will be deploying the Agent in.
    2. Under Subnets, select a private subnet to place the Agent in.
    3. Under Security group, select Use and existing security group.
    4. Remove the default security group and select the security group created in the Service Security Group section.
  7. Under the Load balancing section,
    1. In the Load balancer type dropdown, select Application Load Balancer.
    2. The Container dropdown will prepopulate with the single container defined in the task definition.
    3. Under Application Load Balancer, select Use and existing load balancer.
    4. In the Load balancer dropdown, select the load balancer that you created in the Create Application Load Balancer section.
    5. In the Health check grace period field, enter 15.
    6. Under Listener, select Use an existing listener and select 443:HTTPS in the Listener dropdown.
    7. Under Target group, select Use an existing target group, and select the target group created in the Create Target Group section in the Target group name dropdown.
  8. Select Create.

You should then see the service appear on the management page of the cluster under the Services section. Click on the service and then on the Tasks tab in the service. If tasks are not in a Running state within a few minutes, you can inspect the stopped tasks to look for any errors or failures.

Alias Record for Application Load Balancer in Route 53​

  1. Navigate to Route 53 and click on Hosted zones in the left-hand menu.
  2. Select the hosted zone where you will be deploying the Agent and click Create record.
  3. In the Record name field, enter the subdomain that you would like to use (e.g., "datagrail-rm-agent").
  4. In the Record type dropdown, select A.
  5. Toggle the Alias button ON.
  6. Under Route traffic to,
    1. Select Alias to Application and Classic Load Balancer in the Choose endpoint dropdown.
    2. Select the region where the load balancer was deployed in the Choose Region dropdown.
    3. Select the load balancer in the Choose load balancer dropdown.
    4. Keep Routing policy set to Simple routing and Evaluate target health set to Yes
  7. Click Create records.

Your Agent service will now be reachable at your subdomain!

 

Need help?
If you have any questions, please reach out to your dedicated Account Manager or contact us at support@datagrail.io.

Disclaimer: The information contained in this message does not constitute as legal advice. We would advise seeking professional counsel before acting on or interpreting any material.