Skip to main content

Google Cloud Run

Overview

If you are hosting in Google Cloud Platform, DataGrail recommends using Google Cloud Run to deploy the Request Manager Agent. Deploying in Cloud Run will result in most of the details of load balancing, SSL termination, and service uptime being managed in a simple and standard manner. This toolchain provides simple and robust management of your deployment. To get an overview of Google Cloud Run, check out Google's documentation.

Sourcing the Agent Image

The Request Manager Agent Docker image is hosted in DataGrail's private image registry. Once you have obtained the credentials from your DataGrail representative, you can pull the image using the following command:

Pulling the image
# Authenticate with the DataGrail registry
docker login contairium.datagrail.io -u $DATAGRAIL_SUBDOMAIN -p $DATAGRAIL_API_KEY

# Pull the latest Request Manager Agent image (or specify a version)
docker pull contairium.datagrail.io/datagrail-rm-agent:latest

If you prefer to use a pull-through cache instead of hosting the image yourself, you create an Artifact Registry remote repository.

Environment Variables

Agent Configuration

The Request Manager Agent requires a configuration environment variable named DATAGRAIL_AGENT_CONFIG to define its actions. The configuration contains metadata about the connections, credentials, and other settings required for the Agent to function properly.

For details on the configuration schema, see the DataGrailAgentConfig Schema documentation.

Example DATAGRAIL_AGENT_CONFIG
{
"connections": [
{
"name": "Metrics DB",
"uuid": "272c0934-0a06-4b11-8ec9-7755499001a3",
"capabilities": ["privacy/access","privacy/delete"],
"mode": "live",
"connector_type": "BigQuery",
"queries": {
"access": ["CALL metrics.dsr('access', %(email)s)"],
"delete": ["CALL metrics.dsr('delete', %(email)s)"]
},
"credentials_location": "datagrail-rm-agent-big-query"
}
],
"customer_domain": "acme.datagrail.io",
"datagrail_agent_credentials_location": "datagrail-rm-agent-credentials",
"datagrail_credentials_location": "datagrail-credentials",
"platform": {
"credentials_manager": {
"provider": "GCP",
"options": {
"project_id": "my-project"
}
},
"storage_manager": {
"provider": "GCPCloudStore",
"options": {
"bucket": "acme-datagrail-reports",
"project_id": "my-project"
}
}
}
}

Quick Setup Guide

The following sections contain the core steps for deploying the Agent in GCP with Cloud Run. Please note that depending on your environment's pre-existing configuration, you may need to take additional steps to configure your VPC, subnets, etc. Those are not covered in this document but we are happy to provide you with any assistance we can offer.

Create a Service Identity

Cloud Run will use the Compute Engine default service account if you do not specify a user-managed service account to run the service under. This principal is granted the Editor role, which grants read and write access on all resources in your Google Cloud project. To follow the principle of least privilege, it is recommended that you create a new service account with the most minimal set of permissions.

  1. In the Google Cloud console, navigate to IAM and click Service Account in the left-hand menu, and then Create Service Account in the top bar.
  2. Under Service account details, enter the Service account name (e.g. datagrail-rm-agent), and optionally a Service account description. The Service account ID will automatically be generated based on the Service account name.
  3. In the Grant this service account access to project section, add at least the following three roles to the service account:
    1. Artifact Registry Service Agent - Pulling the Docker image from Artifact Registry.
    2. Secret Manager Secret Accessor - Accessing the various secrets used by the Agent.
    3. Storage Object Creator - Writing the results of an access request to Cloud Storage.

Create Cloud Run Service

Important

As of Request Manager Agent v0.9.0, the container port has changed to 8080. (If using a version older than v0.9.0, set the container port to 80).

  1. In the Google Cloud console, navigate to Cloud Run and click Create Service at the top of the page to start the Service wizard.
  2. Under Container Image URL, enter the container image URL of the uploaded Agent image.
  3. Under Configure, enter a Service name and Region.
  4. Under Authentication, select Allow unauthenticated invocations to enable DataGrail to reach the Agent successfully.
  5. Under CPU Allocation and Pricing, select CPU is always allocated to ensure processing of background tasks.
    Important Setting

    If this step is missed, the Agent will not be able to process background tasks!

  6. Under Service Autoscaling, set minimum number of instances to 1.
  7. Under Ingress Control, if you will be using an Application Load Balancer, select Internal and then Allow traffic from external Application Load Balancers. If not, you can select All which will allow any external traffic to reach the API.
    Best Practice

    We strongly recommend configuring an Application Load Balancer to limit the incoming traffic to only the DataGrail IP address.

  8. Expand the Container(s), Volumes, Networking, Security section, and select the Container tab. Set the Container port to 8080.
  9. Under Resources, set Memory to at least 4 Gib, and CPU to 1.
  10. In the Variables & Secrets tab, select Add Variable and set the DATAGRAIL_AGENT_CONFIG environment variable. The value should be set to the configuration JSON object.
  11. Under Requests, set the timeout to 300, and maximum concurrent requests to 80.
  12. Under Revision Autoscaling, set both the minimum and maximum number of instances to 1 to avoid any automatic autoscaling actions.
  13. With everything configured, click Create at the bottom of the wizard and your service will launch!

Create an Application Load Balancer

The Agent service does not support internal SSL termination, configuring a load balancer or another form of SSL termination is required. TLS 1.2+ is required to provide secure communication between services.

  1. In the Google Cloud console, navigate to the Load balancing page
  2. Click Create load balancer.

Start your Configuration

  1. Under Type of load balancer, select Application Load Balancer (HTTP/HTTPS), and click Next.
  2. Under Public facing or internal, select Public facing (external), and click Next.
  3. Under Global or single region deployment, select Best for regional workloads, and click Next.
  4. Under Create load balancer, ensure that the load balancer has both Public facing (external) and Regional features.
  5. Click Configure.
    Important Setting

    In the next step, ensure that the Network you choose has a proxy-only subnet. For more information, see the documentation.

  6. On the following page, enter the Load Balancer Name, Region, and Network.

Frontend Configuration

  1. Before proceeding, make sure that you have a valid SSL certificate. For more information, see the documentation.
  2. Under New Frontend IP and port:
    1. Optionally enter a Name.
    2. In the Protocol dropdown list, select HTTPS (includes HTTP/2 and HTTP/3).
    3. Under Certificate, either select your certificate, or Create a new certificate.

Backend Configuration

  1. Under Create or select backend services, select Create a backend service.
  2. In the Create backend service window, enter a Name.
  3. Under Backend type, select Serverless network endpoint group.
  4. In the Backends > New backend dropdown list, select Create new serverless network endpoint group.
    1. In the Create Serverless network endpoint group window, enter a Name.
    2. Under Region, the region of your Application Load Balancer should be displayed.
    3. Under Serverless network endpoint group type, Cloud Run should be displayed.
    4. In the Select service > Service dropdown list, select the Cloud Run service you created in the Create Cloud Run Service section.
    5. Click Create.
  5. Back in the Create backend service window, also click Create.

Configure Routing Rules

  1. Click Simple host and path rule.
  2. Select the backend service created in the Backend Configuration section from the Backend dropdown list.

Test the Load Balancer

  1. Navigate back to the Load balancing homepage.
  2. You will see your load balancer under the Load balancers section.
  3. Click on the load balancer you just created.
  4. Note the IP Address of the load balancer.
  5. You can test your load balancer using a web browser by going to https://<load-balancer-ip-address>.
  6. Update your domain's DNS records by adding an A record (e.g. datagrail-rm-agent.your-domain.com) with this IP address.

Create Cloud Armor Policy

Best Practice

We strongly recommend creating a Cloud Armor policy to limit the incoming traffic to only the DataGrail IP address.

  1. In the Google Cloud console, navigate to Cloud Armor and click Create policy.
  2. Under Configure policy, give the policy a Name and a Description.
  3. Under the Policy type, select Backend network policy.
  4. Under Scope, select Global and click Next.
  5. Under Add more rules, click New rule.
    1. Enter a description of the rule (e.g. "Limit the incoming traffic to only the DataGrail IP address.")
    2. Under Condition > Mode, select Basic mode (IP addresses/ranges only).
    3. Under match, input 52.36.177.91.
    4. In the Action dropdown list, select Allow.
    5. Under Priority, enter 0.
    6. Click Next step.
  6. In Apply policy to targets > Targets, click Add target.
  7. Under Backend service target 1, select the load balancer backend created in the Backend Configuration section.
  8. Skip Advanced configurations and click Create policy. The policy can take some time to attach, so wait a few minutes before confirming that you are no longer able to access the Cloud Run service from your browser.

 

Need help?
If you have any questions, please reach out to your dedicated Account Manager or contact us at support@datagrail.io.

Disclaimer: The information contained in this message does not constitute as legal advice. We would advise seeking professional counsel before acting on or interpreting any material.