Skip to main content

Quick Start

This guide provides a streamlined approach to deploying the Request Manager Agent, configuring it to connect to your internal systems, and processing DSRs with minimal setup.

Deploy the Agent​

The first half of the guide focuses on configuring and deploying the Request Manager Agent in your environment before connecting it with DataGrail.

Prerequisites

Before performing the steps below, ensure you have the following:

  • DataGrail image registry credentials.
  • Privileges to create and manage containerized workloads in your cloud environment.
  • Access to the DataGrail platform.

Source the Image​

Request Manager Agent images are distributed via DataGrail's self-hosted Docker registry. Accessing the images requires basic authentication credentials provided by your DataGrail representative.

Pulling the image
# Authenticate with the DataGrail registry
docker login contairium.datagrail.io -u $DATAGRAIL_SUBDOMAIN -p $DATAGRAIL_API_KEY

# Pull the latest Request Manager Agent image (or specify a version)
docker pull contairium.datagrail.io/datagrail-rm-agent:latest

If you prefer to use a pull-through cache instead of hosting the image yourself, you can set up your container orchestration platform's deployment manifest to fetch the image directly during the deployment process.

Agent Credentials​

The Request Manager Agent uses a credentials manager to securely store sensitive information required for its operations. Supported providers can be found in the CredentialsManagerProvider list. The choice of credentials manager depends on your infrastructure and security requirements.

  1. Create the OAuth Client Credentials in your credentials manager. Both the client_id and client_secret are arbitrary values that you control.

    OAuth Client Credentials
    {
      "client_id": "<arbitrary identifier for the credential>",
      "client_secret": "<arbitrary password>"
    }

  2. Create the Callback Token in your credentials manager. This token will be provided by your DataGrail representative.

    Callback Token
    {
      "token": "<your DataGrail token>"
    }

Refer to the Storing Credentials Guide for detailed instructions on securely managing credentials using your preferred credentials manager. During the next step, you will specify the locations of these credentials in the Agent's configuration.

Define the Configuration​

The Request Manager Agent requires a configuration environment variable named DATAGRAIL_AGENT_CONFIG to define its behavior and connections to internal systems. The configuration contains metadata about the connections, credentials, and other settings required for the Agent to function properly. Configurations vary based on cloud provider due to differences in credentials and storage management.

warning

Configuration validation is performed at startup, and any errors will prevent the Agent from starting.

Minimum Configuration
{
  "connections": [],
  "customer_subdomain": "<your subdomain>.datagrail.io",
  "datagrail_agent_credentials_location": "<AWS ARN of your OAuth Client Credentials>",
  "datagrail_credentials_location": "<AWS ARN of your Callback Token>",
  "platform": {
    "credentials_manager": {
      "provider": "<AWSSecretsManager|AWSSSMParameterStore>"
    },
    "storage_manager": {
      "provider": "AWSS3",
      "options": {
        "bucket": "<S3 bucket name>"
      }
    }
  }
}

Replace the placeholder values in the Minimum Configuration above with their actual values:

  1. customer_subdomain: Your DataGrail subdomain (e.g., "acme.datagrail.io").
  2. datagrail_agent_credentials_location: The ARN of the OAuth Client Credentials in Secrets Manager or Parameter Store.
  3. datagrail_credentials_location: The ARN of the Callback Token credential in Secrets Manager or Parameter Store.
  4. platform.credentials_manager.options
    • provider: Credentials manager platform (AWSSecretsManager or AWSSSMParameterStore).
  5. platform.storage_manager.options
    • bucket: S3 bucket name.

For detailed information about these parameters and any optional ones, refer to the DataGrailAgentConfig Schema documentation.

Run the Agent​

The Request Manager Agent can be deployed using any container orchestration platform, such as Google Cloud Run, Amazon ECS, Azure Container Apps, or Kubernetes.

Refer to the specific deployment guides for detailed instructions on running the Agent on your chosen platform:

Connect to DataGrail​

Once the Agent is healthy, connect it to DataGrail by following the Internal Systems (OAuth) connection instructions.

Configure the Internal System​

In this half of the guide, you will configure the agent to execute request logic in your internal system.

The examples provided below use a Postgres database containing a users table which will be queried using the Data Subject's verified email address. If you are integrating a different system, use the configuration for that system located in the Connections section.

Prerequisites

Before performing the steps below, ensure the following:

  • Request Manager Agent is deployed and connected in DataGrail.
  • Network is configured to allow the Agent to connect with the PostgreSQL instance.

Create Request Logic​

In this step, create the request logic to be executed in your system for a specific request type.

When a Data Subject requests to access the personal information that your company maintains on them, they will submit an Access request. Your request logic should return all applicable information found in this system.

Example Access Query
SELECT * FROM contacts WHERE email = 'subject@domain.com';
Using Stored Procedures

For ease of maintainability and readability, using stored procedures is recommended instead of individual queries. When you need to modify the underlying queries, there will be no need to modify the Agent configuration nor restart the Agent.

Example Stored Procedure
CALL sp_dsr_access('subject@domain.com');

System Credentials​

In this section, you will create and configure limited scope credentials for the agent to use with the internal system.

  1. In your system, create the credentials for the agent to connect to your system and execute the request logic (e.g., database user and password). Only grant these credentials the minimum necessary permissions required to perform the configured actions.

  2. Using the format specified in the connection documentation, replace any placeholder values with actual values. The following credentials JSON format is used by the Postgres connector

    {
      "user": "<agent user>",
      "password": "<agent password>",
      "host": "<server domain name or IP address>",
      "port": "<port, e.g. 5432>",
      "dbname": "<database name>"
    }

  3. In your credentials manager, create a secret with the credentials JSON.

  4. Confirm that the agent has the necessary permissions to retrieve the value of the new secret.

Create Connection Object​

To define the interactions between the agent and the internal system, configure the Connection object.

  1. Using the empty connection object below, enter a connection name and uuid (v4).

    {
      "name": "",
      "uuid": "",
      "capabilities": [],
      "mode": "",
      "connector_type": "Postgres",
      "queries": {
        "access": [],
        "delete": [],
        "optout": [],
        "identifiers": {}
      },
      "credentials_location": ""
    }

  2. Set the capabilities that correspond to your request logic.

  3. Under queries, add your request logic as a string to the corresponding request type.

  4. The queries object allows query strings to be dynamically formatted with identifiers. Identifiers are passed individually to queries, and are bound to the variables in the operation. Replace any static identifier in the request logic using pyformat parameter formatting (e.g., ...WHERE email=%(email)s).

    Parameter Binding Example
    -- BEFORE
    CALL sp_dsr_access('subject@domain.com');

    -- AFTER
    CALL sp_dsr_access(%(email)s);

  5. Set the credentials_location value to the location of the secret created above (e.g., AWS ARN).

Example Connection Objects

After following the steps above, the resulting connection object should look similar to these examples.

Example Connection Object with Access Capability
{
  "name": "Contacts DB",
  "uuid": "93f0097e-7341-412a-8311-5e54c9360863",
  "capabilities": ["privacy/access"],
  "mode": "live",
  "connector_type": "Postgres",
  "queries": {
    "access": ["CALL sp_dsr_access(%(email)s);"],
    "delete": [],
    "optout": [],
    "identifiers": {}
  },
  "credentials_location": "arn:aws:secretsmanager:Region:AccountId:secret:datagrail-rm-agent-postgres"
}

Update and Restart Agent​

In this section, you will integrate the internal system with DataGrail by updating the agent environment variable and restarting the agent to apply the changes.

Already Processing Requests?

If your DataGrail account is already processing Data Subject Requests, new agent connections may be automatically processed with all new requests. If you are not sure, check with your DataGrail admin or dedicated Account Manager before starting this section.

  1. In the DATAGRAIL_AGENT_CONFIG environment variable, add the new connection object to the connections array.
  2. Save the environment variable, and restart the agent service to apply the changes.
  3. After the restart, confirm that the agent service is healthy, and the logs contain no errors.
  4. In DataGrail, update the integration by navigating to Integrations and select the agent integration.
  5. Select Edit Integration, enter the Client Secret, and then select Save Changes.
  6. To view the new connection, navigate back to Integrations and locate it in the list.
Processing Your First Request

Now that you've successfully added this connection to DataGrail, try it out by submitting and processing a request.

 

Need help?
If you have any questions, please reach out to your dedicated Account Manager or contact us at support@datagrail.io.

Disclaimer: The information contained in this message does not constitute as legal advice. We would advise seeking professional counsel before acting on or interpreting any material.