Integrating API Proxy

Capabilities

DataGrail's API Proxy integration provides the following capabilities:

Product	Capability
Request Manager	Request Types Access, Deletion, Do Not Sell/Share, Identifiers Identifier Categories Any

Before You Start

The API Proxy integration enables you to connect custom or proprietary APIs to DataGrail by routing privacy requests through your infrastructure. To ensure successful integration, the APIs you connect must follow a flexible API contract that defines basic requirements for authentication, request/response patterns, and data formatting.

Requirements Overview

Review the requirements below to verify your API meets the necessary contract before proceeding with the integration.

Authentication

The API Proxy supports both static and dynamic (OAuth) authentication methods. Credentials are stored in your credentials manager and referenced through string interpolation in your API configuration.

Static authentication uses credentials that remain constant and do not require refresh:

• API Keys - Static keys passed in headers or query parameters
• Bearer Tokens - Long-lived tokens included in Authorization headers
• Basic Authentication - Username and password combinations encoded in headers
• Custom Headers - Static tokens or secrets passed in custom header fields

Dynamic authentication (OAuth) acquires short-lived tokens from your API's token endpoint automatically, so you don't need to manually refresh them. This is required when your API issues short-lived OAuth tokens. See OAuth Authentication for configuration details.

Request/Response Flow

The API must support a synchronous request/response pattern where the integration sends a request and receives a complete response over a single HTTP connection. The response must contain all necessary data or confirmation of the operation's success or failure.

Key requirements:

• Synchronous responses - The API must return results immediately within the request lifecycle (no polling, callbacks, or webhooks)
• Single connection - All operations must complete over one open HTTP connection without requiring follow-up requests
• Stateless requests - Each request must be independent and self-contained, performing deletion, identifier retrieval, or data access without relying on context from previous requests
• Complete responses - The response body must include all relevant data for the operation (for access requests) or appropriate status codes indicating success or failure (for deletion/test requests)

Request Timeout

All API requests for a given query type must be completed in under 5 minutes by default. For example, if three endpoints are configured for access requests, then all three must be completed in under 5 minutes (cumulative). The timeout can be configured by setting the RM_JOB_TIMEOUT_SECONDS environment variable in the container.

Response Body Format

The API responses for each request type must follow specific formatting requirements to ensure the RM Agent can properly process the data.

Access Requests

The response body of an access request must be an array of objects. The value of each key can be of any datatype, including arrays and nested objects. Each object in the array will be converted into a separate file for your data subject.

The environment variable LOGLEVEL can be set to DEBUG to get more detailed feedback if responses are malformed. Be aware that this level of logging has the potential to expose sensitive data.

Example Access Response Body

[
  {
    "first_name": "Peter",
    "last_name": "Gibbons",
    "company": "Initech",
    "friends": ["Samir", "Michael"]
  },
  {
    "address_type": "Home",
    "address": {
      "street": "191 N. Lamar",
      "city": "Flander",
      "state": "Illinois",
      "zip_code": "77070"
    }
  }
]

The above example would generate two files for the data subject:

report_1.csv

first_name, Peter
last_name, Gibbons
company, Initech
friends_0, Samir
friends_1, Michael

report_2.csv

address_type, Home
address_street, 191 N. Lamar
address_city, Flander
address_state, Illinois
address_zip_code, 77070

Deletion Requests

The status code is the only signal used to determine if a deletion request was successful. A response body can be included for logging purposes but will not propagate to the DataGrail platform.

Test Requests

The status code is the only signal used to determine if the API is available and healthy. A response body can be included for logging purposes but will not propagate to the DataGrail platform.

Identifier Requests

The response body of an identifier retrieval request must be an array of objects. The key must be the snake_case Identifier Category name, and the value must be the identifier value.

Response Body for Identifier Under User ID Category

[
  {
    "user_id": "3ef6159b-a523-4ae4-a2b8-6b3ddedf1ab4"
  }
]

Identifier Name

The name of the field in the configuration is the name of the identifier in DataGrail in "snake_case". For example, User ID in DataGrail would be user_id in the configuration. Learn more about identifier setup.

Connecting with RM Agent

To configure the API Proxy integration, you'll create an Agent Query Configuration object in the DataGrail application that defines how the RM Agent should interact with your API. This configuration includes the endpoint URLs, authentication headers, request bodies, and response handling for each privacy request type.

Each API endpoint you configure uses the APIProxyQuery schema below to specify the connection details and expected behavior.

Add the Agent Integration

1. In DataGrail, navigate to Agents and select your Agent.
2. In the top right, select Add New Integration and search for API Proxy.
3. Under Enabled Capabilities and Enabled Identifiers, select only those that will be used for this integration.
4. Enter the Credentials Location (e.g. AWS Secrets Manager ARN).
5. Select the Data Retrieval behavior for deletion requests.
   warning

   When using Retrieve Data, the data reviewed may not be exactly what is deleted due to the access and deletion logic executing separately!

6. Under Agent Query Configuration, add request logic to be executed within API Proxy for all enabled request types.
7. Finally, select Configure Integration. Wait a few moments to ensure that the connection is successful. For failed connections, review the Agent container logs for additional details.

Understanding String Interpolation

Before configuring your API queries, it's important to understand how string interpolation works. The schema supports using curly braces {} to dynamically insert values at runtime in the url, headers, and body fields.

This allows you to reference:

• Identifiers from DataGrail (e.g., {email}, {user_id})
• Credentials from your secrets manager (e.g., {api_key}, {credentials})

The examples below show how to use string interpolation in each field type.

APIProxyQuery Schema

When configuring your API endpoints, you'll define a queries array where each query object follows the APIProxyQuery schema below. Each query represents a single API endpoint that the RM Agent will call for a specific request type (access, deletion, identifier retrieval, or test).

Field
url	string (required) The full URL of the API endpoint.
headers	object(Headers) (optional) The headers to include in the API request.
body	string (optional) The body to include in the API request.
verb	string (required) The HTTP method for the API request. All HTTP methods are supported
verify_ssl	string (required) Determines whether to verify the SSL certificate of the URL. Accepted values are "true" or "false".
valid_response_codes[]	integer (required) Status codes that the Agent should consider successful. The default value is [200]. Note: An instance of a data subject not existing should be handled the same as a successful request. If the API returns 404, for example, 404 should be added to the array.
fail_retry_response_codes[]	integer (required) Status codes that the Agent should consider unsuccessful and should be retried.

Headers

Field
<header_name>	string The value of the header.

String Interpolation Examples

The following examples demonstrate how to use string interpolation in different parts of your query configuration.

URL

URLs can support query parameters, which can be used to pass identifiers or other values. For example, if you want to pass an email address as a query parameter, you can use {email} in the URL.

URL

"url": "https://api.acme.com/v2/data-subject-request?email={email}"

Headers

Headers use substitutions to pass credentials from your credentials manager. The credentials manager will be configured to store the credentials in a key-value pair, and the value will be substituted into the header. For example, if you have a header that requires Basic Authentication, you can store the credentials in your credentials manager and reference them in the header like this:

Headers

"headers": {
    "Authorization": "Basic {credentials}"
}

Credentials to authenticate API requests will be stored in JSON format in your credentials manager. The key-value pairs in the secret will be dictated by substitutions you need to make in your headers. If the API uses Basic Authentication, like the above example, the credentials should be stored as the following key-value pair:

Credentials Manager Secret

{ "credentials": "<base64 encoded username:password>" }

Body

The request body can support any content type. If using JSON, the body must be formatted as a JSON string. Single curly braces { } should be used for substitutions, and double curly braces {{ }} to escape literals.

Body

"body": "{{\"email\": \"{email}\"}}"

Test Query Configuration

Test Query Required

The API Proxy connector requires at least one test query for health checks to determine that the API is available and healthy. Test queries follow the same APIProxyQuery schema as other request types.

Test queries are used by the RM Agent to verify connectivity and availability before processing privacy requests. Configure test queries to call a lightweight endpoint that confirms the API is reachable and properly authenticated.

Example test query:

{
  "url": "https://api.acme.com/health",
  "headers": {
    "Authorization": "Bearer {api_key}"
  },
  "verb": "GET",
  "verify_ssl": "true",
  "valid_response_codes": [200],
  "fail_retry_response_codes": [500, 502, 503]
}

OAuth Authentication

If your API requires OAuth, you can configure the RM Agent to automatically acquire and refresh access tokens on your behalf. When an auth_provider block is present in your credentials, the agent acquires a token before each request and makes it available as {access_token} for use in your query templates.

Two OAuth configurations are supported, depending on how your API issues tokens:

• Client Credentials Grant - For any API with a standard OAuth 2.0 token endpoint
• Azure Active Directory - For APIs protected by Azure AD

Credentials Structure

Store one of the following in your credentials manager (e.g., AWS Secrets Manager or Azure Key Vault), depending on your API's OAuth setup:

Client Credentials Grant

Credentials Manager Secret

{
  "auth_provider": {
    "type": "oauth2",
    "token_url": "https://auth.example.com/oauth2/token",
    "client_id": "your-client-id",
    "client_secret": "your-client-secret",
    "scope": "your.api.scope",
    "auth_method": "client_secret_post",
    "headers": {"X-Tenant-Id": "your-tenant-id"}
  }
}

The auth_provider fields are described below:

Field
type	string (required)	The auth provider type. Use "oauth2".
token_url	string (required)	The URL of your API's OAuth token endpoint.
client_id	string (required)	The client ID issued by your API provider.
client_secret	string (required)	The client secret issued by your API provider.
scope	string (optional)	Space-delimited OAuth scopes to request. Omit if your API doesn't require one.
auth_method	string (optional)	How the client ID and secret are sent to token_url. Use "client_secret_post" (default) to send them as form fields, or "client_secret_basic" to send them in the request's Authorization header instead.
headers	object (optional)	Additional headers to send with the token request, for token endpoints that require something like a tenant ID or a vendor-specific header. These are separate from the headers on the API request that uses the resulting {access_token} (see below).

Azure Active Directory

Credentials Manager Secret

{
  "auth_provider": {
    "type": "msal",
    "client_id": "your-azure-app-client-id",
    "client_secret": "your-client-secret",
    "authority": "https://login.microsoftonline.com/your-tenant-id",
    "scopes": ["https://api.example.com/.default"]
  }
}

The auth_provider fields are described below:

Field
type	string (required)	The auth provider type. Use "msal" for Azure AD.
client_id	string (required)	The Application (client) ID from your Azure AD app registration.
client_secret	string (required)	The client secret from your Azure AD app registration.
authority	string (required)	The Azure AD authority URL. Format: https://login.microsoftonline.com/{tenant-id}
scopes	array[string] (required)	The OAuth scopes required by the target API (e.g., ["https://api.example.com/.default"]).

Using the Access Token

Once auth_provider is configured, reference {access_token} in your query headers, url, or body fields just like any other credential substitution:

Query Configuration

{
  "url": "https://api.example.com/users/{email}",
  "verb": "GET",
  "headers": {
    "Authorization": "Bearer {access_token}",
    "Content-Type": "application/json"
  }
}

Token lifecycle: Tokens are acquired on the first request, cached in memory, and automatically refreshed before they expire. No manual refresh configuration is needed.

Examples

API with a standard OAuth 2.0 token endpoint

Credentials:

{
  "auth_provider": {
    "type": "oauth2",
    "token_url": "https://auth.example.com/oauth2/token",
    "client_id": "your-client-id",
    "client_secret": "your-client-secret",
    "scope": "your.api.scope"
  }
}

Query Configuration:

{
  "url": "https://api.example.com/users/{email}",
  "verb": "GET",
  "headers": {
    "Authorization": "Bearer {access_token}",
    "Content-Type": "application/json"
  }
}

OAuth token endpoint requiring an extra header

Some token endpoints require an additional header, such as a tenant ID, alongside the standard client credentials request. Use the optional headers field to include it:

{
  "auth_provider": {
    "type": "oauth2",
    "token_url": "https://auth.example.com/oauth2/token",
    "client_id": "your-client-id",
    "client_secret": "your-client-secret",
    "headers": {"X-Tenant-Id": "your-tenant-id"}
  }
}

Customer API protected by Azure AD

Credentials:

{
  "auth_provider": {
    "type": "msal",
    "client_id": "12345678-1234-1234-1234-123456789abc",
    "client_secret": "your~secret~value",
    "authority": "https://login.microsoftonline.com/87654321-4321-4321-4321-abcdef123456",
    "scopes": ["api://customer-api/.default"]
  }
}

Query Configuration:

{
  "url": "https://customer-api.example.com/users/{email}",
  "verb": "GET",
  "headers": {
    "Authorization": "Bearer {access_token}",
    "Content-Type": "application/json"
  }
}

Microsoft Graph API

Credentials:

{
  "auth_provider": {
    "type": "msal",
    "client_id": "12345678-1234-1234-1234-123456789abc",
    "client_secret": "your~secret~value",
    "authority": "https://login.microsoftonline.com/common",
    "scopes": ["https://graph.microsoft.com/.default"]
  }
}

Query Configuration:

{
  "url": "https://graph.microsoft.com/v1.0/users/{email}",
  "verb": "GET",
  "headers": {
    "Authorization": "Bearer {access_token}"
  }
}

Custom token header format

If your API uses a non-standard authorization header, reference {access_token} in any header value:

{
  "url": "https://api.example.com/data",
  "verb": "GET",
  "headers": {
    "X-API-Token": "Token {access_token}"
  }
}

This produces: X-API-Token: Token eyJ0eXAi...

---

Troubleshooting

If you are unable to successfully connect the integration, review these common troubleshooting steps:

Agent Unable to Connect to API Proxy

1. Verify that the network is configured to allow the Agent to connect with the API Proxy instance.
2. Verify the Agent has permissions to access the API Proxy credentials stored in your vault.

Agent is Not Connected in DataGrail

1. Confirm that the Agent is running, and logs do not indicate any errors.
2. The DataGrail API Key used by the Agent is valid and not expired.
3. The Agent has permissions to access the DataGrail API Key stored in your vault.
4. Network egress is permitted from the Agent to your DataGrail domain.

OAuth Token Acquisition Fails (Azure AD)

Common causes:

• Incorrect client_id or client_secret in your credentials
• Wrong authority URL — verify the tenant ID is correct
• Client secret has expired — rotate the secret in Azure AD and update your credentials manager
• The app registration has not been granted permissions to the target API

OAuth Token Acquisition Fails (Client Credentials Grant)

Common causes:

• Incorrect client_id or client_secret, or an incorrect token_url
• The auth_method doesn't match what the token endpoint expects — try switching between client_secret_post and client_secret_basic
• The token endpoint requires an additional header that hasn't been set in headers

API Returns 401 Unauthorized After Token is Acquired

The token was acquired successfully but the API rejected it. Common causes:

• The requested scope doesn't match what the API requires — confirm the correct scope with your API provider
• Required permissions haven't been granted or consented to for the app registration or client
• The client hasn't been assigned access to the API

Technical Details

Access Type	Synchronous
Deletion Type	Synchronous (Whole Record)
Opt Out Type	Synchronous