API Proxy Connection

Call an internal or third-party REST APIs to fulfill data subject access, deletion, opt-out, and identifier retrieval requests.

Connection Configuration

The API Proxy connection uses the standard Connection object schema to define the API contract using the APIProxy ConnectorType. This ConnectorType requires an array of APIProxyQuery objects for its queries. Once the object has been created and the credentials have been stored, place the object in the DataGrailAgentConfig connections array.

Example Connection Object

{
  "name": "User Service",
  "uuid": "44d9e703-b8cf-40a8-a138-3cc110319b0d",
  "capabilities": ["privacy/access", "privacy/delete", "privacy/optout", "privacy/identifiers"],
  "mode": "live",
  "connector_type": "APIProxy",
  "queries": {
    "test": [
      {
        "url": "https://api.acme.com/v0/health-check",
        "headers": {
          "Authorization": "Basic {credentials}"
        },
        "verb": "GET",
        "verify_ssl": "true"
      }
    ],
    "identifiers": {
      "primary_phone_number": [
        {
          "url": "https://api.acme.com/v0/identifiers/get-phone?email={email}",
          "headers": {
            "Authorization": "Basic {credentials}",
            "Content-Type": "application/json"
          },
          "verb": "GET",
          "verify_ssl": "true",
          "valid_response_codes": [200]
        }
      ]
    },
    "access": [
      {
        "url": "https://api.acme.com/v0/users?email={email}",
        "headers": {
          "Authorization": "Basic {credentials}",
          "Content-Type": "application/json"
        },
        "verb": "GET",
        "verify_ssl": "true",
        "valid_response_codes": [200]
      }
    ],
    "delete": [
      {
        "url": "https://api.acme.com/v0/users",
        "headers": {
          "Authorization": "Basic {credentials}",
          "Content-Type": "application/json"
        },
        "body": "{{\"email\": \"{email}\"}}",
        "verb": "DELETE",
        "verify_ssl": "true",
        "valid_response_codes": [200, 404]
      }
    ],
    "optout": [
      {
        "url": "https://api.acme.com/v0/users/optout",
        "headers": {
          "Authorization": "Basic {credentials}",
          "Content-Type": "application/json"
        },
        "body": "{{\"email\": \"{email}\"}}",
        "verb": "PUT",
        "verify_ssl": "true",
        "valid_response_codes": [200, 201, 404]
      }
    ]
  },
  "credentials_location": "arn:aws:secretsmanager:Region:AccountId:secret:datagrail-apiproxy"
}

Test Query Required

The API Proxy connector requires an additional test query (or queries) for health checks to determine that the API is available and healthy.

APIProxyQuery Schema

Field
url	string (required) The full URL of the API endpoint.
headers	object(Headers) (optional) The headers to include in the API request.
body	string (optional) The body to include in the API request.
verb	string (required) The HTTP method for the API request. All HTTP methods are supported
verify_ssl	string (required) Determines whether to verify the SSL certificate of the URL. Accepted values are "true" or "false".
valid_response_codes[]	integer (required for v0.10.0+) Status codes that the Agent should consider successful. The default value is [200]. Note: An instance of a data subject not existing should be handled the same as a successful request. If the API returns 404, for example, 404 should be added to the array.
fail_retry_response_codes[]	integer (required for v0.10.0+) Status codes that the Agent should consider unsuccessful and should be retried.

JSON Representation

{
    "url": string,
    "headers": object(Headers),
    "body": string,
    "verb": string,
    "verify_ssl": string,
    "valid_response_codes": [integer]
}

Headers

Field
<header_name>	string The value of the header.

JSON Representation

{
  "<header_name>": string
}

Substitutions

The url, headers, and body parameters support named substitutions through string interpolation to format with identifiers coming from DataGrail, and values stored in your credentials manager. To make a substitution, enclose the variable name in curly braces in the string.

URL

URLs can support query parameters, which can be used to pass identifiers or other values. For example, if you want to pass an email address as a query parameter, you can use {email} in the URL.

URL

"url": "https://api.acme.com/v2/data-subject-request?email={email}"

Headers

Headers use substitutions to pass credentials from your credentials manager. The credentials manager will be configured to store the credentials in a key-value pair, and the value will be substituted into the header. For example, if you have a header that requires Basic Authentication, you can store the credentials in your credentials manager and reference them in the header like this:

Headers

"headers": {
    "Authorization": "Basic {credentials}"
}

Credentials to authenticate API requests will be stored in JSON format in your credentials manager. The key-value pairs in the secret will be dictated by substitutions you need to make in your headers. If the API uses Basic Authentication, like the above example, the credentials should be stored as the following key-value pair:

Credentials Manager Secret

{ "credentials": "<base64 encoded username:password>" }

Body

The request body can support any content type. If using JSON, the body must be formatted as a JSON string. Single curly braces { } should be used for substitutions, and double curly braces {{ }} to escape literals.

Body

"body": "{{\"email\": \"{email}\"}}"

---

API Contract

The APIProxy connection is designed to work with APIs that meet the following requirements:

Authentication

The API must support static token-based authentication and that token must be stored in your credentials manager.

Request/Response Flow

The API must support a synchronous response flow whereby all operations occur over a single open connection.

Each request must be stateless and perform deletion, identifier, or data retrieval without any context of previously executed queries.

Request Timeout

All API requests for a given query type must be completed in under 5 minutes by default. For example, if three endpoints are configured for access requests, then all three must be completed in under 5 minutes (cumulative). The timeout can be configured by setting the job_timeout_seconds parameter in the DataGrailAgentConfig object.

Response Body Format

Access Requests

The response body of an access request must be an array of objects. The value of each key can be of any datatype, including arrays and nested objects. Each object in the array will be converted into a separate file for your data subject.

The environment variable LOGLEVEL can be set to DEBUG to get more detailed feedback if responses are malformed. Be aware that this level of logging has the potential to expose sensitive data.

Example Access Response Body

[
  {
    "first_name": "Peter",
    "last_name": "Gibbons",
    "company": "Initech",
    "friends": ["Samir", "Michael"]
  },
  {
    "address_type": "Home",
    "address": {
      "street": "191 N. Lamar",
      "city": "Flander",
      "state": "Illinois",
      "zip_code": "77070"
    }
  }
]

The above example would generate two files for the data subject:

report_1.csv

first_name, Peter
last_name, Gibbons
company, Initech
friends_0, Samir
friends_1, Michael

report_2.csv

address_type, Home
address_street, 191 N. Lamar
address_city, Flander
address_state, Illinois
address_zip_code, 77070

Deletion Requests

The status code is the only signal used to determine if a deletion request was successful. A body can be included for logging purposes but will not propagate to the DataGrail platform.

Test Requests

The status code is the only signal used to determine if the API is available and healthy. A body can be included for logging purposes but will not propagate to the DataGrail platform.

Identifier Requests

The response body of an identifier retrieval request must be an array of objects. The key must be the snake_case Identifier Category name, and the value must be the identifier value.

Response Body for Identifier Under User ID Category

[
  {
    "user_id": "3ef6159b-a523-4ae4-a2b8-6b3ddedf1ab4"
  }
]

Identifier Name

The name of the field in the configuration is the name of the identifier in DataGrail in "snake_case". For example, User ID in DataGrail would be user_id in the configuration. Learn more about identifier setup.