API Proxy

Call an internal or third-party REST APIs to fulfill data subject access, deletion, opt-out, and identifier retrieval requests.

Connection Configuration

The API Proxy connection uses the standard Connection object schema to define the API contract using the APIProxy ConnectorType. This ConnectorType requires an array of APIProxyQuery objects for its queries. Once the object has been created and the credentials have been stored, place the object in the RMAgentConfig connections array.

Example Configuration

{
    "name": "User Service",
    "uuid": "44d9e703-b8cf-40a8-a138-3cc110319b0d",
    "capabilities": ["privacy/access", "privacy/delete", "privacy/optout", "privacy/identifiers"],
    "mode": "live",
    "connector_type": "APIProxy",
    "queries": {
        "test": [
            {
                "url": "https://api.acme.com/v0/health-check",
                "headers": {
                    "Authorization": "Basic {credentials}"
                },
                "verb": "GET",
                "verify_ssl": "true"
            }
        ],
        "identifiers": {
            "phone_number": [
                {
                    "url": "https://api.acme.com/v0/identifiers/get-phone",
                    "headers": {
                        "Authorization": "Basic {credentials}",
                        "Content-Type": "application/json"
                    },
                    "body": "{{\"email\": \"{email}\"}}",
                    "verb": "POST",
                    "verify_ssl": "true",
                    "valid_response_codes": [200]
                }
            ]
        },
        "access": [
            {
                "url": "https://api.acme.com/v0/users?{email}",
                "headers": {
                    "Authorization": "Basic {credentials}",
                    "Content-Type": "application/json"
                },
                "verb": "GET",
                "verify_ssl": "true",
                "valid_response_codes": [200]
            }
        ],
        "delete": [
            {
                "url": "https://api.acme.com/v0/users",
                "headers": {
                    "Authorization": "Basic {credentials}",
                    "Content-Type": "application/json"
                },
                "body": "{{\"email\": \"{email}\"}}",
                "verb": "DELETE",
                "verify_ssl": "true",
                "valid_response_codes": [200, 404]
            }
        ],
        "optout": [
            {
                "url": "https://api.acme.com/v0/users/optout",
                "headers": {
                    "Authorization": "Basic {credentials}",
                    "Content-Type": "application/json"
                },
                "body": "{{\"email\": \"{email}\"}}",
                "verb": "PUT",
                "verify_ssl": "true",
                "valid_response_codes": [200, 201, 404]
            }
        ]
    },
    "credentials_location": "arn:aws:secretsmanager:Region:AccountId:secret:datagrail-apiproxy"
}

Test Query Required

The API Proxy connector requires an additional test query (or queries) for health checks to determine that the API is available and healthy.

APIProxyQuery

Field
url	string (required) The full URL of the API endpoint.
headers	object(Headers) (optional) The headers to include in the API request.
body	string (optional) The body to include in the API request.
verb	string (required) The HTTP method for the API request. All HTTP methods are supported
verify_ssl	string (required) Determines whether to verify the SSL certificate of the URL. Accepted values are "true" or "false".
valid_response_codes[]	integer (optional) Status codes that the Agent should consider successful. The default value is [200]. Note: An instance of a data subject not existing should be handled the same as a successful request. If the API returns 404, for example, 404 should be added to the array.

JSON Representation
{ "url": string, "headers": object(Headers), "body": string, "verb": enum(Verb), "verify_ssl": string, "valid_response_codes": [integer] }

Headers

Field
<header_name>	string The value of the header.

Substitutions

The url, headers, and body parameters support named substitutions through string interpolation to format with identifiers coming from DataGrail, and values stored in your credentials manager. To make a substitution, enclose the variable name in curly braces in the string.

URL

"url": "https://api.acme.com/v2/data-subject-request?{email}"

Headers

"headers": {
    "Authorization": "Basic {credentials}"
}

Credentials to authenticate API requests will be stored in JSON format in your credentials manager. The key-value pairs in the secret will be dictated by substitutions you need to make in your headers. If the API uses Basic Authentication, like the above example, the credentials should be stored as the following key-value pair:

{ "credentials": "<base64 encoded username:password>" }

Body

The body of the request should be in JSON format where the value must be enclosed in double curly braces so that they are escaped.

"body": "{{\"email\": \"{email}\"}}"

System Requirements

The API Proxy connector requires the following criteria of the target system to be met.

Authentication

The API must support static token-based authentication and the token must be stored in your credentials manager.

Synchronous Flow

The API must support a synchronous response flow whereby all operations occur over a single open connection in under 5 minutes.

Each request must be stateless and perform deletion, identifier, or data retrieval without any context of previously executed queries.

Request Timeout

All API requests for a given query type must be completed in under 5 minutes. For example, if three endpoints are configured for access requests, then all three must be completed in under 5 minutes (cumulative).

Response Body Format

Access Requests

The response body of an access request must be an array of objects. The value of each key can be of any datatype, including arrays and nested objects. Each object in the array will be converted into a separate file for your data subject.

The environment variable LOGLEVEL can be set to DEBUG to get more detailed feedback if responses are malformed. Be aware that this level of logging has the potential to expose sensitive data.

[
  {
    "first_name": "Peter",
    "last_name": "Gibbons",
    "company": "Initech",
    "friends": ["Samir", "Michael"]
  },
  {
    "address_type": "Home",
    "address": {
      "street": "191 N. Lamar",
      "city": "Flander",
      "state": "Illinois",
      "zip_code": "77070"
    }
  }
]

Deletion Requests

The status code is the only signal used to determine if a deletion request was successful. A body can be included for logging purposes but will not propagate to the DataGrail platform.

Test Requests

The status code is the only signal used to determine if the API is available and healthy. A body can be included for logging purposes but will not propagate to the DataGrail platform.

Identifier Requests

The response body of an identifier retrieval request must be an array of objects with the below key-value pair.

[
  {
    "user_id": "3ef6159b-a523-4ae4-a2b8-6b3ddedf1ab4"
  }
]

Identifier Name

The name of the field in the configuration is the name of the identifier in DataGrail in "snake_case", e.g. User ID in DataGrail would be user_id in the configuration. Learn more about identifier setup.