Environment Variables

Environment variables defined within the container enable dynamic configuration of all Request Manager Agent properties and operations.

Request Manager Agent Configuration

The Request Manager Agent's runtime behavior is controlled by the DATAGRAIL_AGENT_CONFIG environment variable. This configuration defines the target systems for integration, the locations of associated credentials, and metadata used to parameterize privacy request execution logic. A properly formatted configuration variable is required for successful Agent startup.

Name	Value
DATAGRAIL_AGENT_CONFIG	object(DataGrailAgentConfig) JSON object that defines connectors, credential locations, cloud storage, etc.

DataGrailAgentConfig Schema

Fields
connections[]	object(Connection) (required) Defines the target systems, their capabilities and privacy request logic.
customer_domain	string (required) Your DataGrail domain.
datagrail_agent_credentials_location	string (required) Location of the OAuth Client Credentials.
datagrail_credentials_location	string (required) Location of the Callback Token.
platform	object(Platform) (required) Credentials and cloud storage providers to use.
redis_url	string (optional) Used for multi-node deployments that need to share an external Redis instance.
job_timeout_seconds	integer (optional) Duration in seconds before a job times out. The default is 180 (3 minutes).

JSON Representation

{
    "connections": [
        object(Connection)
    ],
    "customer_domain": string,
    "datagrail_agent_credentials_location": string,
    "datagrail_credentials_location": string,
    "platform": object(Platform),
    "redis_url": string,
    "job_timeout_seconds": integer
}

---

Connection Schema

The Connection object defines an integration with an internal system, including all metadata regarding the system type, capabilities, and data privacy request logic for different request types. Many Connection objects can be defined in the connections array to support connecting to multiple systems of varying types. While all Connection configurations adhere to the same schema and required parameters, the syntax for parameter binding in queries, and the schema of credentials for each connection will vary. For detailed information, consult the specific documentation for each connection type.

Fields
name	string (required) Name of the target system used for the integration in the platform. Characters should be ASCII-only.
uuid	string (required) An arbitrary and unique version 4 UUID created by you to identify the connection.
capabilities[]	enum(Capability) (required) The capabilities of the connection.
mode	enum(Mode) (required) The status of the connection.
connector_type	enum(ConnectorType) (required) The supported connection to use.
queries	object(Queries) (required) The associated queries of each capability.
credentials_location	string (required) The location of the connection credentials (e.g. Amazon ARN).

JSON Representation

{
    "name": string,
    "uuid": string,
    "capabilities": [
        enum(Capability)
    ],
    "mode": enum(Mode),
    "connector_type": enum(ConnectionType),
    "queries": object(Queries),
    "credentials_location": string
}

Capability

The supported privacy capabilities for a given Connection.

Enums
privacy/access	string Supports data subject access requests.
privacy/delete	string Supports data subject deletion requests.
privacy/optout	string Supports opt-out requests.
privacy/identifiers	string Supports identifier retrieval requests.

Mode

Defines whether the Connection should be included in data subject requests.

Enums
live	string The connection should be included in data subject requests.
test	string The connection should not be included in data subject requests.

ConnectorType

The type of system to connect to.

Enums
APIProxy	string Call a REST API.
BigQuery	string Connect to a BigQuery data warehouse.
DynamoDB	string Connect to a DynamoDB database.
MySQL	string Connect to a MySQL database.
Oracle	string Connect to an Oracle database.
Postgres	string Connect to a Postgres database.
Redshift	string Connect to a Redshift data warehouse.
Snowflake	string Connect to a Snowflake data warehouse.
SQLServer	string Connect to a SQLServer database.
SSH	string Create an SSH tunnel and execute a command line argument.

Queries

The business logic to be executed for the different request types.

Fields
identifiers	object(Identifier) (optional) Queries to retrieve additional data subject identifiers.
access[]	string or object(APIProxyQuery) (optional) Queries to execute an access request.
delete[]	string or object(APIProxyQuery) (optional) Queries to execute a deletion request.
optout[]	string or object(APIProxyQuery) (optional) Queries to execute an opt-out request.
test[]	object(APIProxyQuery) (optional) Required health check query for the APIProxy connector.

JSON Representation

{
    "identifiers": {
        object(Identifier)
    },
    "access": [
        string
    ],
    "delete": [
        string
    ],
    "optout": [
        string
    ]
}

Identifier

The logic to retrieve a configured identifier.

Fields
<identifier_api_name>[]	string Queries to retrieve additional data subject identifiers.

JSON Representation

{
    "<identifier_api_name>": [string]
}

Identifier Name

The API name of the identifier is the name of the identifier in DataGrail in "snake_case", e.g. User ID in DataGrail would be user_id in the configuration. Learn more about identifier setup.

---

Platform Schema

The Platform object defines metadata about the credentials manager and cloud storage bucket used by the Agent. A credentials manager is used to store client credentials, connection strings, PEM keys, API tokens, etc. that authenticate the Agent's various requests, and the cloud storage bucket is used to persist the results of access and identifier retrieval requests so that Personally Identifiable Information (PII) never leaves your private network.

Fields
credentials_manager	object(CredentialsManager) (required) Credentials manager settings.
storage_manager	object(StorageManager) (required) Cloud storage settings.

JSON Representation

{
    "credentials_manager": object(CredentialsManager),
    "storage_manager": object(StorageManager)
}

CredentialsManager

Fields
provider	enum(CredentialsManagerProvider) (required) The credentials manager provider.
options	object(CredentialsManagerOptions) (optional) Metadata about the credentials manager.

JSON Representation

{
    "provider": enum(CredentialsManagerProvider),
    "options": object(CredentialsManagerOptions)
}

CredentialsManagerProvider

Defines the credentials manager provider used by the Agent.

Enums
AWSSSMParameterStore	Amazon Web Services Parameter Store.
AWSSecretsManager	Amazon Web Services Secrets Manager.
AzureKeyVault	Microsoft Azure Key Vault.
GCP	Google Secret Manager.
JSONFile	JSON file for secrets injection.

CredentialsManagerOptions

Defines the options for the CredentialsManagerProvider. The required options vary based on the provider used.

Fields
secret_vault	string (optional) Specifies the Azure Key Vault name. Required when using AzureKeyVault.
project_id	string (optional) Specifies the Google Cloud project ID. Required when using GCP.
filename	string (optional) Specifies the credentials file location/name. Required when using JSONFile.

JSON Representation

{
    "secret_vault": string,
    "project_id": string,
    "filename": string
}

StorageManager

Fields
provider	enum(StorageManagerProvider) (required) The cloud storage manager provider.
options	object(StorageManagerOptions) (optional) Metadata about the storage manager. Required for some providers.

JSON Representation

{
    "provider": enum(StorageManagerProvider),
    "options": object(StoragesManagerOptions)
}

StorageManagerProvider

Defines the cloud storage provider used by the Agent.

Enums
AWSS3	string Amazon Web Services S3
AzureBlob	string Microsoft Azure Blob
BackblazeB2	string Backblaze B2
GCPCloudStore	string Google Cloud Storage

StorageManagerOptions

Defines the options for the StorageManagerProvider. The required options vary based on the provider used.

Fields
bucket	string (required) The name of the cloud storage bucket
project_id	string (optional) Required when using AzureBlob or GCPCloudStore to denote the project ID.
region	string (optional) Required when using BackblazeB2 to denote the region.
endpoint	string (optional) Required when using BackblazeB2 to denote the endpoint.

JSON Representation

{
    "bucket": string,
    "project_id": string,
    "region": string,
    "endpoint": string
}

---

Cloud Provider Credentials

Set long-lived cloud provider credentials when role-based access controls are unavailable, such as in a local development environment.

Best Practice

Using role-based access controls is recommended so long-lived credentials are not stored in plain text in the container's environment.

Amazon Web Services

Name	Value
AWS_ACCESS_KEY_ID	AWS access key associated with an IAM account.
AWS_SECRET_ACCESS_KEY	Secret key associated with the access key. This is essentially the "password" for the access key.
AWS_REGION	The AWS Region to send the request to.

Google Cloud Platform

Name	Value
GOOGLE_APPLICATION_CREDENTIALS_JSON	Extracted Google credentials file JSON.

Microsoft Azure

Name	Value
AZURE_TENANT_ID	The Azure Active Directory tenant (directory) ID.
AZURE_CLIENT_ID	The client (application) ID of an App Registration in the tenant.
AZURE_CLIENT_SECRET	The client secret for the App Registration.