Storing Credentials

The Request Manager Agent uses a secrets manager to securely store the various credentials required for authenticating with internal systems and DataGrail. This guide provides instructions for storing credentials in the supported platforms.

Amazon Web Services

Secrets Manager

Console

1. Navigate to the AWS Secrets Manager Console.
2. Select Store a new secret.
3. Choose Other type of secrets and enter the required key-value pairs for the secret type.
4. Click Next and provide a name for the secret, e.g., datagrail_agent_credentials.
5. Optionally, add a description and tags.
6. Select Next, review the settings, and then click Store.

CLI

1. To minimize the risk of exposing credentials in the command shell history, create a temporary JSON file to store them securely following the schema for the secret type.

   Example for OAuth Client Credentials

   {
       "client_id": "datagrail-rm-agent",
       "client_secret": "Shhh!ThisIsASecret"
   }

2. Create the secret using the AWS CLI.

   Create the secret

     aws secretsmanager create-secret \
         --name "${CREDENTIAL_NAME}" \
         --description "${CREDENTIAL_DESCRIPTION}" \
         --secret-string file://"${CREDENTIAL_FILE}"

Terraform

Example for OAuth Client Credentials

resource "aws_secretsmanager_secret" "datagrail_agent_credentials" {
  name = "datagrail_agent_credentials"
  description = "Client ID and Secret for DataGrail Request Manager Agent"
}

resource "aws_secretsmanager_secret_version" "datagrail_agent_credentials" {
  secret_id     = aws_secretsmanager_secret.datagrail_agent_credentials.id
  secret_string = jsonencode({
    client_id     = var.client_id,
    client_secret = var.client_secret
  })
}

Parameter Store

Console

1. Navigate to the AWS Systems Manager Parameter Store Console.
2. Select Create parameter.
3. Enter a name for the parameter, e.g., /datagrail/agent/credentials.
4. Choose SecureString as the type.
5. Enter the value of the secret with the required key-value pairs for the secret type.
6. Optionally, add a description and tags.
7. Select Create parameter.

CLI

1. To minimize the risk of exposing credentials in the command shell history, create a temporary JSON file to store them securely following the schema for the secret type.

   Example for OAuth Client Credentials

   {
       "client_id": "datagrail-rm-agent",
       "client_secret": "Shhh!ThisIsASecret"
   }

2. Create the parameter using the AWS CLI.

   Create the parameter

   aws ssm put-parameter \
       --name "${CREDENTIAL_NAME}" \
       --description "${CREDENTIAL_DESCRIPTION}" \
       --type "SecureString" \
       --value file://"${CREDENTIAL_FILE}"

Terraform

Example for OAuth Client Credentials

resource "aws_ssm_parameter" "datagrail_agent_credentials" {
  name        = "/datagrail/agent/credentials"
  description = "Client ID and Secret for DataGrail Request Manager Agent"
  type        = "SecureString"
  value       = jsonencode({
    client_id     = var.client_id,
    client_secret = var.client_secret
  })
}

Google Cloud Platform

Secret Manager

Console

1. Navigate to the Google Cloud Secret Manager Console.
2. Select Create Secret.
3. Enter a name for the secret, e.g., datagrail_agent_credentials.
4. Enter the value of the secret with the required key-value pairs for the secret type.
5. Optionally, add a description and labels.
6. Click Create.

CLI

1. To minimize the risk of exposing credentials through the command shell history, create a temporary JSON file to store them securely following the schema for the secret type.

   Example for OAuth Client Credentials

   {
       "client_id": "datagrail-rm-agent",
       "client_secret": "Shhh!ThisIsASecret"
   }

2. Create the secret using the GCP CLI.

   Create the secret

   gcloud secrets create "${CREDENTIAL_NAME}" \
       --replication-policy="automatic" \
       --data-file="${CREDENTIAL_FILE}"

Terraform

Example for OAuth Client Credentials

resource "google_secret_manager_secret" "datagrail_agent_credentials" {
    secret_id = "datagrail_agent_credentials"
    replication {
        automatic = true
    }
}

resource "google_secret_manager_secret_version" "datagrail_agent_credentials" {
    secret = google_secret_manager_secret.datagrail_agent_credentials.id
    secret_data = jsonencode({
        client_id     = var.client_id,
        client_secret = var.client_secret
    })
}

Microsoft Azure

Key Vault

Console

1. Navigate to the Azure Key Vault Console.
2. Select your Key Vault or create a new one.
3. In the left menu, select Secrets and then click + Generate/Import.
4. Enter a name for the secret, e.g., datagrail_agent_credentials.
5. Choose Manual for the upload options and enter the value of the secret with the required key-value pairs for the secret type.
6. Optionally, add a description and tags.
7. Click Create.

CLI

1. To minimize the risk of exposing credentials through the command shell history, create a temporary JSON file to store them securely following the schema for the secret type.

   Example for OAuth Client Credentials

   {
       "client_id": "datagrail-rm-agent",
       "client_secret": "Shhh!ThisIsASecret"
   }

2. Create the secret using the Azure CLI.

   Create the secret

   az keyvault secret set \
       --vault-name "${KEY_VAULT_NAME}" \
       --name "${CREDENTIAL_NAME}" \
       --file "${CREDENTIAL_FILE}"

Terraform

Example for OAuth Client Credentials

resource "azurerm_key_vault_secret" "datagrail_agent_credentials" {
  name         = "datagrail_agent_credentials"
  value        = jsonencode({
    client_id     = var.client_id,
    client_secret = var.client_secret
  })
  key_vault_id = azurerm_key_vault.datagrail_agent.id
}

JSONFile

Credentials can be stored in a JSON file mounted in a volume. This approach is suitable when using Kubernetes Secrets or HashiCorp Vault to store credentials.

The file adheres to a schema where each top-level key represents the location of a specific credential in the file.

JSONFile Schema

{
  "<first_credential>": {
    "<key_1>": "<value_1>",
    "<key_2>": "<value_2>",
    "<key_n>": "<value_n>"
  },
  "<second_credential>": {
    "<key_1>": "<value_1>",
    "<key_2>": "<value_2>",
    "<key_n>": "<value_n>"
  },
  "<nth_credential>": {
    "<key_1>": "<value_1>",
    "<key_2>": "<value_2>",
    "<key_n>": "<value_n>"
  }
}

Once set, your JSON file might look something like this:

JSONFile Example

{
  "datagrail_agent_credentials": {
    "client_id": "datagrail",
    "client_secret": "my-super-secret-password"
  },
  "datagrail_credentials": {
    "token": "dg_ap_api_key.2weATHF4h2KTWERkJrGdsY2PRscfwTbt"
  },
  "postgres_credentials": {
    "user": "datagrail-rm-agent",
    "password": "xindbDX8NUF8oEkYCEKiYFvUKthP4Vmt",
    "host": "staging.4Pe4q8dMtLVB.us-west-2.rds.amazonaws.com",
    "port": 5432,
    "dbname": "Customers"
  }
}