Integrating BigQuery

Capabilities

DataGrail's BigQuery integration provides the following capabilities:

Product	Capability
Request Manager	Request Types Access, Deletion, Do Not Sell/Share, Identifiers Identifier Categories Any
Live Data Map	Data Discovery

Before You Start

To successfully configure this integration, please ensure you have sufficient privileges:

• DataGrail User Role: Super Admin, Connections Manager
• BigQuery User Role: Admin
• Secrets Manager: Write Access
• IAM Roles: Write Access

Connecting with RM Agent

The Request Manager Agent allows you to automate Data Subject Requests by connecting to internal systems within your network, and without requiring ingress from the public network.

The Agent connects to your BigQuery using least privileged IAM roles that you will configure. Additional connection details will be stored securely in a secrets manager. When configuring the BigQuery integration in DataGrail, only the location of that secrets manager entry will be referenced (e.g., Google Cloud Secret Manager resource name), which ensures that no secrets are shared directly with DataGrail.

Prerequisites

Before you can connect to BigQuery, ensure the following:

• RM Agent is deployed and connected in DataGrail.
• Network is configured to allow the Agent to connect with the BigQuery instance.

Store Connection Details

1. To specify the target project, configure the following JSON key-value pairs:

   {
     "project_id": "<project ID>"
   }

2. Store the JSON value in your secrets manager with a name like datagrail.rm-agent.bigquery.

3. Ensure that the Agent has the necessary permissions to access this vault entry.

Add the Agent Integration

1. In DataGrail, navigate to Agents and select your Agent.
2. In the top right, select Add New Integration and search for BigQuery.
3. Under Enabled Capabilities and Enabled Identifiers, select only those that will be used for this integration.
4. Enter the Connection Details Location (e.g. Google Cloud Secret Manager resource name).
5. Select the Data Retrieval behavior for deletion requests.
   warning

   When using Retrieve Data, the data reviewed may not be exactly what is deleted due to the access and deletion logic executing separately!

6. Under Agent Query Configuration, add request logic to be executed within BigQuery for all enabled request types.
   Query Parameter Format

   Use scalar paramstyle when formatting query parameters with identifiers (e.g., email, user_id).

   Example:

   SELECT * FROM users WHERE email = @email

7. Finally, select Configure Integration. Wait a few moments to ensure that the connection is successful. For failed connections, review the Agent container logs for additional details.

---

Connecting with RDD Agent

The Responsible Data Discovery Agent allows you to securely perform data classification by connecting to internal systems within your network, and without requiring ingress from the public network.

For the Agent to scan BigQuery, IAM roles are used to allow the Agent to connect and perform the necessary operations. Additional connection details will be stored securely in a vault on your network. When configuring the BigQuery integration in DataGrail, only the location of that vault entry will be referenced (e.g., Google Cloud Secret Manager resource name), which ensures that no secrets are shared directly with DataGrail.

About BigQuery Block Sampling

The Agent uses BigQuery TABLESAMPLE to randomly sample blocks that are each roughly 1GB in size. In an effort to manage costs, the Agent by default only uses up to 5 blocks and then randomly samples within those blocks.

Given BigQuery pricing, scans will cost roughly up to $100 per about 2,000 tables (assuming all tables are at least 5GB in size). To further manage the block sampling, set the optional environment variable MAX_BIGQUERY_BLOCKS. We recommend no more than 10.

Before Connecting

In order to start scanning BigQuery, ensure the following:

• RDD Agent is deployed and connected in DataGrail.
• Network is configured to allow the Agent to connect with the BigQuery instance.

Store Connection Details in Vault

1. To specify the target project, configure the following JSON key-value pairs:

   {
     "project_id": "<project ID>"
   }

2. Store the JSON value in your vault with a name like datagrail-rdd-bigquery.

3. Ensure that the Agent has the necessary permissions to access this vault entry.

Add Service Account Roles

To grant the Agent access to BigQuery, add the following roles to the Agent service account:

• BigQuery Data Viewer
• BigQuery Job User

Add the Agent Integration

1. In DataGrail, navigate to Agents under Integration network.
2. Select your Agent.
3. In the top right, select Add New Integration.
4. Search for BigQuery, then select Configure.
5. Enter an Integration Name, and only enable the Data Discovery capability.
6. Enter the Connection Details Location (e.g. Google Cloud Secret Manager resource name).
7. (optional) Choose the Business Processes, Region, and System Location.
8. Finally, select Configure Integration. Wait a few moments to ensure that the connection is successful. For failed connections, review the Agent container logs for additional details.

Troubleshooting

If you are unable to successfully connect the integration, review these common troubleshooting steps:

Agent Unable to Connect to BigQuery

1. Verify that the network is configured to allow the Agent to connect with the BigQuery instance.
2. Verify the Agent has permissions to access the BigQuery credentials stored in your vault.

Agent is Not Connected in DataGrail

1. Confirm that the Agent is running, and logs do not indicate any errors.
2. The DataGrail API Key used by the Agent is valid and not expired.
3. The Agent has permissions to access the DataGrail API Key stored in your vault.
4. Network egress is permitted from the Agent to your DataGrail domain.

Technical Details

Access Type	Synchronous
Deletion Type	Synchronous (Whole Record)
Opt Out Type	Synchronous