Integrating direct Google BigQuery usage with Dagster Cloud Insights (Experimental)#

External metrics, such as Google BigQuery usage, can be integrated into the Dagster Insights UI. The dagster-cloud package contains utilities for capturing and submitting external metrics about data operations to Dagster Cloud via an API.

If you use the BigQuery resource, use this guide to integrate BigQuery metrics into the Insights UI. For instructions on integrating usage of dbt models that run in BigQuery, see Integrating BigQuery + dbt with Dagster Cloud Insights.

Prerequisites#

To complete the steps in this guide, you'll need:

• A Dagster Cloud account on the Pro plan

• Access to the Dagster Cloud Insights feature

• BigQuery credentials that have access to the INFORMATION_SCHEMA.JOBS table. For more information on granting access to this table, see the BigQuery documentation.

• To install the following libraries:

  pip install dagster dagster-cloud dagster-gcp
  

  Note: If you already have dagster-cloud installed, make sure you're using version 1.7.0 or newer.

Step 1: Replace your BigQuery resources#

The first step is to replace any existing BigQuery resources with InsightsBigQueryResource. This resource is a drop-in replacement for the BigQueryResource resource, but it also emits BigQuery usage metrics to the Dagster Cloud Insights API.

from dagster_cloud.dagster_insights import InsightsBigQueryResource

defs = Definitions(
  resources={
    "bigquery": InsightsBigQueryResource(project="my-project")
  }
)

Step 2: View BigQuery usage in the Dagster UI#

Once the pipeline runs, BigQuery usage will be visible in the Insights tab in the Dagster UI:

Related#