GCP BigQuery

Server Source code Package

The GCP destination package provides server-side integration for streaming events from walkerOS to Google BigQuery for data warehousing, analytics, and machine learning workloads.

Where This Fits

GCP BigQuery is a server destination in the walkerOS flow:

Streams events to Google BigQuery for data warehousing, analytics dashboards, and machine learning workloads.

Installation

Prerequisites

Google Cloud account with billing enabled
gcloud CLI installed and authenticated (includes bq command)

GCP Setup

Enable BigQuery API

Create BigQuery Dataset

Cost optimization

Physical storage billing charges based on compressed size. This is set at dataset creation and applies to all tables.

Create Service Account

Authentication

Service Account Key
Workload Identity

For environments where you need explicit credentials (Docker containers, external platforms):

Set the environment variable to use the key:

caution

Keep key files secure. Never commit them to version control or include in public Docker images.

Environment Variables

Variable	Description	Default
`GCP_PROJECT_ID`	Your GCP project ID	Required
`BQ_DATASET`	BigQuery dataset name	`walkerOS`
`BQ_TABLE`	BigQuery table name	`events`
`BQ_LOCATION`	BigQuery dataset location	`EU`
`GOOGLE_APPLICATION_CREDENTIALS`	Path to service account key	Required (unless using Workload Identity)

Setup

Integrated
Bundled

Add to your flow.json destinations:

CLI reference →

Configuration reference

Property	Type	Description
`client`	`any`	Google Cloud BigQuery client instance
`projectId*`	`string`	Google Cloud Project ID
`datasetId*`	`string`	BigQuery dataset ID where events will be stored
`tableId*`	`string`	BigQuery table ID for event storage
`location*`	`string`	Geographic location for the BigQuery dataset
`bigquery`	`any`	Additional BigQuery client configuration options

* Required fields

Default table schema

By default, the destination sends the full walkerOS event to BigQuery. All object and array fields are JSON stringified before insertion.

Column	Type	Description
`name`	STRING	Full event name ("entity action")
`id`	STRING	Unique event ID
`entity`	STRING	Entity name
`action`	STRING	Action name
`data`	STRING	JSON stringified data object
`context`	STRING	JSON stringified context
`globals`	STRING	JSON stringified globals
`custom`	STRING	JSON stringified custom data
`user`	STRING	JSON stringified user object
`nested`	STRING	JSON stringified nested entities
`consent`	STRING	JSON stringified consent
`trigger`	STRING	Event trigger
`timestamp`	TIMESTAMP	Event timestamp
`timing`	FLOAT64	Event timing
`group`	STRING	Event group
`count`	INT64	Event count
`version`	STRING	JSON stringified version
`source`	STRING	JSON stringified source
`createdAt`	TIMESTAMP	Row insertion time (BigQuery metadata)

Create table query

Use this SQL query to create the default table schema in BigQuery:

Query optimization

Partitioning by day reduces query costs since BigQuery only scans relevant partitions. Always include a timestamp filter in your queries.

Custom schema mapping

You can send a custom schema by using the data configuration to map specific fields. This is useful when you only need a subset of the event data.

Example: Simple Schema

This example sends only name, id, data, and timestamp:

Integrated
Bundled

With the corresponding simpler table:

Cleanup

To remove BigQuery resources:

Delete the BigQuery dataset
Remove service account IAM bindings from the dataset
Delete the service account
Remove any downloaded key files

Docker Deployment

For Docker deployments outside of GCP infrastructure, use mounted credentials.

Step 1: Create Flow Configuration

Use the ${VAR} syntax for environment variables:

Step 2: Bundle the Flow

Step 3: Run with Docker

Docker Compose Example

Security

Never pass GCP private keys as environment variables. Always mount the credential file at runtime and never bake credentials into Docker images.

💡 Need Professional Support?

Need professional support with your walkerOS implementation? Check out our services.

Installation​

Prerequisites​

GCP Setup​

Enable BigQuery API​

Create BigQuery Dataset​

Create Service Account​

Authentication​

Environment Variables​

Setup​

Configuration reference​

Default table schema​

Create table query​

Custom schema mapping​

Example: Simple Schema​

Cleanup​

Docker Deployment​

Step 1: Create Flow Configuration​

Step 2: Bundle the Flow​

Step 3: Run with Docker​

Docker Compose Example​