Skip to main content

GCP BigQuery

Server Source code Package

The GCP destination package provides server-side integration for streaming events from walkerOS to Google BigQuery for data warehousing, analytics, and machine learning workloads.

Where this fits

GCP BigQuery is a server destination in the walkerOS flow:

Source
Collector
Destination
BigQuery

Streams events to Google BigQuery for data warehousing, analytics dashboards, and machine learning workloads.

Installation​

Loading...
Loading...

Configuration​

This destination uses the standard destination config wrapper (consent, data, env, id, ...). For the shared fields see destination configuration. Package-specific fields live under config.settings and are listed below.

Settings​

PropertyTypeDescriptionMore
clientanyGoogle Cloud BigQuery client instance
projectId*stringGoogle Cloud Project ID
datasetId*stringBigQuery dataset ID where events will be stored
tableId*stringBigQuery table ID for event storage
location*stringGeographic location for the BigQuery dataset
bigqueryanyAdditional BigQuery client configuration options
* Required fields

Mapping​

Per-event rules under config.mapping. For the standard rule fields (consent, condition, data, batch, name, policy) see mapping.

PropertyTypeDescriptionMore

Examples

page view

Event
Out

purchase

Event
Out

Prerequisites​

GCP setup​

Enable BigQuery API​

Loading...

Create BigQuery dataset​

Loading...
Cost optimization

Physical storage billing charges based on compressed size. This is set at dataset creation and applies to all tables.

Create service account​

Loading...

Authentication​

For environments where you need explicit credentials (Docker containers, external platforms):

Loading...

Set the environment variable to use the key:

Loading...
caution

Keep key files secure. Never commit them to version control or include in public Docker images.

Environment variables​

VariableDescriptionDefault
GCP_PROJECT_IDYour GCP project IDRequired
BQ_DATASETBigQuery dataset namewalkerOS
BQ_TABLEBigQuery table nameevents
BQ_LOCATIONBigQuery dataset locationEU
GOOGLE_APPLICATION_CREDENTIALSPath to service account keyRequired (unless using Workload Identity)

Default table schema​

By default, the destination sends the full walkerOS event to BigQuery. All object and array fields are JSON stringified before insertion.

ColumnTypeDescription
nameSTRINGFull event name ("entity action")
idSTRINGUnique event ID
entitySTRINGEntity name
actionSTRINGAction name
dataSTRINGJSON stringified data object
contextSTRINGJSON stringified context
globalsSTRINGJSON stringified globals
customSTRINGJSON stringified custom data
userSTRINGJSON stringified user object
nestedSTRINGJSON stringified nested entities
consentSTRINGJSON stringified consent
triggerSTRINGEvent trigger
timestampTIMESTAMPEvent timestamp
timingFLOAT64Event timing
groupSTRINGEvent group
countINT64Event count
versionSTRINGJSON stringified version
sourceSTRINGJSON stringified source
createdAtTIMESTAMPRow insertion time (BigQuery metadata)

Create table query​

Use this SQL query to create the default table schema in BigQuery:

Loading...
Query optimization

Partitioning by day reduces query costs since BigQuery only scans relevant partitions. Always include a timestamp filter in your queries.

Custom schema mapping​

You can send a custom schema by using the data configuration to map specific fields. This is useful when you only need a subset of the event data.

Example: simple schema​

This example sends only name, id, data, and timestamp:

Loading...

With the corresponding simpler table:

Loading...

Cleanup​

To remove BigQuery resources:

  • Delete the BigQuery dataset
  • Remove service account IAM bindings from the dataset
  • Delete the service account
  • Remove any downloaded key files
💡 Need implementation support?
elbwalker offers hands-on support: setup review, measurement planning, destination mapping, and live troubleshooting. Book a 2-hour session (€399)