> Part of the walkerOS documentation. Project overview and full index: # Google BigQuery [Server](#)[ ](https://github.com/elbwalker/walkerOS/tree/main/packages/server/destinations/gcp) [Source code](https://github.com/elbwalker/walkerOS/tree/main/packages/server/destinations/gcp)[ ](https://www.npmjs.com/package/@walkeros/server-destination-gcp) [Package](https://www.npmjs.com/package/@walkeros/server-destination-gcp) Server-side event streaming to [Google BigQuery](https://cloud.google.com/bigquery) via the Storage Write API for low-latency analytics, machine learning workloads, and data warehousing. The `@walkeros/server-destination-gcp` package also ships [`destinationPubSub`](https://www.walkeros.io/docs/destinations/server/pubsub.md) for publishing events to Pub/Sub topics; this page covers BigQuery only. Where this fits GCP BigQuery is a **server destination** in the walkerOS flow: Streams events to Google BigQuery for data warehousing, analytics dashboards, and machine learning workloads. ## Installation[​](#installation "Direct link to Installation") ``` npm install @walkeros/server-destination-gcp ``` * Integrated * Bundled ``` import { startFlow } from '@walkeros/collector'; import { destinationBigQuery } from '@walkeros/server-destination-gcp'; await startFlow({ destinations: { bigquery: { code: destinationBigQuery, config: { settings: { projectId: 'YOUR_PROJECT_ID', datasetId: 'YOUR_DATASET_ID', tableId: 'YOUR_TABLE_ID', }, }, }, }, }); ``` Add to your `flow.json` destinations: ``` "destinations": { "bigquery": { "package": "@walkeros/server-destination-gcp", "import": "destinationBigQuery", "config": { "settings": { "projectId": "YOUR_PROJECT_ID", "datasetId": "YOUR_DATASET_ID", "tableId": "YOUR_TABLE_ID" } } } } ``` [CLI reference](https://www.walkeros.io/docs/apps/cli.md) ## Configuration[​](#configuration "Direct link to Configuration") This destination uses the standard destination config wrapper (consent, data, env, id, ...). For the shared fields see [destination configuration](https://www.walkeros.io/docs/destinations.md#configuration). Package-specific fields live under `config.settings` and are listed below. ## Settings[​](#settings "Direct link to Settings") | Property | Type | Description | More | | ------------ | -------- | ------------------------------------------------ | ---- | | `client` | `any` | Google Cloud BigQuery client instance | | | `projectId*` | `string` | Google Cloud Project ID | | | `datasetId` | `string` | BigQuery dataset ID where events will be stored | | | `tableId` | `string` | BigQuery table ID for event storage | | | `location` | `string` | Geographic location for the BigQuery dataset | | | `bigquery` | `any` | Additional BigQuery client configuration options | | \* Required fields ## Mapping[​](#mapping "Direct link to Mapping") This package does not define custom rule-level settings. For the standard rule fields (consent, condition, data, batch, name, policy) see [mapping](https://www.walkeros.io/docs/mapping.md). ## Examples ### Page view A page view is appended as one row through the BigQuery Storage Write API JSONWriter. Nested objects/arrays in data, source, etc. are JSON-stringified by eventToRow. Event ``` { "name": "page view", "data": { "title": "Documentation", "url": "https://example.com/docs" }, "context": { "dev": [ "test", 1 ] }, "globals": { "pagegroup": "docs" }, "custom": { "completely": "random" }, "user": { "id": "us3r", "device": "c00k13", "session": "s3ss10n" }, "nested": [ { "entity": "child", "data": { "is": "subordinated" } } ], "consent": { "functional": true }, "id": "15bf0bc1e3c614ac", "trigger": "load", "entity": "page", "action": "view", "timestamp": 1700001100, "timing": 3.14, "source": { "count": 1, "trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9", "type": "express", "platform": "server" } } ``` Out ``` appendRows([ { "name": "page view", "data": "{\"title\":\"Documentation\",\"url\":\"https://example.com/docs\"}", "context": "{\"dev\":[\"test\",1]}", "globals": "{\"pagegroup\":\"docs\"}", "custom": "{\"completely\":\"random\"}", "user": "{\"id\":\"us3r\",\"device\":\"c00k13\",\"session\":\"s3ss10n\"}", "nested": "[{\"entity\":\"child\",\"data\":{\"is\":\"subordinated\"}}]", "consent": "{\"functional\":true}", "id": "15bf0bc1e3c614ac", "trigger": "load", "entity": "page", "action": "view", "timestamp": 1700001100000, "timing": 3.14, "source": "{\"count\":1,\"trace\":\"0a1b2c3d4e5f60718293a4b5c6d7e8f9\",\"type\":\"express\",\"platform\":\"server\"}" } ]) ``` ### Purchase An order event is appended as a single row through JSONWriter.appendRows. The entire nested data object (including arrays like items) is JSON-stringified into the data column via eventToRow(). Event ``` { "name": "order complete", "data": { "id": "ORD-500", "total": 199.99, "items": [ { "sku": "SKU-1", "qty": 2 } ] }, "context": { "shopping": [ "complete", 0 ] }, "globals": { "pagegroup": "shop" }, "custom": { "completely": "random" }, "user": { "id": "us3r", "device": "c00k13", "session": "s3ss10n" }, "nested": [ { "entity": "product", "data": { "id": "ers", "name": "Everyday Ruck Snack", "color": "black", "size": "l", "price": 420 }, "context": { "shopping": [ "complete", 0 ] }, "nested": [] }, { "entity": "product", "data": { "id": "cc", "name": "Cool Cap", "size": "one size", "price": 42 }, "context": { "shopping": [ "complete", 0 ] }, "nested": [] }, { "entity": "gift", "data": { "name": "Surprise" }, "context": { "shopping": [ "complete", 0 ] }, "nested": [] } ], "consent": { "functional": true }, "id": "974b385b0493bdad", "trigger": "load", "entity": "order", "action": "complete", "timestamp": 1700001101, "timing": 3.14, "source": { "count": 1, "trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9", "type": "express", "platform": "server" } } ``` Out ``` appendRows([ { "name": "order complete", "data": "{\"id\":\"ORD-500\",\"total\":199.99,\"items\":[{\"sku\":\"SKU-1\",\"qty\":2}]}", "context": "{\"shopping\":[\"complete\",0]}", "globals": "{\"pagegroup\":\"shop\"}", "custom": "{\"completely\":\"random\"}", "user": "{\"id\":\"us3r\",\"device\":\"c00k13\",\"session\":\"s3ss10n\"}", "nested": "[{\"entity\":\"product\",\"data\":{\"id\":\"ers\",\"name\":\"Everyday Ruck Snack\",\"color\":\"black\",\"size\":\"l\",\"price\":420},\"context\":{\"shopping\":[\"complete\",0]},\"nested\":[]},{\"entity\":\"product\",\"data\":{\"id\":\"cc\",\"name\":\"Cool Cap\",\"size\":\"one size\",\"price\":42},\"context\":{\"shopping\":[\"complete\",0]},\"nested\":[]},{\"entity\":\"gift\",\"data\":{\"name\":\"Surprise\"},\"context\":{\"shopping\":[\"complete\",0]},\"nested\":[]}]", "consent": "{\"functional\":true}", "id": "974b385b0493bdad", "trigger": "load", "entity": "order", "action": "complete", "timestamp": 1700001101000, "timing": 3.14, "source": "{\"count\":1,\"trace\":\"0a1b2c3d4e5f60718293a4b5c6d7e8f9\",\"type\":\"express\",\"platform\":\"server\"}" } ]) ``` ## Prerequisites[​](#prerequisites "Direct link to Prerequisites") * [Google Cloud account](https://cloud.google.com/) with billing enabled * [gcloud CLI](https://cloud.google.com/sdk/docs/install) installed and authenticated (includes `bq` command) ## Setup lifecycle[​](#setup-lifecycle "Direct link to Setup lifecycle") Provision the dataset and table once per environment with the CLI: ``` walkeros setup destination.bigquery ``` Output: a narrated `setup: ok destination.bigquery` line. Add `--json` to also emit a structured envelope reporting `{ datasetCreated, tableCreated }` for `jq` piping. The command is idempotent, safe to re-run. `config.setup` controls provisioning: * omitted or `false`: narrated skip, no provisioning. Operator runs setup explicitly to provision. * `true`: provision with the defaults below. * object matching the `Setup` interface: provision with the declared overrides. See the `Setup` interface in the package for the full shape. ### Defaults[​](#defaults "Direct link to Defaults") | Field | Value | | --------------------- | ------------------------------------------ | | `datasetId` | `walkerOS` (note capital O, S) | | `tableId` | `events` | | `location` | `EU` | | `storageBillingModel` | `PHYSICAL` (cheaper for compressible JSON) | | Partitioning | Day partitioning on `timestamp` | | Clustering | `(name, entity, action)` | Cost optimization Physical storage billing charges based on compressed size. Day partitioning and the `(name, entity, action)` clustering reduce scan costs for typical analytics queries. Always include a `timestamp` filter. ### Drift handling[​](#drift-handling "Direct link to Drift handling") If the existing table's partitioning, clustering, or schema differs from the declared configuration, setup logs `WARN setup.drift {...}` and continues. There is no auto-mutation. Migrations are an operator decision. ## GCP setup[​](#gcp-setup "Direct link to GCP setup") ### Enable BigQuery API[​](#enable-bigquery-api "Direct link to Enable BigQuery API") ``` gcloud services enable bigquery.googleapis.com ``` ### Create service accounts[​](#create-service-accounts "Direct link to Create service accounts") The provisioning step (setup) and the runtime push path need different permissions. We recommend separating them. **Operator (setup) service account**, used by `walkeros setup`: * `bigquery.datasets.create` * `bigquery.tables.create` * `bigquery.datasets.get` (for drift detection) * `bigquery.tables.get` **Runtime service account**, used by the running flow: * `bigquery.tables.updateData` (Storage Write API append) ``` # Runtime service account gcloud iam service-accounts create walkeros-flow \ --display-name="walkerOS Flow Runtime" # Grant Storage Write API append access gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \ --member="serviceAccount:walkeros-flow@YOUR_PROJECT_ID.iam.gserviceaccount.com" \ --role="roles/bigquery.dataEditor" ``` ### Authentication[​](#authentication "Direct link to Authentication") * Service Account Key * Workload Identity For environments where you need explicit credentials (Docker containers, external platforms): ``` gcloud iam service-accounts keys create ./sa-bigquery.json \ --iam-account=walkeros-flow@YOUR_PROJECT_ID.iam.gserviceaccount.com ``` Set the environment variable to use the key: ``` export GOOGLE_APPLICATION_CREDENTIALS=/path/to/sa-bigquery.json ``` caution Keep key files secure. Never commit them to version control or include in public Docker images. For GCP-native platforms (Cloud Run, GKE, Compute Engine), attach the service account directly to your workload. No key file needed. **Cloud Run example:** ``` gcloud run deploy walkeros-flow \ --service-account=walkeros-flow@YOUR_PROJECT_ID.iam.gserviceaccount.com ``` See [Workload Identity documentation](https://cloud.google.com/iam/docs/workload-identity-federation) for other platforms. Inline credentials Instead of relying on `GOOGLE_APPLICATION_CREDENTIALS`, set service account credentials on `config.credentials` (a JSON string or parsed object, `$env` -resolvable). These merge into the BigQuery client and apply to both the control plane (setup, metadata) and the data plane (Storage Write API ingestion). For lower-level control you can still pass auth options through the raw `settings.bigquery` passthrough (for example `keyFilename` or `credentials`). A pre-built `settings.client` authenticates the control plane only; supply `config.credentials` or `settings.bigquery` for the data plane to use non-ADC credentials. ## Environment variables[​](#environment-variables "Direct link to Environment variables") | Variable | Description | Default | | -------------------------------- | --------------------------- | ----------------------------------------- | | `GCP_PROJECT_ID` | Your GCP project ID | Required | | `BQ_DATASET` | BigQuery dataset name | `walkerOS` | | `BQ_TABLE` | BigQuery table name | `events` | | `BQ_LOCATION` | BigQuery dataset location | `EU` | | `GOOGLE_APPLICATION_CREDENTIALS` | Path to service account key | Required (unless using Workload Identity) | ## Storage Write API (data plane)[​](#storage-write-api-data-plane "Direct link to Storage Write API (data plane)") The destination uses BigQuery's [Storage Write API](https://cloud.google.com/bigquery/docs/write-api) for data ingestion. This replaces the legacy `tabledata.insertAll` path. * **Cost**: $25/TB after the 2 TiB/month free tier (vs \~$50/TB for the legacy path). Most low-volume deployments fit entirely in the free tier. * **Batching**: `pushBatch` is implemented. Set `config.batch` on the destination (a bare number is the debounce `wait` in ms) to flush all events in a window as a single `appendRows` call. No `'* *'` wildcard mapping rule is needed. The batch path awaits the write and surfaces row errors: a failed append fails the batch, which routes the events to the dead-letter buffer and the `failed` count rather than being logged and ignored. EXPERIMENTAL SDK The upstream `@google-cloud/bigquery-storage` package self-marks as `EXPERIMENTAL` (subject to change). Pinned at `^5.1.0`. ## Default table schema[​](#default-table-schema "Direct link to Default table schema") The default 15-column schema follows the [walkerOS Event v4](https://www.walkeros.io/docs/getting-started/event-model.md) canonical order. Object fields use the native `JSON` BigQuery type. Only `name` is `REQUIRED`; all other columns are `NULLABLE` for resilience against partial events. | Column | Type | Mode | | ----------- | --------- | -------- | | `name` | STRING | REQUIRED | | `data` | JSON | NULLABLE | | `context` | JSON | NULLABLE | | `globals` | JSON | NULLABLE | | `custom` | JSON | NULLABLE | | `user` | JSON | NULLABLE | | `nested` | JSON | NULLABLE | | `consent` | JSON | NULLABLE | | `id` | STRING | NULLABLE | | `trigger` | STRING | NULLABLE | | `entity` | STRING | NULLABLE | | `action` | STRING | NULLABLE | | `timestamp` | TIMESTAMP | NULLABLE | | `timing` | FLOAT64 | NULLABLE | | `source` | JSON | NULLABLE | There is no `createdAt` column. Use `timestamp` (event time) for partition filters. Query optimization Partitioning by day on `timestamp` and clustering on `(name, entity, action)` reduces scan costs for typical analytics queries. Always include a `timestamp` filter. ## Custom schema mapping[​](#custom-schema-mapping "Direct link to Custom schema mapping") You can send a custom schema by using the `data` configuration to map specific fields. This is useful when you only need a subset of the event data. ### Example: simple schema[​](#example-simple-schema "Direct link to Example: simple schema") This example sends only `name`, `id`, `data`, and `timestamp`: * Integrated * Bundled ``` import { startFlow } from '@walkeros/collector'; import { destinationBigQuery } from '@walkeros/server-destination-gcp'; await startFlow({ destinations: { bigquery: { code: destinationBigQuery, config: { settings: { projectId: 'YOUR_PROJECT_ID', datasetId: 'YOUR_DATASET_ID', tableId: 'events_simple', }, data: { map: { name: 'name', id: 'id', data: 'data', timestamp: 'timestamp', }, }, }, }, }, }); ``` ``` { "destinations": { "bigquery": { "package": "@walkeros/server-destination-gcp", "import": "destinationBigQuery", "config": { "settings": { "projectId": "YOUR_PROJECT_ID", "datasetId": "YOUR_DATASET_ID", "tableId": "events_simple" }, "data": { "map": { "name": "name", "id": "id", "data": "data", "timestamp": "timestamp" } } } } } } ``` With the corresponding simpler table: ``` CREATE TABLE IF NOT EXISTS `YOUR_PROJECT.walkeros.events_simple` ( name STRING, id STRING, data STRING, timestamp INT64 ); ``` ## Cleanup[​](#cleanup "Direct link to Cleanup") To remove BigQuery resources: * Delete the BigQuery dataset * Remove service account IAM bindings from the dataset * Delete the service account * Remove any downloaded key files ## Pub/Sub[​](#pubsub "Direct link to Pub/Sub") The same `@walkeros/server-destination-gcp` package also exports `destinationPubSub` for publishing events to a Pub/Sub topic. See the [Pub/Sub destination page](https://www.walkeros.io/docs/destinations/server/pubsub.md) for full settings, mapping, ordering, attributes, setup, and authentication reference.