Skip to main content
Ask your AI

GCS

Server Source code Package

Google Cloud Storage store with zero runtime dependencies. It uses raw fetch against the GCS JSON API with built-in auth: Application Default Credentials (ADC) on Cloud Run / GKE, or explicit service account JWT for non-GCP environments.

Installation​

npm install @walkeros/server-store-gcs
import { startFlow } from '@walkeros/collector';
import { storeGcsInit } from '@walkeros/server-store-gcs';

await startFlow({
  stores: {
    assets: {
      code: storeGcsInit,
      config: {
        // Omit config.credentials for ADC on Cloud Run/GKE
        settings: {
          bucket: 'my-assets',
          prefix: 'public',
        },
      },
    },
  },
});

Configuration​

This store uses the standard store config wrapper (consent, data, env, id, ...). For the shared fields see store configuration. Package-specific fields live under config.settings and are listed below.

Settings​

PropertyTypeDescriptionMore
bucket*stringGCS bucket name
prefixstringKey prefix prepended to all store keys for scoping
credentialsstring | objectService account JSON (string or object). Omit for ADC on Cloud Run/GKE (deprecated: use config.credentials)
* Required fields

Mapping​

This package does not define custom rule-level settings. For the standard rule fields (consent, condition, data, batch, name, policy) see mapping.

Examples

Prefix scoping

Key "walker.js" with prefix "public/" resolves to GCS path "public/walker.js"

Event
{
  "operation": "get",
  "key": "walker.js",
  "settings": {
    "bucket": "my-assets",
    "prefix": "public"
  }
}
Out
get("public/walker.js", "Bytes<...>")

Read with ADC

Read object from GCS bucket using ADC (no credentials needed), bytes byte-exact

Event
{
  "operation": "get",
  "key": "walker.js"
}
Out
get("walker.js", "Bytes<(function(){...})()>")

When it fails

  • failed read: The request fails (no connection, a refused or forbidden read): get throws a StepError instead of answering a miss, and the collector reports it. Reported as failed with code transport.

Provisioning​

The package ships an idempotent setup() lifecycle that creates the GCS bucket described in flow config. It is invoked only by the explicit operator command, on a store with config.setup set (true or an object):

walkeros setup store.<id>

It never runs automatically and never alters an existing bucket.

caution

Today the command stops before this setup runs, with Package @walkeros/server-store-gcs has no default export: walkeros setup reads setup from the package's default export, and this package's default export is its init function.

Setup options​

OptionTypeDefaultNotes
projectIdstring(resolved)GCP project that owns the bucket. Resolution order below.
locationstring'EU'Multi-region or regional location.
storageClass'STANDARD' | 'NEARLINE' | 'COLDLINE' | 'ARCHIVE''STANDARD'Default object storage class.
versioningbooleanfalseObject versioning. Off by default; opt in.
lifecycle{ rule: unknown[] }(none)Applied at create. Drift detection not included for lifecycle.
kmsKeyNamestring(none)Customer-managed encryption key (CMEK) at create time.
labelsRecord<string, string>(none)Cost-allocation labels.

bucket is taken from settings.bucket and is not duplicated under setup.

projectId resolution​

The GCS create call requires a project. Resolution order:

  1. Explicit setup.projectId.
  2. project_id field inside the config.credentials service-account JSON.
  3. process.env.GOOGLE_CLOUD_PROJECT (Cloud Run / GKE convention).
  4. Throws with an actionable error if none of the above is available.

Behavior​

  • Idempotent: HTTP 409 (bucket exists) is treated as success. The setup never patches or mutates an existing bucket.
  • Drift detection: when the bucket already exists, setup performs a GET /b/<bucket> and logs one setup.drift warning ({ field, declared, actual }) on the logger it receives for each differing field among location, storageClass, versioning, iamConfiguration (uniform bucket-level access, public access prevention) and labels. Drift is logged, never auto-fixed.
  • Defaults enforced at create: uniform bucket-level access on, public access prevention enforced. The package sets both on every bucket it creates.

Runtime hard-fail​

At runtime, the first get / set / delete call issues a single HEAD /b/<bucket> per process per bucket. On 404, it throws with an actionable message:

GCS bucket not found: <bucket> in project <projectId>. Run "walkeros setup store.<id>" to create it.

Operators see the error pointing at the exact command to fix it. Subsequent operations in the same process skip the check via an in-memory cache.

Authentication​

Cloud Run / GKE (ADC)​

When running on GCP infrastructure, omit credentials. The store fetches access tokens from the metadata server automatically, with no configuration needed.

Non-GCP (service account)​

Pass a service account JSON via config.credentials with a $secret. reference. The store signs JWTs locally using node:crypto and exchanges them for access tokens via Google's OAuth2 endpoint.

"config": {
  "credentials": "$secret.GCS_SA_KEY",
  "settings": { "bucket": "my-assets", "prefix": "public" }
}

The GCS_SA_KEY environment variable should contain the full service account JSON (with client_email and private_key fields). When it is missing, the flow stops at startup with required secret "GCS_SA_KEY" is not set. A credentials value without client_email and private_key is ignored without a log line, and the store uses ADC instead.

File serving pattern​

The primary use case is serving static files via the file transformer. The express source maps the request path into ingest.path and runs the transformer in its next chain. Set file: true so the store persists and returns bytes byte-exact:

{
  "stores": {
    "assets": {
      "package": "@walkeros/server-store-gcs",
      "config": {
        "settings": {
          "bucket": "my-assets",
          "prefix": "public"
        },
        "file": true
      }
    }
  },
  "sources": {
    "http": {
      "package": "@walkeros/server-source-express",
      "config": {
        "settings": { "paths": ["/collect", "/static/*path"] },
        "ingest": { "map": { "path": { "key": "path" } } }
      },
      "next": "file"
    }
  },
  "transformers": {
    "file": {
      "package": "@walkeros/server-transformer-file",
      "config": { "settings": { "prefix": "/static" } },
      "env": { "store": "$store.assets" }
    }
  }
}

A request to /static/walker.js looks up public/walker.js in the my-assets bucket. Omit file for a structured key-value store instead.

Security​

  • Key validation: Keys with a .. segment or a leading / or \ are rejected: get answers a miss, set and delete do nothing. Each rejection is a WARN line, which the default log level (ERROR) does not show
  • Prefix scoping: The prefix setting restricts all operations to a subdirectory
  • Credentials: ADC comes from the GCP metadata server only, so off GCP the store needs an explicit service account JSON; a GOOGLE_APPLICATION_CREDENTIALS file is not read

Structured vs file mode​

The GCS store has two modes, decided once at init by config.file:

  • Structured (default): values are structured StoreValue data, serialized by the shared core codec and stored with Content-Type: application/json. Use for session state, lookups, and cached responses.
  • File (file: true): values persist as raw bytes byte-exact, stored with the real mime derived from the key (or application/octet-stream when unknown). set() accepts a Uint8Array or string; get() hands the exact bytes back. Use for serving assets such as walker.js.

One store instance is exactly one mode.

API​

const file = await store.get('walker.js'); // StoreValue | undefined
await store.set('data.json', { ok: true }); // structured value
await store.set(
  'walker.js',
  new Uint8Array([
    /*…*/
  ]),
); // file mode: raw bytes
await store.delete('old-file.txt'); // void

In file mode, get() returns the bytes as a Uint8Array leaf and set() rejects non-Uint8Array, non-string values with a clear error.

💡 Need implementation support?
elbwalker offers hands-on support: setup review, measurement planning, destination mapping, and live troubleshooting. Book a 2-hour session (€399)