GCS
Google Cloud Storage store with zero runtime dependencies. It uses raw fetch against the GCS JSON API with built-in auth: Application Default Credentials (ADC) on Cloud Run / GKE, or explicit service account JWT for non-GCP environments.
Installation
npm install @walkeros/server-store-gcs- Integrated
- Bundled
import { startFlow } from '@walkeros/collector';
import { storeGcsInit } from '@walkeros/server-store-gcs';
await startFlow({
stores: {
assets: {
code: storeGcsInit,
config: {
// Omit config.credentials for ADC on Cloud Run/GKE
settings: {
bucket: 'my-assets',
prefix: 'public',
},
},
},
},
});Add to your flow.json:
"stores": {
"assets": {
"package": "@walkeros/server-store-gcs",
"config": {
"settings": {
"bucket": "my-assets",
"prefix": "public"
}
}
}
}Configuration
This store uses the standard store config wrapper (consent, data, env, id, ...). For the shared fields see store configuration. Package-specific fields live under config.settings and are listed below.
Settings
| Property | Type | Description | More |
|---|---|---|---|
bucket | string | GCS bucket name | |
prefix | string | Key prefix prepended to all store keys for scoping | |
credentials | string | object | Service account JSON (string or object). Omit for ADC on Cloud Run/GKE (deprecated: use config.credentials) |
Mapping
This package does not define custom rule-level settings. For the standard rule fields (consent, condition, data, batch, name, policy) see mapping.
Examples
Prefix scoping
Key "walker.js" with prefix "public/" resolves to GCS path "public/walker.js"
{
"operation": "get",
"key": "walker.js",
"settings": {
"bucket": "my-assets",
"prefix": "public"
}
}get("public/walker.js", "Bytes<...>")Read with ADC
Read object from GCS bucket using ADC (no credentials needed), bytes byte-exact
{
"operation": "get",
"key": "walker.js"
}get("walker.js", "Bytes<(function(){...})()>")When it fails
- failed read: The request fails (no connection, a refused or forbidden read): get throws a StepError instead of answering a miss, and the collector reports it. Reported as
failedwith codetransport.
Provisioning
The package ships an idempotent setup() lifecycle that creates the GCS bucket described in flow config. It is invoked only by the explicit operator command, on a store with config.setup set (true or an object):
walkeros setup store.<id>It never runs automatically and never alters an existing bucket.
Today the command stops before this setup runs, with Package @walkeros/server-store-gcs has no default export: walkeros setup reads setup from the package's default export, and this package's default export is its init function.
Setup options
| Option | Type | Default | Notes |
|---|---|---|---|
projectId | string | (resolved) | GCP project that owns the bucket. Resolution order below. |
location | string | 'EU' | Multi-region or regional location. |
storageClass | 'STANDARD' | 'NEARLINE' | 'COLDLINE' | 'ARCHIVE' | 'STANDARD' | Default object storage class. |
versioning | boolean | false | Object versioning. Off by default; opt in. |
lifecycle | { rule: unknown[] } | (none) | Applied at create. Drift detection not included for lifecycle. |
kmsKeyName | string | (none) | Customer-managed encryption key (CMEK) at create time. |
labels | Record<string, string> | (none) | Cost-allocation labels. |
bucket is taken from settings.bucket and is not duplicated under setup.
projectId resolution
The GCS create call requires a project. Resolution order:
- Explicit
setup.projectId. project_idfield inside theconfig.credentialsservice-account JSON.process.env.GOOGLE_CLOUD_PROJECT(Cloud Run / GKE convention).- Throws with an actionable error if none of the above is available.
Behavior
- Idempotent: HTTP 409 (bucket exists) is treated as success. The setup never patches or mutates an existing bucket.
- Drift detection: when the bucket already exists, setup performs a
GET /b/<bucket>and logs onesetup.driftwarning ({ field, declared, actual }) on the logger it receives for each differing field amonglocation,storageClass,versioning,iamConfiguration(uniform bucket-level access, public access prevention) andlabels. Drift is logged, never auto-fixed. - Defaults enforced at create: uniform bucket-level access on, public access prevention enforced. The package sets both on every bucket it creates.
Runtime hard-fail
At runtime, the first get / set / delete call issues a single HEAD /b/<bucket> per process per bucket. On 404, it throws with an actionable message:
GCS bucket not found: <bucket> in project <projectId>. Run "walkeros setup store.<id>" to create it.Operators see the error pointing at the exact command to fix it. Subsequent operations in the same process skip the check via an in-memory cache.
Authentication
Cloud Run / GKE (ADC)
When running on GCP infrastructure, omit credentials. The store fetches access tokens from the metadata server automatically, with no configuration needed.
Non-GCP (service account)
Pass a service account JSON via config.credentials with a $secret. reference. The store signs JWTs locally using node:crypto and exchanges them for access tokens via Google's OAuth2 endpoint.
"config": {
"credentials": "$secret.GCS_SA_KEY",
"settings": { "bucket": "my-assets", "prefix": "public" }
}The GCS_SA_KEY environment variable should contain the full service account JSON (with client_email and private_key fields). When it is missing, the flow stops at startup with required secret "GCS_SA_KEY" is not set. A credentials value without client_email and private_key is ignored without a log line, and the store uses ADC instead.
File serving pattern
The primary use case is serving static files via the file transformer. The express source maps the request path into ingest.path and runs the transformer in its next chain. Set file: true so the store persists and returns bytes byte-exact:
{
"stores": {
"assets": {
"package": "@walkeros/server-store-gcs",
"config": {
"settings": {
"bucket": "my-assets",
"prefix": "public"
},
"file": true
}
}
},
"sources": {
"http": {
"package": "@walkeros/server-source-express",
"config": {
"settings": { "paths": ["/collect", "/static/*path"] },
"ingest": { "map": { "path": { "key": "path" } } }
},
"next": "file"
}
},
"transformers": {
"file": {
"package": "@walkeros/server-transformer-file",
"config": { "settings": { "prefix": "/static" } },
"env": { "store": "$store.assets" }
}
}
}A request to /static/walker.js looks up public/walker.js in the my-assets bucket. Omit file for a structured key-value store instead.
Security
- Key validation: Keys with a
..segment or a leading/or\are rejected:getanswers a miss,setanddeletedo nothing. Each rejection is a WARN line, which the default log level (ERROR) does not show - Prefix scoping: The
prefixsetting restricts all operations to a subdirectory - Credentials: ADC comes from the GCP metadata server only, so off GCP the store needs an explicit service account JSON; a
GOOGLE_APPLICATION_CREDENTIALSfile is not read
Structured vs file mode
The GCS store has two modes, decided once at init by config.file:
- Structured (default): values are structured
StoreValuedata, serialized by the shared core codec and stored withContent-Type: application/json. Use for session state, lookups, and cached responses. - File (
file: true): values persist as raw bytes byte-exact, stored with the real mime derived from the key (orapplication/octet-streamwhen unknown).set()accepts aUint8Arrayorstring;get()hands the exact bytes back. Use for serving assets such as walker.js.
One store instance is exactly one mode.
API
const file = await store.get('walker.js'); // StoreValue | undefined
await store.set('data.json', { ok: true }); // structured value
await store.set(
'walker.js',
new Uint8Array([
/*…*/
]),
); // file mode: raw bytes
await store.delete('old-file.txt'); // voidIn file mode, get() returns the bytes as a Uint8Array leaf and set() rejects non-Uint8Array, non-string values with a clear error.