Skip to main content
Ask your AI

S3

Server Source code Package

S3-compatible object storage store using s3mini (about 26 KB minified, no dependencies). It works with AWS S3, Cloudflare R2, Scaleway, DigitalOcean Spaces, Backblaze B2, MinIO, and any S3-compatible provider.

Installation​

npm install @walkeros/server-store-s3
import { startFlow } from '@walkeros/collector';
import { storeS3Init } from '@walkeros/server-store-s3';

await startFlow({
  stores: {
    assets: {
      code: storeS3Init,
      config: {
        settings: {
          bucket: 'my-assets',
          endpoint: 'https://s3.eu-west-1.amazonaws.com',
          accessKeyId: process.env.S3_ACCESS_KEY,
          secretAccessKey: process.env.S3_SECRET_KEY,
          region: 'eu-west-1',
          prefix: 'public',
        },
      },
    },
  },
});

Bundled flows​

In a bundled flow, name the init export with "import": "storeS3Init", as the examples on this page do. Without it the flow does not start: the bundler uses the package's default export as the store's code, this package's default export is an object, and startup fails with a TypeError (... is not a function). walkeros setup reads that default export instead, so a setup run needs the step without import (see Provisioning).

Configuration​

This store uses the standard store config wrapper (consent, data, env, id, ...). For the shared fields see store configuration. Package-specific fields live under config.settings and are listed below.

Settings​

PropertyTypeDescriptionMore
bucket*stringS3 bucket name
endpoint*stringS3-compatible endpoint URL
accessKeyId*stringS3 access key ID
secretAccessKey*stringS3 secret access key
regionstringAWS region for SigV4 signing
prefixstringKey prefix prepended to all store keys for scoping
* Required fields

Mapping​

This package does not define custom rule-level settings. For the standard rule fields (consent, condition, data, batch, name, policy) see mapping.

Examples

Prefix scoping

Key "walker.js" with prefix "public/" resolves to S3 path "public/walker.js"

Event
{
  "operation": "get",
  "key": "walker.js",
  "settings": {
    "bucket": "my-assets",
    "prefix": "public"
  }
}
Out
get("public/walker.js", "Bytes<...>")

Read from S3

Read object from S3 and receive its raw bytes byte-exact

Event
{
  "operation": "get",
  "key": "walker.js"
}
Out
get("walker.js", "Bytes<(function(){...})()>")

When it fails

  • failed read: The request fails (no connection, a refused or forbidden read): get throws a StepError instead of answering a miss, and the collector reports it. Reported as failed with code transport.

Provider examples​

ProviderEndpointNotes
AWS S3https://s3.<region>.amazonaws.comSet region to your actual region
Cloudflare R2https://<account>.r2.cloudflarestorage.com
Scalewayhttps://s3.<region>.scw.cloud
DigitalOceanhttps://<region>.digitaloceanspaces.com
Backblaze B2https://s3.<region>.backblazeb2.com
MinIOhttp://localhost:9000Self-hosted

Credentials​

Reference the keys with $secret. so they never sit in the flow file. A server bundle reads them from the environment at runtime, and a missing one stops the flow at startup with required secret "S3_ACCESS_KEY" is not set:

"accessKeyId": "$secret.S3_ACCESS_KEY",
"secretAccessKey": "$secret.S3_SECRET_KEY"

Unlike the AWS SDK, s3mini has no implicit credential chain: accessKeyId and secretAccessKey are always required.

File serving pattern​

The primary use case is serving static files via the file transformer. This is the recommended pattern for managed deployments where files live in a bucket rather than needing to be baked into a Docker image. The express source maps the request path into ingest.path and runs the transformer in its next chain. Set file: true so the store persists and returns bytes byte-exact:

{
  "stores": {
    "assets": {
      "package": "@walkeros/server-store-s3",
      "import": "storeS3Init",
      "config": {
        "settings": {
          "bucket": "my-assets",
          "endpoint": "https://s3.eu-west-1.amazonaws.com",
          "accessKeyId": "$secret.S3_ACCESS_KEY",
          "secretAccessKey": "$secret.S3_SECRET_KEY",
          "prefix": "public"
        },
        "file": true
      }
    }
  },
  "sources": {
    "http": {
      "package": "@walkeros/server-source-express",
      "config": {
        "settings": { "paths": ["/collect", "/static/*path"] },
        "ingest": { "map": { "path": { "key": "path" } } }
      },
      "next": "file"
    }
  },
  "transformers": {
    "file": {
      "package": "@walkeros/server-transformer-file",
      "config": { "settings": { "prefix": "/static" } },
      "env": { "store": "$store.assets" }
    }
  }
}

A request to /static/walker.js looks up public/walker.js in the my-assets bucket. Omit file for a structured key-value store instead.

Provisioning​

The package ships an idempotent setup() lifecycle, invoked only by the explicit operator command:

walkeros setup store.<id>

It never runs automatically. It checks whether the bucket exists and creates it if not. Re-running setup is a no-op when the bucket already exists in your account.

Setup options​

OptionTypeDefaultNotes
regionstringeu-central-1Region the bucket is created in (LocationConstraint). Falls back to settings.region when concrete (not auto).

bucket is taken from settings.bucket and is not duplicated under setup.

Enable provisioning​

Set setup: true in the component config to enable provisioning with defaults, or pass an object to override:

{
  "stores": {
    "assets": {
      "package": "@walkeros/server-store-s3",
      "config": {
        "settings": {
          "bucket": "my-assets",
          "endpoint": "https://s3.eu-central-1.amazonaws.com",
          "accessKeyId": "$secret.S3_ACCESS_KEY",
          "secretAccessKey": "$secret.S3_SECRET_KEY",
          "region": "eu-central-1"
        },
        "setup": true
      }
    }
  }
}

This step has no import: walkeros setup reads setup from the package's default export, and on a step with "import": "storeS3Init" it stops with has no export "storeS3Init". Run setup from a config without import, and the flow with it (see Bundled flows).

What setup does NOT apply​

s3mini is a minimal S3 client that exposes createBucket and bucketExists only. It does not expose PutBucketEncryption, PutPublicAccessBlock, PutBucketVersioning, PutBucketLifecycleConfiguration, or PutBucketTagging. Configure those once via the AWS Console or aws s3api.

Behavior​

  • Idempotent create: BucketAlreadyOwnedByYou (concurrent caller, your account) is treated as success and returns { bucketCreated: false }. BucketAlreadyExists (different AWS account owns the global name) fails loud with an actionable message so you pick a different name.
  • Region resolution: explicit setup.region wins; otherwise settings.region is used when concrete (not auto); otherwise the EU default eu-central-1.

Runtime hard-fail​

The bucket is checked at the first get, set or delete, not when the flow starts: that operation probes bucketExists() once. On a missing bucket it throws with an actionable message:

S3 bucket not found: my-assets at https://s3.eu-central-1.amazonaws.com. Run "walkeros setup store.assets" to create it.

Run walkeros setup store.<id> once to provision the bucket. A failed check is repeated at the next operation, so the running flow picks up the new bucket without a restart.

Security​

  • Key validation: Keys with a .. segment or a leading / or \ are rejected: get answers a miss, set and delete do nothing. Each rejection is a WARN line, which the default log level (ERROR) does not show
  • Prefix scoping: The prefix setting restricts all operations to a subdirectory
  • No credential chain: Credentials must be explicitly provided (no ambient AWS credentials)

Structured vs file mode​

The S3 store has two modes, decided once at init by config.file:

  • Structured (default): values are structured StoreValue data, serialized by the shared core codec and stored with Content-Type: application/json. Use for session state, lookups, and cached responses.
  • File (file: true): values persist as raw bytes byte-exact, stored with the real mime derived from the key (or application/octet-stream when unknown). set() accepts a Uint8Array or string; get() hands the exact bytes back. Use for serving assets such as walker.js.

One store instance is exactly one mode.

API​

const file = await store.get('walker.js'); // StoreValue | undefined
await store.set('data.json', { ok: true }); // structured value
await store.set(
  'walker.js',
  new Uint8Array([
    /*…*/
  ]),
); // file mode: raw bytes
await store.delete('old-file.txt'); // void

In file mode, get() returns the bytes as a Uint8Array leaf and set() rejects non-Uint8Array, non-string values with a clear error.

💡 Need implementation support?
elbwalker offers hands-on support: setup review, measurement planning, destination mapping, and live troubleshooting. Book a 2-hour session (€399)