S3
S3-compatible object storage store using s3mini (about 26 KB minified, no dependencies). It works with AWS S3, Cloudflare R2, Scaleway, DigitalOcean Spaces, Backblaze B2, MinIO, and any S3-compatible provider.
Installation
npm install @walkeros/server-store-s3- Integrated
- Bundled
import { startFlow } from '@walkeros/collector';
import { storeS3Init } from '@walkeros/server-store-s3';
await startFlow({
stores: {
assets: {
code: storeS3Init,
config: {
settings: {
bucket: 'my-assets',
endpoint: 'https://s3.eu-west-1.amazonaws.com',
accessKeyId: process.env.S3_ACCESS_KEY,
secretAccessKey: process.env.S3_SECRET_KEY,
region: 'eu-west-1',
prefix: 'public',
},
},
},
},
});Add to your flow.json:
"stores": {
"assets": {
"package": "@walkeros/server-store-s3",
"import": "storeS3Init",
"config": {
"settings": {
"bucket": "my-assets",
"endpoint": "https://s3.eu-west-1.amazonaws.com",
"accessKeyId": "$secret.S3_ACCESS_KEY",
"secretAccessKey": "$secret.S3_SECRET_KEY",
"region": "eu-west-1",
"prefix": "public"
}
}
}
}Bundled flows
In a bundled flow, name the init export with "import": "storeS3Init", as the examples on this page do. Without it the flow does not start: the bundler uses the package's default export as the store's code, this package's default export is an object, and startup fails with a TypeError (... is not a function). walkeros setup reads that default export instead, so a setup run needs the step without import (see Provisioning).
Configuration
This store uses the standard store config wrapper (consent, data, env, id, ...). For the shared fields see store configuration. Package-specific fields live under config.settings and are listed below.
Settings
| Property | Type | Description | More |
|---|---|---|---|
bucket | string | S3 bucket name | |
endpoint | string | S3-compatible endpoint URL | |
accessKeyId | string | S3 access key ID | |
secretAccessKey | string | S3 secret access key | |
region | string | AWS region for SigV4 signing | |
prefix | string | Key prefix prepended to all store keys for scoping |
Mapping
This package does not define custom rule-level settings. For the standard rule fields (consent, condition, data, batch, name, policy) see mapping.
Examples
Prefix scoping
Key "walker.js" with prefix "public/" resolves to S3 path "public/walker.js"
{
"operation": "get",
"key": "walker.js",
"settings": {
"bucket": "my-assets",
"prefix": "public"
}
}get("public/walker.js", "Bytes<...>")Read from S3
Read object from S3 and receive its raw bytes byte-exact
{
"operation": "get",
"key": "walker.js"
}get("walker.js", "Bytes<(function(){...})()>")When it fails
- failed read: The request fails (no connection, a refused or forbidden read): get throws a StepError instead of answering a miss, and the collector reports it. Reported as
failedwith codetransport.
Provider examples
| Provider | Endpoint | Notes |
|---|---|---|
| AWS S3 | https://s3.<region>.amazonaws.com | Set region to your actual region |
| Cloudflare R2 | https://<account>.r2.cloudflarestorage.com | |
| Scaleway | https://s3.<region>.scw.cloud | |
| DigitalOcean | https://<region>.digitaloceanspaces.com | |
| Backblaze B2 | https://s3.<region>.backblazeb2.com | |
| MinIO | http://localhost:9000 | Self-hosted |
Credentials
Reference the keys with $secret. so they never sit in the flow file. A server bundle reads them from the environment at runtime, and a missing one stops the flow at startup with required secret "S3_ACCESS_KEY" is not set:
"accessKeyId": "$secret.S3_ACCESS_KEY",
"secretAccessKey": "$secret.S3_SECRET_KEY"Unlike the AWS SDK, s3mini has no implicit credential chain: accessKeyId and secretAccessKey are always required.
File serving pattern
The primary use case is serving static files via the file transformer. This is the recommended pattern for managed deployments where files live in a bucket rather than needing to be baked into a Docker image. The express source maps the request path into ingest.path and runs the transformer in its next chain. Set file: true so the store persists and returns bytes byte-exact:
{
"stores": {
"assets": {
"package": "@walkeros/server-store-s3",
"import": "storeS3Init",
"config": {
"settings": {
"bucket": "my-assets",
"endpoint": "https://s3.eu-west-1.amazonaws.com",
"accessKeyId": "$secret.S3_ACCESS_KEY",
"secretAccessKey": "$secret.S3_SECRET_KEY",
"prefix": "public"
},
"file": true
}
}
},
"sources": {
"http": {
"package": "@walkeros/server-source-express",
"config": {
"settings": { "paths": ["/collect", "/static/*path"] },
"ingest": { "map": { "path": { "key": "path" } } }
},
"next": "file"
}
},
"transformers": {
"file": {
"package": "@walkeros/server-transformer-file",
"config": { "settings": { "prefix": "/static" } },
"env": { "store": "$store.assets" }
}
}
}A request to /static/walker.js looks up public/walker.js in the my-assets bucket. Omit file for a structured key-value store instead.
Provisioning
The package ships an idempotent setup() lifecycle, invoked only by the explicit operator command:
walkeros setup store.<id>It never runs automatically. It checks whether the bucket exists and creates it if not. Re-running setup is a no-op when the bucket already exists in your account.
Setup options
| Option | Type | Default | Notes |
|---|---|---|---|
region | string | eu-central-1 | Region the bucket is created in (LocationConstraint). Falls back to settings.region when concrete (not auto). |
bucket is taken from settings.bucket and is not duplicated under setup.
Enable provisioning
Set setup: true in the component config to enable provisioning with defaults, or pass an object to override:
{
"stores": {
"assets": {
"package": "@walkeros/server-store-s3",
"config": {
"settings": {
"bucket": "my-assets",
"endpoint": "https://s3.eu-central-1.amazonaws.com",
"accessKeyId": "$secret.S3_ACCESS_KEY",
"secretAccessKey": "$secret.S3_SECRET_KEY",
"region": "eu-central-1"
},
"setup": true
}
}
}
}This step has no import: walkeros setup reads setup from the package's default export, and on a step with "import": "storeS3Init" it stops with has no export "storeS3Init". Run setup from a config without import, and the flow with it (see Bundled flows).
What setup does NOT apply
s3mini is a minimal S3 client that exposes createBucket and bucketExists only. It does not expose PutBucketEncryption, PutPublicAccessBlock, PutBucketVersioning, PutBucketLifecycleConfiguration, or PutBucketTagging. Configure those once via the AWS Console or aws s3api.
Behavior
- Idempotent create:
BucketAlreadyOwnedByYou(concurrent caller, your account) is treated as success and returns{ bucketCreated: false }.BucketAlreadyExists(different AWS account owns the global name) fails loud with an actionable message so you pick a different name. - Region resolution: explicit
setup.regionwins; otherwisesettings.regionis used when concrete (notauto); otherwise the EU defaulteu-central-1.
Runtime hard-fail
The bucket is checked at the first get, set or delete, not when the flow starts: that operation probes bucketExists() once. On a missing bucket it throws with an actionable message:
S3 bucket not found: my-assets at https://s3.eu-central-1.amazonaws.com. Run "walkeros setup store.assets" to create it.Run walkeros setup store.<id> once to provision the bucket. A failed check is repeated at the next operation, so the running flow picks up the new bucket without a restart.
Security
- Key validation: Keys with a
..segment or a leading/or\are rejected:getanswers a miss,setanddeletedo nothing. Each rejection is a WARN line, which the default log level (ERROR) does not show - Prefix scoping: The
prefixsetting restricts all operations to a subdirectory - No credential chain: Credentials must be explicitly provided (no ambient AWS credentials)
Structured vs file mode
The S3 store has two modes, decided once at init by config.file:
- Structured (default): values are structured
StoreValuedata, serialized by the shared core codec and stored withContent-Type: application/json. Use for session state, lookups, and cached responses. - File (
file: true): values persist as raw bytes byte-exact, stored with the real mime derived from the key (orapplication/octet-streamwhen unknown).set()accepts aUint8Arrayorstring;get()hands the exact bytes back. Use for serving assets such as walker.js.
One store instance is exactly one mode.
API
const file = await store.get('walker.js'); // StoreValue | undefined
await store.set('data.json', { ok: true }); // structured value
await store.set(
'walker.js',
new Uint8Array([
/*…*/
]),
); // file mode: raw bytes
await store.delete('old-file.txt'); // voidIn file mode, get() returns the bytes as a Uint8Array leaf and set() rejects non-Uint8Array, non-string values with a clear error.