Redis Streams
Server-side event streaming to Redis Streams via the ioredis client. Each event is appended to a configurable stream via XADD, with optional approximate MAXLEN trimming for bounded memory usage, JSON or flat serialization modes, and graceful shutdown via destroy().
Redis Streams is a server destination in the walkerOS flow:
Receives events server-side from the collector, serializes them as JSON (or flat fields), and appends them to a Redis Stream for downstream consumers (stream processors, worker queues, pub/sub fan-out, XREAD consumer groups).
Installation
npm install @walkeros/collector @walkeros/server-destination-redis- Integrated
- Bundled
import { startFlow } from '@walkeros/collector';
import destinationRedis from '@walkeros/server-destination-redis';
await startFlow({
destinations: {
redis: {
code: destinationRedis,
config: {
settings: {
redis: {
streamKey: 'walkeros:events',
},
},
},
},
},
});
A complete flow.json with this destination:
{
"version": 4,
"flows": {
"default": {
"config": {
"platform": "server"
},
"destinations": {
"redis": {
"package": "@walkeros/server-destination-redis",
"config": {
"settings": {
"redis": {
"streamKey": "walkeros:events"
}
}
}
}
}
}
}
}Configuration
This destination uses the standard destination config wrapper (consent, data, env, id, ...). For the shared fields see destination configuration. Package-specific fields live under config.settings and are listed below.
Settings
| Property | Type | Description | More |
|---|---|---|---|
redis | redis | Redis Streams configuration | |
streamKey | string | Redis stream key name (like 'walkeros:events'). All events are appended to this stream via XADD. | |
url | string | Redis connection URL (like 'redis://localhost:6379' or 'rediss://:password@host:6380'). Supports redis:// and rediss:// (TLS) protocols. | |
options | Record<string, any> | ioredis connection options. Used when url is not provided. Supports host, port, password, db, tls, and all other ioredis options. | |
maxLen | integer | Maximum stream length. Enables approximate MAXLEN trimming on every XADD to bound memory usage (like 50000). | |
exactTrimming | boolean | Use exact MAXLEN instead of approximate (~). Not recommended for production. Default: false. | |
serialization | 'json' | 'flat' | Serialization mode. 'json' stores the full event as a single 'event' field (default). 'flat' stores top-level event fields as separate stream entry fields. |
Mapping
Per-event rules under config.mapping. For the standard rule fields (consent, condition, data, batch, name, policy) see mapping.
| Property | Type | Description | More |
|---|---|---|---|
streamKey | string | Override Redis stream key for this rule. Takes precedence over settings.redis.streamKey. |
Examples
Default stream
An event is appended to the configured Redis stream via XADD with the full event JSON as a single field.
{
"name": "page view",
"data": {
"domain": "www.example.com",
"title": "walkerOS documentation",
"referrer": "https://www.walkeros.io/",
"search": "?foo=bar",
"hash": "#hash",
"id": "/docs/"
},
"context": {
"dev": [
"test",
1
]
},
"globals": {
"pagegroup": "docs"
},
"custom": {
"completely": "random"
},
"user": {
"id": "us3r",
"device": "c00k13",
"session": "s3ss10n"
},
"nested": [
{
"entity": "child",
"data": {
"is": "subordinated"
}
}
],
"consent": {
"functional": true
},
"id": "f0cc5f6fec3d8eeb",
"trigger": "load",
"entity": "page",
"action": "view",
"timestamp": 1700000100000,
"timing": 3.14,
"source": {
"count": 1,
"trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9",
"type": "collector",
"schema": "4"
}
}client.xadd([
"walkeros:events",
"*",
"event",
"json:event"
])Order event
An order complete event is appended to the Redis stream alongside other event types for downstream consumers.
{
"name": "order complete",
"data": {
"id": "ORD-400",
"value": 99.99,
"currency": "EUR"
},
"context": {
"shopping": [
"complete",
0
]
},
"globals": {
"pagegroup": "shop"
},
"custom": {
"completely": "random"
},
"user": {
"id": "us3r",
"device": "c00k13",
"session": "s3ss10n"
},
"nested": [
{
"entity": "product",
"data": {
"id": "ers",
"name": "Everyday Ruck Snack",
"color": "black",
"size": "l",
"price": 420
},
"context": {
"shopping": [
"complete",
0
]
},
"nested": []
},
{
"entity": "product",
"data": {
"id": "cc",
"name": "Cool Cap",
"size": "one size",
"price": 42
},
"context": {
"shopping": [
"complete",
0
]
},
"nested": []
},
{
"entity": "gift",
"data": {
"name": "Surprise"
},
"context": {
"shopping": [
"complete",
0
]
},
"nested": []
}
],
"consent": {
"functional": true
},
"id": "235d0d7bd7a213f2",
"trigger": "load",
"entity": "order",
"action": "complete",
"timestamp": 1700000101000,
"timing": 3.14,
"source": {
"count": 1,
"trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9",
"type": "collector",
"schema": "4"
}
}client.xadd([
"walkeros:events",
"*",
"event",
"json:event"
])Stream key override
A mapping rule routes the event to a dedicated Redis stream instead of the destination default.
{
"name": "order complete",
"data": {
"id": "ORD-500",
"value": 42
},
"context": {
"shopping": [
"complete",
0
]
},
"globals": {
"pagegroup": "shop"
},
"custom": {
"completely": "random"
},
"user": {
"id": "us3r",
"device": "c00k13",
"session": "s3ss10n"
},
"nested": [
{
"entity": "product",
"data": {
"id": "ers",
"name": "Everyday Ruck Snack",
"color": "black",
"size": "l",
"price": 420
},
"context": {
"shopping": [
"complete",
0
]
},
"nested": []
},
{
"entity": "product",
"data": {
"id": "cc",
"name": "Cool Cap",
"size": "one size",
"price": 42
},
"context": {
"shopping": [
"complete",
0
]
},
"nested": []
},
{
"entity": "gift",
"data": {
"name": "Surprise"
},
"context": {
"shopping": [
"complete",
0
]
},
"nested": []
}
],
"consent": {
"functional": true
},
"id": "235d0d7bd7a213f2",
"trigger": "load",
"entity": "order",
"action": "complete",
"timestamp": 1700000104000,
"timing": 3.14,
"source": {
"count": 1,
"trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9",
"type": "collector",
"schema": "4"
}
}{
"settings": {
"streamKey": "walkeros:orders"
}
}client.xadd([
"walkeros:orders",
"*",
"event",
"json:event"
])Exact trim
XADD uses exact MAXLEN trimming to enforce a precise Redis stream length at the cost of extra work.
{
"name": "page view",
"data": {
"domain": "www.example.com",
"title": "walkerOS documentation",
"referrer": "https://www.walkeros.io/",
"search": "?foo=bar",
"hash": "#hash",
"id": "/docs/"
},
"context": {
"dev": [
"test",
1
]
},
"globals": {
"pagegroup": "docs"
},
"custom": {
"completely": "random"
},
"user": {
"id": "us3r",
"device": "c00k13",
"session": "s3ss10n"
},
"nested": [
{
"entity": "child",
"data": {
"is": "subordinated"
}
}
],
"consent": {
"functional": true
},
"id": "f0cc5f6fec3d8eeb",
"trigger": "load",
"entity": "page",
"action": "view",
"timestamp": 1700000103000,
"timing": 3.14,
"source": {
"count": 1,
"trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9",
"type": "collector",
"schema": "4"
}
}client.xadd([
"walkeros:events",
"MAXLEN",
5000,
"*",
"event",
"json:event"
])MAXLEN trim
XADD uses approximate MAXLEN trimming to cap the Redis stream length, discarding older entries efficiently.
{
"name": "product view",
"data": {
"id": "SKU-123",
"name": "Widget"
},
"context": {
"shopping": [
"detail",
0
]
},
"globals": {
"pagegroup": "shop"
},
"custom": {
"completely": "random"
},
"user": {
"id": "us3r",
"device": "c00k13",
"session": "s3ss10n"
},
"nested": [],
"consent": {
"functional": true
},
"id": "db22fb16c70f663d",
"trigger": "load",
"entity": "product",
"action": "view",
"timestamp": 1700000102000,
"timing": 3.14,
"source": {
"count": 1,
"trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9",
"type": "collector",
"schema": "4"
}
}client.xadd([
"walkeros:events",
"MAXLEN",
"~",
50000,
"*",
"event",
"json:event"
])The destination creates a single long-lived ioredis client during init(). With url set, options is ignored; with neither, ioredis connects to localhost:6379. On flow hot-swap or server shutdown, destroy() calls client.quit() to flush in-flight commands and close the TCP connection gracefully. User-provided clients (wired in via _client) are left untouched.
A failed XADD fails the push with a transport error: the event reaches the dead-letter queue and the collector logs one ERROR line per cause and minute. A lost connection is logged at WARN and counted in status.connectionErrors, and shutdown does not wait on a disconnected client. A rule with silent: true appends nothing.
Stream entry format
Each event is appended via XADD <streamKey> [MAXLEN [~] <n>] * <field> <value> ...:
- streamKey: from
settings.redis.streamKey(ormapping.settings.streamKeyoverride) *: auto-generated entry ID (ms-timestamp + sequence)- fields: in
jsonmode, a singleeventfield with the full event JSON-stringified; inflatmode, top-level event keys as separate fields (nested objects JSON-encoded) - MAXLEN: when
settings.redis.maxLenis set, trims the stream to ~N entries. Approximate (~) by default; setexactTrimming: trueto trim exactly (slower)
Use mapping.settings.streamKey to route specific events to dedicated streams (e.g. orders to walkeros:orders, identities to walkeros:identities).
Serialization modes
JSON (default)
Stores the full walkerOS event as a single event field. Easiest to consume, decode with JSON.parse().
XADD walkeros:events * event '{"name":"page view","data":{"title":"Home"},...}'
Flat
Stores top-level event fields as separate stream entry fields. Nested objects are JSON-encoded. Useful when downstream consumers want to filter or project specific fields without parsing the full event.
XADD walkeros:events * name 'page view' data '{"title":"Home"}' ... entity page action view timestamp 1700000000000 ...
Configure via settings.redis.serialization: 'flat'.
Authentication
Redis Cloud / Upstash (TLS + password)
{
"redis": {
"streamKey": "walkeros:events",
"url": "$secret.REDIS_URL"
}
}Store the whole rediss:// URL in the secret, password included (for example
rediss://default:<password>@<host>:<port>): a $secret. reference must be
the entire value (secrets).
AWS ElastiCache (in-VPC, no auth)
{
"redis": {
"streamKey": "walkeros:events",
"url": "redis://my-cluster.cache.amazonaws.com:6379"
}
}Custom options (ACL user, non-default db)
{
"redis": {
"streamKey": "walkeros:events",
"options": {
"host": "redis.example.com",
"port": 6379,
"username": "walkeros",
"password": "$secret.REDIS_PASSWORD",
"db": 1,
"tls": {}
}
}
}MAXLEN trimming
Redis Streams grow unbounded by default. For high-volume pipelines, cap stream length with maxLen:
{
"redis": {
"streamKey": "walkeros:events",
"url": "redis://localhost:6379",
"maxLen": 50000
}
}Approximate trimming (~) is used by default. Redis trims to "about N" entries with minimal overhead. For exact trimming, set exactTrimming: true (slower, use only when strict bounds are required).