Collector
The collector is the processing engine of walkerOS: it receives events from sources, enriches them with additional data, applies consent rules and routes them to destinations.
What it does
The collector validates and normalizes incoming events and applies privacy rules and the user's consent preferences. It adds its globals, user and consent state to every event, then sends the processed events to the configured destinations.
Role in architecture
In the walkerOS data flow, the collector sits between sources and destinations:
Sources capture events and send them to the collector, which processes and routes them to your chosen destinations like Google Analytics, custom APIs, or data warehouses.
Installation
npm install @walkeros/collectorBasic setup
import { startFlow } from '@walkeros/collector';
import { sourceBrowser } from '@walkeros/web-source-browser';
const { collector, elb } = await startFlow({
consent: { functional: true },
sources: {
browser: {
code: sourceBrowser,
config: {
settings: {
pageview: true,
},
},
},
},
destinations: {
console: {
code: {
type: 'console',
config: {},
push: (event) => console.log('Event:', event),
},
},
},
});Settings
| Property | Type | Description | More |
|---|---|---|---|
run | boolean | Whether to run collector automatically on initialization | |
globalsStatic | WalkerOS.Properties | Static global properties that persist across collector runs | |
sessionStatic | Collector.SessionStatic | Static session data that persists across collector runs | |
logger | Logger.Config | Logger configuration (level, handler) | |
queueMax | number | Maximum events retained in collector.queue (late-registration replay). FIFO drop on overflow. Default 1000. | |
consentDenied | 'queue' | 'skip' | What a destination does with a consent-denied event: 'queue' (default) keeps it until consent is granted, 'skip' skips it (counted in status.destinations[id].skipped). | |
name | string | Flow name; keys this flow's entry in event.source.release and the observer flowId. | |
release | string | Config release id stamped into event.source.release for this flow. | |
next | Route | Collector chain: runs once per completed event before the destination fan-out. A stop drops the event for every destination. | |
consent | WalkerOS.Consent | Initial consent state | |
user | WalkerOS.User | Initial user data | |
globals | WalkerOS.Properties | Initial global properties | |
sources | Source.InitSources | Source configurations | |
destinations | Destination.InitDestinations | Destination configurations | |
transformers | Transformer.Configs | Transformer configurations | |
stores | Store.Configs | Store configurations | |
custom | WalkerOS.Properties | Initial custom implementation-specific properties | |
hooks | Collector.Hooks | Pipeline observation hooks | |
observe | Observe | Observation session connect config | |
observers | Collector.Observers | Caller-supplied advisory observer functions |
Advanced setup
import { startFlow } from '@walkeros/collector';
import { sourceBrowser } from '@walkeros/web-source-browser';
import { destinationAPI } from '@walkeros/web-destination-api';
const { collector, elb } = await startFlow({
consent: { functional: true },
sources: {
browser: {
code: sourceBrowser,
config: {
settings: {
pageview: true,
},
},
},
},
destinations: {
api: {
code: destinationAPI,
config: {
settings: {
url: 'https://analytics.example.com/events',
},
},
},
},
logger: {
level: 'DEBUG', // Show all logs (ERROR, WARN, INFO, DEBUG)
},
});Event transformation
The collector uses mapping to transform events as they flow through the system. Sources and destinations each have their own mapping; a destination's mapping controls how walkerOS events are converted to vendor-specific formats.
For example, transforming a product add event into GA4's add_to_cart:
destinations: {
ga4: {
code: destinationGtag,
config: {
settings: { ga4: { measurementId: 'G-XXXXXXXXXX' } },
mapping: {
product: {
add: {
name: 'add_to_cart',
data: {
map: {
currency: { value: 'USD' },
value: 'data.price',
items: { loop: ['nested', { map: { item_id: 'data.id' } }] }
}
}
}
}
}
}
}
}Collector chain (next)
The collector's next is a transformer chain that runs once per event, after
the collector completed the event and before it hands the event to the
destinations. Every destination receives what the chain produces, so it is the
place for work all destinations share, such as validation or bot scoring.
import { startFlow } from '@walkeros/collector';
import { transformerBot } from '@walkeros/server-transformer-bot';
import { transformerValidate } from '@walkeros/transformer-validate';
await startFlow({
transformers: {
bot: { code: transformerBot },
validate: { code: transformerValidate },
},
next: [
{
match: { key: 'event.name', operator: 'eq', value: 'product impression' },
stop: true,
},
'bot',
'validate',
],
destinations: {
/* ... */
},
});next accepts the same Route as every other chain field (see
Chain resolution); in a flow
file it is collector.next. Its routes read { ingest, event } with the
completed event. What the chain does reaches every destination:
- A
stop(or a transformer returningfalse) drops the event for all destinations. To drop an event for one destination only, put thestopin that destination'sbefore. - A
manyfork delivers each copy to the destinations on its own, each with its ownevent.id. - Consent-queued events, late destinations, and replayed pre-run events receive the chain's output; the chain never runs twice for one event.
walker configcan replacenextfor later events.
Status
The collector tracks delivery metrics on collector.status:
const { collector } = await startFlow({
/* ... */
});
// After events have been processed
console.log(collector.status);
// {
// startedAt: 1707580800000,
// in: 100, // Total events received
// out: 95, // Total events delivered
// failed: 5, // Total failures
// sources: {
// browser: { count: 100, lastAt: 1707580900000, duration: 1200 }
// },
// destinations: {
// ga4: { count: 95, failed: 0, skipped: 0, lastAt: 1707580900000,
// duration: 800, queuePushSize: 0, dlqSize: 0 },
// meta: { count: 0, failed: 5, lastAt: 1707580850000, duration: 400,
// queuePushSize: 0, dlqSize: 5 }
// },
// // Process-wide drop counters keyed by stepId. See `stepId()` in
// // `@walkeros/core`. Entries are created lazily on first drop.
// dropped: {
// // "collector" is the reserved stepId for the collector-level buffer.
// // "destination.<id>" keys each destination's drop counters.
// // Each entry exposes optional { queue?, dlq? } counts.
// }
// }
// Compute average destination push time
const ga4 = collector.status.destinations.ga4;
const avgMs = ga4.duration / (ga4.count + ga4.failed);What a push answers
elb and collector.push answer a PushResult. For each destination the
event ends in one of three places:
done: the destination delivered the event,queued: the event is waiting, with areason:consent(the destination waits for a grant),batch(it sits in a batch until the flush) orheld(a state delivery to the destination, such as its consent handler, is running or failed; the destination retries once per window),failed: the destination did not deliver it, with areasonthat carries acode, amessageandretryable(see the table in Logger).
An event a destination skips on purpose, because a rule has ignore: true
or because consent is denied on a server collector, appears under none of the
three and is counted in status.destinations.<id>.skipped.
ok is false when any destination failed, and also when a step before the
destinations failed: reason then names the cause, for example queue-full
(the collector has not run yet and its queue is full), transformer (a
transformer threw or is not ready) or mapping (a rule condition threw).
Unmet consent is never a failure, so a consent wait answers ok: true with
queued.
A source that answers a producer reads this result: see Producer acknowledgement.
What failed counts
collector.status.failed is the single counter for any walkerOS-internal
pipeline failure. Each destination also counts its own failures in
status.destinations.<id>.failed. Contributing sites:
- a destination's
pushorpushBatchthrew, or the destination reported an error throughcontext.reportError, - a destination could not start or did not become ready:
initthrew or did not settle withinconfig.timeout, an open circuit breaker, a destination held past one retry window, a full queue, - an exception escaped the inner pipeline of
collector.pushorcollector.command(any uncaught error inside the boundary), - a transformer threw or is not ready,
- a rule
conditionthrew (the event fails withmapping), - source startup failures (
code()threw,init()threw, queued-on flush threw), - transformer init failures.
Rejected client input does not increment status.failed. An event
the pipeline declares invalid (a missing or malformed name, or an event
that cannot be serialized to JSON) resolves as
{ ok: false, invalid: true, error }, logs one error line without a stack
per source and minute, and counts on
collector.status.sources.<id>.rejected. That keeps failed a measure of
walkerOS faults rather than of whatever the public internet posts at a
collector endpoint. Pass values, not DOM nodes or framework objects: in a
React app a DOM node's own keys lead into cycles, so such an event is
refused as a whole.
User-supplied callback throws in on subscriptions (destination.on,
source.on, consent rules, ready, run, session, generic) log at error
level but do not increment status.failed. A mapping fn, validate or
value-level callback that throws drops only that value; the event still goes
out and the collector logs one ERROR line per cause. A destination whose
on('consent') handler fails is reported as consent-not-applied.
Every failure is written to the log by the collector, once per cause. See Logger for the line format, the rate limit and the codes.
A source whose init() throws stays with config.init === false
instead of being marked initialized, so source.config.init shows that the
source did not start.
Alarm on the ratio of status.failed to status.in. The structured fields of
each line (step, code, status, retryable) are enough to filter
destination vs boundary failures downstream.
Personal data in logs. Failure lines carry the step id, the cause code,
the HTTP status, the event name, the mapping key and the error message
(capped at 300 characters), never the event's data, user, consent,
globals, context or custom, an ingest value, a vendor response body or a
setting value. A vendor response body stays on the error object.
Consent-denied events on a server
A collector holds an event that a destination's consent, or its rule's, does not allow, and
sends it when consent is granted later (consentDenied: 'queue', the
default). A server has no later grant for an event's own consent, so server
flows built by the CLI default to consentDenied: 'skip': the event is not
queued, the destination counts it in skipped, and the collector writes a
DEBUG line. Set the value in the flow's collector block to choose the other
mode; an authored value always wins. Unmet consent is never an error or a
warning.
"collector": {
"consentDenied": "queue"
}With 'queue' on a server, which is also the default of a collector you start
with startFlow in your own code, every user's denied events wait in the same
destination queue. A later event whose own consent allows the destination is
sent while the same user's earlier denied event stays queued, and one
walker consent command sends the queued events of every user, including
users who never consented. Pass consentDenied: 'skip' to startFlow on a
server to skip these events instead.
A held event keeps the consent it was admitted with: when its destination's hold ends it is delivered even if the collector's consent changed meanwhile.
Flushing batches
walker flush sends every pending destination batch at once and leaves the
collector running (see Commands). The browser source
sends it when the page goes away.
Fatal errors
Throw FatalError (exported from @walkeros/core) for invariant
violations or operator-initiated aborts that must crash the host process.
Standard Error is absorbed by the boundary catch, logged, and counted.
FatalError bypasses the boundary catch of collector.push and
collector.command and propagates, so a supervisor (CLI runner, Express
server, container orchestrator) can terminate cleanly. It propagates when
thrown in a source (its factory or init), in a transformer's init or in an
on handler. Thrown in a destination's init or push, or in a
transformer's push, it is handled like any other error thrown there and does
not propagate. Thrown in a mapping callback of a destination, it does not
propagate either, but it fails that destination's push and the event goes to
its dead-letter queue. An ordinary error in a value callback drops only that
value; in a rule condition it fails the push the same way, with the reason
mapping.
import { FatalError } from '@walkeros/core';
// in a source's or transformer's init
if (!config.apiKey) {
throw new FatalError('apiKey missing, refusing to start');
}Queue sizes and DLQ sizes can be read directly from destination instances, or from the status snapshot:
const dest = collector.destinations['meta'];
const queueSize = dest.queuePush?.length || 0;
const dlqSize = dest.dlq?.length || 0;
// Or from status (point-in-time snapshots refreshed after each push pass):
const metaStatus = collector.status.destinations['meta'];
metaStatus.queuePushSize; // events waiting for consent
metaStatus.dlqSize; // failed pushes retained for triage
// Drop counts live on the process-wide `status.dropped` map, keyed by
// stepId (use `stepId()` from `@walkeros/core`):
import { stepId } from '@walkeros/core';
const metaDrops = collector.status.dropped[stepId('destination', 'meta')];
metaDrops?.queue ?? 0; // monotonic count of evicted consent-queued events
metaDrops?.dlq ?? 0; // monotonic count of evicted DLQ entriesBuffer bounds
The collector keeps four internal buffers per process. Each is size-bounded
with a configurable cap, and the dropped or failed counter moves on
overflow. What happens to the entry and how loud it is depends on the buffer:
collector.queueevicts the oldest entry and logs once per overflow window at INFO, because it only feeds destinations added later.collector.preRunQueuerefuses the newest event. That push answersok: falsewith the reasonqueue-full, counts as failed and logs an ERROR line.destination.queuePushevicts the oldest consent-waiting event first and counts it indropped; it logs once per overflow window at DEBUG, because unmet consent is never an error. When the destination is held and the queue holds only held events, the newest event is refused and fails withqueue-full.destination.dlqevicts the oldest failed entry (it was reported failed already, only its replay material is lost) and logs an ERROR line.
| Buffer | Purpose | Default cap | Config |
|---|---|---|---|
collector.queue | Replay buffer for late-registered destinations | 1000 | Collector.Config.queueMax |
collector.preRunQueue | Events pushed before the collector runs | 1000 | Collector.Config.queueMax |
destination.queuePush | Per-destination consent-denied buffer | 1000 | Destination.Config.queueMax |
destination.dlq | Per-destination dead-letter queue of failed pushes | 100 | Destination.Config.dlqMax |
Each step has its own cap, and the collector's value does not carry over to its destinations. Set the collector cap on the collector, and per-destination caps on each destination:
const { collector } = await startFlow({
// Collector replay buffer cap
queueMax: 5_000,
destinations: {
bigquery: {
code: bigqueryDestination,
config: {
// Keep more failed rows for triage on this destination
dlqMax: 1_000,
},
},
},
});Operators alarm on the dropped counters to detect sustained overflow.
Counters live on collector.status.dropped, keyed by stepId (build the
key with stepId() from @walkeros/core):
import { stepId } from '@walkeros/core';
// Collector replay buffer drops: traffic bursts outrunning destination
// registration.
const collectorDrops =
collector.status.dropped[stepId('collector')]?.queue ?? 0;
// Events refused before the collector ran are not drops: they count in
// collector.status.failed, and their push result has the reason queue-full.
// Destination drops by buffer:
// - queue: consent-denied events evicted from the destination's queuePush
// - dlq: failed-push entries evicted from the destination's DLQ
const ga4Drops = collector.status.dropped[stepId('destination', 'ga4')];
ga4Drops?.queue ?? 0;
ga4Drops?.dlq ?? 0; // sustained non-zero rate signals destination outageIn the complete example
The server collector of flow-complete.json, the tested example flow, sets a logger and
runs its chain once per event:
{
"logger": {
"level": "$var.logLevel"
},
"next": [
"fingerprint",
"bot",
{
"match": {
"key": "event.trigger",
"operator": "regex",
"value": "^(impression|visible)$",
"not": true
},
"next": "enrich"
}
]
}tourThe web collector carries the page defaults. globals can be overridden by
the page, globalsStatic survives a run reset, custom holds values that are
not event data:
{
"language": "en"
}web-entry{
"site": "$env.SITE_ID:shop"
}web-entry{
"environment": "$env.ENVIRONMENT"
}web-entrySee also
- Operating modes: integrated vs bundled approaches
- CLI documentation: configure with JSON and build with CLI
- Sources: available event capture sources
- Destinations: available event delivery destinations