Skip to main content
Ask your AI

Collector

The collector is the processing engine of walkerOS: it receives events from sources, enriches them with additional data, applies consent rules and routes them to destinations.

What it does​

The collector validates and normalizes incoming events and applies privacy rules and the user's consent preferences. It adds its globals, user and consent state to every event, then sends the processed events to the configured destinations.

Role in architecture​

In the walkerOS data flow, the collector sits between sources and destinations:

Source
Collector
Destination
123
1Receive: Incoming mapped events from sources2Process: State management, consent handling, enrichment3Route: Map events to destination-specific requirements

Sources capture events and send them to the collector, which processes and routes them to your chosen destinations like Google Analytics, custom APIs, or data warehouses.

Installation​

npm install @walkeros/collector

Basic setup​

import { startFlow } from '@walkeros/collector';
import { sourceBrowser } from '@walkeros/web-source-browser';

const { collector, elb } = await startFlow({
  consent: { functional: true },
  sources: {
    browser: {
      code: sourceBrowser,
      config: {
        settings: {
          pageview: true,
        },
      },
    },
  },
  destinations: {
    console: {
      code: {
        type: 'console',
        config: {},
        push: (event) => console.log('Event:', event),
      },
    },
  },
});

Settings​

PropertyTypeDescriptionMore
runbooleanWhether to run collector automatically on initialization
globalsStaticWalkerOS.PropertiesStatic global properties that persist across collector runs
sessionStaticCollector.SessionStaticStatic session data that persists across collector runs
loggerLogger.ConfigLogger configuration (level, handler)
queueMaxnumberMaximum events retained in collector.queue (late-registration replay). FIFO drop on overflow. Default 1000.
consentDenied'queue' | 'skip'What a destination does with a consent-denied event: 'queue' (default) keeps it until consent is granted, 'skip' skips it (counted in status.destinations[id].skipped).
namestringFlow name; keys this flow's entry in event.source.release and the observer flowId.
releasestringConfig release id stamped into event.source.release for this flow.
nextRouteCollector chain: runs once per completed event before the destination fan-out. A stop drops the event for every destination.
consentWalkerOS.ConsentInitial consent state
userWalkerOS.UserInitial user data
globalsWalkerOS.PropertiesInitial global properties
sourcesSource.InitSourcesSource configurations
destinationsDestination.InitDestinationsDestination configurations
transformersTransformer.ConfigsTransformer configurations
storesStore.ConfigsStore configurations
customWalkerOS.PropertiesInitial custom implementation-specific properties
hooksCollector.HooksPipeline observation hooks
observeObserveObservation session connect config
observersCollector.ObserversCaller-supplied advisory observer functions

Advanced setup​

import { startFlow } from '@walkeros/collector';
import { sourceBrowser } from '@walkeros/web-source-browser';
import { destinationAPI } from '@walkeros/web-destination-api';

const { collector, elb } = await startFlow({
  consent: { functional: true },
  sources: {
    browser: {
      code: sourceBrowser,
      config: {
        settings: {
          pageview: true,
        },
      },
    },
  },
  destinations: {
    api: {
      code: destinationAPI,
      config: {
        settings: {
          url: 'https://analytics.example.com/events',
        },
      },
    },
  },
  logger: {
    level: 'DEBUG', // Show all logs (ERROR, WARN, INFO, DEBUG)
  },
});

Event transformation​

The collector uses mapping to transform events as they flow through the system. Sources and destinations each have their own mapping; a destination's mapping controls how walkerOS events are converted to vendor-specific formats.

For example, transforming a product add event into GA4's add_to_cart:

destinations: {
  ga4: {
    code: destinationGtag,
    config: {
      settings: { ga4: { measurementId: 'G-XXXXXXXXXX' } },
      mapping: {
        product: {
          add: {
            name: 'add_to_cart',
            data: {
              map: {
                currency: { value: 'USD' },
                value: 'data.price',
                items: { loop: ['nested', { map: { item_id: 'data.id' } }] }
              }
            }
          }
        }
      }
    }
  }
}

Learn more about mapping

Collector chain (next)​

The collector's next is a transformer chain that runs once per event, after the collector completed the event and before it hands the event to the destinations. Every destination receives what the chain produces, so it is the place for work all destinations share, such as validation or bot scoring.

Source
Collector
Transformer(s)
GA4
Meta
1
1Runs once per event, before the fan-out
import { startFlow } from '@walkeros/collector';
import { transformerBot } from '@walkeros/server-transformer-bot';
import { transformerValidate } from '@walkeros/transformer-validate';

await startFlow({
  transformers: {
    bot: { code: transformerBot },
    validate: { code: transformerValidate },
  },
  next: [
    {
      match: { key: 'event.name', operator: 'eq', value: 'product impression' },
      stop: true,
    },
    'bot',
    'validate',
  ],
  destinations: {
    /* ... */
  },
});

next accepts the same Route as every other chain field (see Chain resolution); in a flow file it is collector.next. Its routes read { ingest, event } with the completed event. What the chain does reaches every destination:

  • A stop (or a transformer returning false) drops the event for all destinations. To drop an event for one destination only, put the stop in that destination's before.
  • A many fork delivers each copy to the destinations on its own, each with its own event.id.
  • Consent-queued events, late destinations, and replayed pre-run events receive the chain's output; the chain never runs twice for one event.
  • walker config can replace next for later events.

Status​

The collector tracks delivery metrics on collector.status:

const { collector } = await startFlow({
  /* ... */
});

// After events have been processed
console.log(collector.status);
// {
//   startedAt: 1707580800000,
//   in: 100,       // Total events received
//   out: 95,       // Total events delivered
//   failed: 5,     // Total failures
//   sources: {
//     browser: { count: 100, lastAt: 1707580900000, duration: 1200 }
//   },
//   destinations: {
//     ga4:  { count: 95, failed: 0, skipped: 0, lastAt: 1707580900000,
//             duration: 800, queuePushSize: 0, dlqSize: 0 },
//     meta: { count: 0, failed: 5, lastAt: 1707580850000, duration: 400,
//             queuePushSize: 0, dlqSize: 5 }
//   },
//   // Process-wide drop counters keyed by stepId. See `stepId()` in
//   // `@walkeros/core`. Entries are created lazily on first drop.
//   dropped: {
//     // "collector" is the reserved stepId for the collector-level buffer.
//     // "destination.<id>" keys each destination's drop counters.
//     // Each entry exposes optional { queue?, dlq? } counts.
//   }
// }

// Compute average destination push time
const ga4 = collector.status.destinations.ga4;
const avgMs = ga4.duration / (ga4.count + ga4.failed);

What a push answers​

elb and collector.push answer a PushResult. For each destination the event ends in one of three places:

  • done: the destination delivered the event,
  • queued: the event is waiting, with a reason: consent (the destination waits for a grant), batch (it sits in a batch until the flush) or held (a state delivery to the destination, such as its consent handler, is running or failed; the destination retries once per window),
  • failed: the destination did not deliver it, with a reason that carries a code, a message and retryable (see the table in Logger).

An event a destination skips on purpose, because a rule has ignore: true or because consent is denied on a server collector, appears under none of the three and is counted in status.destinations.<id>.skipped.

ok is false when any destination failed, and also when a step before the destinations failed: reason then names the cause, for example queue-full (the collector has not run yet and its queue is full), transformer (a transformer threw or is not ready) or mapping (a rule condition threw). Unmet consent is never a failure, so a consent wait answers ok: true with queued.

A source that answers a producer reads this result: see Producer acknowledgement.

What failed counts​

collector.status.failed is the single counter for any walkerOS-internal pipeline failure. Each destination also counts its own failures in status.destinations.<id>.failed. Contributing sites:

  • a destination's push or pushBatch threw, or the destination reported an error through context.reportError,
  • a destination could not start or did not become ready: init threw or did not settle within config.timeout, an open circuit breaker, a destination held past one retry window, a full queue,
  • an exception escaped the inner pipeline of collector.push or collector.command (any uncaught error inside the boundary),
  • a transformer threw or is not ready,
  • a rule condition threw (the event fails with mapping),
  • source startup failures (code() threw, init() threw, queued-on flush threw),
  • transformer init failures.

Rejected client input does not increment status.failed. An event the pipeline declares invalid (a missing or malformed name, or an event that cannot be serialized to JSON) resolves as { ok: false, invalid: true, error }, logs one error line without a stack per source and minute, and counts on collector.status.sources.<id>.rejected. That keeps failed a measure of walkerOS faults rather than of whatever the public internet posts at a collector endpoint. Pass values, not DOM nodes or framework objects: in a React app a DOM node's own keys lead into cycles, so such an event is refused as a whole.

User-supplied callback throws in on subscriptions (destination.on, source.on, consent rules, ready, run, session, generic) log at error level but do not increment status.failed. A mapping fn, validate or value-level callback that throws drops only that value; the event still goes out and the collector logs one ERROR line per cause. A destination whose on('consent') handler fails is reported as consent-not-applied.

Every failure is written to the log by the collector, once per cause. See Logger for the line format, the rate limit and the codes.

A source whose init() throws stays with config.init === false instead of being marked initialized, so source.config.init shows that the source did not start.

Alarm on the ratio of status.failed to status.in. The structured fields of each line (step, code, status, retryable) are enough to filter destination vs boundary failures downstream.

Personal data in logs. Failure lines carry the step id, the cause code, the HTTP status, the event name, the mapping key and the error message (capped at 300 characters), never the event's data, user, consent, globals, context or custom, an ingest value, a vendor response body or a setting value. A vendor response body stays on the error object.

A collector holds an event that a destination's consent, or its rule's, does not allow, and sends it when consent is granted later (consentDenied: 'queue', the default). A server has no later grant for an event's own consent, so server flows built by the CLI default to consentDenied: 'skip': the event is not queued, the destination counts it in skipped, and the collector writes a DEBUG line. Set the value in the flow's collector block to choose the other mode; an authored value always wins. Unmet consent is never an error or a warning.

"collector": {
  "consentDenied": "queue"
}

With 'queue' on a server, which is also the default of a collector you start with startFlow in your own code, every user's denied events wait in the same destination queue. A later event whose own consent allows the destination is sent while the same user's earlier denied event stays queued, and one walker consent command sends the queued events of every user, including users who never consented. Pass consentDenied: 'skip' to startFlow on a server to skip these events instead.

A held event keeps the consent it was admitted with: when its destination's hold ends it is delivered even if the collector's consent changed meanwhile.

Flushing batches​

walker flush sends every pending destination batch at once and leaves the collector running (see Commands). The browser source sends it when the page goes away.

Fatal errors​

Throw FatalError (exported from @walkeros/core) for invariant violations or operator-initiated aborts that must crash the host process. Standard Error is absorbed by the boundary catch, logged, and counted. FatalError bypasses the boundary catch of collector.push and collector.command and propagates, so a supervisor (CLI runner, Express server, container orchestrator) can terminate cleanly. It propagates when thrown in a source (its factory or init), in a transformer's init or in an on handler. Thrown in a destination's init or push, or in a transformer's push, it is handled like any other error thrown there and does not propagate. Thrown in a mapping callback of a destination, it does not propagate either, but it fails that destination's push and the event goes to its dead-letter queue. An ordinary error in a value callback drops only that value; in a rule condition it fails the push the same way, with the reason mapping.

import { FatalError } from '@walkeros/core';

// in a source's or transformer's init
if (!config.apiKey) {
  throw new FatalError('apiKey missing, refusing to start');
}

Queue sizes and DLQ sizes can be read directly from destination instances, or from the status snapshot:

const dest = collector.destinations['meta'];
const queueSize = dest.queuePush?.length || 0;
const dlqSize = dest.dlq?.length || 0;

// Or from status (point-in-time snapshots refreshed after each push pass):
const metaStatus = collector.status.destinations['meta'];
metaStatus.queuePushSize; // events waiting for consent
metaStatus.dlqSize; // failed pushes retained for triage

// Drop counts live on the process-wide `status.dropped` map, keyed by
// stepId (use `stepId()` from `@walkeros/core`):
import { stepId } from '@walkeros/core';
const metaDrops = collector.status.dropped[stepId('destination', 'meta')];
metaDrops?.queue ?? 0; // monotonic count of evicted consent-queued events
metaDrops?.dlq ?? 0; // monotonic count of evicted DLQ entries

Buffer bounds​

The collector keeps four internal buffers per process. Each is size-bounded with a configurable cap, and the dropped or failed counter moves on overflow. What happens to the entry and how loud it is depends on the buffer:

  • collector.queue evicts the oldest entry and logs once per overflow window at INFO, because it only feeds destinations added later.
  • collector.preRunQueue refuses the newest event. That push answers ok: false with the reason queue-full, counts as failed and logs an ERROR line.
  • destination.queuePush evicts the oldest consent-waiting event first and counts it in dropped; it logs once per overflow window at DEBUG, because unmet consent is never an error. When the destination is held and the queue holds only held events, the newest event is refused and fails with queue-full.
  • destination.dlq evicts the oldest failed entry (it was reported failed already, only its replay material is lost) and logs an ERROR line.
BufferPurposeDefault capConfig
collector.queueReplay buffer for late-registered destinations1000Collector.Config.queueMax
collector.preRunQueueEvents pushed before the collector runs1000Collector.Config.queueMax
destination.queuePushPer-destination consent-denied buffer1000Destination.Config.queueMax
destination.dlqPer-destination dead-letter queue of failed pushes100Destination.Config.dlqMax

Each step has its own cap, and the collector's value does not carry over to its destinations. Set the collector cap on the collector, and per-destination caps on each destination:

const { collector } = await startFlow({
  // Collector replay buffer cap
  queueMax: 5_000,
  destinations: {
    bigquery: {
      code: bigqueryDestination,
      config: {
        // Keep more failed rows for triage on this destination
        dlqMax: 1_000,
      },
    },
  },
});

Operators alarm on the dropped counters to detect sustained overflow. Counters live on collector.status.dropped, keyed by stepId (build the key with stepId() from @walkeros/core):

import { stepId } from '@walkeros/core';

// Collector replay buffer drops: traffic bursts outrunning destination
// registration.
const collectorDrops =
  collector.status.dropped[stepId('collector')]?.queue ?? 0;

// Events refused before the collector ran are not drops: they count in
// collector.status.failed, and their push result has the reason queue-full.

// Destination drops by buffer:
//  - queue: consent-denied events evicted from the destination's queuePush
//  - dlq:   failed-push entries evicted from the destination's DLQ
const ga4Drops = collector.status.dropped[stepId('destination', 'ga4')];
ga4Drops?.queue ?? 0;
ga4Drops?.dlq ?? 0; // sustained non-zero rate signals destination outage

In the complete example​

The server collector of flow-complete.json, the tested example flow, sets a logger and runs its chain once per event:

/flows/server/collector
{
  "logger": {
    "level": "$var.logLevel"
  },
  "next": [
    "fingerprint",
    "bot",
    {
      "match": {
        "key": "event.trigger",
        "operator": "regex",
        "value": "^(impression|visible)$",
        "not": true
      },
      "next": "enrich"
    }
  ]
}
The collector sits between sources and destinations and runs its own chain once per event. Guide chapter tour

The web collector carries the page defaults. globals can be overridden by the page, globalsStatic survives a run reset, custom holds values that are not event data:

/flows/web/collector/globals
{
  "language": "en"
}
globals.language is the default the page can override; the contract requires it. Guide chapter web-entry
/flows/web/collector/globalsStatic
{
  "site": "$env.SITE_ID:shop"
}
globalsStatic survives a run reset; the site id belongs there. Guide chapter web-entry
/flows/web/collector/custom
{
  "environment": "$env.ENVIRONMENT"
}
custom carries values for the pipeline that are not event data. Guide chapter web-entry

See also​

💡 Need implementation support?
elbwalker offers hands-on support: setup review, measurement planning, destination mapping, and live troubleshooting. Book a 2-hour session (€399)