Skip to main content
Ask your AI

Create Your Own Source

A source captures events from an environment (browser, server, third-party API) and forwards them to the walkerOS collector.

The source interface​

Sources are async functions that receive a context object and return a source instance:

type Source.Init<Types> = (
  context: Source.Context<Types>,
) => Source.Instance<Types> | Promise<Source.Instance<Types>>;

The context contains:

interface Source.Context<Types> {
  config: Partial<Source.Config<Types>>;  // Settings, mapping, options
  env: Types['env'];                       // Environment (push, logger)
  logger: Logger;                          // Logging functions
  id: string;                              // Source identifier
  collector: Collector.Instance;           // Collector reference
  withScope: Source.Context['withScope'];  // Per-request scope (server sources)
}

The returned instance must implement:

interface Source.Instance {
  type: string; // Unique identifier
  config: Source.Config; // Merged configuration
  push: Elb.Fn; // Outward entry, usually env.elb
  init?(): void | Promise<void>; // Optional setup, runs once every source is registered
  destroy?(context): void | Promise<void>; // Optional cleanup
  on?(event, context): void | Promise<void>; // Optional reactive hooks
}

Events the source captures go through env.push, the source pipeline: its mapping, its before and next chains and its entry in collector.status.sources. env.elb is the collector's public interface for events and walker commands, and it skips the source pipeline. When the source is primary, startFlow returns its push as elb, so set push to env.elb unless the source exposes its own signature, such as an HTTP handler. With env.push in that slot, a walker command answers { ok: false, invalid: true }.

Types bundle​

Sources use a Types interface to bundle all TypeScript types:

import type { Source, Elb } from '@walkeros/core';

// 1. Define your settings
interface Settings {
  captureClicks?: boolean;
  prefix?: string;
}

// 2. Define environment dependencies
interface Env extends Source.BaseEnv {
  customAPI?: YourAPIType;
}

// 3. Bundle them together
interface Types extends Source.Types<Settings, never, Elb.Fn, Env> {}

The 4 type parameters:

  1. Settings: Source configuration options
  2. Mapping: Event mapping (usually never for sources)
  3. Push: External push signature (what source.push exposes)
  4. Env: Internal dependencies the source reads from env

Context destructuring​

The context parameter contains everything your source needs:

export const mySource: Source.Init<Types> = async (context) => {
  // Destructure what you need from context
  const { config = {}, env, logger, id } = context;
  const { push: envPush, customAPI } = env;

  // customAPI validation only needed if required by your source
  if (!customAPI) {
    throw new Error('Source requires customAPI in environment');
  }

  // Use env.push to send events to collector
  await envPush({ name: 'my event', data: { value: 'example' } });

  // ...then return the source instance (see the minimal example below)
};

env is the author's dependency bag: you provide platform and vendor dependencies (like window, document, custom APIs) when configuring the source. The collector provides five capabilities into that bag, and they always win: push, elb, command, logger and sources are runtime-owned, so setting one of them in env has no effect.

A factory that throws, for example over a missing dependency, is logged at ERROR (source factory failed) and the source is not registered.

Ending the pipeline elsewhere: terminus​

A source registration that needs the pipeline to end somewhere other than the collector declares terminus next to code. A terminus receives the event exactly as the source emitted it, and the entire pipeline is skipped: no before/next chains, no source cache, no state, no mapping (including ignore and consent), no minted span id, no per-source ingest, no respond, no status.sources counting and no observability records. The bypass is total by design, with no stage excepted. It exists for deterministic capture at the source→collector boundary (step examples, dev tooling); production flows leave it unset, and stored flow configs cannot carry it (validation rejects unknown source keys).

Per-scope context (server sources)​

A single source factory instance handles many concurrent invocations: an Express server processes overlapping requests, a Lambda may be reused across invocations, a queue consumer processes batches. Each logical unit of work, an HTTP request, a queue message, a websocket frame, is a scope. To keep ingest metadata and response delegates isolated between concurrent scopes, server sources wrap each invocation with context.withScope:

const push = async (req: Request, res: Response): Promise<void> => {
  const respond = createRespond((options) => {
    // wire options into res.send / res.json
  });

  // Normalize the platform request into the shared scope shape first, so
  // `config.ingest` paths resolve the same here as on every other source.
  const scope = buildScopeFromNodeRequest(req, {
    body: req.body,
    defaultProtocol: 'http',
  });

  // Each call to withScope builds a fresh Ingest from the scope via
  // `config.ingest` mapping, captures `respond`, and runs the body
  // with a per-scope env. Concurrent scopes never crosstalk.
  await context.withScope(scope, respond, async (env) => {
    // toEventList decides how many events the body carries; each push
    // result becomes the outcome the caller is answered with.
    const outcomes: EventOutcome[] = [];
    for (const event of toEventList(scope.body))
      outcomes.push(pushResultToOutcome(await env.push(event)));
    const { status, body } = batchResponse(outcomes);
    respond({ status, body });
  });
};

A new server source must build a scope and use toEventList, so config.ingest paths and accepted body forms match every other source. pushResultToOutcome turns each push result into an outcome: an error with status 400 for an invalid event, an error with status 500 when a resend could fix the failure, and the event id when it was delivered or failed for a reason a resend cannot fix (still visible in the destination's failed counter and an ERROR line). batchResponse answers 207 with per-index errors when any outcome is an error and 200 otherwise, so in the snippet above a single event gets the batch answer too; a source that answers single-event requests itself uses the outcome's status. See producer acknowledgement. buildScopeFromNodeRequest, normalizeHeaders, normalizeQuery, normalizeBody, toEventList, isBatchBody, pushResultToOutcome and batchResponse all come from @walkeros/core.

Inside the body, env.push carries the scope's ingest and respond all the way to destinations. Transformers and destinations read env.respond to delegate the HTTP response and context.ingest for source-extracted metadata. The scope ends when the body resolves.

Browser sources (and other single-scope sources) skip withScope. A browser tab is a single logical scope for its entire lifetime; calling env.push directly on the factory env is correct. Only server sources handling concurrent inbound work need withScope.

Minimal example​

import type { Source, Elb } from '@walkeros/core';

interface Settings {
  prefix?: string;
}

interface Env extends Source.BaseEnv {}

interface Types extends Source.Types<Settings, never, Elb.Fn, Env> {}

export const sourceCustom: Source.Init<Types> = async (context) => {
  const { config = {}, env } = context;

  const settings: Source.Settings<Types> = {
    prefix: 'custom',
    ...config?.settings,
  };

  const fullConfig: Source.Config<Types> = {
    ...config,
    settings,
  };

  return {
    type: 'custom',
    config: fullConfig,
    push: env.elb, // startFlow's elb when this source is primary
  };
};

Complete example: Event API source​

Capturing events from a third-party API with event listeners:

import type { Source, Elb, WalkerOS } from '@walkeros/core';

// Your external API interface
interface ExternalAPI {
  on(event: string, handler: (data: WalkerOS.Properties) => void): void;
  off(event: string, handler: (data: WalkerOS.Properties) => void): void;
}

interface Settings {
  captureInteractions?: boolean;
  captureErrors?: boolean;
  prefix?: string;
}

interface Env extends Source.BaseEnv {
  api?: ExternalAPI;
}

export interface Types extends Source.Types<Settings, never, Elb.Fn, Env> {}

export const sourceEventAPI: Source.Init<Types> = async (context) => {
  const { config = {}, env } = context;
  const { push: envPush, api } = env;

  if (!api) throw new Error('Source requires api instance');

  const settings: Source.Settings<Types> = {
    captureInteractions: true,
    captureErrors: true,
    prefix: 'app',
    ...config?.settings,
  };

  const handlers = new Map<string, (data: WalkerOS.Properties) => void>();

  // Register event handler
  const register = (
    event: string,
    transform: (data: WalkerOS.Properties) => WalkerOS.DeepPartialEvent,
  ) => {
    const handler = (data: WalkerOS.Properties) => {
      // The source type keeps events attributed to this source
      envPush({ ...transform(data), source: { type: 'event-api' } });
    };
    api.on(event, handler);
    handlers.set(event, handler);
  };

  // Set up event listeners based on settings
  if (settings.captureInteractions) {
    register('userAction', (data) => ({
      name: `${settings.prefix} interaction`,
      data,
    }));
  }

  if (settings.captureErrors) {
    register('error', (data) => ({
      name: `${settings.prefix} error`,
      data,
    }));
  }

  return {
    type: 'event-api',
    config: { ...config, settings },
    push: env.elb,
    destroy: async () => {
      handlers.forEach((handler, event) => api.off(event, handler));
    },
  };
};

Using your source​

import { startFlow } from '@walkeros/collector';
import { sourceEventAPI } from './sourceEventAPI';

const externalAPI = getYourAPI();

const { elb } = await startFlow({
  sources: {
    eventAPI: {
      code: sourceEventAPI,
      config: {
        settings: {
          captureInteractions: true,
          prefix: 'myapp',
        },
      },
      env: {
        api: externalAPI, // The collector adds push, elb, command and logger
      },
    },
  },
});

Testing your source​

Test Utilities​

// __tests__/test-utils.ts

export function createMockPush() {
  const mock = jest.fn();
  mock.mockResolvedValue({
    event: { id: 'test-id' },
    ok: true,
  });
  return mock;
}

export function createMockAPI() {
  return {
    on: jest.fn(),
    off: jest.fn(),
  };
}

Test Example​

import { sourceEventAPI, type Types } from '../index';
import type { Source, Collector } from '@walkeros/core';
import { createMockLogger } from '@walkeros/core';
import { createMockPush, createMockAPI } from './test-utils';

// Helper to create source context for testing
function createSourceContext(
  config: Partial<Source.Config<Types>> = {},
  env: Partial<Types['env']> = {},
): Source.Context<Types> {
  return {
    config,
    env: env as Types['env'],
    logger: env.logger || createMockLogger(),
    id: 'test-event-api',
    collector: {} as Collector.Instance,
    withScope: () => Promise.reject(new Error('Not used by this source')),
  };
}

describe('Event API Source', () => {
  it('validates required dependencies', async () => {
    const push = createMockPush();

    await expect(
      sourceEventAPI(createSourceContext({}, { push })),
    ).rejects.toThrow('requires api');
  });

  it('registers event listeners', async () => {
    const api = createMockAPI();
    const push = createMockPush();

    await sourceEventAPI(
      createSourceContext(
        { settings: { captureInteractions: true } },
        { push, api },
      ),
    );

    expect(api.on).toHaveBeenCalledWith('userAction', expect.any(Function));
  });

  it('transforms and forwards events', async () => {
    const api = createMockAPI();
    const push = createMockPush();

    await sourceEventAPI(createSourceContext({}, { push, api }));

    // Simulate event
    const handler = api.on.mock.calls[0][1];
    handler({ action: 'click' });

    expect(push).toHaveBeenCalledWith({
      name: 'app interaction',
      data: { action: 'click' },
      source: { type: 'event-api' },
    });
  });

  it('cleans up on destroy', async () => {
    const api = createMockAPI();
    const push = createMockPush();

    const context = createSourceContext({}, { push, api });
    const source = await sourceEventAPI(context);
    await source.destroy?.(context);

    expect(api.off).toHaveBeenCalled();
  });
});

Reading a step back from the collector​

When an integration test needs to call a step's raw push directly through the collector (a source's push(req, res), a destination's push(event, ctx), etc.), the bag types (collector.sources, collector.destinations, collector.transformers, collector.stores) erase the per-step generic. The push signature collapses to Elb.Fn on read.

Use the typed accessors from @walkeros/core to recover the narrow type without casts:

import { Source } from '@walkeros/core';
import type { Types as ExpressTypes } from '@walkeros/server-source-express';

const expressSource = Source.getSource<ExpressTypes>(collector, 'express');
await expressSource.push(req, res); // typed!

Symmetric helpers exist for every step kind: Source.getSource, Destination.getDestination, Transformer.getTransformer, Store.getStore. Each throws <Kind> not found: <id> when the id is unknown. The helpers are opt-in: they do nothing unless you call them.

Setup lifecycle (optional)​

Sources may implement an optional setup() lifecycle for one-time, operator-time provisioning, for example registering a webhook callback or creating a Pub/Sub subscription. Setup runs only when an operator explicitly invokes walkeros setup source.<name>; the runtime never auto-invokes it. Opt in via config.setup in the flow config (true or an object).

See Setup lifecycle on the destinations page for the full concept, and walkeros setup CLI command for the operator-facing command.

Conditional activation with require​

Sources can declare dependencies on collector events. A source with require won't receive lifecycle events (via on()) until all specified events have fired:

await startFlow({
  sources: {
    dataLayer: {
      code: sourceDataLayer,
      config: { require: ['consent'] },
    },
  },
});

Here the dataLayer source still captures new dataLayer.push() calls at once, from its init(). Only the replay of entries pushed before it started, which it runs on the run event, waits for the first consent command.

How it works:

  • Sources are registered in collector.sources immediately, and the collector runs Instance.init() on each one eagerly after registration. Source.Config.init flips to true once init has run.
  • require does not gate code execution or capture. It gates on() delivery. Lifecycle events targeted at a source whose require is unmet are buffered in the source's Instance.queueOn array.
  • After each collector event (consent, user, session, run, etc.), the collector decrements the matching entry from each source's require list. When a source's require is empty (and config.init === true), its queued lifecycle events are replayed by calling source.on(type, data) for each entry, then the queue is cleared.
  • A source without an on() method is not affected by require. The session source, for example, detects sessions from its init() and starts regardless of require.

Use require for a source whose work happens in on(): ['consent'] waits for the first consent command, ['user'] for identity, ['session'] for session detection, ['consent', 'run'] for both consent and the collector run.

Common patterns​

Polling for API Readiness​

When the external API isn't immediately available:

async function waitForAPI<T>(getter: () => T | null): Promise<T> {
  return new Promise((resolve) => {
    const check = () => {
      const instance = getter();
      if (instance) resolve(instance);
      else setTimeout(check, 100);
    };
    check();
  });
}

// Usage
const api = await waitForAPI(() => window.myAPI || null);

Source as adapter pattern​

Sources bridge external systems and the collector:

External System ←→ Source (Adapter) ←→ Collector

Two interfaces:

  1. External (source.push): Platform-specific signature

    • Event sources (browser, dataLayer, your own): Elb.Fn, events and walker commands → Returns Promise<PushResult>
    • Server: push(req, res) → Returns Promise<void> (writes HTTP response)
    • Your choice: Match your environment's needs
  2. Internal (env.push): The source pipeline

    • Always Collector.PushFn - same across all sources
    • Sources translate external inputs → standard events → env.push(event)

Example signatures:

// HTTP handler source
export type Push = (req: Request, res: Response) => Promise<void>;

// Event source
export type Push = Elb.Fn;

The Push type parameter defines what your source exposes externally. Internally, every source sends its events through env.push.

Key concepts​

  • Context pattern: Sources receive a single context object with config, env, logger, id
  • Validate custom dependencies: Only check optional env properties your source needs
  • Types bundle: Use 4-parameter pattern for full type safety
  • Adapter pattern: External push adapts to environment, internal env.push stays standard
  • Cleanup: Implement destroy() to remove listeners
  • Stateless: Let collector manage state

Package convention​

Every walkerOS package includes machine-readable metadata for tooling and discovery.

walkerOS field in package.json​

{
  "walkerOS": { "type": "source", "platform": "web" },
  "keywords": ["walkerOS", "walkerOS-source"]
}
FieldRequiredDescription
walkerOSYesObject with type and platform metadata

Build-time generation​

Use buildDev() from the shared tsup config to auto-generate walkerOS.json:

import { buildDev } from '@walkeros/config/tsup';

This file contains your package's JSON Schemas and examples, enabling MCP tools and the CLI to validate configurations without installing your package.

buildDev() fails the build when the package publishes no schemas.settings. A package with two or more walkerOS.exports also exports exportExamples and exportSchemas from src/dev.ts, each listing exactly those exports, the default included, with settings in every exportSchemas entry; a package with fewer exports sets neither.

Optional: Hints​

Packages can export a hints record from src/dev.ts to provide context beyond schemas and examples, such as capture timing, event formats or troubleshooting tips. Hints are serialized into walkerOS.json and surfaced via MCP tools. See the walkeros-create-source skill for details.

Publishing checklist​

  • walkerOS field in package.json
  • Keywords include walkerOS and walkerOS-source
  • buildDev() in tsup.config.ts
  • schemas.settings exported from src/dev.ts (per export for multi-export packages)
  • dist/walkerOS.json generated on build
  • npm run test passes
  • npm run lint passes

Next steps​

💡 Need implementation support?
elbwalker offers hands-on support: setup review, measurement planning, destination mapping, and live troubleshooting. Book a 2-hour session (€399)