Create Your Own Source
A source captures events from an environment (browser, server, third-party API) and forwards them to the walkerOS collector.
The source interface
Sources are async functions that receive a context object and return a source instance:
type Source.Init<Types> = (
context: Source.Context<Types>,
) => Source.Instance<Types> | Promise<Source.Instance<Types>>;The context contains:
interface Source.Context<Types> {
config: Partial<Source.Config<Types>>; // Settings, mapping, options
env: Types['env']; // Environment (push, logger)
logger: Logger; // Logging functions
id: string; // Source identifier
collector: Collector.Instance; // Collector reference
withScope: Source.Context['withScope']; // Per-request scope (server sources)
}The returned instance must implement:
interface Source.Instance {
type: string; // Unique identifier
config: Source.Config; // Merged configuration
push: Elb.Fn; // Outward entry, usually env.elb
init?(): void | Promise<void>; // Optional setup, runs once every source is registered
destroy?(context): void | Promise<void>; // Optional cleanup
on?(event, context): void | Promise<void>; // Optional reactive hooks
}Events the source captures go through env.push, the source pipeline: its
mapping, its before and next chains and its entry in
collector.status.sources. env.elb is the collector's public interface for
events and walker commands, and it skips the source pipeline. When the source
is primary, startFlow returns its push as elb, so set push to env.elb
unless the source exposes its own signature, such as an HTTP handler. With
env.push in that slot, a walker command answers { ok: false, invalid: true }.
Types bundle
Sources use a Types interface to bundle all TypeScript types:
import type { Source, Elb } from '@walkeros/core';
// 1. Define your settings
interface Settings {
captureClicks?: boolean;
prefix?: string;
}
// 2. Define environment dependencies
interface Env extends Source.BaseEnv {
customAPI?: YourAPIType;
}
// 3. Bundle them together
interface Types extends Source.Types<Settings, never, Elb.Fn, Env> {}The 4 type parameters:
- Settings: Source configuration options
- Mapping: Event mapping (usually
neverfor sources) - Push: External push signature (what
source.pushexposes) - Env: Internal dependencies the source reads from
env
Context destructuring
The context parameter contains everything your source needs:
export const mySource: Source.Init<Types> = async (context) => {
// Destructure what you need from context
const { config = {}, env, logger, id } = context;
const { push: envPush, customAPI } = env;
// customAPI validation only needed if required by your source
if (!customAPI) {
throw new Error('Source requires customAPI in environment');
}
// Use env.push to send events to collector
await envPush({ name: 'my event', data: { value: 'example' } });
// ...then return the source instance (see the minimal example below)
};env is the author's dependency bag: you provide platform and vendor dependencies (like window, document, custom APIs) when configuring the source. The collector provides five capabilities into that bag, and they always win: push, elb, command, logger and sources are runtime-owned, so setting one of them in env has no effect.
A factory that throws, for example over a missing dependency, is logged at ERROR (source factory failed) and the source is not registered.
Ending the pipeline elsewhere: terminus
A source registration that needs the pipeline to end somewhere other than the collector declares terminus next to code. A terminus receives the event exactly as the source emitted it, and the entire pipeline is skipped: no before/next chains, no source cache, no state, no mapping (including ignore and consent), no minted span id, no per-source ingest, no respond, no status.sources counting and no observability records. The bypass is total by design, with no stage excepted. It exists for deterministic capture at the source→collector boundary (step examples, dev tooling); production flows leave it unset, and stored flow configs cannot carry it (validation rejects unknown source keys).
Per-scope context (server sources)
A single source factory instance handles many concurrent invocations: an Express server processes overlapping requests, a Lambda may be reused across invocations, a queue consumer processes batches. Each logical unit of work, an HTTP request, a queue message, a websocket frame, is a scope. To keep ingest metadata and response delegates isolated between concurrent scopes, server sources wrap each invocation with context.withScope:
const push = async (req: Request, res: Response): Promise<void> => {
const respond = createRespond((options) => {
// wire options into res.send / res.json
});
// Normalize the platform request into the shared scope shape first, so
// `config.ingest` paths resolve the same here as on every other source.
const scope = buildScopeFromNodeRequest(req, {
body: req.body,
defaultProtocol: 'http',
});
// Each call to withScope builds a fresh Ingest from the scope via
// `config.ingest` mapping, captures `respond`, and runs the body
// with a per-scope env. Concurrent scopes never crosstalk.
await context.withScope(scope, respond, async (env) => {
// toEventList decides how many events the body carries; each push
// result becomes the outcome the caller is answered with.
const outcomes: EventOutcome[] = [];
for (const event of toEventList(scope.body))
outcomes.push(pushResultToOutcome(await env.push(event)));
const { status, body } = batchResponse(outcomes);
respond({ status, body });
});
};A new server source must build a scope and use toEventList, so config.ingest paths and accepted body forms match every other source. pushResultToOutcome turns each push result into an outcome: an error with status 400 for an invalid event, an error with status 500 when a resend could fix the failure, and the event id when it was delivered or failed for a reason a resend cannot fix (still visible in the destination's failed counter and an ERROR line). batchResponse answers 207 with per-index errors when any outcome is an error and 200 otherwise, so in the snippet above a single event gets the batch answer too; a source that answers single-event requests itself uses the outcome's status. See producer acknowledgement. buildScopeFromNodeRequest, normalizeHeaders, normalizeQuery, normalizeBody, toEventList, isBatchBody, pushResultToOutcome and batchResponse all come from @walkeros/core.
Inside the body, env.push carries the scope's ingest and respond all the way to destinations. Transformers and destinations read env.respond to delegate the HTTP response and context.ingest for source-extracted metadata. The scope ends when the body resolves.
Browser sources (and other single-scope sources) skip withScope. A browser tab is a single logical scope for its entire lifetime; calling env.push directly on the factory env is correct. Only server sources handling concurrent inbound work need withScope.
Minimal example
import type { Source, Elb } from '@walkeros/core';
interface Settings {
prefix?: string;
}
interface Env extends Source.BaseEnv {}
interface Types extends Source.Types<Settings, never, Elb.Fn, Env> {}
export const sourceCustom: Source.Init<Types> = async (context) => {
const { config = {}, env } = context;
const settings: Source.Settings<Types> = {
prefix: 'custom',
...config?.settings,
};
const fullConfig: Source.Config<Types> = {
...config,
settings,
};
return {
type: 'custom',
config: fullConfig,
push: env.elb, // startFlow's elb when this source is primary
};
};Complete example: Event API source
Capturing events from a third-party API with event listeners:
import type { Source, Elb, WalkerOS } from '@walkeros/core';
// Your external API interface
interface ExternalAPI {
on(event: string, handler: (data: WalkerOS.Properties) => void): void;
off(event: string, handler: (data: WalkerOS.Properties) => void): void;
}
interface Settings {
captureInteractions?: boolean;
captureErrors?: boolean;
prefix?: string;
}
interface Env extends Source.BaseEnv {
api?: ExternalAPI;
}
export interface Types extends Source.Types<Settings, never, Elb.Fn, Env> {}
export const sourceEventAPI: Source.Init<Types> = async (context) => {
const { config = {}, env } = context;
const { push: envPush, api } = env;
if (!api) throw new Error('Source requires api instance');
const settings: Source.Settings<Types> = {
captureInteractions: true,
captureErrors: true,
prefix: 'app',
...config?.settings,
};
const handlers = new Map<string, (data: WalkerOS.Properties) => void>();
// Register event handler
const register = (
event: string,
transform: (data: WalkerOS.Properties) => WalkerOS.DeepPartialEvent,
) => {
const handler = (data: WalkerOS.Properties) => {
// The source type keeps events attributed to this source
envPush({ ...transform(data), source: { type: 'event-api' } });
};
api.on(event, handler);
handlers.set(event, handler);
};
// Set up event listeners based on settings
if (settings.captureInteractions) {
register('userAction', (data) => ({
name: `${settings.prefix} interaction`,
data,
}));
}
if (settings.captureErrors) {
register('error', (data) => ({
name: `${settings.prefix} error`,
data,
}));
}
return {
type: 'event-api',
config: { ...config, settings },
push: env.elb,
destroy: async () => {
handlers.forEach((handler, event) => api.off(event, handler));
},
};
};Using your source
import { startFlow } from '@walkeros/collector';
import { sourceEventAPI } from './sourceEventAPI';
const externalAPI = getYourAPI();
const { elb } = await startFlow({
sources: {
eventAPI: {
code: sourceEventAPI,
config: {
settings: {
captureInteractions: true,
prefix: 'myapp',
},
},
env: {
api: externalAPI, // The collector adds push, elb, command and logger
},
},
},
});Testing your source
Test Utilities
// __tests__/test-utils.ts
export function createMockPush() {
const mock = jest.fn();
mock.mockResolvedValue({
event: { id: 'test-id' },
ok: true,
});
return mock;
}
export function createMockAPI() {
return {
on: jest.fn(),
off: jest.fn(),
};
}Test Example
import { sourceEventAPI, type Types } from '../index';
import type { Source, Collector } from '@walkeros/core';
import { createMockLogger } from '@walkeros/core';
import { createMockPush, createMockAPI } from './test-utils';
// Helper to create source context for testing
function createSourceContext(
config: Partial<Source.Config<Types>> = {},
env: Partial<Types['env']> = {},
): Source.Context<Types> {
return {
config,
env: env as Types['env'],
logger: env.logger || createMockLogger(),
id: 'test-event-api',
collector: {} as Collector.Instance,
withScope: () => Promise.reject(new Error('Not used by this source')),
};
}
describe('Event API Source', () => {
it('validates required dependencies', async () => {
const push = createMockPush();
await expect(
sourceEventAPI(createSourceContext({}, { push })),
).rejects.toThrow('requires api');
});
it('registers event listeners', async () => {
const api = createMockAPI();
const push = createMockPush();
await sourceEventAPI(
createSourceContext(
{ settings: { captureInteractions: true } },
{ push, api },
),
);
expect(api.on).toHaveBeenCalledWith('userAction', expect.any(Function));
});
it('transforms and forwards events', async () => {
const api = createMockAPI();
const push = createMockPush();
await sourceEventAPI(createSourceContext({}, { push, api }));
// Simulate event
const handler = api.on.mock.calls[0][1];
handler({ action: 'click' });
expect(push).toHaveBeenCalledWith({
name: 'app interaction',
data: { action: 'click' },
source: { type: 'event-api' },
});
});
it('cleans up on destroy', async () => {
const api = createMockAPI();
const push = createMockPush();
const context = createSourceContext({}, { push, api });
const source = await sourceEventAPI(context);
await source.destroy?.(context);
expect(api.off).toHaveBeenCalled();
});
});Reading a step back from the collector
When an integration test needs to call a step's raw push directly through the collector (a source's push(req, res), a destination's push(event, ctx), etc.), the bag types (collector.sources, collector.destinations, collector.transformers, collector.stores) erase the per-step generic. The push signature collapses to Elb.Fn on read.
Use the typed accessors from @walkeros/core to recover the narrow type without casts:
import { Source } from '@walkeros/core';
import type { Types as ExpressTypes } from '@walkeros/server-source-express';
const expressSource = Source.getSource<ExpressTypes>(collector, 'express');
await expressSource.push(req, res); // typed!Symmetric helpers exist for every step kind: Source.getSource, Destination.getDestination, Transformer.getTransformer, Store.getStore. Each throws <Kind> not found: <id> when the id is unknown. The helpers are opt-in: they do nothing unless you call them.
Setup lifecycle (optional)
Sources may implement an optional setup() lifecycle for one-time, operator-time provisioning, for example registering a webhook callback or creating a Pub/Sub subscription. Setup runs only when an operator explicitly invokes walkeros setup source.<name>; the runtime never auto-invokes it. Opt in via config.setup in the flow config (true or an object).
See Setup lifecycle on the destinations page for the full concept, and walkeros setup CLI command for the operator-facing command.
Conditional activation with require
Sources can declare dependencies on collector events. A source with require won't receive lifecycle events (via on()) until all specified events have fired:
await startFlow({
sources: {
dataLayer: {
code: sourceDataLayer,
config: { require: ['consent'] },
},
},
});Here the dataLayer source still captures new dataLayer.push() calls at once, from its init(). Only the replay of entries pushed before it started, which it runs on the run event, waits for the first consent command.
How it works:
- Sources are registered in
collector.sourcesimmediately, and the collector runsInstance.init()on each one eagerly after registration.Source.Config.initflips totrueonce init has run. requiredoes not gate code execution or capture. It gateson()delivery. Lifecycle events targeted at a source whoserequireis unmet are buffered in the source'sInstance.queueOnarray.- After each collector event (
consent,user,session,run, etc.), the collector decrements the matching entry from each source'srequirelist. When a source's require is empty (andconfig.init === true), its queued lifecycle events are replayed by callingsource.on(type, data)for each entry, then the queue is cleared. - A source without an
on()method is not affected byrequire. The session source, for example, detects sessions from itsinit()and starts regardless ofrequire.
Use require for a source whose work happens in on(): ['consent'] waits for the first consent command, ['user'] for identity, ['session'] for session detection, ['consent', 'run'] for both consent and the collector run.
Common patterns
Polling for API Readiness
When the external API isn't immediately available:
async function waitForAPI<T>(getter: () => T | null): Promise<T> {
return new Promise((resolve) => {
const check = () => {
const instance = getter();
if (instance) resolve(instance);
else setTimeout(check, 100);
};
check();
});
}
// Usage
const api = await waitForAPI(() => window.myAPI || null);Source as adapter pattern
Sources bridge external systems and the collector:
External System ←→ Source (Adapter) ←→ Collector
Two interfaces:
-
External (
source.push): Platform-specific signature- Event sources (browser, dataLayer, your own):
Elb.Fn, events andwalkercommands → ReturnsPromise<PushResult> - Server:
push(req, res)→ ReturnsPromise<void>(writes HTTP response) - Your choice: Match your environment's needs
- Event sources (browser, dataLayer, your own):
-
Internal (
env.push): The source pipeline- Always
Collector.PushFn- same across all sources - Sources translate external inputs → standard events →
env.push(event)
- Always
Example signatures:
// HTTP handler source
export type Push = (req: Request, res: Response) => Promise<void>;
// Event source
export type Push = Elb.Fn;The Push type parameter defines what your source exposes externally. Internally, every source sends its events through env.push.
Key concepts
- Context pattern: Sources receive a single context object with config, env, logger, id
- Validate custom dependencies: Only check optional env properties your source needs
- Types bundle: Use 4-parameter pattern for full type safety
- Adapter pattern: External push adapts to environment, internal env.push stays standard
- Cleanup: Implement
destroy()to remove listeners - Stateless: Let collector manage state
Package convention
Every walkerOS package includes machine-readable metadata for tooling and discovery.
walkerOS field in package.json
{
"walkerOS": { "type": "source", "platform": "web" },
"keywords": ["walkerOS", "walkerOS-source"]
}| Field | Required | Description |
|---|---|---|
walkerOS | Yes | Object with type and platform metadata |
Build-time generation
Use buildDev() from the shared tsup config to auto-generate walkerOS.json:
import { buildDev } from '@walkeros/config/tsup';This file contains your package's JSON Schemas and examples, enabling MCP tools and the CLI to validate configurations without installing your package.
buildDev() fails the build when the package publishes no schemas.settings. A package with two or more walkerOS.exports also exports exportExamples and exportSchemas from src/dev.ts, each listing exactly those exports, the default included, with settings in every exportSchemas entry; a package with fewer exports sets neither.
Optional: Hints
Packages can export a hints record from src/dev.ts to provide context beyond schemas and examples, such as capture timing, event formats or troubleshooting tips. Hints are serialized into walkerOS.json and surfaced via MCP tools. See the walkeros-create-source skill for details.
Publishing checklist
-
walkerOSfield in package.json - Keywords include
walkerOSandwalkerOS-source -
buildDev()in tsup.config.ts -
schemas.settingsexported fromsrc/dev.ts(per export for multi-export packages) -
dist/walkerOS.jsongenerated on build -
npm run testpasses -
npm run lintpasses
Next steps
- Review Browser Source for DOM patterns
- Review DataLayer Source for interception patterns
- Learn about creating destinations