Apache Kafka
Server-side event streaming to Apache Kafka via the kafkajs client. Each event is serialized as JSON, keyed for partition-friendly ordering, and produced to a configurable topic. It supports TLS and SASL authentication (PLAIN, SCRAM), gzip compression, per-rule topic and key overrides, and graceful shutdown via destroy().
Kafka is a server destination in the walkerOS flow:
Receives events server-side from the collector, serializes them as JSON with a partition-friendly message key, and produces them to a Kafka topic for downstream stream processing (Flink, Spark, ksqlDB, Kafka Connect, consumers).
Installation
npm install @walkeros/collector @walkeros/server-destination-kafka- Integrated
- Bundled
import { startFlow } from '@walkeros/collector';
import destinationKafka from '@walkeros/server-destination-kafka';
await startFlow({
destinations: {
kafka: {
code: destinationKafka,
config: {
settings: {
kafka: {
brokers: [
'localhost:9092',
],
topic: 'walkeros-events',
},
},
},
},
},
});
A complete flow.json with this destination:
{
"version": 4,
"flows": {
"default": {
"config": {
"platform": "server"
},
"destinations": {
"kafka": {
"package": "@walkeros/server-destination-kafka",
"config": {
"settings": {
"kafka": {
"brokers": [
"localhost:9092"
],
"topic": "walkeros-events"
}
}
}
}
}
}
}
}Configuration
This destination uses the standard destination config wrapper (consent, data, env, id, ...). For the shared fields see destination configuration. Package-specific fields live under config.settings and are listed below.
Settings
| Property | Type | Description | More |
|---|---|---|---|
kafka | kafka | Kafka connection and producer settings. | |
brokers | Array<string> | Kafka broker addresses (host:port). At least one required. | |
clientId | string | Kafka client ID. Default: walkeros. | |
ssl | boolean | object | TLS configuration. Set true for default TLS, or provide a tls.ConnectionOptions object for mTLS. | |
sasl | sasl | SASL authentication config (like mechanism plain for Confluent Cloud). | |
mechanism | SASL authentication mechanism. oauthbearer needs a token provider function, which these settings cannot hold, so it does not authenticate. | ||
username | string | Username for plain/scram mechanisms. | |
password | string | Password for plain/scram mechanisms. | |
accessKeyId | string | AWS access key ID for the kafkajs mechanism aws. | |
secretAccessKey | string | AWS secret access key for the kafkajs mechanism aws. | |
sessionToken | string | AWS session token for temporary credentials (mechanism: aws). | |
authorizationIdentity | string | AWS authorization identity, required by kafkajs for mechanism aws. | |
connectionTimeout | integer | Connection timeout in ms. Default: 1000. | |
requestTimeout | integer | Request timeout in ms. Default: 30000. | |
topic | string | Target Kafka topic name. | |
acks | integer | Acknowledgement level. -1 = all replicas, 0 = fire-and-forget, 1 = leader only. Default: -1. | |
timeout | integer | Broker response timeout in ms. Default: 30000. | |
compression | Message compression codec. Default: gzip. snappy, lz4 and zstd need a codec registered in kafkajs CompressionCodecs in the same process; without one every send fails. | ||
idempotent | boolean | Enable idempotent producer for exactly-once delivery. Default: false. | |
allowAutoTopicCreation | boolean | Allow auto-creation of topics on the broker. Default: false. | |
key | string | Mapping value path for message key derivation (e.g. user.id, data.userId). Default: entity_action. | |
headers | Record<string, string> | Static headers added to every message. | |
retry | retry | Retry configuration for transient failures. | |
maxRetryTime | integer | Max total retry wait in ms. Default: 30000. | |
initialRetryTime | integer | First retry delay in ms. Default: 300. | |
retries | integer | Max retry count. Default: 5. |
Mapping
Per-event rules under config.mapping. For the standard rule fields (consent, condition, data, batch, name, policy) see mapping.
| Property | Type | Description | More |
|---|---|---|---|
key | string | Override message key mapping path for this rule (e.g. data.id). Takes precedence over settings.kafka.key. | |
topic | string | Override Kafka topic for this rule. Takes precedence over settings.kafka.topic. |
Examples
Default event
An event is produced to the configured Kafka topic with the full JSON body and entity_action as the message key.
{
"name": "page view",
"data": {
"domain": "www.example.com",
"title": "walkerOS documentation",
"referrer": "https://www.walkeros.io/",
"search": "?foo=bar",
"hash": "#hash",
"id": "/docs/"
},
"context": {
"dev": [
"test",
1
]
},
"globals": {
"pagegroup": "docs"
},
"custom": {
"completely": "random"
},
"user": {
"id": "us3r",
"device": "c00k13",
"session": "s3ss10n"
},
"nested": [
{
"entity": "child",
"data": {
"is": "subordinated"
}
}
],
"consent": {
"functional": true
},
"id": "f0cc5f6fec3d8eeb",
"trigger": "load",
"entity": "page",
"action": "view",
"timestamp": 1700000100000,
"timing": 3.14,
"source": {
"count": 1,
"trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9",
"type": "collector",
"schema": "4"
}
}producer.send({
"topic": "walkeros-events",
"messages": [
{
"key": "page_view",
"value": "json:event",
"headers": {
"content-type": "application/json"
},
"timestamp": "1700000100000"
}
],
"acks": -1,
"compression": 1
})Key from user id
A settings.kafka.key path resolves the message key from the event, here using user.id for per-user partitioning.
{
"name": "user signup",
"data": {
"plan": "pro"
},
"context": {
"dev": [
"test",
1
]
},
"globals": {
"lang": "elb"
},
"custom": {
"completely": "random"
},
"user": {
"id": "usr-789"
},
"nested": [
{
"entity": "child",
"data": {
"is": "subordinated"
}
}
],
"consent": {
"functional": true
},
"id": "f66dbc3bf5e573df",
"trigger": "test",
"entity": "user",
"action": "signup",
"timestamp": 1700000103000,
"timing": 3.14,
"source": {
"count": 1,
"trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9",
"type": "collector",
"schema": "4"
}
}producer.send({
"topic": "walkeros-events",
"messages": [
{
"key": "usr-789",
"value": "json:event",
"headers": {
"content-type": "application/json"
},
"timestamp": "1700000103000"
}
],
"acks": -1,
"compression": 1
})Mapped payload
A data mapping transforms the event payload before producing it as the Kafka message value.
{
"name": "order complete",
"data": {
"id": "ORD-400",
"value": 99.99,
"currency": "EUR"
},
"context": {
"shopping": [
"complete",
0
]
},
"globals": {
"pagegroup": "shop"
},
"custom": {
"completely": "random"
},
"user": {
"id": "us3r",
"device": "c00k13",
"session": "s3ss10n"
},
"nested": [
{
"entity": "product",
"data": {
"id": "ers",
"name": "Everyday Ruck Snack",
"color": "black",
"size": "l",
"price": 420
},
"context": {
"shopping": [
"complete",
0
]
},
"nested": []
},
{
"entity": "product",
"data": {
"id": "cc",
"name": "Cool Cap",
"size": "one size",
"price": 42
},
"context": {
"shopping": [
"complete",
0
]
},
"nested": []
},
{
"entity": "gift",
"data": {
"name": "Surprise"
},
"context": {
"shopping": [
"complete",
0
]
},
"nested": []
}
],
"consent": {
"functional": true
},
"id": "235d0d7bd7a213f2",
"trigger": "load",
"entity": "order",
"action": "complete",
"timestamp": 1700000102000,
"timing": 3.14,
"source": {
"count": 1,
"trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9",
"type": "collector",
"schema": "4"
}
}{
"name": "purchase",
"data": {
"map": {
"order_id": "data.id",
"revenue": "data.value",
"currency": "data.currency"
}
}
}producer.send({
"topic": "walkeros-events",
"messages": [
{
"key": "purchase",
"value": "json:data",
"headers": {
"content-type": "application/json"
},
"timestamp": "1700000102000"
}
],
"acks": -1,
"compression": 1
})Renamed event
A mapping renames the event which also changes the default Kafka message key used for partitioning.
{
"name": "order complete",
"data": {
"id": "0rd3r1d",
"currency": "EUR",
"value": 462,
"tax": 73.76,
"shipping": 5.22,
"total": 467.22
},
"context": {
"shopping": [
"complete",
0
]
},
"globals": {
"pagegroup": "shop"
},
"custom": {
"completely": "random"
},
"user": {
"id": "us3r",
"device": "c00k13",
"session": "s3ss10n"
},
"nested": [
{
"entity": "product",
"data": {
"id": "ers",
"name": "Everyday Ruck Snack",
"color": "black",
"size": "l",
"price": 420
},
"context": {
"shopping": [
"complete",
0
]
},
"nested": []
},
{
"entity": "product",
"data": {
"id": "cc",
"name": "Cool Cap",
"size": "one size",
"price": 42
},
"context": {
"shopping": [
"complete",
0
]
},
"nested": []
},
{
"entity": "gift",
"data": {
"name": "Surprise"
},
"context": {
"shopping": [
"complete",
0
]
},
"nested": []
}
],
"consent": {
"functional": true
},
"id": "235d0d7bd7a213f2",
"trigger": "load",
"entity": "order",
"action": "complete",
"timestamp": 1700000101000,
"timing": 3.14,
"source": {
"count": 1,
"trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9",
"type": "collector",
"schema": "4"
}
}{
"name": "purchase"
}producer.send({
"topic": "walkeros-events",
"messages": [
{
"key": "purchase",
"value": "json:event",
"headers": {
"content-type": "application/json"
},
"timestamp": "1700000101000"
}
],
"acks": -1,
"compression": 1
})Topic override
A mapping rule overrides the destination topic so specific events are routed to a dedicated stream.
{
"name": "order complete",
"data": {
"id": "ORD-500",
"value": 42
},
"context": {
"shopping": [
"complete",
0
]
},
"globals": {
"pagegroup": "shop"
},
"custom": {
"completely": "random"
},
"user": {
"id": "us3r",
"device": "c00k13",
"session": "s3ss10n"
},
"nested": [
{
"entity": "product",
"data": {
"id": "ers",
"name": "Everyday Ruck Snack",
"color": "black",
"size": "l",
"price": 420
},
"context": {
"shopping": [
"complete",
0
]
},
"nested": []
},
{
"entity": "product",
"data": {
"id": "cc",
"name": "Cool Cap",
"size": "one size",
"price": 42
},
"context": {
"shopping": [
"complete",
0
]
},
"nested": []
},
{
"entity": "gift",
"data": {
"name": "Surprise"
},
"context": {
"shopping": [
"complete",
0
]
},
"nested": []
}
],
"consent": {
"functional": true
},
"id": "235d0d7bd7a213f2",
"trigger": "load",
"entity": "order",
"action": "complete",
"timestamp": 1700000104000,
"timing": 3.14,
"source": {
"count": 1,
"trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9",
"type": "collector",
"schema": "4"
}
}{
"settings": {
"topic": "orders-stream"
}
}producer.send({
"topic": "orders-stream",
"messages": [
{
"key": "order_complete",
"value": "json:event",
"headers": {
"content-type": "application/json"
},
"timestamp": "1700000104000"
}
],
"acks": -1,
"compression": 1
})The destination creates a single long-lived kafkajs producer and connects it at the first push, so a flow starts without a reachable broker. kafkajs retries a failed connect or send as settings.kafka.retry allows. When a send still fails, the event fails with a transport error, marked retryable unless no retry can fix it (such as MESSAGE_TOO_LARGE), and goes to the dead-letter queue (when delivery fails). A topic that does not exist fails every event (below). On flow hot-swap or server shutdown, destroy() calls producer.disconnect() to close the connections.
Message format
Events are serialized as JSON and produced with the following structure:
- topic, from
settings.kafka.topic(ormapping.settings.topicoverride) - key, resolved from
settings.kafka.key(ormapping.settings.keyoverride) mapping path; defaults to the event name with spaces replaced by_(e.g.page_view,order_complete) for partition-based ordering. A key path whose value is not a non-empty string (a number, for example) falls back to that default - value, the mapped
datawhen adatamapping resolves to an object with at least one field, otherwise the full walkerOS event,JSON.stringify()-ed - headers,
content-type: application/jsonplus any staticsettings.kafka.headers - timestamp, event timestamp as string (ms since epoch)
Use mapping.settings.topic to route specific events to dedicated topics (e.g. orders to orders-stream, identities to identity-stream). Use mapping.settings.key to set a key path per rule (e.g. data.order_id for order events).
settings.kafka.compression defaults to gzip, which kafkajs ships. snappy, lz4 and zstd need a codec registered in kafkajs's CompressionCodecs in the same process. The destination registers none, so register the codec in your own code before the first send (integrated mode). Without one, every send fails with a transport error.
Setup
This destination's setup requires explicit numPartitions and replicationFactor.
There is no universally correct default for either: replicationFactor must be
less than or equal to broker count, and numPartitions is a function of
expected throughput and consumer parallelism. The boolean form setup: true is
rejected with an error listing the required fields. Only the object form is
valid.
Provision a topic once per environment with the CLI:
walkeros setup destination.kafkaThe command prints setup: ok destination.kafka; with --json it prints a JSON
result with { topicCreated, schemaRegistered }. The command is idempotent and
safe to re-run. Drift on numPartitions, replicationFactor, or
configEntries is logged as a setup.drift warning with the field, the
declared and the actual value, and never auto-mutates the broker.
config.setup:
- omitted or
false(default): no provisioning; the command printssetup: skipped destination.kafka (config.setup is unset)(oris false). true: the command fails with an error that names the required fields. There are no safe defaults for partition count or replication factor.{ numPartitions, replicationFactor, ... }: object form is the only valid form. See theSetupinterface in the package for full options.
Required fields
| Field | Type | Notes |
|---|---|---|
numPartitions | number | Required at runtime. No safe default. |
replicationFactor | number | Required at runtime. Must be <= broker count. |
Optional fields
| Field | Type | Notes |
|---|---|---|
topic | string | Falls back to settings.kafka.topic when omitted. |
configEntries | object | Topic-level config, e.g. { "retention.ms": "604800000" }. |
schemaRegistry | object | Confluent Schema Registry binding (see below). |
validateOnly | boolean | kafkajs broker-side dry-run. No topic is created. |
Example
{
"destinations": {
"kafka": {
"package": "@walkeros/server-destination-kafka",
"config": {
"settings": {
"kafka": { "brokers": ["broker:9092"], "topic": "walkeros-events" }
},
"setup": {
"numPartitions": 6,
"replicationFactor": 3,
"configEntries": { "retention.ms": "604800000" }
}
}
}
}
}Schema Registry (optional)
{
"setup": {
"numPartitions": 6,
"replicationFactor": 3,
"schemaRegistry": {
"url": "https://schema-registry.example.com",
"subject": "walkeros-events-value",
"schemaType": "JSON",
"schema": "{ \"type\": \"object\" }",
"compatibility": "BACKWARD"
}
}
}The schema is registered via the Confluent Schema Registry REST API. The
optional compatibility level is set on the subject after registration.
Runtime error when the topic is missing
When setup was not run and the topic does not exist on the cluster, every
event fails with Kafka push failed, a retryable transport error, once the
kafkajs retries are used up. kafkajs hands over the broker's
UNKNOWN_TOPIC_OR_PARTITION inside its retry error, so in the event's
dead-letter entry it is the cause of the error's cause. Run setup with explicit
numPartitions and replicationFactor to provision the topic.
Authentication
Confluent Cloud (SASL/PLAIN)
{
"kafka": {
"brokers": ["pkc-xxxxx.us-east-1.aws.confluent.cloud:9092"],
"topic": "walkeros-events",
"ssl": true,
"sasl": {
"mechanism": "plain",
"username": "$secret.CONFLUENT_API_KEY",
"password": "$secret.CONFLUENT_API_SECRET"
}
}
}SASL/SCRAM
{
"kafka": {
"brokers": ["broker.example.com:9096"],
"topic": "walkeros-events",
"ssl": true,
"sasl": {
"mechanism": "scram-sha-512",
"username": "$secret.KAFKA_USERNAME",
"password": "$secret.KAFKA_PASSWORD"
}
}
}mechanism: "aws" is kafkajs's own AWS mechanism: it needs
authorizationIdentity next to accessKeyId and secretAccessKey, and a
broker that offers the AWS SASL mechanism. oauthbearer needs a token
provider function, which these settings cannot hold, so it does not
authenticate.