Skip to main content
Ask your AI

Apache Kafka

Server Source code Package Beta

Server-side event streaming to Apache Kafka via the kafkajs client. Each event is serialized as JSON, keyed for partition-friendly ordering, and produced to a configurable topic. It supports TLS and SASL authentication (PLAIN, SCRAM), gzip compression, per-rule topic and key overrides, and graceful shutdown via destroy().

Where this fits

Kafka is a server destination in the walkerOS flow:

Receives events server-side from the collector, serializes them as JSON with a partition-friendly message key, and produces them to a Kafka topic for downstream stream processing (Flink, Spark, ksqlDB, Kafka Connect, consumers).

Installation​

npm install @walkeros/collector @walkeros/server-destination-kafka
import { startFlow } from '@walkeros/collector';
import destinationKafka from '@walkeros/server-destination-kafka';

await startFlow({
  destinations: {
    kafka: {
      code: destinationKafka,
      config: {
        settings: {
          kafka: {
            brokers: [
              'localhost:9092',
            ],
            topic: 'walkeros-events',
          },
        },
      },
    },
  },
});

Configuration​

This destination uses the standard destination config wrapper (consent, data, env, id, ...). For the shared fields see destination configuration. Package-specific fields live under config.settings and are listed below.

Settings​

PropertyTypeDescriptionMore
kafka*kafkaKafka connection and producer settings.
brokers*Array<string>Kafka broker addresses (host:port). At least one required.
clientIdstringKafka client ID. Default: walkeros.
sslboolean | objectTLS configuration. Set true for default TLS, or provide a tls.ConnectionOptions object for mTLS.
saslsaslSASL authentication config (like mechanism plain for Confluent Cloud).
mechanism*SASL authentication mechanism. oauthbearer needs a token provider function, which these settings cannot hold, so it does not authenticate.
usernamestringUsername for plain/scram mechanisms.
passwordstringPassword for plain/scram mechanisms.
accessKeyIdstringAWS access key ID for the kafkajs mechanism aws.
secretAccessKeystringAWS secret access key for the kafkajs mechanism aws.
sessionTokenstringAWS session token for temporary credentials (mechanism: aws).
authorizationIdentitystringAWS authorization identity, required by kafkajs for mechanism aws.
connectionTimeoutintegerConnection timeout in ms. Default: 1000.
requestTimeoutintegerRequest timeout in ms. Default: 30000.
topic*stringTarget Kafka topic name.
acksintegerAcknowledgement level. -1 = all replicas, 0 = fire-and-forget, 1 = leader only. Default: -1.
timeoutintegerBroker response timeout in ms. Default: 30000.
compressionMessage compression codec. Default: gzip. snappy, lz4 and zstd need a codec registered in kafkajs CompressionCodecs in the same process; without one every send fails.
idempotentbooleanEnable idempotent producer for exactly-once delivery. Default: false.
allowAutoTopicCreationbooleanAllow auto-creation of topics on the broker. Default: false.
keystringMapping value path for message key derivation (e.g. user.id, data.userId). Default: entity_action.
headersRecord<string, string>Static headers added to every message.
retryretryRetry configuration for transient failures.
maxRetryTimeintegerMax total retry wait in ms. Default: 30000.
initialRetryTimeintegerFirst retry delay in ms. Default: 300.
retriesintegerMax retry count. Default: 5.
* Required fields

Mapping​

Per-event rules under config.mapping. For the standard rule fields (consent, condition, data, batch, name, policy) see mapping.

PropertyTypeDescriptionMore
keystringOverride message key mapping path for this rule (e.g. data.id). Takes precedence over settings.kafka.key.
topicstringOverride Kafka topic for this rule. Takes precedence over settings.kafka.topic.

Examples

Default event

An event is produced to the configured Kafka topic with the full JSON body and entity_action as the message key.

Event
{
  "name": "page view",
  "data": {
    "domain": "www.example.com",
    "title": "walkerOS documentation",
    "referrer": "https://www.walkeros.io/",
    "search": "?foo=bar",
    "hash": "#hash",
    "id": "/docs/"
  },
  "context": {
    "dev": [
      "test",
      1
    ]
  },
  "globals": {
    "pagegroup": "docs"
  },
  "custom": {
    "completely": "random"
  },
  "user": {
    "id": "us3r",
    "device": "c00k13",
    "session": "s3ss10n"
  },
  "nested": [
    {
      "entity": "child",
      "data": {
        "is": "subordinated"
      }
    }
  ],
  "consent": {
    "functional": true
  },
  "id": "f0cc5f6fec3d8eeb",
  "trigger": "load",
  "entity": "page",
  "action": "view",
  "timestamp": 1700000100000,
  "timing": 3.14,
  "source": {
    "count": 1,
    "trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9",
    "type": "collector",
    "schema": "4"
  }
}
Out
producer.send({
  "topic": "walkeros-events",
  "messages": [
    {
      "key": "page_view",
      "value": "json:event",
      "headers": {
        "content-type": "application/json"
      },
      "timestamp": "1700000100000"
    }
  ],
  "acks": -1,
  "compression": 1
})

Key from user id

A settings.kafka.key path resolves the message key from the event, here using user.id for per-user partitioning.

Event
{
  "name": "user signup",
  "data": {
    "plan": "pro"
  },
  "context": {
    "dev": [
      "test",
      1
    ]
  },
  "globals": {
    "lang": "elb"
  },
  "custom": {
    "completely": "random"
  },
  "user": {
    "id": "usr-789"
  },
  "nested": [
    {
      "entity": "child",
      "data": {
        "is": "subordinated"
      }
    }
  ],
  "consent": {
    "functional": true
  },
  "id": "f66dbc3bf5e573df",
  "trigger": "test",
  "entity": "user",
  "action": "signup",
  "timestamp": 1700000103000,
  "timing": 3.14,
  "source": {
    "count": 1,
    "trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9",
    "type": "collector",
    "schema": "4"
  }
}
Out
producer.send({
  "topic": "walkeros-events",
  "messages": [
    {
      "key": "usr-789",
      "value": "json:event",
      "headers": {
        "content-type": "application/json"
      },
      "timestamp": "1700000103000"
    }
  ],
  "acks": -1,
  "compression": 1
})

Mapped payload

A data mapping transforms the event payload before producing it as the Kafka message value.

Event
{
  "name": "order complete",
  "data": {
    "id": "ORD-400",
    "value": 99.99,
    "currency": "EUR"
  },
  "context": {
    "shopping": [
      "complete",
      0
    ]
  },
  "globals": {
    "pagegroup": "shop"
  },
  "custom": {
    "completely": "random"
  },
  "user": {
    "id": "us3r",
    "device": "c00k13",
    "session": "s3ss10n"
  },
  "nested": [
    {
      "entity": "product",
      "data": {
        "id": "ers",
        "name": "Everyday Ruck Snack",
        "color": "black",
        "size": "l",
        "price": 420
      },
      "context": {
        "shopping": [
          "complete",
          0
        ]
      },
      "nested": []
    },
    {
      "entity": "product",
      "data": {
        "id": "cc",
        "name": "Cool Cap",
        "size": "one size",
        "price": 42
      },
      "context": {
        "shopping": [
          "complete",
          0
        ]
      },
      "nested": []
    },
    {
      "entity": "gift",
      "data": {
        "name": "Surprise"
      },
      "context": {
        "shopping": [
          "complete",
          0
        ]
      },
      "nested": []
    }
  ],
  "consent": {
    "functional": true
  },
  "id": "235d0d7bd7a213f2",
  "trigger": "load",
  "entity": "order",
  "action": "complete",
  "timestamp": 1700000102000,
  "timing": 3.14,
  "source": {
    "count": 1,
    "trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9",
    "type": "collector",
    "schema": "4"
  }
}
Mapping
{
  "name": "purchase",
  "data": {
    "map": {
      "order_id": "data.id",
      "revenue": "data.value",
      "currency": "data.currency"
    }
  }
}
Out
producer.send({
  "topic": "walkeros-events",
  "messages": [
    {
      "key": "purchase",
      "value": "json:data",
      "headers": {
        "content-type": "application/json"
      },
      "timestamp": "1700000102000"
    }
  ],
  "acks": -1,
  "compression": 1
})

Renamed event

A mapping renames the event which also changes the default Kafka message key used for partitioning.

Event
{
  "name": "order complete",
  "data": {
    "id": "0rd3r1d",
    "currency": "EUR",
    "value": 462,
    "tax": 73.76,
    "shipping": 5.22,
    "total": 467.22
  },
  "context": {
    "shopping": [
      "complete",
      0
    ]
  },
  "globals": {
    "pagegroup": "shop"
  },
  "custom": {
    "completely": "random"
  },
  "user": {
    "id": "us3r",
    "device": "c00k13",
    "session": "s3ss10n"
  },
  "nested": [
    {
      "entity": "product",
      "data": {
        "id": "ers",
        "name": "Everyday Ruck Snack",
        "color": "black",
        "size": "l",
        "price": 420
      },
      "context": {
        "shopping": [
          "complete",
          0
        ]
      },
      "nested": []
    },
    {
      "entity": "product",
      "data": {
        "id": "cc",
        "name": "Cool Cap",
        "size": "one size",
        "price": 42
      },
      "context": {
        "shopping": [
          "complete",
          0
        ]
      },
      "nested": []
    },
    {
      "entity": "gift",
      "data": {
        "name": "Surprise"
      },
      "context": {
        "shopping": [
          "complete",
          0
        ]
      },
      "nested": []
    }
  ],
  "consent": {
    "functional": true
  },
  "id": "235d0d7bd7a213f2",
  "trigger": "load",
  "entity": "order",
  "action": "complete",
  "timestamp": 1700000101000,
  "timing": 3.14,
  "source": {
    "count": 1,
    "trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9",
    "type": "collector",
    "schema": "4"
  }
}
Mapping
{
  "name": "purchase"
}
Out
producer.send({
  "topic": "walkeros-events",
  "messages": [
    {
      "key": "purchase",
      "value": "json:event",
      "headers": {
        "content-type": "application/json"
      },
      "timestamp": "1700000101000"
    }
  ],
  "acks": -1,
  "compression": 1
})

Topic override

A mapping rule overrides the destination topic so specific events are routed to a dedicated stream.

Event
{
  "name": "order complete",
  "data": {
    "id": "ORD-500",
    "value": 42
  },
  "context": {
    "shopping": [
      "complete",
      0
    ]
  },
  "globals": {
    "pagegroup": "shop"
  },
  "custom": {
    "completely": "random"
  },
  "user": {
    "id": "us3r",
    "device": "c00k13",
    "session": "s3ss10n"
  },
  "nested": [
    {
      "entity": "product",
      "data": {
        "id": "ers",
        "name": "Everyday Ruck Snack",
        "color": "black",
        "size": "l",
        "price": 420
      },
      "context": {
        "shopping": [
          "complete",
          0
        ]
      },
      "nested": []
    },
    {
      "entity": "product",
      "data": {
        "id": "cc",
        "name": "Cool Cap",
        "size": "one size",
        "price": 42
      },
      "context": {
        "shopping": [
          "complete",
          0
        ]
      },
      "nested": []
    },
    {
      "entity": "gift",
      "data": {
        "name": "Surprise"
      },
      "context": {
        "shopping": [
          "complete",
          0
        ]
      },
      "nested": []
    }
  ],
  "consent": {
    "functional": true
  },
  "id": "235d0d7bd7a213f2",
  "trigger": "load",
  "entity": "order",
  "action": "complete",
  "timestamp": 1700000104000,
  "timing": 3.14,
  "source": {
    "count": 1,
    "trace": "0a1b2c3d4e5f60718293a4b5c6d7e8f9",
    "type": "collector",
    "schema": "4"
  }
}
Mapping
{
  "settings": {
    "topic": "orders-stream"
  }
}
Out
producer.send({
  "topic": "orders-stream",
  "messages": [
    {
      "key": "order_complete",
      "value": "json:event",
      "headers": {
        "content-type": "application/json"
      },
      "timestamp": "1700000104000"
    }
  ],
  "acks": -1,
  "compression": 1
})

The destination creates a single long-lived kafkajs producer and connects it at the first push, so a flow starts without a reachable broker. kafkajs retries a failed connect or send as settings.kafka.retry allows. When a send still fails, the event fails with a transport error, marked retryable unless no retry can fix it (such as MESSAGE_TOO_LARGE), and goes to the dead-letter queue (when delivery fails). A topic that does not exist fails every event (below). On flow hot-swap or server shutdown, destroy() calls producer.disconnect() to close the connections.

Message format​

Events are serialized as JSON and produced with the following structure:

  • topic, from settings.kafka.topic (or mapping.settings.topic override)
  • key, resolved from settings.kafka.key (or mapping.settings.key override) mapping path; defaults to the event name with spaces replaced by _ (e.g. page_view, order_complete) for partition-based ordering. A key path whose value is not a non-empty string (a number, for example) falls back to that default
  • value, the mapped data when a data mapping resolves to an object with at least one field, otherwise the full walkerOS event, JSON.stringify()-ed
  • headers, content-type: application/json plus any static settings.kafka.headers
  • timestamp, event timestamp as string (ms since epoch)

Use mapping.settings.topic to route specific events to dedicated topics (e.g. orders to orders-stream, identities to identity-stream). Use mapping.settings.key to set a key path per rule (e.g. data.order_id for order events).

settings.kafka.compression defaults to gzip, which kafkajs ships. snappy, lz4 and zstd need a codec registered in kafkajs's CompressionCodecs in the same process. The destination registers none, so register the codec in your own code before the first send (integrated mode). Without one, every send fails with a transport error.

Setup​

No safe defaults

This destination's setup requires explicit numPartitions and replicationFactor. There is no universally correct default for either: replicationFactor must be less than or equal to broker count, and numPartitions is a function of expected throughput and consumer parallelism. The boolean form setup: true is rejected with an error listing the required fields. Only the object form is valid.

Provision a topic once per environment with the CLI:

walkeros setup destination.kafka

The command prints setup: ok destination.kafka; with --json it prints a JSON result with { topicCreated, schemaRegistered }. The command is idempotent and safe to re-run. Drift on numPartitions, replicationFactor, or configEntries is logged as a setup.drift warning with the field, the declared and the actual value, and never auto-mutates the broker.

config.setup:

  • omitted or false (default): no provisioning; the command prints setup: skipped destination.kafka (config.setup is unset) (or is false).
  • true: the command fails with an error that names the required fields. There are no safe defaults for partition count or replication factor.
  • { numPartitions, replicationFactor, ... }: object form is the only valid form. See the Setup interface in the package for full options.

Required fields​

FieldTypeNotes
numPartitionsnumberRequired at runtime. No safe default.
replicationFactornumberRequired at runtime. Must be <= broker count.

Optional fields​

FieldTypeNotes
topicstringFalls back to settings.kafka.topic when omitted.
configEntriesobjectTopic-level config, e.g. { "retention.ms": "604800000" }.
schemaRegistryobjectConfluent Schema Registry binding (see below).
validateOnlybooleankafkajs broker-side dry-run. No topic is created.

Example​

{
  "destinations": {
    "kafka": {
      "package": "@walkeros/server-destination-kafka",
      "config": {
        "settings": {
          "kafka": { "brokers": ["broker:9092"], "topic": "walkeros-events" }
        },
        "setup": {
          "numPartitions": 6,
          "replicationFactor": 3,
          "configEntries": { "retention.ms": "604800000" }
        }
      }
    }
  }
}

Schema Registry (optional)​

{
  "setup": {
    "numPartitions": 6,
    "replicationFactor": 3,
    "schemaRegistry": {
      "url": "https://schema-registry.example.com",
      "subject": "walkeros-events-value",
      "schemaType": "JSON",
      "schema": "{ \"type\": \"object\" }",
      "compatibility": "BACKWARD"
    }
  }
}

The schema is registered via the Confluent Schema Registry REST API. The optional compatibility level is set on the subject after registration.

Runtime error when the topic is missing​

When setup was not run and the topic does not exist on the cluster, every event fails with Kafka push failed, a retryable transport error, once the kafkajs retries are used up. kafkajs hands over the broker's UNKNOWN_TOPIC_OR_PARTITION inside its retry error, so in the event's dead-letter entry it is the cause of the error's cause. Run setup with explicit numPartitions and replicationFactor to provision the topic.

Authentication​

Confluent Cloud (SASL/PLAIN)​

{
  "kafka": {
    "brokers": ["pkc-xxxxx.us-east-1.aws.confluent.cloud:9092"],
    "topic": "walkeros-events",
    "ssl": true,
    "sasl": {
      "mechanism": "plain",
      "username": "$secret.CONFLUENT_API_KEY",
      "password": "$secret.CONFLUENT_API_SECRET"
    }
  }
}

SASL/SCRAM​

{
  "kafka": {
    "brokers": ["broker.example.com:9096"],
    "topic": "walkeros-events",
    "ssl": true,
    "sasl": {
      "mechanism": "scram-sha-512",
      "username": "$secret.KAFKA_USERNAME",
      "password": "$secret.KAFKA_PASSWORD"
    }
  }
}

mechanism: "aws" is kafkajs's own AWS mechanism: it needs authorizationIdentity next to accessKeyId and secretAccessKey, and a broker that offers the AWS SASL mechanism. oauthbearer needs a token provider function, which these settings cannot hold, so it does not authenticate.

💡 Need implementation support?
elbwalker offers hands-on support: setup review, measurement planning, destination mapping, and live troubleshooting. Book a 2-hour session (€399)