Skip to main content
Ask your AI

Cache

Cache is a built-in config property available on every step type (source, transformer, destination). It prevents redundant processing by caching results and serving them on subsequent matching requests.

Configuration​

Add cache to any step in your flow config:

"sources": {
"express": {
"package": "@walkeros/server-source-express",
"config": {
"ingest": { "map": { "method": "method", "path": "path" } }
},
"cache": {
"rules": [
{
"match": { "key": "ingest.method", "operator": "eq", "value": "GET" },
"key": ["ingest.method", "ingest.path"],
"ttl": 300
}
]
}
}
}

ingest.method and ingest.path exist because config.ingest lifts them from the request. Without it the rule's match and key find nothing, and nothing is cached (see Key Resolution).

Cache Rule Properties​

PropertyTypeDescription
matchMatchExpressionWhen to apply this cache rule
keystring[]Dot-paths to build the cache key from
ttlnumberTime-to-live in seconds
updateRecord<string, Value>Source caches only: changes to the cached response, applied on HIT and MISS

match is optional, omit it for an always-match rule.

Cache Properties​

PropertyTypeDefaultDescription
stopbooleanfalse (true for sources)Stop flow on cache HIT and skip remaining steps, returning the cached value
storestringBuilt-in memoryReference to a store in flow.stores
namespacestring(none)Optional key prefix. Omit to write keys directly to the store. Same store + same key + same namespace = same cache entry.
rulesCacheRule[]RequiredAt least one cache rule

walkeros validate does not reject an unknown key in cache or in a rule; such a key has no effect.

Behavior by Step Type​

Source Cache (stop by default)​

Source cache intercepts requests before the pipeline runs. Sources default to stop: true. On cache HIT, the cached respond value is returned immediately and the pipeline is skipped, as in the example above. With stop: false, a HIT changes nothing: the pipeline runs as on a MISS, and update is not applied.

Transformer Cache (step-level)​

Transformer cache skips the transformer's execution on HIT but continues the chain with the cached event. This helps with expensive computations such as API enrichment. With stop: true, the remaining chain stops on HIT and downstream transformers and destinations are skipped.

Halt scopes

Transformer caches with stop: true in a source chain (source.before, source.next) or in the collector chain (collector.next) halt the event on HIT: no destination sees it. In a destination's before chain, a HIT skips the rest of that chain and the destination receives the cached event. Destination caches with stop: true halt only the per-destination delivery; sibling destinations still receive the event. To drop an event by rule rather than by cache, use a route stop (see Dropping an event with stop).

The dedup step of flow-complete.json has no code of its own, only a cache. It runs in the server source's next chain, so a hit with stop: true drops a resent event before the collector:

/flows/server/transformers/dedup/cache
{
  "stop": true,
  "namespace": "dedup",
  "rules": [
    {
      "key": [
        "event.id"
      ],
      "ttl": 3600
    }
  ]
}
A resend with the same event id is stopped for an hour. Guide chapter quality

/flows/server/transformers/dedup/cache/stop: true. stop: true turns a cache hit into a drop. Guide chapter quality

/flows/server/transformers/dedup/cache/namespace: "dedup". namespace keeps cache keys of different steps apart. Guide chapter quality

/flows/server/transformers/dedup/cache/rules
[
  {
    "key": [
      "event.id"
    ],
    "ttl": 3600
  }
]
rules say which events are cached, by which key, for how long. Guide chapter quality
/flows/server/transformers/dedup/cache/rules/0/key
[
  "event.id"
]
key is a list of paths; together they identify a duplicate. Guide chapter quality

/flows/server/transformers/dedup/cache/rules/0/ttl: 3600. ttl in seconds. Guide chapter quality

The first event with this id passes and its id is remembered for an hour. A resend with the same id is stopped (second request, proven in the flow test).

Event
{
  "name": "order complete",
  "entity": "order",
  "action": "complete",
  "data": {
    "id": "ORD-100",
    "value": 149.8,
    "currency": "EUR",
    "coupon": "WELCOME10"
  },
  "id": "ev-order-complete",
  "trigger": "load",
  "timestamp": 1700000000000,
  "consent": {
    "functional": true,
    "marketing": true
  },
  "globals": {
    "language": "en",
    "pageGroup": "checkout"
  },
  "user": {
    "device": "d3v1c3",
    "session": "s3ss10n",
    "id": "cust-42",
    "email": "jane@example.com"
  },
  "source": {
    "type": "express",
    "platform": "server",
    "url": "https://www.example.com/checkout/thanks"
  },
  "nested": [
    {
      "entity": "product",
      "data": {
        "id": "SKU-1",
        "name": "Trail Runner",
        "price": 129.9,
        "quantity": 1
      }
    },
    {
      "entity": "product",
      "data": {
        "id": "SKU-2",
        "name": "Running Socks",
        "price": 9.95,
        "quantity": 2
      }
    }
  ]
}
Out
return {
  "event": {
    "name": "order complete",
    "entity": "order",
    "action": "complete",
    "data": {
      "id": "ORD-100",
      "value": 149.8,
      "currency": "EUR",
      "coupon": "WELCOME10"
    },
    "id": "ev-order-complete",
    "trigger": "load",
    "timestamp": 1700000000000,
    "consent": {
      "functional": true,
      "marketing": true
    },
    "globals": {
      "language": "en",
      "pageGroup": "checkout"
    },
    "user": {
      "device": "d3v1c3",
      "session": "s3ss10n",
      "id": "cust-42",
      "email": "jane@example.com"
    },
    "source": {
      "type": "express",
      "platform": "server",
      "url": "https://www.example.com/checkout/thanks"
    },
    "nested": [
      {
        "entity": "product",
        "data": {
          "id": "SKU-1",
          "name": "Trail Runner",
          "price": 129.9,
          "quantity": 1
        }
      },
      {
        "entity": "product",
        "data": {
          "id": "SKU-2",
          "name": "Running Socks",
          "price": 9.95,
          "quantity": 2
        }
      }
    ]
  }
}
First delivery passes: example firstDelivery of step dedup. Guide chapter quality

Destination Cache (deduplication)​

Destination cache skips the push on HIT, preventing duplicate events from reaching the destination. Other destinations still receive the event. With stop: true, the before transformer chain is also skipped on HIT.

A thank-you page reload sends the same order again with a new event id, so the dedup step above lets it through. Meta dedups on the order id instead, and the rule's match limits the cache to order complete:

/flows/server/destinations/meta/cache
{
  "stop": true,
  "rules": [
    {
      "match": {
        "key": "event.name",
        "operator": "eq",
        "value": "order complete"
      },
      "key": [
        "event.data.id"
      ],
      "ttl": 86400
    }
  ]
}
A thank-you page reload gets a new event id for the same order; Meta gets each order once. Guide chapter quality
/flows/server/destinations/meta/cache/rules/0/match
{
  "key": "event.name",
  "operator": "eq",
  "value": "order complete"
}
A rule match limits the cache to order complete. Guide chapter quality

Key Resolution​

Cache keys use dot-path resolution from a structured context:

  • ingest.method: request method from source ingest
  • ingest.path: request path from source ingest
  • event.name: event name
  • event.data.id: nested event data field
  • event.user.loggedIn: user properties

Ingest holds only the fields the source's config.ingest maps. A source cache sees only ingest: event. paths resolve in transformer and destination caches. The resolved paths are joined with :; when every path of the rule resolves empty, the event is not cached, and nothing reports it.

Update Rules​

In a source cache, use update to modify the response on both HIT and MISS:

"cache": {
"rules": [{
"key": ["ingest.method", "ingest.path"],
"ttl": 300,
"update": {
"headers.X-Cache": { "key": "cache.status" },
"headers.Cache-Control": { "value": "public, max-age=300" }
}
}]
}

cache.status resolves to "HIT" or "MISS". Static values use { "value": "..." }. Transformer and destination caches do not apply update, and walkeros validate does not flag it there.

Custom Store​

By default, cache uses a shared in-memory store. Reference a custom store for persistence:

{
"stores": {
"files": {
"package": "@walkeros/server-store-fs",
"config": { "settings": { "basePath": "./cache" } }
}
},
"sources": {
"express": {
"package": "@walkeros/server-source-express",
"config": {
"ingest": { "map": { "method": "method", "path": "path" } }
},
"cache": {
"store": "files",
"rules": [{ "key": ["ingest.method", "ingest.path"], "ttl": 300 }]
}
}
}
}

The filesystem store keeps entries on the local disk of one instance. To share cache entries across instances, use @walkeros/server-store-s3 or @walkeros/server-store-gcs instead. Do not use the Sheets store as a cache tier (a cache.store target): nothing rejects it, but every cache read and write then costs a Sheets request. A cache can still wrap it.

The filesystem store rejects a key that begins with / (a WARN line, Path traversal rejected, and no entry), so start key with a field such as ingest.method, or set namespace.

A cache write that fails writes no log line and the push still reports ok: true, so a dedup cache on a store that refuses writes lets duplicates through.

A store can carry a cache of its own. In the complete example, reads from the customers store are memoized for five minutes:

/flows/server/stores/customers/cache
{
  "rules": [
    {
      "ttl": 300
    }
  ]
}
Store cache memoizes customer reads for five minutes. Guide chapter state-stores

Custom stores can be either synchronous (in-memory) or asynchronous (file system, S3, GCS, the cache wrapper). Cache lookups await the underlying store.get internally, so HIT/MISS semantics behave identically regardless of which backing you wire.

💡 Need implementation support?
elbwalker offers hands-on support: setup review, measurement planning, destination mapping, and live troubleshooting. Book a 2-hour session (€399)