Reliable event pipeline

Overview

Use a dedicated event pipeline when dropped analytics events would affect Experimentation results or guarded releases.

LaunchDarkly SDKs buffer analytics events in memory and send them to LaunchDarkly on a schedule. An extended outage of events.launchdarkly.com fills that memory buffer and the SDK drops events.

A dedicated event pipeline moves durability off SDK memory and onto disk-backed storage that holds days of backlog.

To learn about the underlying SDK resilience guidance, read Resilient architecture patterns for LaunchDarkly’s SDKs.

How the pipeline works

This diagram shows the pipeline’s architecture:

Point the SDK’s event routing at the pipeline’s ingestion tier instead of LaunchDarkly directly. Flag delivery keeps using LaunchDarkly or LD Relay directly, so flag evaluation keeps working during an event outage.

The pipeline has three stages:

• An ingestion tier accepts the SDK’s event payloads and writes them to a durable broker. This tier holds no state, so you scale it horizontally behind a load balancer.
• A durable broker persists events to disk across multiple replicated nodes. Durability lives here, not in any tier’s memory.
• A forwarder tier reads batches from the broker and delivers them to LaunchDarkly. It retries failed sends until LaunchDarkly acknowledges them and holds no state of its own.

See the pipeline handle an outage

This animation shows the broker holding events during a LaunchDarkly outage:

LaunchDarkly goes offline. The forwarder keeps retrying, and the broker’s queue depth grows since nothing can drain it yet. Once LaunchDarkly recovers, the forwarder delivers the backlog, and the queue depth returns to empty. No events reach LaunchDarkly during the outage, and none get dropped either.

Account for different SDK types

Server-side, browser, and mobile SDKs send events differently on the wire. A pipeline that only handles one shape silently breaks for the others.

This table shows how each SDK type authenticates and where it sends events:

iOS and Android SDKs both use the mobile shape. Browser SDKs never send an Authorization header. Mobile SDKs do send one, but format it differently than server-side SDKs do.

Mobile SDKs also post to a fixed path instead of embedding a credential in the URL. Don’t assume browser and mobile SDKs share one client-side pattern. They don’t.

Design the ingestion tier to accept all three shapes if your applications use more than one SDK type. Capture the credential from whichever location applies to that route. Carry both the credential and its type alongside each event, so the forwarder tier can rebuild the correct outgoing request.

Browser SDKs also require CORS handling, since requests originate directly from a user’s browser. Configure the ingestion tier to answer CORS preflight requests. Echo an appropriate Access-Control-Allow-Origin header, the same way events.launchdarkly.com does today.

Browser and mobile credentials are public by design, unlike a server-side SDK key. Plan for the ingestion tier’s client-facing routes to stay reachable from the public internet. Evaluate rate limiting and abuse protection accordingly.

How the pipeline scales

This diagram shows one pipeline deployment serving multiple applications and environments at once:

Scale the ingestion and forwarder tiers independently of each other and independently of LD Relay. Event-queue durability and flag delivery solve different scaling problems, so scaling one never forces you to scale the other.

The broker scales by adding partitions and nodes, not by adding per-key configuration. One pipeline deployment handles events from any number of LaunchDarkly environments or projects without dedicated setup for each key.

No credentials to configure in advance

The pipeline needs zero credentials configured anywhere in advance, regardless of SDK type.

The ingestion tier captures whichever credential each incoming request carries, from whichever location applies to that route. The forwarder tier groups events by whatever credentials appear in the data and delivers each group under its originating credential.

Adding a new environment is a client-side configuration change only. You never update the pipeline itself.

One shared pipeline deployment safely serves multiple LaunchDarkly projects or environments at the same time. Each environment’s events flow through independently and never get delivered under the wrong key.

Understand delivery guarantees

No pipeline eliminates event loss in every failure scenario.

This pattern reduces loss during transient outages and traffic spikes. It does not guarantee zero loss under all conditions.

Event loss can still happen in the following cases:

• The SDK’s own queue reaches capacity before the pipeline receives requests
• The broker’s disk storage reaches capacity
• Invalid credentials cause persistent 401 or 403 responses
• Invalid payloads cause persistent 4xx responses
• An outage exceeds the broker’s retention window

Consider other broker options

This pattern works with any durable, disk-backed message broker.

You can build it with the following open source options:

• Apache Kafka
• Redpanda
• NATS JetStream
• RabbitMQ

You can also build it with the following managed options:

• Amazon Kinesis
• Confluent Cloud
• Google Pub/Sub
• Azure Event Hubs

Lighter-weight forwarding tools such as Vector and Fluent Bit can also fill the forwarder role. Both tools add their own disk buffering in front of LaunchDarkly.

Related content

To learn more, read Using LD Relay and Key concepts.