Telemetry data flow - detection to dashboard

The lifecycle of a single security event: detected, buffered, batched, encrypted, flushed to core, surfaced in the dashboard, and every place it can be lost.

5 min read · verified against [email protected]

A security event is not delivered the moment it is detected. It moves through a pipeline - buffer, optional encryption, batch flush - before it reaches the Core API and becomes visible in the dashboard. Understanding each step tells you where an event can be lost and how to prevent it.

One event's journey

  1. Detection fires. fastapi-guard's middleware detects an event - a SQLi probe, a rate-limit trip, a behavioral rule match, a ban application.
  2. In-memory buffer append. guard-agent appends the event to the appropriate in-memory buffer (events buffer or metrics buffer). The buffer is bounded. The append is non-blocking; the request path does not wait for delivery.
  3. Flush trigger. Two conditions are evaluated continuously. Size trigger: when the buffer reaches 80% capacity (len(buffer) >= buffer_size * 0.8), a flush fires immediately. Time trigger: every agent_flush_interval seconds, a sentinel flush runs regardless of buffer fill. The first trigger to fire wins; both run concurrently as independent conditions.
  4. Optional encryption. If agent_project_encryption_key is set, guard-agent seals the event batch with AES-256-GCM using the project key before transmission. The wire format is plain nonce + ciphertext ([nonce: 12 bytes][ciphertext+tag: N bytes]), base64-encoded, there is no version marker. See Encryption key rotation.
  5. POST to Core API. The agent posts the batch to the Core API - POST /api/v1/events (plain) or POST /api/v1/events/encrypted (encrypted). Authenticated with X-API-Key. Transport is TLS. On failure, the agent retries with exponential backoff.
  6. Stored. The Core API authenticates, scopes to the project, and decrypts the batch with the project's single active key, then persists events. Sensitive fields are re-encrypted at rest under the master KEK; those at-rest ciphertexts carry the master-key version marker (a separate mechanism from the telemetry payload, which has none).
  7. Dashboard. The dashboard queries the Core API and surfaces events on the Events page, the Threats view, and the Metrics charts.

Buffering and batching

Events do not travel one at a time. guard-agent maintains in-memory buffers and flushes them in batches. This decouples detection (on the hot request path) from delivery (background task), so a network hiccup or core outage does not inflate request latency or lose events immediately.

The flush triggers - size at 80% capacity OR time at agent_flush_interval - give you both low latency under load and predictable delivery during quiet periods. Size-triggered flushes fire early enough (20% headroom) that a slow flush has room to complete before the buffer overflows.

On flush failure, the agent retries with exponential backoff: delay = backoff_factor × 2^attempt, capped at 60 seconds. Default is 3 retry attempts. A 429 from the Core API puts the agent into self-throttling mode rather than immediate retry.

The full buffer configuration - sizing formula, overflow policies, Redis spillover for delivery durability - is covered in Agent buffering & backpressure.

Encryption in transit

The encryption layer is optional but recommended for production. When agent_project_encryption_key is set, every event payload is sealed with AES-256-GCM before it leaves your process. This sits on top of TLS, useful when a TLS-terminating proxy is in the path and you want the payload to remain opaque to the proxy.

The agent selects the endpoint automatically: POST /api/v1/events/encrypted when the key is set, POST /api/v1/events when it is not. You do not choose the endpoint directly.

The project encryption key is held in process memory (loaded from GUARD_PROJECT_ENCRYPTION_KEY at startup). After a key rotation, every agent instance must restart to pick up the new key. Rotation overwrites the project key with no historic fallback, so events emitted before the restart are sealed with the old key and the Core API cannot decrypt them. Those batches fail AES-GCM authentication and are rejected with HTTP 400. See Encryption key rotation and API keys & encryption setup for the full key lifecycle.

What reaches the dashboard

The agent maintains two independent buffers: events and metrics. They fill independently, flush independently, and post to separate endpoints (/api/v1/events and /api/v1/metrics).

Disabling one does not affect the other. Setting agent_enable_metrics=False stops the metrics buffer without touching event delivery. This independence means you can debug the events stream in isolation by disabling metrics - no cross-contamination.

Events buffer: detections, bans, rate-limit hits, behavioral rule trips. High-cardinality signal; one entry per security decision.

Metrics buffer: aggregated counts and timings, pre-aggregated before emission. Lower cardinality.

Both surface in the dashboard, but on different views - events appear in the Events and Threats pages; metrics feed the Metrics charts and dashboard widgets.

Every place an event can be lost

Buffer overflow drop (default policy). If events arrive faster than the agent can flush - sustained traffic spike, prolonged Core API outage - the buffer fills. The default "drop" policy silently evicts the oldest entry. No error is logged, no metric is incremented. The first signal is gaps in the dashboard. Size your buffer to make this rare: buffer_size >= peak_events_per_second × flush_interval × 2. See Agent buffering & backpressure.

Core API outage beyond buffer capacity. If the Core API is unreachable for longer than the buffer can absorb (outage duration × event rate > buffer capacity), events overflow and are dropped under the default policy. Optional Redis spillover (agent_events:* hashes, TTL 3600s) can extend effective capacity - see Agent buffering & backpressure.

Wrong endpoint or key mismatch. If agent_endpoint points at the wrong URL, or agent_api_key doesn't match the project, or the encryption key does not match the project's current active key, events are rejected at the Core API and do not appear in the dashboard. The agent retries and eventually drops after exhausting retry attempts.

If events are not appearing, a decision tree for diagnosing which loss point is active is in Events not appearing in the dashboard.

Related deep-dives