Encrypted events rejected after key rotation

Why telemetry stops the moment you rotate the project encryption key, and the rotation sequence that avoids the gap.

4 min read · verified against [email protected]

Rotating the project encryption key is a single API call. The gap it creates - events emitted after rotation but before the agent restarts - is not a bug. It is a predictable consequence of when the new key takes effect on each side of the pipeline. Knowing the sequence prevents it.

Symptom: events stop after rotating the key

Events were flowing before the rotation. Immediately after calling the rotation endpoint, new events stop appearing in the dashboard. There are no 401 errors in logs. The agent appears to be flushing normally. The dashboard just shows nothing new.

This is the mismatch window. The server's active key version changed at rotation time. The agent still holds the previous plaintext key in process memory, loaded at startup, never refreshed. Every event the agent emits is sealed with the old key version. Because rotation makes the new key version the active ingest version immediately, events arriving sealed under the old key version are rejected at ingestion and dropped before they reach the dashboard. Historic events remain readable. Each ciphertext carries the version it was sealed with, and the server retains the matching historic cipher for old versions, but new events submitted by a stale agent are not accepted at ingest.

Historical events are not affected. Each ciphertext carries a one-byte version marker. The server retains historic cipher versions and can decrypt old events on demand. Rotation does not re-encrypt historical data. It only changes what version new events seal with.

The rotation order that breaks things

The common mistake is rotating the key on the server first and redeploying the agent later, with a gap in between. During that gap, the agent seals events with a key version the server no longer accepts as the active ingest version.

1. Rotate key on server  ← server now expects v_new
2.   [gap: agent still running with v_old]
3. Events emitted here are sealed with v_old → rejected
4. Redeploy agent        ← agent picks up v_new at startup

The gap can be seconds (rolling deploy that starts immediately) or hours (manual deploy that gets delayed). Every event in that window is lost.

Correct rotation sequence

The safe sequence, derived from the five-step playbook in Encryption key rotation:

  1. Rotate - call the rotation endpoint. The server generates and activates the new key version. This step is instant.

  2. Capture - read plaintext_key from the response. It is shown once and not stored server-side in recoverable form. Copy it to your secrets manager before closing the page or discarding the response. If you lose it, you must rotate again.

  3. Update the secret - replace GUARD_PROJECT_ENCRYPTION_KEY (or however the plaintext is injected into your deployment) with the new value. Do this before triggering the deploy.

  4. Roll - trigger a rolling deploy or process restart of every agent instance. Each instance reads the env var at startup and loads the new plaintext into memory. New events seal with the new version.

  5. Verify - confirm fresh events appear in the dashboard. If events still do not appear after a full flush interval, check whether all replicas have restarted. An instance that did not pick up the new env var is still sealing with the old key.

The key property of this sequence: the env var is updated before the deploy fires, so by the time a restarted process begins emitting events, it has the correct key in memory. The mismatch window is reduced to the time between the rotation API call and the first restarted replica, typically a few seconds in a rolling deploy.

Verifying the agent picked up the new key

After the deploy, check that fresh events are appearing in the dashboard. If the dashboard stays silent:

  • Confirm the env var was updated in your secrets manager and that the deploy picked up the new value (not a cached or stale image).
  • If you have multiple replicas, confirm all of them restarted. A partial rollout leaves some instances on the old key.
  • Wait one agent_flush_interval after a restart before concluding events are missing. The buffer needs at least one flush cycle to deliver.

If events still do not appear after verifying the above, work through the full decision tree in Events not appearing in the dashboard.

The agent-side key lifecycle - how the plaintext is loaded, where it lives in memory, and why a live reload is not supported - is covered in Encryption key rotation.

Related