Encryption key rotation

Rotating your project encryption key without losing in-flight events or breaking the agent.

5 min read · verified against [email protected]

Rotation is the one cryptographic operation you should treat as routine. The mechanics are designed to be safe; the schedule is up to you.

Three reasons, three urgencies

Hygiene - quarterly or per release train. Limits the blast radius of a leaked key. Every active encryption key is one accidental log-line away from disclosure. Rotation puts a hard upper bound on how long any one key matters.

Suspected compromise - now. Key material may have leaked: accidental git commit, exposed env dump, departed engineer with prod access. Rotate first, investigate after. Old ciphertexts remain readable to you; the leaked key only encrypts events from the rotation backwards.

Compliance - per framework, typically yearly minimum. SOC 2, ISO 27001, PCI, and similar frameworks expect documented key-rotation schedules. The cadence is part of the audit narrative; the evidence is the rotation timestamps and the version markers on subsequent events.

The trigger - one endpoint, one new key

Rotation is a single API call.

POST /api/v1/encryption/rotate
Authorization: Bearer <project-token>

→ 200 OK
{
  "project_id": "proj_...",
  "key_version": 7,
  "plaintext_key": "..."   // shown ONCE
}

Server-side:

  1. Generate fresh 256-bit key
  2. Wrap with the master key
  3. Replace the active project payload key
  4. Bump key_version on the project record
  5. Return plaintext to caller (once)

Versioning - each ciphertext carries the version that sealed it

Every encrypted payload starts with a one-byte version marker. On decrypt, the server reads the marker to pick the right historic cipher. Old events stay readable after rotation. They're sealed under the version they were ingested with.

payload structure:  [v: 1 byte][ciphertext: N bytes]

Before rotation: v=6 on every new event. All emissions carry the current version. Decryption is uniform - one key, one cipher.

After rotation: v=7 on new events, v=6 on historical. Mixed-version state. Both decryptable.

Historical events - sealed under their original version, forever

Rotation does NOT re-encrypt historical events. They remain in their original ciphertext, with their original version marker, decryptable on demand for as long as the server retains the matching historic cipher.

An auditor pulls a 6-month-old detection event from the dashboard. Event carries v=6. Server looks up the v=6 historic cipher, decrypts it, returns plaintext. From the auditor's perspective, rotation is invisible. The dashboard "just works."

Agent side - the agent holds the plaintext key in memory

The server has the new key. The agent does not. Until the agent picks up the new plaintext, every event it emits is sealed with the OLD key, and the dashboard can't decrypt them.

Option A - manual redeploy (confirmed). Set GUARD_PROJECT_ENCRYPTION_KEY to the new plaintext from the rotation response. Redeploy or restart the process. The new key is loaded into memory at startup; new emissions seal with it. Reliable. Requires a deploy. Brief mismatch window before all replicas pick up the new key.

Option B - dynamic-rules push (unconfirmed). In principle, the rotation could propagate via the dynamic-rules pipeline (see Dynamic rules) - the agent picks up the new key without a redeploy. NOT confirmed in the verified code path. Confirm with the platform owner before relying on this. Until then, treat manual redeploy as the only documented path.

The playbook - five steps

  1. Rotate - call POST /api/v1/encryption/rotate (dashboard button or direct API). Server generates new key.
  2. Capture - read plaintext_key from the response. This is the only time it's shown. Store in your secrets manager before closing the page.
  3. Update env - set the new plaintext as your deployment's encryption-key secret. Same env var, new value.
  4. Roll - trigger a rolling deploy or process restart of every agent instance. Each picks up the new key at startup.
  5. Verify - confirm fresh events in the dashboard show the bumped key version. If old version persists, an agent didn't pick up the new key.

Total elapsed time: a few minutes for the API call, however long your rolling deploy takes for the rest. Plan for a brief mismatch window.

Wiring

project_encryption_key = os.environ.get("GUARD_PROJECT_ENCRYPTION_KEY")

SecurityConfig(
    agent_project_encryption_key=project_encryption_key,
)
SurfaceRotation footprint
App codeThe SecurityConfig line is stable across rotations - no change.
Env / secrets managerReplace GUARD_PROJECT_ENCRYPTION_KEY with the new plaintext.
Running processEnv vars are read at startup. Live reload of this value is not supported. Restart required.

Troubleshooting

SymptomLikely causeFix
After rotation, the agent emits but the dashboard can't decryptAgent still using old key in memory. New emissions sealed with old key. Replicas haven't rolled yet, or env var update didn't take effect.Restart the agent (or full rolling deploy). Verify env var is the new plaintext. Fresh events should decrypt cleanly.
Old events suddenly unreadable after rotationHistoric version marker references a cipher version the server no longer holds. Typically happens when the server-side master key is rotated without preserving historic versions - not the project key rotation covered here.Stop. Contact the platform owner - historic key material may need restoration. Do NOT rotate the master key in production without a coordinated procedure.
Lost the new plaintext key from the rotation responseResponse shows plaintext ONCE. Closed the tab, didn't capture.Rotate again - call the endpoint a second time. It supersedes the previous rotation. Capture this time.