500s on healthy requests after deploy

When fail_secure turns an infrastructure hiccup into a 500. How to confirm it's the security pipeline, and why the fix is the dependency, not the flag.

4 min read · verified against [email protected]

A deploy goes out, monitoring flips red, and healthy requests start returning 500. The business logic is fine - tests pass, the database is up. The security pipeline is returning 500 before the request reaches the handler.

Symptom

Every request, or a significant fraction of them, returns 500 immediately after deploy. The error text is "Security check failed" or similar. No change was made to business logic. The issue started at deploy time and affects requests that were clean before.

Most likely: fail_secure with a broken dependency

fail_secure=True is the default since fastapi-guard 7.0.0. Its job is to return HTTP 500 when any check in the security pipeline raises an exception - Redis unreachable, detector timeout, rate-limit counter unavailable.

The intent is explicit: with fail_secure=True, an infrastructure failure surfaces as a visible 500 rather than silently allowing the request through. This is the safe posture. An attacker cannot bypass detection by inducing an infra failure if the pipeline hard-fails.

When fail_secure=False, a failing check is skipped and the request proceeds as if the check returned clean. That prevents 500s, but it means a flapping Redis or a detector timeout quietly opens a window where no enforcement runs.

Since guard-core 3.5.0 there is a narrower lever for Redis specifically:

redis_fail_openbool

On GuardRedisError (Redis unreachable), skip the failing check and let the request through instead of honoring fail_secure. Defaults to False so fail_secure is the single source of truth for every check failure, including Redis outages. Set True to opt into treating Redis outages as an availability concern distinct from other check failures.

default: False

With redis_fail_open=True, a check that fails with a Redis error is skipped and the request proceeds, while every other pipeline failure still returns 500 under fail_secure. It defaults to False, so unless you have set it, fail_secure alone decides.

See Passive mode vs fail-secure for the full mechanics.

Confirm it

Check two things before changing any config.

1. Is the error coming from the security pipeline?

The middleware returns 500 with the message "Security check failed" when fail_secure=True triggers. Look for that exact string in the response body or logs. If the 500 comes with a different message or a Python traceback from your route handler, this is a different issue.

2. Which dependency is failing?

The most common cause is Redis. The pipeline depends on Redis for rate-limit counters, ban checks, and behavioral state. If Redis is unreachable at deploy time - wrong DSN after a migration, a firewall rule that didn't propagate, an auth change - every request that needs a Redis check will fail.

Check the Redis connection:

  • Confirm redis_url in SecurityConfig points to the correct host and port.
  • Try connecting directly: redis-cli -u <redis_url> ping should return PONG.
  • Check logs for ConnectionRefusedError, TimeoutError, or similar from the Redis client.

The failure modes that trigger fail_secure are covered in Redis backend & failure modes.

Where a 500 fits in the pipeline - it fires before the handler, at the check stage - is covered in How a request flows through fastapi-guard.

Fix the dependency, not the flag

The correct fix is to restore the dependency. fail_secure=True is doing its job: it is telling you that a check cannot run. Disabling the flag to stop the 500s does not fix the problem. It hides it.

Common fixes by failure type:

DependencyWhat to check
Redis unreachableDSN (redis_url), port, auth credentials, firewall rules, Redis process status
Redis restarted without persistenceBan keys lost; AOF (appendonly yes) prevents this on restart
Rate-limit / ban check erroringRedis key schema mismatch after a version upgrade - check redis_prefix
Redis flapping (transient blips, not a hard outage)redis_retries (default 1, exponential backoff) and the 2.0s socket timeouts absorb transient errors - see Connection resilience knobs

After fixing the underlying issue, deploy again. Requests should succeed once the pipeline can complete its checks.

When you must temporarily disable

There are cases where letting requests through is the right short-term choice: a production incident, a dependency that cannot be restored quickly, a canary stage where availability must be preserved while investigating.

If the broken dependency is Redis, prefer redis_fail_open=True over fail_secure=False. It has a narrower blast radius. Only checks failing with Redis errors are skipped; a crashing detector or any other pipeline failure still surfaces as a 500 instead of being silently swallowed.

If you set fail_secure=False (or redis_fail_open=True) temporarily:

  • Set a calendar reminder to re-enable it.
  • Document why it was changed and when it will be reverted.
  • Watch the dashboard actively during the window, detections still emit, but enforcement is silent.

Do not leave fail_secure=False in production as a permanent posture. The flag exists to make infrastructure failures visible; disabling it indefinitely removes that signal.

The four-stage rollout matrix in Passive mode vs fail-secure shows that the only production-recommended config is fail_secure=True.

Related