Troubleshooting - agent offline or not reporting

Step-by-step diagnostics when an agent goes stale (stops reporting) or never connects. Covers configuration, network, and runtime issues.

6 min read · verified against [email protected]

When an agent is in stale state (stopped reporting) or never-connected (never sent telemetry), follow this guide to diagnose and fix the issue.

Quick checklist

Before diving into detailed diagnostics, verify these basics:

  • ✅ The application with the agent is running
  • ✅ The agent library is installed (pip list | grep fastapi-guard)
  • ✅ The agent is initialized in your application startup (before first request)
  • ✅ The agent has a valid API key and project ID in SecurityConfig
  • ✅ The Core API endpoint is reachable from your application (no firewall blocking port 443 to api.guard-core.com)
  • ✅ There are no obvious errors in application stderr/stdout

If all are true and the issue persists, continue to detailed diagnostics.

Diagnosis: Is the issue agent-side or network-side?

Check agent local logs

Enable agent debug logging to see what's happening locally:

import logging

# Set guard agent logger to DEBUG
logging.getLogger("guard").setLevel(logging.DEBUG)
# or for verbose output:
logging.getLogger("guard.agent").setLevel(logging.DEBUG)

Look for these patterns in the logs:

If you see:

  • "Agent initialized successfully" - agent started OK
  • "Buffer flush: X events, Y metrics" - agent is buffering telemetry
  • "POST /api/v1/events response 200"` - agent is successfully sending data

→ Agent-side is working. Issue is likely on the dashboard side (data not persisted) or network timing. Wait 1-2 minutes for the dashboard to refresh and check again.

If you see:

  • "Failed to initialize agent", "ConfigError", or "Invalid API key" - configuration error
  • "Connection refused", "Timeout" - network unreachable
  • "HTTP 401", "HTTP 403" - authentication failed

→ Configuration or network issue. See relevant section below.

Check if heartbeats are being sent

The agent sends heartbeats even when there are zero events. If the agent is running but stale:

  1. Generate traffic to your application (make a request)
  2. Check agent logs for "Heartbeat sent" or "Buffer flush" messages
  3. Wait 30 seconds and refresh the dashboard

If heartbeats are being logged but the dashboard is not updating, the issue is likely network-related (see below).

Common issues and fixes

Issue 1: Configuration error (invalid API key or project ID)

Symptoms:

  • ConfigError in agent logs
  • "Invalid API key" or "Unknown project" in logs
  • Agent never enters healthy state

How to fix:

  1. Verify the API key is correct:

    # Get the key from the dashboard at /settings/api-keys
echo $GUARD_API_KEY  # or however you pass it to the agent

  2. Verify the project ID is correct:

    # Get the external_id from /projects or /projects/[id]
# It should be a short identifier like "proj_abcd1234"

  3. Ensure the API key belongs to the right project (keys are per-project, not global)

  4. If the key was recently rotated or created, the dashboard may still be reflecting old data. Wait 30 seconds and refresh.

  5. Regenerate a fresh API key if unsure:

    • Go to /settings/api-keys
    • Select the project
    • Create a new key
    • Copy it immediately (it's only shown once)
    • Restart your application with the new key

Issue 2: Network unreachable (firewall, proxy, DNS)

Symptoms:

  • "Connection refused" or "Timeout connecting to api.guard-core.com" in logs
  • Agent initialized but no data ever sent
  • Occasional stale periods followed by recovery

How to fix:

  1. Test DNS resolution:

    # From the machine running the agent
nslookup api.guard-core.com
# or
dig api.guard-core.com

    Should return one or more IP addresses.

  2. Test connectivity to the API endpoint:

    # HTTPS (port 443, what the agent uses)
curl -v https://api.guard-core.com/health
# Should return HTTP 200 or 401 (not connection refused)

  3. Check firewall / security groups:

    • Ensure outbound HTTPS (port 443) is allowed
    • If behind a corporate proxy, configure the agent to use it: 
      config = SecurityConfig(
    http_proxy="http://proxy.corp.com:8080",
    https_proxy="https://proxy.corp.com:8080",
)

  4. Check application environment:

    • If running in Kubernetes, check NetworkPolicy rules
    • If running in a container, check network settings
    • If running behind a NAT / load balancer, verify outbound routes

  5. Temporarily disable TLS verification (for testing only):

    # This is insecure; only for debugging!
import os
os.environ["GUARD_INSECURE_SKIP_TLS"] = "true"
# Then test. If it works, TLS certificate validation is the issue.
# (e.g., custom CA in your environment, missing root certs in container)

Issue 3: Agent crashes or gets killed

Symptoms:

  • Agent was healthy, then suddenly stale
  • Application logs show exception or out-of-memory
  • dmesg shows process killed (OOMKiller in Linux)

How to fix:

  1. Check if the agent process is still running:

    ps aux | grep python
# Look for your application process

  2. Check application crash logs:

    tail -100 /var/log/myapp.log | grep -i error

  3. Monitor memory usage:

    # While the app is running
top -p $(pgrep -f "your_app")
# Look for VIRT (virtual) and RES (resident) memory

  4. If memory is the issue:

    • Increase buffer timeouts so the agent flushes more frequently: 
      config = SecurityConfig(
    agent_event_buffer_max_time_sec=5,  # flush every 5 sec instead of 30
    agent_event_buffer_max_size=50,     # flush at 50 events instead of 100
)
    • Or reduce what's being buffered: 
      config = SecurityConfig(
    enable_metrics=False,  # disable metrics if only events matter
)

  5. If the agent thread crashes silently:

    • Enable agent debug logging and watch for exceptions
    • Check if there's a background thread exception handler in your application

Issue 4: Application is in passive mode

Symptoms:

  • Agent is healthy and sending events
  • Dashboard shows events in the Events page
  • But agent table shows zero events in 24h

Explanation: If passive_mode=True, detections are computed but not blocking. Telemetry is still sent, so the agent should be healthy. If the event count is zero, either:

  • No traffic is reaching the application
  • No detections are firing (all traffic is clean)
  • Metrics are disabled (enable_metrics=False)

This is not a bug; it's expected behavior. To verify the agent is working:

  • Check the Events page (/events) to see if any events are there
  • Enable passive_mode=False to see blocking events (changes behavior, only for testing)
  • Check application access logs to verify traffic is arriving

Issue 5: Dashboard lag or caching

Symptoms:

  • Agent logs show heartbeats being sent
  • Connectivity test passes
  • But dashboard still shows stale status

How to fix:

  1. Hard refresh the browser:

    • Cmd+Shift+R (Mac) or Ctrl+Shift+R (Windows/Linux)

  2. Clear browser cache:

    • Open DevTools (F12)
    • Right-click the refresh button → "Empty cache and hard refresh"

  3. Check dashboard for a known issue:

    • The dashboard updates agent status every 30 seconds
    • If the agent just went stale, it may take up to 2 minutes to reflect
    • Wait 2 minutes and refresh again

  4. Check the Core API directly:

    # If you have API access, query the agent status endpoint directly
curl -H "Authorization: Bearer $YOUR_API_KEY" \
  https://api.guard-core.com/api/v1/agents/status
# Look for your project and its last_seen timestamp

Still stuck?

If none of the above resolve the issue:

  1. Gather diagnostics:

    • Full agent debug logs (last 100 lines)
    • Application startup output
    • Network trace (curl -v output to the Core API)
    • Agent library version (pip show fastapi-guard)
    • Application runtime environment (Python version, OS, container info)

  2. Contact support:

    • Go to /settings and find the Support section
    • Include the diagnostics above

Next steps