Your /health Endpoint Is Lying

To make the failure concrete, imagine an internal platform where one user request can depend on an API, a background worker, an enterprise integration, and data that arrives through a scheduled synchronization job. Each component has its own health signal. The user, however, depends on the entire path.

Four services are running. Every container is green. The health dashboard says 100%.

Users still cannot complete the job they came to do.

The API accepts their request. A worker has stopped making progress. One enterprise credential expired overnight. The data behind another feature has not refreshed since yesterday. None of that changes this endpoint:

@app.get("/health")
async def health() -> dict[str, str]:
    return {"status": "ok"}

The endpoint is not technically lying. The process is alive, and it returned exactly what the code told it to return.

It is lying in the way that matters. People read “healthy” as “the product works,” while the endpoint proves only “this process can answer one easy request.”

I learned to separate those claims while working on a platform with that shape. A fluent response at the front of the system could hide an expired credential or stale source behind it. Later work on an enterprise expense platform made the same lesson concrete around database readiness, worker progress, and deferred jobs.

A useful health model is not one boolean. It is a set of questions with different consequences.

First ask what will consume the answer

A health check should not begin with an endpoint. It should begin with the decision somebody will make from its result.

Different consumers need different answers:

One response cannot answer all five well.

If a database outage makes liveness fail, an orchestrator may restart every API instance. The database remains unavailable. The restarts add connection pressure and remove useful diagnostics.

If the liveness path stays responsive while the application's real work loop is stuck, a broken process can remain in service forever.

If one optional provider makes readiness fail, the load balancer can remove every otherwise useful instance. A feature-level problem becomes a full outage.

The right check depends on the action attached to failure.

Liveness means “restart me”

The cleanest definition of liveness is operational:

If this check keeps failing, restarting this process is likely to help.

That makes liveness deliberately narrow. It can check that the process and event loop are responsive. It can detect an unrecoverable internal deadlock. It should not usually call every dependency.

@app.get("/health/live")
async def live() -> dict[str, str]:
    return {"status": "ok"}

That simple handler is useful if reaching it proves the server can schedule and run work. Some runtimes need a deeper internal progress marker. The principle stays the same: test the process, not the world around it.

The official Kubernetes probe documentation says liveness determines when to restart a container. It also warns that bad liveness checks can create cascading failures under load.

A database connection does not belong in liveness merely because the application uses a database. Restarting the API will not repair PostgreSQL. A third-party outage does not belong there either. Restarting all clients can make that provider's recovery harder.

Keep liveness cheap, local, and difficult to fail for an external reason.

Readiness means “send me traffic”

Readiness asks a different question:

Can this instance serve the traffic it is about to receive correctly enough?

A failed readiness check should remove the instance from traffic without killing it. Kubernetes does exactly that for Pods behind a Service.

For a typical API, readiness may include:

• the application finished startup;
• required configuration loaded and passed validation;
• the primary database is reachable within a short timeout;
• the expected database schema is present;
• the process is using the intended database role;
• local resources are not exhausted beyond a safe limit.

@app.get("/health/ready")
async def ready() -> Response:
    checks = await run_required_checks(timeout_seconds=1)

    if not checks.all_required_ok:
        return JSONResponse(
            status_code=503,
            content={"status": "not_ready"},
        )

    return JSONResponse(
        status_code=200,
        content={"status": "ready"},
    )

This is sample code. Real checks should run concurrently where they are independent, use strict timeouts, and avoid expensive queries.

There is a trap here. A readiness endpoint that calls six remote services on every probe creates a high-frequency dependency fan-out. The health system can become meaningful production traffic. It can also mark every API instance unready because one optional integration is slow.

Only a dependency required for the instance's core traffic should usually gate readiness. Everything else belongs in capability health.

Startup is its own state

Some applications need time to start. They may load a model, verify a schema, warm a cache, or build a local index.

Using a large initial delay for liveness guesses how long startup will take. The guess is either too short on a slow day or needlessly long on a fast one.

A startup check answers whether initialization has completed. Kubernetes waits for a startup probe to succeed before it begins liveness and readiness probes. That protects a slow but valid startup from being mistaken for a dead process.

Startup failure should still be bounded. “The application is warming up” cannot mean “wait forever while a broken migration loops.” Report the current phase internally and fail after a deliberate limit.

Dependencies are not one shared fate

Most real applications depend on several things:

• a primary database;
• object storage;
• a message broker or worker queue;
• email or notification delivery;
• an identity provider;
• external business systems;
• scheduled data ingestion;
• model or search providers.

Treating all of them as one healthy: true value throws away the information operators need.

I prefer a capability view:

{
  "status": "degraded",
  "capabilities": {
    "claims": "available",
    "document_upload": "available",
    "notifications": "delayed",
    "directory_sync": "stale"
  }
}

This response describes the product more honestly. Core writes may still work while notifications are delayed. Existing users may still sign in while directory synchronization is stale. A read-only report may remain available during a provider outage.

The public endpoint should expose only safe, coarse states. Detailed provider names, hostnames, error text, credential dates, and internal topology belong in an authenticated operator view.

The distinction also helps with graceful degradation. If an optional recommendation service is down, the application can hide that feature or return a clear partial response. It does not need to reject every request.

A worker needs a progress check, not an HTTP check

Background workers often have no meaningful request endpoint. A worker process can respond to /health while its queue loop is stuck.

Worker health comes from durable progress:

• when it last polled for work;
• when it last acquired a lease;
• when it last completed work;
• age of the oldest ready item;
• number of expired leases;
• retry and dead-letter rates;
• work created versus work completed.

The most useful signal is often age, not count.

A queue with 500 items can be healthy if work is arriving and completing quickly. A queue with one item can be unhealthy if that item has waited for an hour.

queue depth             12
oldest ready age        18 seconds
last successful work     4 seconds ago
expired active leases    0

That tells a much better story than worker_up 1.

Worker checks should distinguish “no work arrived” from “work arrived and nothing consumed it.” A last-success timestamp alone becomes stale during a quiet period and can create a false alarm. Pair it with queue activity and polling evidence.

Credentials can be unhealthy before they fail

Many enterprise integrations do not break because the remote server is down. They break because a token, certificate, client secret, or delegated grant expired.

A connectivity probe can stay green until the exact moment of expiry. By then users are already affected.

Credential health needs a time dimension:

valid      credential works and is not near expiry
warning    credential works but rotation is due soon
failed     credential is expired, revoked, or rejected
unknown    status could not be verified safely

This is not a reason to expose expiry timestamps publicly. It is a reason to alert the team that can rotate the credential before failure.

The same idea applies to certificates, signing keys, storage quotas, and subscription limits. Health should include approaching failure when action is possible, not only current failure after the fact.

Freshness is different from availability

A data source can answer every request and still be wrong for the product because its data is stale.

Imagine a knowledge or employee directory sync that last succeeded two days ago. The database is up. The API is fast. Search returns results. The results no longer reflect the source system.

Availability asks whether data can be read. Freshness asks whether it is recent enough to trust.

A freshness check needs three values:

• the last successful source watermark;
• the expected update cadence;
• the maximum acceptable age for this use case.

Different data deserves different thresholds. A country list may remain useful for months. Access assignments may become unsafe within minutes. Do not create one global “stale after 24 hours” rule because it is easy to remember.

Freshness also needs an unknown state. If the source does not expose a trustworthy watermark, pretending the data is current is worse than admitting that freshness could not be proven.

Semantic health asks whether the answer is usable

Infrastructure checks can prove that the request completed. They cannot prove that the result makes sense.

A search system can return zero results because an index is empty. A document pipeline can mark work complete while extracting no text. A model provider can return syntactically valid output that fails the product's quality checks.

Semantic health is domain-specific. Examples include:

• a required catalogue contains plausible records;
• an index contains the expected classes of documents;
• recent outputs pass a bounded quality evaluation;
• a generated report contains all mandatory sections;
• synchronization did not silently drop most source records.

These checks should rarely sit inside a high-frequency readiness probe. They are better as scheduled monitors, release checks, or low-frequency synthetic evaluations.

The goal is not to prove that every answer is correct. It is to catch large failures that infrastructure health cannot see.

The highest layer is the user journey

The strongest health check asks whether a person can complete the task the system exists to support.

For an expense platform, a safe synthetic journey might create a disposable draft, read it back, and clean it up. For a helpdesk platform, it might submit a test question through an isolated account and verify that the expected workflow finishes.

This check crosses boundaries:

browser or client
    → edge
    → API
    → database
    → worker
    → required provider
    → visible result

It is also more expensive and more likely to have side effects. Run it at a lower frequency. Use dedicated test identities and data. Make cleanup reliable. Keep it away from real users and notifications.

A synthetic journey does not replace component metrics. It tells you that the product failed. Component health, logs, metrics, and traces help explain where.

That is where Tracing a Request You're Not Allowed to Log becomes relevant. Health detects a broad condition. A trace follows one run through the system without requiring the raw sensitive payload.

The seven layers I now look for

The complete model I use is this:

Not every product needs all seven on day one. The table is a way to find blind spots, not a demand for seven dashboards.

Start with liveness and readiness. Add worker progress when work moves out of the request. Add credential and freshness checks when external systems enter the product. Add synthetic journeys for the paths whose failure would matter most.

Health responses should be boring and safe

Public health endpoints are easy to overfill. Detailed errors feel helpful during development, then expose internal information in production.

A public response often needs only this:

{
  "status": "ready"
}

An authenticated operator response can include bounded detail:

{
  "status": "degraded",
  "checks": [
    {"name": "database", "status": "ok"},
    {"name": "notification_worker", "status": "delayed"},
    {"name": "directory_freshness", "status": "warning"}
  ],
  "observed_at": "2026-08-30T10:30:00Z"
}

Avoid raw exception messages. Do not include credentials, connection strings, internal URLs, tenant or employee identifiers, queue payloads, or provider responses.

Checks also need strict time budgets. A health endpoint that waits thirty seconds for a dependency can consume threads and connections during the exact outage when the service is under pressure.

Cache slow diagnostic results when appropriate, but expose their observation time. A cached check with no timestamp looks current even when the checker itself stopped.

Stable failures make alerts useful

One provider outage can break hundreds of jobs. Alerting once per failed job creates hundreds of pages for one cause.

I group recurring failures by a safe signature. The signature can include a stable error class, dependency category, operation, and sanitized failure code. It should not include the user, payload, or random exception text.

signature = hash(
    dependency_kind,
    operation,
    safe_error_code,
    failure_class,
)

The alert can then track first seen, last seen, occurrence count, affected capability, and whether somebody acknowledged it. A new signature creates a new incident signal. The same signature updates the existing one.

This reduces noise without hiding scope. The dashboard can still show how many operations failed. The pager does not need to repeat the same root condition 500 times.

Alerts should also reflect user impact. A single failed optional refresh may need a ticket. A stalled core workflow or failed readiness across all instances may need a page.

Google's SRE guidance on monitoring distributed systems recommends simple monitoring tied to clear purposes such as alerting, diagnosis, and trend analysis. That is a useful standard. A complex health score with no operational decision behind it is decoration.

Test the consequences

Health tests should verify more than response bodies.

• When liveness fails repeatedly, does the platform restart only the unhealthy process?
• When readiness fails, does traffic stop without restarting the instance?
• Does a database outage leave enough healthy capacity to recover?
• Does an optional provider failure degrade one capability instead of the whole product?
• Does a stuck worker raise backlog age even while its process remains alive?
• Does a near-expiry credential warn before users are affected?
• Does stale data become visible without leaking source details?
• Does the synthetic journey use isolated data and clean up safely?
• Are probe timeouts shorter than the systems calling them?

Run these tests under failure, not only in a healthy local environment. A readiness check that works when every dependency is fast may collapse when one dependency hangs.

Also test the health system itself. If the monitor stops running, can you tell? If its last observation is old, does the dashboard show that age instead of displaying the last green result forever?

What a green endpoint can honestly mean

/health is not useless. It is only too vague.

Give each endpoint one clear promise:

• /health/live: this process is responsive enough that a restart is not currently justified;
• /health/ready: this instance can accept its required traffic now;
• authenticated capability health: these product functions are available, degraded, stale, or failed;
• worker metrics: deferred work is making progress;
• synthetic journeys: a representative task can complete across the system.

No single response proves the whole product is correct. That is fine. Honest partial claims are more useful than one confident green light.

The endpoint at the beginning returned 200 while users were stuck. The fix is not to make that one endpoint check everything.

The fix is to stop asking one question and calling its answer “health.”