Observability: Studio, Pulse and Sentinel

B11 was "when something goes wrong, nobody can tell me what happened". The honest version of that problem is that the information existed, in a log file, on a server, in a format nobody could search, during an hour when the person who could read it was asleep.

A generated project mounts three dashboards and a health endpoint. This chapter is what each one answers, and the discipline that makes them useful rather than decorative.

Four questions, four tools

• /studio
• What is actually in the database?
• /pulse
• What is this request doing, and why is it slow?
• /sentinel
• Who is attacking us, and what did we do about it?
• /health
• Is everything this process depends on alive?

They answer different questions, and reaching for the wrong one is how a debugging session takes three hours. "The invoice shows the wrong total" is Studio. "The invoice page takes four seconds" is Pulse.

Studio: the database, as it is

The embedded database browser at /studio lists tables, browses rows, follows foreign keys and runs queries. It is the fastest way to answer a question of the form "did that actually save".

The habit worth forming: before debugging the code, look at the row. Half of all "the API is broken" reports are a row that is exactly what the code wrote, and the bug is upstream in what was sent.

Pulse: where the time went

Pulse mounts at /pulse and records request traces, database queries, runtime metrics and errors. It is the tool for the question "why is this slow", and it answers it in the only way that helps: by showing which part took the time.

The three views that earn their keep:

The request trace. One request, with its queries listed and timed. An endpoint taking 800ms turns out to be forty queries of 20ms, which is the N+1 from chapter twenty two, now visible rather than suspected.

The slow query list. Sorted by total time rather than by worst case, because a 5ms query run ten thousand times costs more than a 2 second report nobody runs.

Errors with their context. Not just the stack, but the request that caused it.

pulse.WithExcludePaths("/studio/*", "/sentinel/*",
    "/docs/*", "/pulse/*")

The dashboards exclude each other, which sounds obvious and is a real bug otherwise: an observability tool observing its own polling fills itself with its own traffic.

Pulse can keep its data in SQLite so traces survive a restart, or stay in an in-memory ring buffer for peak throughput. The trade is the usual one: the ring buffer costs nothing and forgets, the file costs writes and remembers.

Sentinel: the attacks you were not watching

Sentinel is the security suite: a web application firewall, rate limiting, an auth shield and anomaly detection, with a dashboard at /sentinel.

What it changes is not that you are attacked. It is that you can see it. A login endpoint takes credential stuffing from the day it is public, and without a dashboard the only evidence is a slightly busier log.

Two configuration details are worth knowing before you meet them.

Development gets relaxed limits. A rate limit tight enough for production blocks you while you are testing, and a developer who has been blocked by their own firewall usually turns the whole thing off.

Its storage is its own. Sentinel persists the threat log, blocked IPs and audit trail through its own adapter, not the application's database handle. Left unset it falls back to a local SQLite file, which is ephemeral inside a container, so every redeploy drops the threat log and the blocked-IP list. Point it at the real database in production.

Health: the one endpoint a load balancer reads

{
  "status": "degraded",
  "components": {
    "api":      { "state": "ok" },
    "database": { "state": "ok", "latency_ms": 2 },
    "redis":    { "state": "off",
                  "detail": "REDIS_URL is empty, so caching, jobs
                             and cron are off on purpose" },
    "jobs":     { "state": "unknown" }
  }
}

The distinction that makes this endpoint useful is between off and broken. Redis absent because nobody configured it is a working project without caching. Redis configured and unreachable is an incident. A health check that reports both as "red" gets ignored within a week, and a health check that reports both as "green" is worse.

The logs, and the ID that joins everything

Every request carries a request ID, returned in a header and attached to every log line it produces. That is what turns "it failed around two o'clock" into one search.

grit logs                # the tail of this run
grit logs --errors       # only failures, with stack traces
grit logs --grep 500     # lines matching something

grit start records everything the API and the frontends print into.grit/logs/dev.log and keeps the previous run beside it, which matters more than it sounds: the interesting output is usually from the run that just crashed and scrolled away.

Making it a habit

Observability is not installed, it is practised. Four habits that cost nothing:

Read Pulse after building a feature, not after a complaint. The N+1 you find on the day you write the code costs ten minutes.

Check health after every deploy. Not "the site loads", the endpoint.

Look at Sentinel weekly. You will learn what normal traffic looks like, which is the only way to recognise abnormal traffic.

Put the request ID in your error messages. Then a screenshot from a user is actionable.

grit seed Invoice --count 5000
grit start

Assignment

Commit your work with the message Observability - Assignment One, exactly as written, so the leaderboard can score it.