Close the observability gap the incident exposed
The question nobody could answer during the incident, made answerable before the next one.
The ticket
Add the instrumentation needed to answer the question the team could not answer during the incident.
Acceptance criteria
- The question can be answered from a dashboard or a query, without reading code
- The instrumentation carries enough context to narrow by tenant or request
- The query is saved and linked from the runbook
- Cardinality is bounded so the new fields do not blow up storage
What lands as proof
The saved query returning an answer for the incident window, which proves the gap is actually closed.
Why teams defer it
- The gap only exists during an incident, and after one nobody wants to look at it again for a while.
- Adding fields has a cost in storage and cardinality that needs somebody to own the tradeoff.
Questions
- What does the agent actually change?
- The ticket is scoped to one outcome: add the instrumentation needed to answer the question the team could not answer during the incident. Work that serves that outcome is in scope, and anything outside it is left for a separate ticket, so the pull request stays reviewable.
- How do I know the work is done?
- The pull request carries the evidence, not only the diff. Here that means the question nobody could answer at 3am is now answerable, so a reviewer can confirm the result without reproducing the work locally.
- How much oversight does this need?
- The run stops once the plan is written. Somebody reads the approach and approves it before any code exists, which is the cheapest moment to redirect the work.