Investigate one request across usage, traces, and logs
Follow a single request from the Odock UI into Tempo and Loki.
Investigate one request across usage, traces, and logs
Use this workflow when an application team gives you a request id, a failing response, or a narrow incident window and you need precise evidence.
Start in Usage Records when possible.
Find the request row, then open the record detail page.
Copy the key identifiers from the record.
The useful fields are:
Request IDTrace IDif present- provider and model
- timestamp and latency
- routing outcome
Open the matching trace in Grafana or Tempo.
If the UI exposes a direct trace link, use it. Otherwise open Tempo and search for the copied Trace ID.
Inspect the span tree.
Start with:
http.requestfor total durationgateway.route.resolvefor routing behaviorgateway.provider.requestfor upstream latency or errorsgateway.plugin.executefor plugin overheadgateway.security.modulefor allow or block decisions
If the trace is missing, remember that traces may be sampled. Continue with logs and the usage record.
Open Loki and filter by request_id.
This usually gives the fastest textual evidence for the same request across gateway stages.
Compare the three surfaces before drawing a conclusion.
Use:
- Usage Records for the exact final outcome, tokens, and cost
- Traces for path and latency inside the gateway
- Logs for detailed stage-by-stage events and error text
Common Conclusions
| Evidence pattern | Likely conclusion |
|---|---|
| Usage record shows reroute, trace shows repeated provider attempts | Upstream provider instability or route fallback |
Trace shows high gateway.plugin.execute duration | Plugin overhead inside the gateway |
Logs show decision=block in a security module | Expected enforcement, not an infrastructure outage |
| Usage record exists but no trace is found | Trace sampling or trace pipeline issue |
Next, if the issue looks provider-specific, continue with Investigate provider latency or errors.