Troubleshooting

Gateway Fails At Startup

Check:

• POSTGRES_DSN is set.
• Postgres is reachable.
• Redis is reachable.
• HTTP_PORT is valid.
• OBSERVABILITY_SAMPLE_RATE is between 0 and 1.
• Provider key encryption key ID and private key are either both set or both empty.
• TLS certificate and key are set when HTTP_ENABLE_TLS=true.

UI Fails At Startup

Check:

• DATABASE_URL is set.
• Better Auth secret is set.
• GitHub OAuth credentials are set.
• Prisma migrations have run.
• NEXT_PUBLIC_PROVIDER_KEY_ENCRYPTION_* values were present at build time if provider key encryption is used.

Login Redirect Loops

Check:

• public app URL and auth base URL match the browser URL,
• cookies are valid for the host,
• reverse proxy headers are correct,
• GitHub OAuth callback URL matches deployment URL.

API Route Returns 403

For admin routes:

• only SUPER_ADMIN can use /api/admin/*.

For organisation routes:

• non-super admins must have session.user.organisationId equal to the organisation ID in the path.
• the route must map to an RBAC resource.
• the action must be permitted by the user's role.
• manager and user permissions may require team membership or self-target conditions.

Gateway Returns 401

Check:

• API key is sent in Authorization: Bearer, X-API-Key, x-goog-api-key, or Gemini key query parameter.
• API key exists.
• API key is not revoked.
• API key has not expired.
• gateway auth cache has been invalidated after changes or TTL has elapsed.

Gateway Returns model_not_allowed

Check:

• requested model name matches the model name, not necessarily the upstream slug;
• model belongs to the API key's organisation;
• ApiKeyModelAccess grant exists for the key and model;
• model access cache was invalidated after changes.

Gateway Returns Provider Config Errors

Common checks:

• model has a provider;
• provider is enabled;
• model has a provider API key;
• provider API key is not revoked;
• encrypted provider key can be decrypted by the gateway private key;
• UI public key ID matches gateway private key ID;
• provider type is supported by the gateway.

Rate Limits Trigger Unexpectedly

Check:

• global policy in RATELIMIT_GLOBAL_POLICY;
• organisation, team, API key, model, and MCP policies;
• client IP and trusted proxy configuration;
• whether max_tokens exceeds policy payload cap;
• Redis availability;
• stale policy cache.

Budget Or Quota Blocks Requests

402 usually indicates budget exceeded. 429 usually indicates quota exceeded.

Check:

• active budgets/quotas for organisation, team/user, and API key scopes;
• period and timezone;
• amount or limit;
• current window used and reserved;
• stale reservations;
• model pricing if cost budgets are expected to reserve cost.

Usage Records Missing

Check:

• Redis is available for the usage collector;
• collector enqueue or flush errors in metrics/logs;
• request reached the usage stage;
• API key ID is present;
• provider returned usage or estimator/normalizer can infer usage;
• Postgres insert errors.

MCP Request Fails

Check:

• MCP server ID or slug in URL;
• server is enabled;
• API key has MCP access;
• allowed/blocked tool lists;
• transport configuration;
• HTTP endpoint reachability or stdio command availability;
• MCP policies and payload size;
• MCP auth config.

Smart Routing Does Not Run

Check:

• organisation routing enabled flag is true;
• API key has policies.routing;
• requested model type has a matching routing policy;
• candidates are the same model type;
• API key has access to candidate models;
• candidate models have valid provider config;
• routing cache was invalidated or TTL elapsed.

No Metrics

Check:

• OBSERVABILITY_PROMETHEUS_ENABLED=true;
• /metrics is reachable;
• Prometheus target odock-server is up;
• gateway is not exporting duplicate/conflicting OTLP metrics.

No Traces

Check:

• OBSERVABILITY_OTEL_TRACES_EXPORTER=otlphttp;
• OBSERVABILITY_OTEL_ENDPOINT points to the collector;
• collector health endpoint is up;
• Tempo is healthy;
• sample rate is not 0.

No Logs In Loki

Check:

• gateway logger outputs to stdout;
• Promtail is running;
• Promtail can read Docker container logs;
• Loki is healthy;
• Grafana Loki datasource is healthy.

Provider Key Encryption Fails In Browser

Check:

• page is served from HTTPS or localhost;
• crypto.subtle is available;
• public key env var is present in the built UI bundle;
• public key PEM is valid;
• key ID is present.

For LAN HTTP testing, browsers may block WebCrypto. Use HTTPS or a browser development override for that origin.