LLM Gateway Local Development

Status: Active local runbook Last updated: 2026-07-16

Sidecar integration reference:

• llm-gateway/docs/sidecar-agent-registration.md

1. Prerequisites

• Go 1.23+
• just
• curl
• python3
• docker + docker compose (for local Postgres)

From repo root:

cd /path/to/gateway
just --justfile llm-gateway/justfile --list

2. Local Modes

2.1 Postgres gateway mode

Start DB:

just --justfile llm-gateway/justfile audit-db-up

Run gateway with registry backend and registration endpoint enabled:

just --justfile llm-gateway/justfile gateway-registry

Or run from local TOML runtime config:

just --justfile llm-gateway/justfile gateway-config

This mode enables:

• trusted registry in Postgres
• POST /v1/agents/registrations
• audit writes to Postgres (default in gateway-registry)
• Anthropic/OpenAI/ChatGPT backend endpoint routing

3. Generate signing key

SEED=$(just --justfile llm-gateway/justfile gen-key | awk -F= '/seed_b64url/{print $2; exit}')

4. Register agent in registry mode

Create provisioning token from the local registration token secret:

TOKEN=$(just --justfile llm-gateway/justfile reg-token | awk -F= '/^token=/{print $2; exit}')

Register the agent (uses SEED public key):

just --justfile llm-gateway/justfile register-agent "$TOKEN" "$SEED"

5. Build signed headers and run E2E

5.1 Registry mode

just --justfile llm-gateway/justfile e2e-prepare-registry "$TOKEN" "$SEED"
just --justfile llm-gateway/justfile e2e-success
just --justfile llm-gateway/justfile e2e-replay
just --justfile llm-gateway/justfile e2e-missing-signature

6. Relay (/v1/relay)

/v1/relay is the identity-firewall path for non-LLM HTTP. A registered agent signs a request that names an absolute destination in X-Viper-Upstream-URL. The gateway verifies the signature (destination must be covered), runs proxy.pre plugins, forwards the call, and writes an audit row with traffic_kind=relay plus structured upstream_host / upstream_url.

6.1 Sign the destination

guardian-sigtool sign covers the destination when you pass -upstream-url:

cd llm-gateway/service
GOWORK=off go run ./cmd/guardian-sigtool sign \
  -method POST \
  -url 'http://localhost:10000/v1/relay' \
  -keyid "$DID" \
  -seed-b64url "$SEED" \
  -content-type 'application/json' \
  -body '{"hello":"relay"}' \
  -state-cid "$STATE_CID" \
  -upstream-url 'https://httpbingo.org/post'

Sending X-Viper-Upstream-URL without covering it in the signature fails with 400 destination_not_signed.

Do not use http://127.0.0.1:… as the destination against a real gateway: the SSRF floor returns 403 destination_blocked. Prefer a public echo such as https://httpbingo.org/post, or an RFC1918 host that is a legitimate internal API. just mock-relay-upstream (:18081) is only useful when the relay transport bypasses that floor (unit tests).

6.2 Operator recipes

Prerequisites: gateway-registry on :10000, and preferably control-plane on :10010 sharing the same Postgres (for audit asserts).

TOKEN=$(just --justfile llm-gateway/justfile reg-token | awk -F= '/^token=/{print $2; exit}')
SEED=$(just --justfile llm-gateway/justfile gen-key | awk -F= '/seed_b64url/{print $2; exit}')
# or reuse a stable seed you already registered

just --justfile llm-gateway/justfile e2e-relay-suite "$TOKEN" "$SEED"

The suite registers the agent, signs a relay request, then checks:

Case	Expect
success	HTTP 200; audit proxied / verified
unsigned destination	HTTP 400 destination_not_signed; audit local_deny
untrusted (no Signature)	HTTP 401 sig_missing_headers
plugin deny	skipped when E2E_RELAY_ALLOW_DENY_SKIP=1 (suite default); otherwise required

Each case prints expect/got; the suite ends with a PASS/FAIL/SKIP summary. Audit lines poll control-plane when reachable (auth-disabled local CP needs no bearer). Audit asserts are required by default; set RELAY_E2E_ALLOW_AUDIT_SKIP=1 only for HTTP-only smoke when CP auth is unavailable. e2e-relay-suite soft-skips plugin-deny by default (E2E_RELAY_ALLOW_DENY_SKIP=1); durable deny coverage is gateway-dev-e2e -target relay.

Individual recipes: e2e-relay-prepare-registry, e2e-relay-success, e2e-relay-unsigned-destination, e2e-relay-untrusted.

6.3 CI / gateway-dev-e2e

Durable regression coverage lives in gateway-dev-e2e (not the just suite):

cd llm-gateway/service
export GUARDIAN_GATEWAY_URL=http://localhost:10000
export GUARDIAN_CONTROL_PLANE_URL=http://localhost:10010
# Local auth-disabled CP: any non-empty placeholder satisfies config validation.
export GUARDIAN_CONTROL_PLANE_BEARER_TOKEN=local

GOWORK=off go run ./cmd/gateway-dev-e2e -target relay

Optional: E2E_RELAY_UPSTREAM (default https://httpbingo.org/post). Plugin host deny needs --allow-shared-dev-mutations plus the plugin signer env used by blocking targets.

6.4 Reading audit

Console: open the agent, filter Logs / Destination for traffic_kind=relay.

API:

curl -sS "http://localhost:10010/v1/agents/$AGENT_ID/audit-exchanges?traffic_kind=relay&limit=5"
# If CP auth is enabled, add: -H "Authorization: Bearer $GUARDIAN_CONTROL_PLANE_TOKEN"

Expect upstream_host as hostname-only on successful route_type=proxied rows. Unsigned-destination denials keep traffic_kind=relay with empty upstream fields.

6.5 Viper path

Agent-side: viper proxy --relay and egress capture (viper claude --egress-capture). Go binaries on macOS (gh) need viper ca trust (login keychain, SSL-only); see GAT-265. This runbook stays on gateway operator flows.

6.6 Deny a host

Install a proxy.pre plugin that rejects relay.host (see llm-gateway/docs/local-plugin-e2e.md and the gateway-dev-e2e relay plugin-deny scenario). Denied calls audit as route_type=local_deny with the plugin’s decision code (unit tests use relay_host_denied).

6.7 Observability

Relay spans/metrics carry gateway.traffic_kind and relay.destination_host. See llm-gateway/docs/otel-lifecycle.md. Operator e2e does not hard-assert OTEL artifacts.

7. Useful commands

Gateway health:

just --justfile llm-gateway/justfile gateway-health

Show migrations applied:

just --justfile llm-gateway/justfile audit-db-migrations

Run tests:

just --justfile llm-gateway/justfile test

8. OTEL Validation Workflow (OBS-01.8 + OBS-02.3)

Run the fast headless validation flow (collector + artifact contract check with propagation matrix assertions):

just --justfile llm-gateway/justfile obs-otel-validate

Expected success output includes:

• PASS: telemetry contract validated
• propagation scenarios validated: 9
• artifact paths:
  • llm-gateway/observability/.artifacts/traces.jsonl
  • llm-gateway/observability/.artifacts/logs.jsonl

Run the UI-backed validation flow (Jaeger + terminal-correlated logs):

just --justfile llm-gateway/justfile obs-otel-validate-ui

This deterministic UI validation run:

• generates proxy telemetry via the collector export test harness
• defaults to 3 agents with 5 requests each (15 total)
• validates telemetry contract artifacts
• renders span/log correlation plus per-agent request counts in terminal
• prints Jaeger URL (http://localhost:16686/search)

UI artifacts are written to:

• llm-gateway/observability/.artifacts/traces.jsonl
• llm-gateway/observability/.artifacts/logs.jsonl

Run both validation flows in sequence:

just --justfile llm-gateway/justfile obs-otel-validate-all

Run the fast observability contract test (topology + required fields) directly:

just --justfile llm-gateway/justfile obs-otel-contract-test

Run propagation contract tests directly (including upgrade-handshake propagation contract checks):

just --justfile llm-gateway/justfile obs-otel-propagation-contract-test

Run p95 instrumentation overhead check (delta <= 5ms by default):

just --justfile llm-gateway/justfile obs-otel-overhead-check

Override samples or threshold:

just --justfile llm-gateway/justfile obs-otel-overhead-check samples=300 max_delta_ms=4.5

Run the reproducible OBS-02.3 regression flow in one command:

just --justfile llm-gateway/justfile obs-otel-regression-check

This runs:

1. obs-otel-validate (artifact + propagation contract validation)
2. obs-otel-propagation-contract-test (Go propagation contract checks)
3. obs-otel-overhead-check (p95 delta <= 5ms)

This command writes a report artifact at:

• llm-gateway/observability/.artifacts/overhead-report.json

When recording OBS-02.3 evidence in an issue/PR note, include:

• command run (obs-otel-regression-check)
• pass/fail summary from validation output
• overhead report summary from overhead-report.json (delta_p95_ms, max_allowed_delta_ms, pass)

Collector lifecycle helpers:

just --justfile llm-gateway/justfile obs-otel-collector-up
just --justfile llm-gateway/justfile obs-otel-collector-down

Compatibility alias:

just --justfile llm-gateway/justfile obs-otel-demo

obs-otel-demo maps to obs-otel-validate-ui.

Re-render existing UI/demo artifacts without generating new telemetry:

just --justfile llm-gateway/justfile obs-otel-demo-show

Override UI/demo traffic volume:

OBS_OTEL_DEMO_AGENT_COUNT=4 OBS_OTEL_DEMO_REQUESTS_PER_AGENT=8 \
  just --justfile llm-gateway/justfile obs-otel-validate-ui

UI stack lifecycle helpers:

just --justfile llm-gateway/justfile obs-otel-demo-up
just --justfile llm-gateway/justfile obs-otel-demo-down

9. Local config files

Tracked templates:

• llm-gateway/service/configs/runtime.example.toml
• llm-gateway/docs/local-plugin-e2e.md

Local mutable files (gitignored):

• llm-gateway/service/configs/runtime.local.toml

10. Troubleshooting

• invalid config: registry-db-dsn is required when registry-backend=postgres

  • pass -registry-db-dsn, or use just --justfile llm-gateway/justfile gateway / gateway-registry

• registration_invalid_token

  • ensure token was minted with the configured registration token secret
  • ensure token not expired

• registration_conflict

  • agent_id conflicts with existing records, or active key ownership conflict in registry

• sig_invalid

  • ensure registered key matches signing seed
  • ensure keyid uses did:key:z...

• obs-otel-validate fails before telemetry validation

  • rerun with shell trace for more detail: bash -x llm-gateway/observability/run_otel_validation.sh llm-gateway
  • ensure Docker is running and ports 4317/4318 are free for the local collector
  • ensure go test ./internal/gatewayhttp -run '^TestProxyCollectorExportValidation$' can reach OBS_OTEL_COLLECTOR_ENDPOINT (default 127.0.0.1:4318)
  • if overriding collector volume env vars, ensure OBS_OTEL_COLLECTOR_AGENT_COUNT * OBS_OTEL_COLLECTOR_REQUESTS_PER_AGENT >= 9 so the propagation scenario matrix is fully exercised

• obs-otel-validate-ui (or obs-otel-demo) fails before telemetry validation

  • rerun with shell trace for more detail: bash -x llm-gateway/observability/run_otel_demo.sh llm-gateway
  • ensure Docker is running and ports 14317/14318/16686 are free for demo collector + Jaeger
  • ensure go test ./internal/gatewayhttp -run '^TestProxyCollectorExportValidation$' can reach OBS_OTEL_DEMO_COLLECTOR_ENDPOINT (default 127.0.0.1:14318)
  • if overriding demo traffic env vars, ensure OBS_OTEL_DEMO_AGENT_COUNT * OBS_OTEL_DEMO_REQUESTS_PER_AGENT >= 9

11. Codex/OpenAI local sidecar profile

Run the thin test sidecar in Codex profile:

cd llm-gateway/service
GOWORK=off go run ./scripts/test_sidecar.go \
  -listen 127.0.0.1:18082 \
  -gateway-base-url http://127.0.0.1:10000 \
  -seed-b64url "$SEED" \
  -register-token "$TOKEN" \
  -client-profile codex

Optional fallback key injection (only used when incoming Authorization is missing):

GOWORK=off go run ./scripts/test_sidecar.go \
  -listen 127.0.0.1:18082 \
  -gateway-base-url http://127.0.0.1:10000 \
  -seed-b64url "$SEED" \
  -register-token "$TOKEN" \
  -client-profile codex \
  -openai-api-key "$OPENAI_API_KEY"

Project-local env (example):

export OPENAI_BASE_URL=http://127.0.0.1:18082/v1
export OPENAI_API_KEY=sk-...

client-profile=codex rewrites OpenAI-style request paths to gateway provider-prefix routes under /v1/openai_api/v1/....