Helix

Overview

Helix is a full-stack observability platform for LLM-powered applications. A lightweight SDK wraps multi-provider LLM calls (Anthropic, OpenAI, Google, and 300+ models via OpenRouter), captures inference metadata like latency, token usage, errors, and session IDs, and ships it asynchronously to an ingestion pipeline backed by PostgreSQL. A Next.js chatbot demonstrates the SDK, and Grafana dashboards expose real-time latency, throughput, and error metrics.

The design goal drove every architectural decision: production-grade observability over LLM calls without adding latency to the user-facing response path. The response path and the log write path are fully decoupled, so a broker outage never blocks a reply.

300+

Models via one OpenRouter key

Providers behind one client

0ms

Added to response path

1 cmd

Full-stack Docker boot

Architecture

A Turborepo pnpm monorepo: three apps and three shared packages.

apps/
  web/         Next.js 16 chat UI (App Router, Turbopack)
  api/         Fastify API gateway: conversation CRUD, SSE streaming
  ingestion/   Kafka consumer + PostgreSQL writer
packages/
  sdk/         Unified LLM client (Anthropic, OpenAI, Gemini, OpenRouter) + redaction
  db/          Drizzle schema + TimescaleDB hypertable SQL
  types/       Shared Zod schemas + inferred TypeScript types
infra/
  docker-compose.yml   one-run local stack
  k8s/                 Helm charts for EKS (one per service)
  grafana/             dashboards + provisioning

Data flows one way to the user and out-of-band to storage:

User -> Next.js (SSE) -> Fastify API Gateway -> SDK Wrapper -> LLM Provider
                                                    |
                                         Kafka: llm.inference.logs
                                                    |
                                       Ingestion Service -> PostgreSQL -> Grafana

Decoupled write path

The SDK emits one Kafka event per call, fire-and-forget, and ingestion consumes and persists it out-of-band. The gateway never writes inference_logs directly. Kafka is the buffer: an ingestion slowdown grows consumer lag rather than dropping data or stalling replies, and consumers scale independently in one consumer group.

Ingestion Flow

1
Call completes in the SDK
The SDK measures firstTokenMs and latencyMs, builds one versioned KafkaLogPayload envelope (schemaVersion, eventId, emittedAt plus call metadata), and redacts the previews.
2
Fire-and-forget emit
The payload is emitted to the llm.inference.logs topic with a 2000ms producer timeout. If Kafka is slow or down, the SDK logs to stderr and moves on: the user already has their response.
3
Validate and redact
apps/ingestion consumes the topic and runs the pipeline: JSON parse, KafkaLogPayloadSchema Zod validation, a second PII redaction layer, then insert via Drizzle.
4
Persist or dead-letter
The DB write is retried up to 3 times. Validation failures or persistent write failures route to the llm.inference.dlq topic with x-dlq-* headers carrying the original raw bytes.
5
Never stall
processMessage never throws: every path resolves to a write or a DLQ route, so the consumer offset always advances and the loop never stalls. Grafana reflects new calls within seconds.

Key Features

Unified multi-provider SDK

One LLMClient interface over Anthropic, OpenAI, and Gemini direct SDKs plus OpenRouter, which reuses the openai SDK against openrouter.ai/api/v1 for one key and 300+ models. Pure TypeScript, no HTTP layer of its own.

Zero-latency observability

Inference metadata is captured and shipped without touching the response path. The emit is asynchronous and fire-and-forget, so logging can never slow or block the user reply.

TimescaleDB hypertable

inference_logs is a hypertable partitioned on request_at. Time-bucketed dashboard queries (p50/p95/p99 latency, throughput per minute) run orders of magnitude faster than on a plain table, with no change to the query API.

Streaming end to end

SSE from the browser through the gateway to the SDK, which intercepts the stream to measure firstTokenMs. A client disconnect mid-stream aborts the upstream LLM call and emits a cancelled log.

Two-layer PII redaction

Email, phone, SSN, and credit-card patterns are redacted once in the SDK before emit and again in ingestion before the DB write. A raw payload never reaches the database even if one layer is bypassed.

Graceful provider gating

hasProvider(name) lets the gateway refuse a request before hijacking the SSE response, returning 503 provider_not_configured so the client shows a friendly message instead of a silent stream close.

Redis context cache

A trailing 10-message window is read from ctx:{conversation_id} (24h TTL), removing a DB round-trip per message. A cold miss rebuilds the window from redacted DB history.

Conversation lifecycle

List, resume, and archive conversations from the UI. Cancel closes the SSE stream, persists partial output, and sets status='cancelled'. Cancel never deletes: a cancelled conversation stays viewable and posting to it returns 409.

Data Model

Four PostgreSQL tables, Drizzle schema as the source of truth:

conversations: one row per chat session, holding provider, model, status.
messages: every user, assistant, and system turn, content always redacted.
inference_logs: one row per LLM API call, a TimescaleDB hypertable.
providers: provider configuration, name, base URL, active flag.

No primary key on the hypertable

TimescaleDB hypertables cannot carry a primary key that excludes the partitioning column, so inference_logs keeps an id with a default but no PK constraint. Idempotency is handled upstream via the payload eventId rather than a database uniqueness guarantee.

Fire-and-forget tradeoff

Guaranteeing the user response is never blocked means a Kafka outage drops log events rather than queueing them, so observability data can be lost during an incident. Acceptable because observability is non-critical relative to the user reply. A durable disk spool or outbox is the planned improvement.

Outcome

Helix demonstrates a production-shaped LLM observability system: a unified multi-provider SDK, an event-driven ingestion pipeline with Zod validation and a dead-letter queue, a TimescaleDB hypertable tuned for time-series dashboard queries, and Grafana visibility into latency, throughput, and errors. The central invariant holds throughout: the log write path never blocks the user response. Stateless services, Kafka backpressure, and Helm charts for AWS EKS make the horizontal-scaling story concrete rather than aspirational.

Overview

Architecture

Ingestion Flow

Call completes in the SDK

Fire-and-forget emit

Validate and redact

Persist or dead-letter

Never stall