AI Observability in PostHog Web

The PostHog web app is where you actually read your AI product's traces. Open AI Observability in PostHog and you land on a dashboard of cost, latency, token usage, and error rate, with tabs for traces, sessions, users, tools, and errors alongside it.

From there you can open any trace to see the full conversation – every generation and span in order, with inputs, outputs, model, tokens, and cost on each one. Traces carry the person who triggered them, so you can jump straight to their session replay or any exception the request threw.

Before you start

1. Set up AI Observability – install a PostHog LLM SDK or provider integration so traces are captured.
2. Open the web app – no install needed, just sign in to PostHog.

What you can do here

• Analyze LLM performance – track cost, latency, tokens, and error rate across models, features, and users.
• Read traces and sessions – follow a single request end to end, or a whole multi-visit session.
• Test prompts in the playground – rerun a prompt against a different model or wording without redeploying.
• Cluster similar traces – group conversations by what users were actually trying to do, so failure patterns stand out.
• Check sentiment – see which conversations left users frustrated, classified automatically.
• Summarize a trace – get a written summary and flow diagram of a long agent run instead of reading every message.
• Analyze traces with PostHog AI – ask PostHog AI, right here in the web app, about spend, latency, and errors across your trace data, and have it write custom parsers for trace formats it doesn't recognize. It's an interaction mode inside the web app, not a separate surface.

Self-driving reports

A scout watches your trace data, and when a slice of it regresses, it writes up what it found as a report in the Self-driving inbox. To get these reports, set up Self-driving.

Unlike error tracking, this scout stops at the report. It does not open a draft pull request.

How a regression becomes a report

The scout looks for five kinds of change, sliced by the dimensions it finds in your data – model, provider, feature, product area, and any custom properties you attach:

• Cost – spend on a model or feature stepping above its own baseline.
• Latency – a per-model latency band widening, rather than an aggregate that averages fast and slow models together.
• Errors – a rise in failed calls, or a new error class appearing.
• Volume – generation counts moving in a way the user count doesn't explain.
• Evaluation performance – a scored behavior getting worse, which evaluations catch as they run.

It only files a report when it has localized the change to a specific slice and can show the numbers behind it.

Why there is no automatic pull request

A cost step, a latency band, or a failing evaluation is an investigation, not a known fix. The cause might be a model change, a prompt, a retry loop, a new customer with unusual inputs, or nothing wrong at all. There's no safe code change to guess at.

So this scout files each finding as an investigation report and routes it to a reviewer – the person who owns that model, feature, or evaluation – rather than kicking off a coding agent. You decide what, if anything, to change.

What you can do with a report

• Read the report – the slice that regressed, when it moved, and the traces, generations, and query results behind it.
• Ask questions – start a discussion with an agent about the report to dig further before you act.
• Follow the activity – the report's activity log shows what happened to it and when.
• Decide what happens next – change a prompt, roll back a model, adjust an evaluation, or dismiss the report. It's your call, and nothing happens until you make it.

Related

• Start with the data model if traces, spans, and generations are new to you.
• Query the same data from your editor over MCP.
• Drive the same things from your own code with the API.
• Score your traces automatically with evaluations.