Langfuse Review: Six Months of LLM Tracing in Production

The verdict first

I am still running Langfuse, and I would run it again tomorrow. It is the one piece of infrastructure in my stack that removed a category of problem rather than adding a feature.

When a tenant tells me an article came out wrong, I do not guess. I open the trace and see the exact prompt, the retrieved chunks in context, the model that answered, the token counts and the score my judge assigned. That takes seconds. Before Langfuse it took an afternoon of log archaeology, and the honest answer was usually “I think the retrieval missed.”

The caveat: the open-source build is free to run — but not free to operate. It wants a relational database, a columnar store, a cache and a worker, and those four have to be upgraded, backed up and watched on your schedule. If you have that capacity it is an excellent deal. If you do not, the hosted tier is saner — and check current pricing, because it meters on trace volume rather than seats.

What I tested on

• Duration: roughly six months in production, plus two months running the hosted version side by side.
• Stack: Python FastAPI generation layer, .NET 8 API in front, Langfuse self-hosted on one Linux box with my other containers.
• Load: tens of traces per generated article — every retrieval, rerank, generation and compliance check emits a span, so volume tracks pipeline depth, not article count.
• Integration: SDK instrumentation in Python, OpenTelemetry spans from the .NET service into the same collector.
• Datasets: a few dozen curated cases, plus a larger set sampled from production failures.

What it does well

• Trace fidelity for multi-step RAG. My pipeline is retrieve, rerank, generate, check, sometimes retry. Langfuse nests those as observations under one trace, so I see the whole attempt rather than a flat list of requests — the shape the RAG pipeline guide describes.
• Cost accounting that survives retries. Cost attaches to the observation, not the request, so a retried generation shows what it actually cost. The LLM cost breakdown covers what to measure first.
• Datasets that connect to production. Pull a real trace, save it into a dataset, change the prompt, re-run, compare. I wrote the scoring rules, but Langfuse runs the runs and shows both sides.
• Provider-agnostic instrumentation. Nothing in the trace layer is tied to one vendor, which matters when you route between models by tenant and task.
• A judge model good enough to be a smoke detector. LLM-as-a-judge is not a measurement, but a cheap judge that flags regressions gets used.

Where it fell short

• Self-hosting is four services, not one. The compose file brings up the app, a relational database for metadata and auth, a columnar store for the traces, plus a cache and an ingestion worker. I first ran it on the same modest server as everything else, and the ingest worker competed with my generation workers at peak.
• Upgrade risk is real and unevenly documented. Pulling a new image can mean a schema migration, a data migration, or both. I hit a release where the upgrade needed an explicit migration command buried in one line of a changelog. I now snapshot the database before every bump.
• Cost figures drift from real invoices. The platform applies a price table to convert tokens into money, and that table is a snapshot. When a provider changes rates, or I use a model it does not cover, reported cost drifts from the invoice.
• Online scoring on every trace is a trap. Running a judge against production traffic looks thorough and is quietly expensive, because every scored trace costs a second model call. Score a sampled subset instead — the discipline the evaluation guide argues for.
• Payload limits bite on real documents. Long contexts get rejected, truncated or quietly dropped, and the trace makes the model look lazy rather than the payload heavy. Multi-tenancy compounds this: one project per tenant gives isolation but many projects to manage, while one shared project with a tenant tag gives no real data boundary. I started shared and regretted it.
• No anomaly detection. Metrics and webhook alerts cover “error rate went up”, not “the retrieval score distribution shifted and nobody was paged”.

The setup

Instrumentation is small. In Python it is a decorator per call site plus a client configured once:

from langfuse import Langfuse
from langfuse.decorators import observe

client = Langfuse(
    public_key=os.environ["LANGFUSE_PUBLIC_KEY"],
    secret_key=os.environ["LANGFUSE_SECRET_KEY"],
    host=os.environ["LANGFUSE_HOST"],   # self-hosted: http://langfuse:3000
)

@observe(name="generate-article")
def generate_article(tenant_id: str, brief: str, chunks: list[str]) -> str:
    return llm.complete(build_prompt(brief, chunks))

The decorator records the span and pulls token counts from the response; the same pattern exists in the other official SDKs. Self-hosting is a compose file with those four services, and two settings are not optional:

services:
  langfuse-worker:
    environment:
      DATABASE_URL: postgresql://langfuse:secret@langfuse-db:5432/langfuse
      CLICKHOUSE_URL: http://langfuse-clickhouse:8123
      SALT: ${LANGFUSE_SALT}
      ENCRYPTION_KEY: ${LANGFUSE_ENCRYPTION_KEY}
    command: worker

The salt and encryption key must be stable across restarts, because changing them invalidates stored credentials and you re-authenticate every integration by hand. The worker also needs resource limits, or a burst of ingestion starves the pipeline next to it. For the .NET service I exported OpenTelemetry spans rather than using the native SDK; pick one path per service, never both, because mixing them duplicates spans.

What it cost

The open-source build has no licence fee. That is the entire software cost. What it has instead is an infrastructure cost people underestimate, because the tracing database is only the visible part.

My breakdown is a relational database with a modest footprint, a columnar store that wants real memory, a cache, a worker, and disk for retention — trace history grows continuously and row count is not what fills the disk.

The hidden cost nobody budgets for is the operational tax: upgrades, backups, and the fact that somebody has to own it. I now have a runbook for restoring the database from a snapshot, and I have needed it once.

What broke along the way

Traces disappeared for one tenant only. They were visible in the dashboard but never in that tenant’s own project view. The cause was mine: I rolled out per-tenant project keys to the Python services and forgot the .NET service, still writing to the shared project with a tenant tag. One environment variable fixed it. Multi-tenant observability needs a test asserting “this tenant’s trace appears in this project”, not a manual check.

Cost totals ran well below my provider invoices. This looked like a bug in the tracing tool and was not. The price table lagged the provider’s rates, and one of my endpoints was missing from it, so it counted as free. I now pin my own rate card in code and compute cost from raw token counts.

Ingest fell behind and the queue backed up. After I moved the generation pipeline onto the same host, traces arrived minutes late at peak — the columnar store and the generation workers were fighting for memory. Moving the tracing stack to its own host fixed it, and it is the change I would make on day one.

Would I use it again?

Yes, with one condition: I would only self-host if I already have infrastructure I am comfortable owning, and I would put the tracing stack on its own host rather than sharing memory with the pipeline. With no infrastructure at all, I would start on the hosted tier and revisit self-hosting once the volume made the bill visible.

I would recommend it to anyone whose pipeline makes more than one model call per user-visible output, because that is exactly where “the answer was bad” stops being diagnosable. I would not recommend it for a single-call application.

Pros and cons

What it gets you

• Multi-step traces instead of isolated log lines — the difference between diagnosing and guessing
• Dataset experiments run against real production failures rather than invented examples
• Token and cost visibility per step, per tenant, including retries
• Instrumentation that survives a model switch

What it costs

• Four services to run, upgrade and back up, plus the disk and memory they need
• A per-tenant structure decision that is expensive to change once you have traffic
• Cost numbers that are estimates until reconciled against your own invoice
• Upgrade reading as part of your job, for as long as you run it

Who this is for

This is for the person whose pipeline has more moving parts than they can hold in their head, and who has already been burned by a bug invisible in the logs. It suits teams running multiple providers, tenants and stages, where “which of these four calls caused this bad output” gets asked regularly and needs a real answer.

It is not for a single-prompt application, and not for someone who wants a dashboard in an afternoon. The self-hosted path is a weekend of work before you see a useful trace; if your pipeline is one call deep, a structured log with a request id gets you most of the way.

FAQ

Q: Is Langfuse free to use in production? A: The open-source build carries no per-seat licence fee, but running it is not free — you pay in infrastructure. The hosted tier has a free allowance before it switches to usage-based billing, so check current pricing before assuming either way.

Q: Do I need the paid plan to use evaluations? A: No. Datasets, experiments and LLM-as-a-judge scoring all work in the open-source version. The paid hosted tier adds convenience — managed infrastructure, higher retention allowances and team features — and I ran my evaluation loop on the self-hosted build.

Q: Should I self-host Langfuse or use the cloud version? A: Self-host if you already run stateful infrastructure and your prompts contain customer content you would rather not send to a third party. Use the cloud version if observability is not your job and you would rather not maintain a database, a columnar store and a queue.

Q: How much does tracing slow down the pipeline? A: With SDK-based instrumentation the overhead is small, roughly a few percent on a typical generation call. The bigger latency cost comes from running an LLM judge on live traffic, since that adds a second model call and can dominate response time.

Q: What is the single biggest gotcha when self-hosting? A: Upgrades. Container images move forward faster than the migration notes always explain, and the columnar store in particular needs a deliberate migration rather than a blind pull and restart. Read the release notes and back up the database first.