LangSmith

LangSmith is the observability + eval platform from the LangChain team — the trace-and-score dashboard that turns "did this agent improve?" into a chart. It does the exact job Langfuse does, and does it well: the eval/experiment UX is genuinely polished. The catch is that it's a closed-source SaaS — your traces leave the machine and the good tiers are paid. This page is the orient-and-decide surface — official docs at docs.smith.langchain.com own the SDK contract.

The house observability lane is already Langfuse — self-hosted, MIT, free. So this is an evaluated-and-chose-otherwise page, not a setup guide. The reason to skip is the self-host + $0 + data-locality preference, not quality.

What it is

A LLM-application observability and evaluation platform. Closed source; you cannot read the platform, fork it, or run it in your own VPC without an Enterprise contract. Framework-agnostic despite the Lang* naming — it traces apps built on the OpenAI SDK, Anthropic SDK, Vercel AI SDK, LlamaIndex, or no framework at all, and as of 2026 ships end-to-end OpenTelemetry support both directions. Five core surfaces:

• Tracing — every LLM call gets logged with input, output, model, latency, token cost. Multi-step agent runs nest into a tree; 2026 added AI helpers that summarize large traces and surface common failure modes once a run spans dozens of steps.
• Datasets + experiments — build test-case datasets, define evaluators, and run experiments to compare versions side by side with regression flags. This is the strongest part of the product.
• LLM-as-judge — run LLM-based, code-based, or multi-turn evaluators on real production traces, and calibrate the judge to match human preferences.
• Annotation queues — route production traces to humans for review inside the platform, no custom UI required.
• Prompt Hub — a versioned prompt repository with tagged versions, public/private prompts, and pull/push from the SDKs.

The pitch is real: when "non-determinism is the regression suite," you need a queryable record of every run plus a good eval loop. LangSmith is a clean answer to that. It's just a hosted answer.

When to use it

Reach for it when:

• The project is already deep into LangChain / LangGraph and you want the tightest-integrated tracing with zero wiring.
• The eval/experiment UX matters more than data locality — LangSmith's side-by-side experiment view and calibrated LLM-judges are best-in-class and worth the tour.
• A team needs a hosted annotation queue today and has no appetite to stand up and babysit a self-hosted stack.
• Someone else is paying, procurement wants a vendor with a SLA and AWS Marketplace availability, and the traces living off-box is acceptable.

Skip it when (the Langfuse-instead case):

• You want the data on your own box. Langfuse self-hosted keeps every trace in a Postgres/ClickHouse stack you own — same traces, same scores, same datasets, same LLM-as-judge, zero data leaving the machine. This is the GL default.
• The budget is $0. Langfuse self-hosted is MIT and free forever; LangSmith's free Developer tier caps at 5K traces/month with 14-day retention, and self-hosting is Enterprise-only.
• Vendor lock-in on the observability layer is a non-starter — a closed platform you can't fork is a dependency Langfuse doesn't impose.
• You're not on LangChain anyway. Both are framework-agnostic, so LangSmith's one structural advantage (native LangChain integration) doesn't apply to a DSPy-first stack.

At a glance

Core concepts

• Trace / Run — one request from start to finish; child runs nest as spans (retrieval, tool call, generation).
• Dataset — a versioned set of (input, expected_output) examples to run experiments against.
• Experiment — a dataset run against a version of your app, scored by evaluators, diffable side by side with regression flags.
• Evaluator — LLM-as-judge, code-based, or multi-turn; calibratable to human preference.
• Annotation queue — a review lane that routes traces to humans and captures structured feedback.
• Prompt (Hub) — versioned prompt stored in LangSmith; code fetches by name and tag.

Distribution + pricing

• Developer (free) — 5K traces/month, 14-day retention, 1 seat, 1 workspace. Overage ~$0.50/1K traces.
• Plus (~$39/seat/mo) — 10K base traces included, 3 workspaces; overage ~$2.50/1K base traces (14-day) or ~$5/1K extended (400-day retention).
• Enterprise (custom) — the only tier with self-hosting / VPC deployment, plus compliance and support. Self-host runs are a heavy infra + licensing commitment, not a docker compose up.
• SDKs — Python and TypeScript (langsmith-sdk), plus OpenTelemetry ingest/export. The SDK is open; the platform behind it is not.

How to integrate

If a build genuinely needs LangSmith over the house Langfuse lane, the order is:

1. Sign up + grab keys. Create a workspace at smith.langchain.com; set LANGSMITH_API_KEY and LANGSMITH_TRACING=true in the environment.
2. Trace. For LangChain/LangGraph apps tracing is automatic once the env vars are set. For everything else, wrap calls with the @traceable decorator or export via OpenTelemetry to the LangSmith endpoint.
3. Build a dataset. Upload (input, expected_output) examples; this is the seed for every experiment.
4. Define evaluators. Start with an LLM-as-judge on the dimension that matters, then calibrate it against a handful of human annotations so the judge tracks real preference.
5. Run experiments. Execute the dataset against each app version; read the side-by-side diff and regression flags to decide whether a change actually helped.
6. Add an annotation queue for the traces where automated scoring isn't trustworthy yet — human labels feed back into dataset + judge calibration.

Note the shape: this is the same six-step loop the Langfuse page describes. The workflow ports cleanly between the two — which is exactly why the choice comes down to hosted-vs-self-hosted, not features.

In the GL stack

Honest framing: there is no greenfield LangSmith need here. The observability lane is already filled by Langfuse self-hosted. Everywhere LangSmith could go, it would be a swap — trading self-host + $0 + data-locality for LangSmith's polish. The slots below are where that swap would land if it ever made sense.

builddaily.io

• Chat-bridge + drafter eval already targets Langfuse (traces per chat call, scores per post_writer compile). LangSmith would cover the identical surface with a nicer experiment diff — but every trace would then live on LangChain's servers, and the public build-in-public ethos leans self-hosted. No swap planned.
• Fine-tune A/B would map to LangSmith experiments cleanly (two dataset runs, side-by-side scores). Langfuse charts already do this for free.

paiddaily.io

• Pendle classifier + Aerodrome explainer traces are Langfuse-bound. LangSmith's calibrated LLM-judge is a genuine draw for the classifier eval loop — the one place its polish is tempting — but the corrections-as-eval-items pattern works fine in Langfuse, and paiddaily's data-sensitivity argues for keeping traces on-box.

sagedaily.io

• Per-reading traces + user feedback scores land in Langfuse today. LangSmith would be pure substitution; readings are personal user data, which reinforces the self-hosted default rather than a hosted swap.

Gotchas

• Free tier is a trial, not a home. 5K traces/month with 14-day retention burns fast on any multi-step agent — a single run that fans out to 10 sub-calls is 10 traces. Past that you're on the paid clock.
• Self-hosting is Enterprise-gated. Unlike Langfuse, there's no free self-host path — "keep the data local" costs a custom contract plus real infra spend.
• Seats, not viewers. Every person who needs access is a full paid seat; there's no cheaper read-only tier.
• Extended retention is a 9–10x line item. Base traces expire at 14 days; keeping 400-day history multiplies the per-trace cost. Decide retention deliberately.
• Framework-agnostic, but the docs assume LangChain. Non-LangChain wiring (decorator / OTel) works well but is the less-trodden path in the guides.

Risks

• Data locality. Traces, prompts, and eval inputs leave your machine and sit on LangChain's servers unless you're on an Enterprise self-host contract. For personal-data surfaces (sagedaily readings, paiddaily positions) that's the deciding factor against it.
• Vendor lock-in. Closed source — you can't fork it, read it, or run it in your own VPC on a normal plan. If pricing or terms shift, migration is a project. Langfuse's MIT license makes the same worry disappear: you own the binary and the Postgres.
• Cost creep. The free tier is generous-looking until trace volume, seats, and extended retention stack up; real usage lands on Plus-and-up quickly. The house rule is free/self-hosted by default, and Langfuse satisfies it with no ceiling.

Alternatives

Related

• Langfuse — the chosen alternative. Same job (traces, scores, datasets, LLM-judge, review queue) self-hosted, MIT, and free. This is the GL observability lane.
• LangChain — LangSmith's tightest integration; the one place it beats framework-agnostic rivals. GL is DSPy-first, so that edge doesn't land here.
• LangGraph — agent-graph runs trace natively into LangSmith; the same runs trace into Langfuse just as well.
• DSPy — the brain layer whose compiles need an eval surface. That surface is Langfuse in the GL stack, not LangSmith.