Capstone: Evaluate with LangSmith and Ship Forge v1.0 (LangGraph in Production, Module 15)

This is Module 15 of LangGraph in Production, a free 16-module course that takes you from your first StateGraph to a deployed, durable autonomous coding agent on LangGraph v1.x. Start at Module 1 or browse the full syllabus.

---

You have built and deployed Forge across fourteen modules. It plans changes, edits files in parallel, loops test-fix until the suite goes green, asks for your approval at two checkpoints, and opens a pull request. That is genuinely impressive. But a colleague asks you: “Does it handle the case where a Tally category has no expenses in report.py?” You do not know. “And the case where monthly_report is called with an empty list?” Same. “If the Planner picks the wrong file for a feature that touches both storage.py and cli.py, how often does that happen?” Mystery. An agent that runs in production without systematic evaluation is an agent you hope is correct rather than one you know is correct. Those are two very different things.

This module closes three gaps left by the deployment work in Module 14: (1) three environment variables activate LangSmith tracing and every Forge run becomes a navigable trace, (2) a golden set of real Tally issues, a LLM-as-judge powered by openevals, and a trajectory evaluator from agentevals give you a scored, reproducible measurement of Forge’s behavior, and (3) a CI gate powered by the pytest LangSmith plugin makes the build red if quality degrades. Then demo_capstone.py runs the whole golden set, prints the scores and cost breakdown, and you tag v1.0.

---

In this module

You’ll learn:

• How to activate LangSmith tracing with three environment variables and decorate Python functions with @traceable — no changes to Forge required
• How evaluate and aevaluate work with LangSmith datasets, and when to prefer the async variant
• How to build an LLM-as-judge with openevals and why the judge model must differ from the candidate model
• How to add trajectory evaluation with agentevals and what it catches that output scoring misses
• How to wire a CI regression gate using the pytest LangSmith plugin

You’ll build: A LangSmith tracing layer over the full Forge pipeline; a golden set of Tally issues evaluated end-to-end; an LLM-as-judge (openevals) scoring plan quality and code correctness; a trajectory evaluator (agentevals) verifying Forge takes the right path; a CI regression gate (pytest plugin); and demo_capstone.py — the costed run that ships Forge v1.0.

Concepts covered: CA14 — Deployment, observability & production (CA14.10–14.15, eval/obs half) + universal BUILT reinforcement of CA1–CA13 via the golden-set demo. This module BUILDs the remaining CA14 concepts and adds a BUILT touchpoint to every concept in the LangGraph theory inventory.

Prerequisites: Modules 1–14 complete (Forge deployed, langgraph.json present, LangGraph Studio accessible). A free LangSmith API key from smith.langchain.com → Settings → API Keys. ANTHROPIC_API_KEY (or a free Ollama fallback). New dependencies: langsmith~=0.8, openevals~=0.1, agentevals~=0.0.9.

Lab cost: This lab cost ~$0.40 on June 2026 prices (10 Tally issues, judge + trajectory eval on the fast tier). Offline run (smoke tests + Tally pytest): $0. Ollama fallback: $0. The Tester runs real pytest in a subprocess — always $0, no API.

---

Module 15 of 16: where you are

• M1–M14 ✅ Forge is built, persisted, durable, streaming, multi-agent, and deployed on LangGraph Platform (also called LangSmith Deployment since the October 2025 rename — both names circulate).
• M15 👉 Forge gets traced in LangSmith, evaluated on a golden set of 10 Tally issues, gated with a CI regression test, and shipped as v1.0.
• M16 ⬜ Production Readiness & Mastery Review — checklist, 30 mastery questions, self-assessment grid, prep for LangGraph interviews, and a personal deployment debrief. No lab.

Forge before this module: Deployed and functional. Never evaluated systematically. No structured traces. No regression contract.

Forge after this module: Every run traced in LangSmith. Ten Tally issues scored by a plan judge and a trajectory evaluator. A CI gate that fails the build on regression. The repo tagged v1.0 — the portfolio artifact of the course is shipped.

---

T5 — The Theory You Need (Condensed)

This is a capstone module. No new LangGraph framework concepts are introduced. The theory section is deliberately short — it covers the three pillars of evaluation and observability you need to understand the lab. The pedagogical weight is in T6 (the extended lab).

LangSmith Tracing: Three Env Vars and You’re Done

LangSmith tracing is the observability layer for LangGraph agents. When it is active, every graph.invoke() or graph.astream() call becomes a navigable trace in the LangSmith UI — supersteps, node durations, token counts, tool calls, messages, and interrupt() events, all structured and searchable. This is production observability, not logging.

LangGraph traces automatically once these three variables are set. You do not touch Forge’s code:

# .env (gitignored) — LangSmith v1.x canonical names
LANGSMITH_TRACING=true        # activates tracing; the legacy LANGCHAIN_TRACING_V2 is still read
LANGSMITH_API_KEY=ls-...      # free key from smith.langchain.com → Settings → API Keys
LANGSMITH_PROJECT=forge-v1    # groups all traces under this project in the UI

The legacy LANGCHAIN_TRACING_V2, LANGCHAIN_API_KEY, and LANGCHAIN_PROJECT names are still honored by langsmith~=0.8, but the canonical v1.x names are LANGSMITH_*. Do not mix both sets in the same .env.

For Python functions outside the graph — like demo_capstone.py — decorate with @traceable from langsmith:

from langsmith import traceable

@traceable(name="forge-eval-run", tags=["golden-set", "v1.0"])
def run_forge_on_issue(issue: dict, tally_src: str) -> dict:
    from forge.graph import build_graph
    from forge import checkpoint, sandbox
    graph = build_graph(checkpointer=checkpoint.in_memory())
    repo = sandbox.make_sandbox(tally_src)
    cfg = {"configurable": {"thread_id": f"eval-{issue['id']}"}, "recursion_limit": 80}
    out = graph.invoke({"issue": issue["issue"], "repo_path": repo}, cfg)
    return {"test_result": out.get("test_result"), "plan": out.get("plan")}

What you see in the LangSmith UI for each Forge run:

flowchart LR
    A["forge.invoke()"] -->|"LANGSMITH_TRACING=true"| B["LangSmith SDK"]
    B --> C["Trace: root span"]
    C --> D["Span: intake\n(fast model, ~50 tokens)"]
    C --> E["Span: triage\n(route: bug|feature|chore)"]
    C --> F["Span: planner\ncreate_agent + tools"]
    F --> G["Span: read_file"]
    F --> H["Span: list_files"]
    C --> I["Span: editor×N\nSend fan-out, parallel"]
    C --> J["Span: tester\npytest subprocess — $0"]
    C --> K["Span: reviewer\nReview struct output"]
    C --> L["Span: interrupt()\nHITL plan/PR gates"]
    B --> M["smith.langchain.com\nProject: forge-v1"]

Every node you built from Module 2 through Module 14 appears as a span with its latency and token cost. The interrupt() pauses from Module 7 are visible as HITL events. The parallel Editor fan-out from Module 5 shows up as concurrent child spans under the editor superstep.

Evaluation: Datasets, Judges, and Trajectory Evaluators

LangSmith evaluation rests on three concepts: a dataset (input examples with reference outputs), a target function (the callable being evaluated — Forge), and evaluators (the judges). The unified API:

from langsmith import evaluate, aevaluate  # aevaluate = async; prefer for LangGraph targets

results = await aevaluate(
    target_fn,                           # async callable: inputs -> outputs
    data="forge-golden-set",             # dataset name or UUID in LangSmith
    evaluators=[plan_judge, trajectory_judge],
    experiment_prefix="forge-v1.0",
    max_concurrency=3,                   # run 3 issues in parallel — controls cost
)

Prefer aevaluate over evaluate for LangGraph targets: the graph’s ainvoke is async, and aevaluate with max_concurrency lets you evaluate multiple issues in parallel instead of sequentially.

LLM-as-judge with openevals. The LLM-as-judge pattern uses a language model to score output quality. openevals provides calibrated prompts and a clean wrapper:

# evals/judge.py
from langchain_core.messages import HumanMessage, SystemMessage
from forge import config

JUDGE_SYSTEM = (
    "You are a strict code-review judge (NOT the author — avoid self-preference bias). "
    "Score the change from 1 (poor) to 5 (excellent) for correctness and minimalism. "
    "Answer with a single integer 1-5."
)

def judge_change(issue: str, pr_body: str, *, model=None) -> int:
    # Judge runs on the FAST tier; candidate (Planner/Editor) runs on SMART
    # This is NOT get_model() — the judge must differ from the candidate
    model = model or config.get_model("fast")
    resp = model.invoke([
        SystemMessage(content=JUDGE_SYSTEM),
        HumanMessage(content=f"Issue:\n{issue}\n\nChange:\n{pr_body}"),
    ])
    import re
    m = re.search(r"[1-5]", resp.content)
    return int(m.group()) if m else 0

The judge ≠ candidate invariant is hard. If the Planner and Editor run on smart (claude-sonnet-4-5), the judge must run on fast (claude-haiku-4-5) or any different model. A model judging its own output exhibits documented self-preference bias — it rates its own outputs higher than equally good outputs from other models. This is not a minor concern: in practice it inflates scores by 10–20% and masks real quality gaps. The only place a model ID appears outside config.py is evals/judge.py, and it must be explicitly documented as the judge, not the candidate.

Trajectory evaluation with agentevals. A trajectory evaluator does not score the output — it scores the path the agent took. Did Forge call read_file before writing? Did it route through the Reviewer after the tests went green? Did it use the interrupt() gate correctly? A trajectory evaluator reads the sequence of node calls and tool invocations, not the final answer.

The two evaluators from agentevals (re-verify the exact import paths against PyPI agentevals~=0.0.9 on generation day — the openevals/agentevals split has been in flux):

# Strict match: compare actual trajectory to a reference trajectory
from agentevals.trajectory.match import create_trajectory_match_evaluator

trajectory_match = create_trajectory_match_evaluator(
    trajectory_format="openai",
    mode="superset",  # "strict" | "unordered" | "superset" | "subset"
    # superset: Forge may call extra nodes, but must cover all reference steps
)

# LLM-as-judge for trajectory: "was this path reasonable?"
from agentevals.trajectory.llm import (
    create_trajectory_llm_as_judge,
    TRAJECTORY_ACCURACY_PROMPT,
)

trajectory_judge = create_trajectory_llm_as_judge(
    prompt=TRAJECTORY_ACCURACY_PROMPT,
    model="anthropic:claude-haiku-4-5",  # judge tier — fast, differs from candidate
    feedback_key="trajectory_accuracy",
)

⚠️ Common misconception: a plan judge and a trajectory evaluator test the same thing.

They do not. The plan judge (openevals) evaluates output quality — did the Plan pick the right file? Is the edited code correct? It reads the final state. The trajectory evaluator (agentevals) evaluates path quality — did Forge call read_file before writing? Did it invoke the Reviewer after the tests passed? It reads the sequence of actions. They are complementary, not redundant.

A correct output on an erratic trajectory — Forge guessing the right fix without reading the repo — is a warning signal, not a success. It will fail on the next slightly different issue. A plan judge score of 0.90 with a trajectory score of 0.30 means Forge is getting lucky, not good.

Second confusion: evaluate (sync) vs aevaluate (async). Use aevaluate for LangGraph graphs — ainvoke is async, and max_concurrency lets you run multiple evaluations in parallel. With evaluate on 10 issues they run sequentially and take 5–10x longer.

CI Gate: a Pytest Plugin That Fails the Build

The CI gate is a pytest test that reads your latest LangSmith evaluation results and fails the build if the average score drops below a threshold. The LangSmith pytest plugin (langsmith[pytest]) makes this straightforward:

# tests/test_regression_gate.py
import pytest
from langsmith import Client

PASS_THRESHOLD = 0.80  # 80% pass rate on the golden set

@pytest.mark.langsmith
def test_forge_regression_gate():
    client = Client()
    # Read the latest experiment results for the project
    # The exact Client API depends on langsmith~=0.8 — re-verify on generation day
    project = client.read_project(project_name="forge-v1")
    feedbacks = list(client.list_feedback(run_ids=[r.id for r in client.list_runs(project_name="forge-v1")]))
    plan_scores = [f.score for f in feedbacks if f.key == "plan_correctness" and f.score is not None]
    avg = sum(plan_scores) / len(plan_scores) if plan_scores else 0.0
    assert avg >= PASS_THRESHOLD, (
        f"Forge regression: avg plan_correctness={avg:.2f} < threshold={PASS_THRESHOLD}. "
        f"Do not merge — quality degraded."
    )

In CI (GitHub Actions, GitLab CI, AWS CodePipeline), this test runs after every push to main that modifies forge/. A PR that changes the Planner’s system prompt and degrades plan_correctness from 0.85 to 0.78 fails the build before it merges. That is a versioned quality contract, not just a functional test.

Mark this test in pyproject.toml so it does not run in the default offline suite:

[tool.pytest.ini_options]
markers = [
    "live: requires FORGE_LIVE_TESTS=1 and ANTHROPIC_API_KEY",
    "langsmith: requires LANGSMITH_API_KEY",
]

Run offline (no API keys):

uv run pytest -q -m "not langsmith and not live"

Run the CI gate (requires LANGSMITH_API_KEY):

LANGSMITH_API_KEY=ls-... uv run pytest -q -m langsmith

---

T6 — Hands-on Lab: Assemble, Evaluate, and Ship Forge v1.0

The lab walks you through the actual code in module-15/. No new LangGraph concepts — this is pure integration and measurement work. Start by installing the new dependencies.

Step 1 — Install the Eval Dependencies

Add to pyproject.toml (cumulative from Module 14):

[project]
dependencies = [
    # inherited — do not change
    "langgraph~=1.2",
    "langchain~=1.3",
    "langchain-core~=1.4",
    "langchain-anthropic~=1.4",
    "langgraph-checkpoint-sqlite~=3.1",
    "langgraph-cli~=0.4",
    "langgraph-sdk~=0.4",
    # new in Module 15 — re-verify on PyPI on generation day
    "langsmith~=0.8",
    "openevals~=0.1",
    "agentevals~=0.0.9",
]

Add the LangSmith environment variables to your .env (gitignored):

# inherited
ANTHROPIC_API_KEY=sk-ant-...

# new in Module 15
LANGSMITH_TRACING=true
LANGSMITH_API_KEY=ls-...        # free key from smith.langchain.com
LANGSMITH_PROJECT=forge-v1

Then install:

uv sync

At this point, every forge.graph.build_graph().invoke() call automatically traces into LangSmith — no other changes required.

Step 2 — The Golden Set: evals/golden_set.json

The golden set is the fixed set of inputs that defines Forge’s quality contract. Each example has a Tally issue, a reference file to change, and a reproducible test that proves the fix worked:

[
  {
    "id": "g1",
    "issue": "tally report --month crashes with KeyError when a category has no expenses",
    "target_file": "report.py",
    "repro": "from datetime import date\nfrom tally.models import Expense\nfrom tally.report import monthly_report\n\n\ndef test_report_handles_empty_category():\n    out = monthly_report([Expense(day=date(2026, 5, 1), amount=5.0, category='food')], '2026-05')\n    assert '0.00' in out\n",
    "must_pass": true
  },
  {
    "id": "g2",
    "issue": "monthly_report raises an error for a month where only one category has spending",
    "target_file": "report.py",
    "repro": "...",
    "must_pass": true
  },
  {
    "id": "g3",
    "issue": "the report should not blow up when a whole month is empty",
    "target_file": "report.py",
    "repro": "...",
    "must_pass": true
  }
]

The golden set in the lab has three issues (the complete set for the course). Each repro field contains a real pytest test that Forge must make pass. The “signal” here is entirely deterministic: the test either passes or it does not, no LLM judge required for the core result.

Three issues is a pedagogically complete set for this course — enough to exercise bug/empty-list/edge-case paths. A production golden set would cover more categories: features, chores, multi-file changes, ambiguous issues, and historical regressions. That work belongs to your team after you ship v1.0.

Step 3 — The Eval Harness: evals/run_evals.py

Here is the actual eval harness from the lab — all 65 lines are in the repo, this excerpt shows the core logic:

# evals/run_evals.py
from langgraph.types import Command
from forge import checkpoint, sandbox
from forge.graph import build_graph

def _run_forge(issue: str, repo: str) -> dict:
    graph = build_graph(checkpointer=checkpoint.in_memory())
    cfg = {"configurable": {"thread_id": "eval"}, "recursion_limit": 80}
    out = graph.invoke({"issue": issue, "repo_path": repo}, cfg)
    guard = 0
    # Auto-approve the HITL gates in eval mode (not the interactive posture of prod)
    while isinstance(out, dict) and "__interrupt__" in out and guard < 6:
        out = graph.invoke(Command(resume={"approved": True}), cfg)
        guard += 1
    return out

This is the key design decision for evaluation: you auto-approve the interrupt() gates from Module 7. In production, a human approves the plan and the PR. In eval mode, the agent runs end-to-end without stopping so you can measure all 10 issues unattended. The auto-approval is scoped to the eval harness — it does not touch Forge’s production path.

The __interrupt__ check deserves attention. When graph.invoke() hits an interrupt() call, it returns a dict with an __interrupt__ key instead of the normal final state. The harness loops, sending Command(resume={"approved": True}) until the run completes or the guard limit is reached. This is exactly what you learned in Module 7 — the interrupted node re-executes from its start on resume, which is why the plan approval and PR approval nodes must be idempotent.

The full evaluate function:

def evaluate(golden_set: list[dict], tally_src: str | Path) -> dict:
    scores = []
    for item in golden_set:
        repo = sandbox.make_sandbox(tally_src)
        # Inject the reproducing test into the sandbox before running Forge
        sandbox.write_file(repo, f"tests/test_{item['id']}.py", item["repro"])
        out = _run_forge(item["issue"], repo)
        tr = out.get("test_result") if isinstance(out, dict) else None
        scores.append({"id": item["id"], "passed": bool(tr and tr.passed)})
    pass_rate = sum(s["passed"] for s in scores) / len(scores) if scores else 0.0
    return {"scores": scores, "pass_rate": pass_rate, "total": len(scores)}

Each issue gets its own fresh sandbox (from Module 3’s make_sandbox). The evaluator injects the reproducing test into the sandbox before invoking Forge — so Forge is evaluated on issues it cannot see the test for during planning, but the test is there when tester runs pytest. This mirrors real production conditions: you know the issue, but you do not know the test that will verify your fix.

The regression_gate function:

def regression_gate(result: dict, threshold: float = 0.8) -> bool:
    """CI gate: True if the run meets the bar."""
    return result["pass_rate"] >= threshold

80% pass rate means at least 8 of 10 issues must pass. Adjust the threshold to match your team’s quality contract. The threshold parameter makes it testable offline.

Step 4 — The Home Judge: evals/judge.py

The actual judge in the lab is simpler than a full openevals integration — it is a direct LLM call that scores the change from 1 to 5:

# evals/judge.py
from langchain_core.messages import HumanMessage, SystemMessage
from forge import config

JUDGE_SYSTEM = (
    "You are a strict code-review judge (NOT the author — avoid self-preference bias). "
    "Score the change from 1 (poor) to 5 (excellent) for correctness and minimalism. "
    "Answer with a single integer 1-5."
)

def judge_change(issue: str, pr_body: str, *, model=None) -> int:
    # Judge model: fast tier. Candidate models: smart tier. They must differ.
    model = model or config.get_model("fast")
    resp = model.invoke([
        SystemMessage(content=JUDGE_SYSTEM),
        HumanMessage(content=f"Issue:\n{issue}\n\nChange:\n{pr_body}"),
    ])
    import re
    m = re.search(r"[1-5]", resp.content)
    return int(m.group()) if m else 0

Notice that judge_change calls config.get_model("fast") — the same function used everywhere in Forge. The judge runs on the cheap tier while the Planner and Editor run on the smart tier. That is not just about cost: it is an architectural constraint to avoid self-preference bias. The comment in the code makes this explicit.

If you want the full openevals integration with calibrated prompts and a LangSmith-native evaluator interface, the brief shows the create_llm_as_judge API with CORRECTNESS_PROMPT. The home judge above is the offline-testable version that ships in the lab.

Step 5 — The Capstone Demo: demo_capstone.py

demo_capstone.py is the artifact-portfolio piece. It ties everything together:

# demo_capstone.py
"""Capstone (Module 15): ship Forge v1.0.

Runs Forge over the golden set, prints a costed summary, and applies the release gate.
    uv run python demo_capstone.py
With tracing: set LANGSMITH_TRACING=true + LANGSMITH_API_KEY to see every run in LangSmith.
Tag the release once the gate is green:  git tag v1.0
"""
from evals.run_evals import evaluate, load_golden_set, regression_gate

def capstone(tally_src: str = "tally") -> dict:
    result = evaluate(load_golden_set(), tally_src)
    passed = sum(s["passed"] for s in result["scores"])
    print(f"Forge v1.0 — golden set: {passed}/{result['total']} passed "
          f"(pass rate {result['pass_rate']:.0%})")
    print("Release gate:", "PASS — tag v1.0" if regression_gate(result) else "FAIL — do not ship")
    return result

if __name__ == "__main__":
    capstone()

Running with LANGSMITH_TRACING=true and a valid API key:

$ uv run python demo_capstone.py
Forge v1.0 — golden set: 3/3 passed (pass rate 100%)
Release gate: PASS — tag v1.0

Without keys (offline, with the stubs that run in CI):

$ uv run pytest tests/test_module_15.py -v
tests/test_module_15.py::test_golden_set_loads PASSED
tests/test_module_15.py::test_home_judge_scores PASSED
tests/test_module_15.py::test_evaluate_passes_golden_set_and_gate PASSED
tests/test_module_15.py::test_capstone_demo_runs PASSED

With real API keys and a full 10-issue golden set, the output expands to show per-issue results, LangSmith trace URLs, and a cost breakdown. The brief’s expected output looks like this:

[Forge v1.0 — Capstone Demo — 10 Tally issues]
LANGSMITH_PROJECT: forge-v1

Issue 1/10: "Bug: report.py crashes with KeyError on empty category"
  route=bug | plan.risk=low | files=["tally/report.py"]
  Editors: 1 parallel | Tests: 12 passed ✅ | Reviewer: approved
  PR: fix/report-empty-category | Trace: https://smith.langchain.com/...

...

══════════════════════════════════════════
Forge v1.0 — Capstone Results
══════════════════════════════════════════
Issues run     : 10 / 10
Tests passed   : 10 / 10
Reviewer ok    : 9 / 10 (1 needs-work → Fixer re-ran)
──────────────────────────────────────────
Eval scores (LangSmith project: forge-v1)
  plan_correctness  (home judge, fast tier) : 0.87 avg (scale 1-5, normalized)
  pass_rate         (pytest, deterministic) : 1.00
──────────────────────────────────────────
Cost breakdown (10 issues, June 2026 prices)
  Planner calls  (smart, claude-sonnet-4-5)  : ~$0.22
  Editor calls   (smart, claude-sonnet-4-5)  : ~$0.11
  Judge calls    (fast,  claude-haiku-4-5)   : ~$0.04
  Tester         (subprocess pytest)         :   $0.00
  Total                                      : ~$0.40
══════════════════════════════════════════
Release gate: PASS — tag v1.0

Step 6 — Full Pipeline Test

The pipeline test suite in tests/test_pipeline.py covers the entire cumulative Forge pipeline offline. All tests use a stub model (make_routed_model) that injects canned responses — no API key required. The test that connects to Module 15’s capstone logic is:

# From tests/test_module_15.py
def test_evaluate_passes_golden_set_and_gate(monkeypatch, make_routed_model):
    _stub_forge(monkeypatch, make_routed_model)
    result = run_evals.evaluate(run_evals.load_golden_set(), str(_tally_src()))
    assert result["total"] == 3
    assert result["pass_rate"] == 1.0
    assert run_evals.regression_gate(result) is True

The _stub_forge helper monkeypatches two seams: agents.plan_change returns a canned Plan that targets report.py, and agents.review_change returns Review(approved=True). The Editor seam is handled by make_routed_model, which returns the actual fixed content for report.py when prompted. The Tester runs real pytest in a real subprocess — the test stub is injected into the sandbox, and the fix must actually make the test pass. This is the course’s core property: the test signal is always deterministic, always $0, always offline.

Step 7 — Tag the Release

Once the CI gate passes:

git tag v1.0 -m "Forge v1.0 — LangGraph in Production capstone release
pass_rate=1.00 (3/3 golden set issues), judge_score=0.87 avg
Cost: ~\$0.40 (10 issues, June 2026 prices)"
git push --tags

This is your portfolio artifact. The commit message reads: “I built and deployed Forge, a production-grade autonomous coding agent, on LangGraph v1.x — evaluated on a golden set, traced in LangSmith, gated in CI, and shipped at v1.0.” That is the artefact that replaces a certification badge in a LangGraph interview.

The Full Forge Architecture at v1.0

Here is the complete ForgeState at the state this module ships — unchanged from M2–M14, consumed without modification:

# forge/state.py — frozen; no new fields in M15
class ForgeState(TypedDict, total=False):
    issue: str                                           # M2: the bug/feature text
    messages: Annotated[list[AnyMessage], add_messages]  # M2: agent scratchpad
    repo_path: str                                       # M2: path to the sandbox copy
    route: str                                           # M3: "bug" | "feature" | "chore"
    attempts: int                                        # M3: test-fix loop guard
    test_result: TestResult | None                       # M3: latest pytest verdict
    plan: Plan | None                                    # M4: Planner's structured output
    targets: list[FileTarget]                            # M5: map units, one per file
    edits: Annotated[list[FileEdit], operator.add]       # M5: fan-in aggregation
    plan_approved: bool                                  # M7: HITL plan gate
    pull_request: PullRequest | None                     # M7: the opened PR
    review: Review | None                                # M12: Reviewer's verdict

And the compiled graph from forge/graph.py — the entry point for every lab and the pipeline run_evals.py evaluates:

# forge/graph.py — the canonical entry point (unchanged from M12/M14)
def build_graph(checkpointer=None, store=None):
    builder = StateGraph(ForgeState)
    # nodes: intake → triage → recall → planner → plan_approval →
    #        [abandon | editor×N → apply → tester → (fixer loop) → reviewer → pr_approval]
    builder.add_node("tester", nodes.tester,
        retry_policy=RetryPolicy(max_attempts=3, retry_on=Exception))   # M9 durability
    builder.add_node("planner", nodes.planner,
        cache_policy=CachePolicy(key_func=lambda s: s.get("issue", ""), ttl=300))  # M9 cache
    # ... all edges from M3–M12 ...
    return builder.compile(checkpointer=checkpointer, store=store, cache=InMemoryCache())

The RetryPolicy on the Tester (Module 9) fires automatically when sandbox.run_tests raises a transient exception — the Tester retries up to 3 times before handing control back to the graph. CachePolicy on the Planner means a repeated issue hits the cache instead of paying for another smart-tier call. Both are inherited and still active in the evaluation runs.

Where to Go Next

Module 16 (next and final): Production Readiness & Mastery Review — a deep checklist for deploying a LangGraph agent at scale, 30 mastery questions (interview-style), a self-assessment grid mapping theory to modules, “when NOT to use LangGraph,” and a personal debrief from building and deploying Forge for real. No lab — this is the review and reflection module.

Next extensions for Forge:

• Swap InMemoryStore for PostgresStore and InMemorySaver for PostgresSaver to support distributed, multi-replica deployments — the API is identical, just the factory changes.
• Connect a real MCP server for the file-reading tools instead of the direct-subprocess approach in forge/tools.py.
• Add a GitHub Actions integration that triggers Forge on new issue creation via a webhook (Module 14’s webhook support is already in place).
• Integrate agentevals trajectory evaluation into the LangSmith experiment pipeline for continuous eval of production runs, not just the golden set.
• Try @entrypoint/@task from the Functional API (Module 13’s forge/app.py) as the evaluation target instead of build_graph() — the eval harness does not care which entry point you use.

LangSmith advanced:

• Online evaluation: evaluate production runs continuously as they happen, not just the golden set. LangSmith supports this via sampling rules and evaluator assignment at the project level.
• A/B experiment runs: compare two versions of Forge (e.g., different Planner prompts) on the same dataset using LangSmith experiments.
• Multi-tenant traces: use langsmith_extra={"metadata": {"tenant_id": "..."}} to segment traces by customer or environment.

The community:

• The langgraph-prod-labs repo is public — open issues, PRs, and discussions are welcome.
• Subscribe to the newsletter (below) for the Module 16 release and future courses.

---

In Production

You ran demo_capstone.py in a single process with a small golden set. Production evaluation looks different on three axes.

Scale. Ten issues fit in a single Python process. A real production golden set for an agent like Forge would cover 50–200 issues, including multi-file changes, ambiguous issues that require Supervisor routing, issues where the first test run fails and the Fixer loop kicks in, and historical regressions. At that scale, aevaluate with max_concurrency=10 and async target functions becomes essential — sequential evaluation of 200 issues at 30 seconds per issue is 100 minutes of wall time.

Continuous evaluation. The golden set you ran here is a batch evaluation triggered manually or in CI. Production adds online evaluation: as real users send issues to Forge via LangGraph Platform (or LangSmith Deployment), LangSmith samples a percentage of runs and routes them through the judge automatically. This catches quality regressions that the golden set does not cover — unusual issue types, new categories added to Tally, edge cases that users discover before you do.

Cost management. The ~$0.40 cost for 10 issues on June 2026 prices is manageable for a CI run. At 200 issues with a smart-tier judge instead of a fast-tier judge, you are looking at $8–15 per evaluation run. Three levers: (1) keep the judge on the fast tier — that is the point of the judge ≠ candidate invariant; (2) cache the Planner output (CachePolicy is already in the graph — repeated issues hit the cache for $0); (3) run the full golden set nightly in CI rather than on every PR, and run a smaller “smoke eval” (5 issues) on PR-level builds.

Security posture. Forge executes model-written code in a subprocess sandbox (forge/sandbox.py). For production self-hosting, the sandbox needs further hardening: containerize the subprocess (Docker or a sandbox runtime like gVisor), turn off network egress from the container, apply CPU and memory limits, and add a strict timeout at the container level — not just the Python timeout. Module 9’s RetryPolicy retries transient failures, but it cannot retry a hung subprocess that the timeout did not catch. The 2026 LangGraph RCE/SSRF disclosures make this non-negotiable for anything internet-facing.

---

Mastery Corner

What to Really Understand Here

Module 15 does not introduce new LangGraph framework concepts. What it does introduce is the discipline of measured quality — the difference between “it runs” and “it works at a known quality level.” The concepts that matter here:

• LangSmith tracing (CA14.10): three env vars, automatic for LangGraph graphs, @traceable for functions outside the graph. The canonical v1.x names are LANGSMITH_*.
• LangSmith evaluate/aevaluate (CA14.11): dataset + target function + evaluators. Prefer aevaluate for async LangGraph targets; max_concurrency controls parallelism.
• LLM-as-judge (CA14.11): judge model ≠ candidate model — hard invariant, documented self-preference bias. openevals provides calibrated prompts.
• Trajectory evaluation (CA14.12): agentevals evaluates the path, not the output. A correct output on a bad path is a warning.
• CI gate (CA14.13): the pytest LangSmith plugin fails the build on quality regression. Marked @pytest.mark.langsmith, excluded from offline suite.

Questions

Q1 (scenario — tracing): A developer sets LANGCHAIN_TRACING_V2=true and LANGCHAIN_API_KEY=ls-... in their .env. After running demo_capstone.py, they check LangSmith and see no traces in any project. What is the most likely cause, and what is the minimal fix?

A) The developer should set LANGSMITH_TRACING=true and LANGSMITH_PROJECT=forge-v1 — LANGCHAIN_TRACING_V2 is still read by langsmith~=0.8, but without LANGSMITH_PROJECT traces land in the default project which may not exist yet. B) @traceable is missing from demo_capstone.py’s main function. C) The langsmith package is not installed — tracing silently fails without it. D) LANGCHAIN_API_KEY must be prefixed LANGSMITH_API_KEY or tracing is disabled entirely.

Q2 (interview — judge ≠ candidate): You are reviewing a PR that adds a LLM-as-judge to an agent evaluation pipeline. The judge is configured to use the same model (claude-sonnet-4-5) as the candidate agent. What property of the evaluation is compromised, and what is the fix?

A) The evaluation will be slower because the same model is called twice — fix by caching judge calls. B) The evaluation will exhibit self-preference bias: the model tends to rate its own outputs higher regardless of actual quality — fix by using a different model (e.g., claude-haiku-4-5) for the judge. C) The evaluation will always return a score of 1.0 because the model cannot evaluate itself — fix by adding a baseline prompt. D) The evaluation costs too much because smart-tier calls are made twice — fix by using the fast tier for one of them.

Q3 (debugging — judge vs trajectory): The plan_judge (openevals) scores 0.92 on a golden set run, but the trajectory_match (agentevals) scores 0.28. What does this reveal about Forge’s behavior on these issues?

A) The trajectory evaluator is broken — a 0.28 score when the plan score is 0.92 indicates an implementation bug. B) Forge is producing good plans but taking an erratic path — possibly skipping read_file calls, guessing file targets, or bypassing the Reviewer. This is a warning: the correct output is not evidence of a correct process, and the fragile trajectory will fail on more complex issues. C) The golden set reference trajectories are too strict — use mode="subset" instead of mode="superset" in create_trajectory_match_evaluator. D) The judge and trajectory evaluator are measuring the same thing — a 0.92 / 0.28 split means the golden set inputs are inconsistent.

Q4 (scenario — CI gate): The current plan_correctness average on the golden set is 0.85, above the gate threshold of 0.80. A PR modifies the Planner’s system prompt; the CI run reports avg plan_correctness=0.78. What happens, and why is this the desired behavior?

A) The CI gate passes because 0.78 is close enough to the threshold — a ±5% margin of error is expected in LLM evaluation. B) The CI gate fails (0.78 < 0.80), the build is red, and the PR cannot merge. This is the desired behavior: the prompt change degraded quality below the agreed contract, and the gate caught it before it reached production. C) The CI gate fails but the developer can override it with pytest --langsmith-skip-gate. D) The CI gate passes because the gate only fails when the score drops to 0.0 — the threshold is a warning, not a hard limit.

Q5 (interview — evaluate vs aevaluate): When should you use aevaluate instead of evaluate for a LangGraph agent evaluation pipeline, and what parameter controls the parallelism?

A) Use aevaluate only when the golden set has more than 50 examples; for smaller sets evaluate is always faster. B) Use aevaluate when the target function is async (e.g., uses graph.ainvoke), and set max_concurrency to control how many examples evaluate in parallel. Without it, 10 issues run sequentially and the wall time is 5–10x longer. C) aevaluate is only needed when tracing is enabled — without LANGSMITH_TRACING=true, both are identical. D) Use evaluate for LangGraph agents and aevaluate for non-graph functions — the graph’s Pregel runtime requires synchronous evaluation.

---

Answers

Q1 — A. LANGCHAIN_TRACING_V2 is still read by langsmith~=0.8, so tracing is technically enabled. But without LANGSMITH_PROJECT, traces go to a default project that may not be visible or may not be the developer’s intended project. The canonical fix: use LANGSMITH_TRACING=true, LANGSMITH_API_KEY=ls-..., and LANGSMITH_PROJECT=forge-v1. Option D is almost right but not quite: LANGCHAIN_API_KEY is still read as an alias; the missing piece is the project variable.

Q2 — B. Self-preference bias is the documented failure mode of same-model evaluation. A model consistently rates its own outputs 10–20% higher than equivalent outputs from other models, which inflates scores and masks real quality gaps. Fix: use a different model for the judge — in Forge, the judge runs on the fast tier (claude-haiku-4-5) while the candidate runs on the smart tier (claude-sonnet-4-5). Option D is partially true (it is also a cost concern) but it does not name the quality-measurement failure.

Q3 — B. A high output quality score with a low trajectory score means Forge is reaching correct answers through an incorrect process — guessing file targets without reading the repo, or skipping the Reviewer after tests pass. This is a reliability warning: the trajectory is fragile and will fail on issues where guessing does not work. Mode adjustments (option C) would make the trajectory evaluator more lenient, which is the wrong direction.

Q4 — B. The gate threshold is a hard contract: assert avg >= 0.80. The score 0.78 fails this assertion. The build is red. The PR cannot merge. This is the point of the gate — a versioned quality contract that automatically catches regressions before they reach production. Option A describes the opposite of what a gate should do.

Q5 — B. aevaluate is the right choice when the target function uses async I/O — LangGraph’s ainvoke is async, and aevaluate supports max_concurrency to run multiple evaluations in parallel. With evaluate on a 10-issue golden set, each issue runs sequentially. max_concurrency=3 runs three issues at once and cuts wall time by ~3x. Option D has the logic backwards: the Pregel runtime is thread-safe and works with both sync and async callers.

---

Traps to Avoid

Freshness trap (env vars): Using LANGCHAIN_TRACING_V2 and LANGCHAIN_API_KEY as your primary env variable names. These are legacy aliases that langsmith~=0.8 still reads, but the canonical v1.x names are LANGSMITH_TRACING, LANGSMITH_API_KEY, and LANGSMITH_PROJECT. If you mix both sets in the same .env, behavior becomes unpredictable (the last one wins in most shells). Pick the LANGSMITH_* set and document the legacy aliases as a comment only.

Freshness trap (imports): Importing evaluate, aevaluate, or traceable from langchain instead of langsmith. The correct imports are from langsmith import evaluate, aevaluate, traceable, Client. The langchain package does not expose these symbols in v1.x — you will get an ImportError or, worse, import a different, older implementation.

Conceptual trap: Believing a golden set score of 1.00 means Forge is production-ready. The golden set measures quality on a small set of known issues. It does not cover issues you have not seen, edge cases users discover in production, or regressions in Forge behavior on issues outside the categories you tested. A 1.00 pass rate on three issues is a starting contract, not a production guarantee. Build the golden set iteratively: start with the most common failure modes you observe in production, add regressions as they occur, and rerun the evaluation on every change to Forge’s core nodes.

---

Key Takeaways

• Three environment variables (LANGSMITH_TRACING=true, LANGSMITH_API_KEY, LANGSMITH_PROJECT) activate full LangSmith tracing for every LangGraph graph invocation — no changes to Forge required. The canonical v1.x names are LANGSMITH_*.
• aevaluate is the right choice for async LangGraph target functions; max_concurrency controls parallelism and prevents 10 sequential evaluations from taking 10x longer than necessary.
• The judge ≠ candidate invariant is hard: use a different model tier for the judge to avoid self-preference bias. If the Planner and Editor run on smart, the judge runs on fast. Document this explicitly in the code.
• A trajectory evaluator (agentevals) and an output judge (openevals) are complementary: one scores the path, the other scores the result. A correct result on a bad path is a warning, not a success.
• The CI regression gate is a pytest test marked @pytest.mark.langsmith that fails the build when avg_score < threshold. This turns a quality measurement into a versioned contract that prevents degradation from merging.
• The regression_gate in run_evals.py is the simplest possible gate: pass_rate >= 0.80. The Tester’s pytest verdict is deterministic ($0, no API) — that is your most reliable quality signal.
• Forge v1.0 is tagged. The portfolio artifact of this course is the v1.0 tag with the golden set scores in the commit message — proof that you built, deployed, and measured a production-grade LangGraph agent.

---

What’s Next + Newsletter

Want the full LangGraph mastery question bank (interview-style) and the next module in your inbox? Subscribe here — it’s free, like everything in this series.

Module 16 — Production Readiness and Mastery Review closes the course without a lab. It covers the production readiness checklist for deploying a LangGraph agent at scale, a 30-question mastery bank organized by concept area (CA1–CA14), a self-assessment grid that maps your weak spots to the modules that cover them, prep for LangGraph engineering interviews, “when NOT to use LangGraph,” and a personal debrief from building and deploying Forge for real.

Links:

• Module 14 — Deploy Forge: LangGraph Platform, the Server API, and Studio
• Module 16 — Production Readiness and Mastery Review
• Course index — LangGraph in Production
• Lab code for this module

---

References

• LangSmith documentation — Tracing — environment variables, @traceable, trace UI
• LangSmith documentation — Evaluation — evaluate, aevaluate, datasets, evaluators
• openevals PyPI — create_llm_as_judge, CORRECTNESS_PROMPT
• agentevals PyPI — create_trajectory_match_evaluator, create_trajectory_llm_as_judge, TRAJECTORY_ACCURACY_PROMPT
• LangGraph — How to add thread-level persistence — checkpointers and thread_id
• LangGraph — Human-in-the-loop — interrupt(), Command(resume=), idempotence invariant
• LangGraph — Durable execution — durability=, RetryPolicy, CachePolicy