Agent Lightning 1.0 has no APO: what we kept

6 min readSimon BudziakBy Simon Budziak

On this page6 sections
Title card reading The critique outlived the tool, with the Agent Lightning logo and name in the top-left corner. A navy background with a single document panel outlined in thin gold, holding a stack of glowing cream lines. Four of those lines have a gap bitten out of the middle, and a fine gold leader line runs from each gap to a small gold ring in the margin, marking the four places the document leaves undefined.

In February 2026 we wired Microsoft Research’s Agent Lightning to a LangGraph scheduling agent to see whether automatic prompt optimization does anything useful. It did, and the useful part was not the score. APO’s durable output was its critique: it named four specification gaps in our prompt that we had not noticed, and fixing those is what changed the agent’s behaviour. Since then the algorithm has been removed from the framework entirely, so this is now a post about what survives the tool that taught it, plus a correction to a number we should not have published the way we did.

What we built, and what APO actually produced

The agent picks a meeting room under real constraints: capacity, equipment, accessibility, booking conflicts, and a soft preference for shorter walks. It is an ordinary LangGraph state machine, an agent node and a tool node with conditional routing. The dataset is 57 hand-written tasks covering capacity edge cases, equipment mismatches, overlapping slots and tie-breaks. Grading is an LLM judge scoring 0.0 to 1.0 against an expected choice. Code: soba-labs/langgraph-agent-lightning-optimization.

Our baseline prompt was one sentence. What APO returned was not a rephrasing, it was a list of things we had left undefined:

  • Capacity semantics. Does a room with capacity 10 satisfy a request for 10 attendees, or is 10 a limit to stay under?
  • Equipment matching. Case sensitive? Are aliases allowed, so that a “TV” satisfies a “display”?
  • Tie-breaking. When several rooms clear every hard constraint, which wins, and by what rule?
  • Overlap. What exactly counts as a booking conflict at the boundaries?

Every one of those is a specification gap, not a wording problem. They are the questions a careful reviewer asks, and we had shipped a prompt that answered none of them. The rewritten prompt was three to four times longer and read like a spec.

That is the part worth keeping, and it does not depend on the tool: a process that reads your failures and names your undefined cases is valuable even if you discard the generated prompt and write the rules yourself.

What our number does and does not say

The original version of this post led with a 37% improvement, roughly 0.65 to 0.89 average score. That framing claimed more than our evidence carries, so here is the correction.

It was one run. Validation was 10 tasks per iteration out of the 57. The figure is a mean LLM-judge score, not a task success rate, and those differ: a judge awarding partial credit moves a mean without a single task flipping from fail to pass. Our public repo records the method and the dataset but no scores, so there is no artifact anyone can check that number against, ourselves included.

The direction was real. The precision was not earned. A defensible figure needs a held-out set you never optimise against, more than one seed, and the run’s outputs committed somewhere.

A three-stage vertical diagram of the automatic prompt optimization loop on a navy card. Evaluate runs rollouts with the current prompt and collects scores from the LLM judge. Critique analyses the traces and writes out the ambiguities it found. Rewrite applies that critique to produce a new prompt template. A note underneath records that the loop optimises against the judge, so the judge's calibration sets the ceiling on everything above it.
The loop optimises against your grader, which makes the grader, not the algorithm, the thing to get right first.

The optimiser optimises against your judge

This is the part we would put first if we wrote the post again. An automatic optimiser maximises whatever your reward function says. If the judge is miscalibrated, the optimiser will faithfully drive the agent toward the judge’s mistakes, and the score will climb while quality does not.

That failure is invisible from inside the loop, because the number keeps going up. It only shows up when a person reads the outputs. We wrote separately about why calibrating an LLM judge against human labels is the actual work, and that post is really the prerequisite for this one. Get the judge agreeing with a human first; only then point an optimiser at it. Evals come before optimisation, not after.

Agent Lightning 1.0 removed APO

Here is what changed, and it is why you cannot follow the original integration today.

Agent Lightning v1.0.0 was, in the project’s own words, a release in which “we have completely refactored the codebase, making it lightweight”, built around “~3,500 lines of code” and aimed at reinforcement-learning training against the real agent harness, with Kubernetes support and a coding-agent example moving SWE-bench Verified from 41.8% to 56.4%.

The refactor did not carry APO across. At tag v0.3.0 the repository ships agentlightning/algorithm/apo/, its .poml prompt templates, dedicated CI workflows and an algorithm-zoo/apo documentation page. At v1.0.1 and on the default branch, none of that exists: the package is client, config, controller, hooks, schemas, server and verl. The @rollout decorator, LitAgent and PromptTemplate that the integration was built on all belonged to the 0.x line.

So if you want the flow this post describes, pin the 0.x line. The platform constraint we hit belongs there too: the 0.3.0 installation guide states Agent Lightning “is officially supported on Linux distributions” and that “macOS and Windows (outside of WSL2) are not supported”, which is what forced our shared-memory workaround on a Mac.

One detail did outlive the change, in a way. We reported in February that APO’s poml dependency was undeclared and should be fixed upstream. Checking the published package today, agentlightning 1.0.1 still declares no poml, and now requires Python 3.12 or newer. The dependency was never fixed; the algorithm that needed it left instead.

What we would do now

Automatic prompt optimization is worth running when you have a real dataset, a grader you have checked against a human, and prompts that have plateaued. The specific tool we used is no longer the way to get it, and that is the ordinary condition of this field rather than a scandal.

What we kept is smaller and more portable than the integration: run your failures through something that will name your undefined cases, and treat the resulting list as a specification review. You can do that with an optimiser, or with a careful prompt and an afternoon. The four gaps APO found in our scheduling prompt were not exotic. They were the questions we should have answered before shipping it.

Sources

Last updated:

How we build on LangChain in production