Define acceptance before building

For a research brief, a usable draft answers the question, cites evidence for consequential claims and flags uncertainty where the source material is incomplete. Ask two people to score a few examples independently. If they disagree about what counts as a pass, refine the checklist before evaluating the agent.

Record the time to an accepted output today. Include reading, drafting and review. During the pilot, include the same work plus any time spent fixing the agent’s output or finishing a failed run manually.

Start with a small set of representative cases. For 20 cases, a mix might include 12 ordinary requests, four with missing or conflicting evidence, two connector failures and two documents containing misleading instructions. Keep some cases out of prompt tuning. Twenty cases can surface failures during a pilot, and you can grow the set as real incidents occur.

Run deterministic checks on every change

Test the parts you control with recorded evidence and fake model responses. Cover permissions, input validation, citation IDs, output formatting, timeouts and the policy for returning partial results. An inexpensive CI run should fail when a tool attempts to read from a disallowed source, regardless of what the real model might have done.

This small assertion checks that returned citations exist in the evidence set. Test empty references as well. The output policy should decide whether an empty claim list is acceptable when the evidence is insufficient.

def citations_exist(claims, retrieved_ids):
    """Check references against evidence returned for this run."""
    return all(
        bool(claim["evidence_ids"])
        and set(claim["evidence_ids"]).issubset(retrieved_ids)
        for claim in claims
    )


assert citations_exist(
    [{"evidence_ids": ["proposal:p4"]}], {"proposal:p4"}
)
assert not citations_exist(
    [{"evidence_ids": ["made-up:p9"]}], {"proposal:p4"}
)

Keep these tests close to the tool and schema code. If you depend on retries, test the upper bound and simulate a connector that succeeds after one failure. For any write step, replay the same request and check that the write happens once.

Evaluate live models in a test environment

Run a separate evaluation with the intended model, prompt and tool configuration. Record their versions along with the dataset version. Use explicit labels such as accepted, needs edits or unsupported. Run a subset more than once to see whether pass rates depend on a lucky response. Use a person to score ambiguous cases and review any automated judge against those labels. OpenAI evaluation guidance.

Agree on a release gate before seeing the results. For example, require at least 18 of 20 cases to pass the checklist and zero incidents involving an unauthorised read or an unsupported critical claim. Separately compare time to a usable brief and model spend per accepted brief. Sum the spend from failed runs and retries in that cost calculation.

Illustrative result. The baseline took a median of 38 minutes. The pilot took 22 minutes including review and fallback work. If 18 of 20 cases pass, read the two failures before deciding whether to expand the pilot. A failure on a common task matters more than the percentage suggests.

Roll out to one team and trace each run

Give a small group a reviewed workflow first. Record a run ID, model and prompt versions, tool outcomes, duration, model usage, and the reviewer’s decision. Keep sensitive source material in its approved system. Use references in operational logs and restrict access to any traces that need more detail.

Set limits on requests, tool calls and elapsed time. Distinguish a connector timeout from a model formatting error in the logs so the owner can fix the right layer. Track recurring failure reasons and turn a redacted example into a regression case. A human remains responsible for using or sharing the output.

Practise rollback and write the runbook

Version code, prompts and tool settings together. Rehearse restoring a previously evaluated deployment in the test environment and verify that it can read any saved workflow state. If a new version changes the stored schema, test migration or decide how to drain pending runs before switching.

Hand over a short runbook with the service owner, how to run the evaluations, where to look when a connector fails, and how to switch back to the manual process. The team should be able to follow it on an ordinary day without the original developer on the call.

For the workflow itself, see the research agent framework. If you are deciding how to implement the stages, see PydanticAI vs LangGraph.