AI Agents Series | Building an Agent Harness with Jev

Back to the AI Agents Series

An agent can spend a surprising amount of effort deciding how to do the work: which model should handle a request, whether a proposed command is acceptable, and whether the final answer meets a rubric. Each decision may be small, but putting a language-model call at every checkpoint adds delay and cost to the whole workflow.

In Building a Harness with Jev, Sydney Runkle from LangChain demonstrates a model designed for these bounded judgments. Jev a System-One model, from TypeSafe AI, reads context and returns typed decisions and probabilities. The surrounding application uses those results to select a model, gate a tool call, or evaluate an answer.

TL;DR

• A harness makes decisions around the agent loop. Routing happens before model execution, tool checks before side effects, and evaluation after an answer or trace is available.
• Jev returns bounded answers. Choice selects a category, Score rates an ordered scale, and Noul returns a yes/no probability. The application still decides what to do with those values.
• One state can support several independent questions. This is useful for classifying a ticket’s owner, urgency, and frustration together. A later question cannot consume an earlier answer within the same request.
• The main LLM still handles generation and open-ended reasoning. Jev supplies decisions at selected points in the workflow.
• Measure the complete workflow. Faster classification helps only if routing preserves task quality, tool checks detect the relevant risks, and evaluation agrees with a useful reference.

0. What is Jev

Jev is a decision model developed by TypeSafe AI, which calls it a System One model. The name borrows Daniel Kahneman’s distinction between fast, intuitive judgments and slower, deliberate reasoning. In this setting, the analogy describes Jev’s intended role: making quick, bounded decisions that software can act on. TypeSafe’s introduction.

A typical chat LLM generates a response token by token, whether that response is prose, code, or JSON. Jev instead returns typed decisions and probabilities without generating a text response. Its outputs include a choice from caller-defined alternatives (Choice), a yes/no probability (Noul), or a rating on an ordered scale (Score). For example, an application could supply a support ticket and ask Jev to choose between billing, technical support, and sales, receiving the selected category and probabilities for the alternatives. Output types.

Functionally, Jev can be used as a general-purpose zero-shot classification and scoring model: the application describes a task and its possible outcomes without supplying task-specific labeled examples or fine-tuning a new classifier. The categories are defined when making a request, rather than fixed to one task. This overlaps with established zero-shot methods, including NLI-based classifiers and GLiClass, which also produce label scores without generating text. The relevant comparison therefore concerns decision quality, probability calibration, latency, and cost.

TypeSafeAI reports 20–200× faster inference and 40–400× lower cost than LLMs on classification-style tasks. However, these figures depend on the workload and comparison model. For example, LangChain’s evaluation announcement reports an average of 0.44 seconds per call for Jev versus 2.16–2.83 seconds for its LLM judges, corresponding to approximately 4.9–6.4× faster inference in that experiment. This supports a speed advantage in that setting, but does not establish the same gains across agent workloads or superiority over specialized zero-shot classifiers.

1. Where Jev fits in the agent loop

The core agent loop includes basic steps: the LLM model executes its tools and gets the observations. The model can continue acting or produce a final response. Sydney introduces two interfaces that make this cycle usable in software: tool calling, which structures action requests, and structured outputs, which give the final answer an expected schema. Watch 00:12–01:38.

Here, the harness is the runtime around that model: it supplies context, dispatches tools, applies execution rules, and records what happened. A model’s proposed action becomes a real action only when this software executes it.

In the process, several places where a small judgment has a large effect:

Jev fits in the agent loop by providing judgments precisely and efficiently. For example, getting the user request and routing criteria then predicting which model is the most suitable for this task.

2. The interface: state, questions, and typed answers

Jev requires State –what the model should examine and Questions –the judgments to make about that state, including the predefined categories. Jev returns answers with predefined categories instead of a generated explanation as normal LLMs. Watch 01:38–04:11.

As shown in the example, the questions field has: is_urgent is a predefined class, belonging to type noul (talked about it later, basically noul means is this true question), and instructions which is a description of the predefined class. Jev returns answers with the noul probability of 0.999, indicating that the state has been classified as is_urgent with 0.999 probability.

Three question types, three different meanings

Jev supports three questions types each with different meaning, as illustrated through example below:

A Choice returns one of the alternatives supplied by the application, along with a probability distribution. Its confidence summarizes how concentrated that distribution is; it is a different quantity from the winning option’s probability. When valid inputs may fall outside the listed categories, an other option gives them a place to go. See the Choice and confidence documentation.

A Score uses ordered descriptions indexed from zero. For calm, frustrated, and very angry, the indices are 0, 1, and 2. The score is the probability-weighted mean of those indices. Thus 1.035 is a position on this scale, not a percentage. Different probability distributions can have the same mean, so the distribution matters when a decision is consequential. This is specified in the Score documentation.

A Noul is a binary classification. It returns a probability whether the state is true. A value near 0.5 expresses uncertainty about yes versus no, rather than medium urgency. Code must supply the threshold or review rule. See the Noul documentation.

3. Example 1 - A minimal LangChain example

Install the integration in your Python environment and configure TYPESAFE_API_KEY with a key from TypeSafe:

uv pip install langchain-typesafe

The following example uses an original incident report and adds application logic after classification:

from langchain_typesafe import Noul, TypeSafeClassifier

incident = "Customers cannot finish checkout after the latest release."
questions = {
    "customer_outage": Noul(
        instructions=(
            "Does this report indicate an ongoing failure "
            "that blocks customer purchases?"
        )
    )
}

classifier = TypeSafeClassifier()
result = classifier.invoke({"state": incident, "questions": questions})
p_outage = result.nouls["customer_outage"].noul

# Illustrative triage thresholds, to be evaluated on your own reports.
if p_outage >= 0.90:
    queue = "incident-triage"
elif p_outage <= 0.10:
    queue = "standard-support"
else:
    queue = "needs-review"

print({"probability": p_outage, "queue": queue})

The call shape and .nouls accessor follow the current integration documentation. The example was checked against that interface but was not executed against the paid API; no particular output probability is asserted here.

Notice the boundary between inference and policy. Jev estimates whether the report describes an outage. The if statement chooses a queue. The two thresholds are illustrative design choices, not Jev defaults or values supplied by the tutorial. Retaining an uncertain region gives the application somewhere to send ambiguous reports.

4. Example 2 - Model routing: choose the model before paying for the work

This section shows another example of Jev - choosing a appropriate model based on input request, e.g., a request to change a label and a request to diagnose an intermittent concurrency bug need different amounts of reasoning. The first use case asks Jev to classify the request against criteria for the available models, then lets the harness invoke the selected model.

The supplied ModelRouterMiddleware has a specific scope: it classifies the latest human message once at the start of a run, then uses that selection for the run’s model calls. It does not automatically reconsider the route after every tool observation. The implementation stores the choice in model_route. These details come from the router implementation, checked on September 22, 2026.

Design consideration: routing needs evaluation at the task level. A useful accounting is:

Total task cost = routing + selected model work + retries or escalation

A cheaper first model can be a poor choice if it repeatedly fails and forces the expensive model to restart. Conversely, routing every short prompt to the inexpensive model may miss a brief but consequential request. Evaluate the criteria against representative tasks, recording completion quality, total spend, elapsed time, and the frequency of escalation.

If later observations reveal that a task is harder than it first appeared, escalation must be implemented explicitly. That is a runtime policy beyond the tutorial’s initial routing example.

5. Example 3 - Tool checks: classify before side effects occur

A tool check occupies a different boundary. The agent has already proposed an operation, but the runtime has not executed it. Jev evaluates whether that action is risky, and middleware can refuse the call. User uses deleting a database or important files as motivating examples. Watch 06:47–07:28.

Sydney says she had disabled an earlier risk check because its delay made her coding agent feel unproductive, then enabled it after Jev reduced the delay. This is a useful observation about repeated checks: their latency affects whether people keep them in a workflow. The anecdote does not measure the classifier’s ability to detect harmful actions.

The current AutoModeMiddleware is experimental and requires the langchain-typesafe[experimental] extra. Its documented behavior is narrower than a general approval system:

• Only explicitly listed tools are checked.
• Refusal returns an error ToolMessage instead of executing the operation.
• Human approval requires a separate human-in-the-loop mechanism.

See the integration documentation.

The current implementation sends the proposed tool details and up to 30 recent messages to the classifier. It refuses calls whose risk probability is at least 0.5. That threshold is an internal constant, not a public threshold= constructor argument. A classification exception propagates before execution rather than approving the call.

Design consideration: define risk in terms of the actual operation and its context. Deleting a disposable test directory and deleting the production backup store may look syntactically similar. Useful state includes the target, arguments, relevant user authorization, and environment. A probability-based check should operate within the permissions and execution boundaries already required by the application; it cannot establish that every dangerous action will be recognized.

6. Example 4 - Online evaluation: give the judge evidence and a rubric

The final use case evaluates an answer after the agent has produced it. User supplies the original question, the answer, and criteria such as correctness, grounding, and completeness. The goal is to make evaluation affordable enough to apply more broadly across agent runs. Watch 07:28–08:47.

The refund example makes the reason for separate criteria clear. An answer might give the correct return period while omitting important conditions. Collapsing those dimensions into a single “good answer” question makes the failure harder to locate.

Design consideration: an evaluator needs evidence for each judgment. Grounding requires the relevant source material; agreement with a reference requires a reference. A judge given only a polished answer cannot recover missing policy details reliably. The presence of a citation is also different from support for the claim being cited.

For a practical rubric, I would keep separate feedback fields for the core answer, source support, and required conditions. Code can check directly observable properties, while a model can assess semantic ones. Disagreements with human review then point to a particular criterion to improve. This connects to Evaluation and Observability.