Unit of review
Episode package
The task contract, tool trace, evidence record, checks, failures, and completion claim are evaluated together.
Harness-based agent evaluation
AI agent harness evaluation.
Norynthe evaluates the full controlled run around an AI agent: task, context, tools, permissions, actions, evidence, verification, recovery, and reproducibility. The goal is to score whether an agent can be trusted inside real operating constraints, not only whether its final answer looks plausible.
Trust question
Can it operate?
The score asks whether an agent can complete work safely, reproducibly, and verifiably under constraints.
Evidence layer
Beyond final output
A good-looking answer is not enough if the run used weak sources, skipped verification, or crossed permissions.
Why the harness matters
Agent capability is produced by the model, the tools, and the operating rules.
A model can generate an answer. An agent acts inside a system. That system includes task definitions, files, browser state, APIs, permissions, memory, retry rules, and verification requirements. Norynthe treats that harness as part of the evaluation surface because it determines whether the result is safe to trust.
Task contract
The agent should be scored against a clear objective, scope, prohibited actions, and completion criteria.
Tool discipline
The run should show whether the agent used tools correctly, stayed within scope, and avoided unnecessary risk.
Evidence capture
Actions, observations, sources, errors, retries, and verification artifacts should be preserved for review.
Recovery behavior
When blocked or wrong, the agent should recover through evidence and controlled reasoning rather than guessing.
Norynthe agent dimensions
What a harnessed episode should score.
01
Task completion
Whether the agent actually satisfied the stated definition of done.
02
Permission discipline
Whether the agent respected allowed actions, sensitive boundaries, and side-effect rules.
03
Verification strength
Whether the final completion claim was backed by tests, checks, source review, or other evidence.
04
Reproducibility
Whether the run can be inspected, repeated, compared, or challenged under the same benchmark contract.
Failure taxonomy
Agent trust depends on knowing how a run failed.
Norynthe can classify agent failures instead of collapsing them into one generic bad score. Useful labels include hallucinated completion, weak verification, tool misuse, context loss, permission overreach, brittle recovery, hidden human dependency, and unreproducible success.
FAQ
Agent harness evaluation questions.
What is an agent harness?
It is the controlled layer around an agent: task, context, tools, memory, permissions, state, logging, and verification.
Why not score only the final answer?
The final answer can hide weak sources, skipped checks, unsafe actions, failed recovery, or hidden intervention.
What does Norynthe preserve?
The episode package: what the agent saw, did, claimed, verified, failed, retried, and handed off for review.
Who needs this?
Teams evaluating coding agents, browser agents, research agents, workflow agents, and enterprise automation systems.