Key points
- Changing the agent harness while keeping the model constant significantly alters task completion rates on AppWorld benchmarks.
- CUGA achieved higher scenario consistency than Hermes or Deep Agents in the controlled experiment involving ninety tasks.
- The study highlights that harness selection should align with specific workload requirements rather than seeking a universal best option.
The claim
The report states that using the same open-weight model across three different harnesses resulted in completion rates ranging from thirty per cent to eighty-one per cent. This variation occurred on ninety tasks within the AppWorld benchmark, which simulates multi-step interactions across various applications.The study shows that the software framework controlling an AI agent can change its success rate more than the AI model itself.Fixing the LLM reveals that changing the harness moves task completion from thirty per cent to eighty-one per cent.
The response
The authors clarify that this is not a definitive ranking of the harnesses. Hermes and Deep Agents were not designed for this specific type of multi-app API work, so their lower scores reflect a mismatch with the test conditions rather than inherent inferiority.
The bottom line for teams
Enterprise agents require strict rule enforcement and operation within secure boundaries, which consumer-focused harnesses may not provide. CUGA is presented as a solution for these specific enterprise needs, including support for sovereign clouds and air-gapped deployments where data must remain isolated.
What the paper shows
The research tests whether the agent harness influences outcomes when the model is held constant. Using GLM-5.3-flash, the team ran CUGA, Hermes, and Deep Agents on ninety AppWorld tasks.reported CUGA passed twenty scenarios completely, compared to eight for Hermes and two for Deep Agents. Limitations include a small sample size, non-identical setups, and a lack of policy enforcement testing.
Evidence: reported
Same Model, Different Harness: CUGA, Hermes, and Deep Agents — When to Pick CUGA?aifleet-huggingface
Why it matters
- Model evaluators can isolate harness performance by fixing the model, revealing infrastructure-level bottlenecks in agent workflows.
- The significant gap in scenario consistency suggests that harness architecture heavily influences reliability in multi-step tasks.
- Enterprise teams must consider deployment constraints like air-gapping when selecting harnesses, as consumer tools may lack these features.
- The study indicates that current benchmarks may not fully capture enterprise-specific requirements like strict rule enforcement.
Topics: agent harnesses, IBM Research, AppWorld, CUGA, model evaluation
Updated
The strongest doubt
Counterpoint from the sources
The results may not generalize to other tasks due to the limited ninety-task sample size and non-identical harness configurations.
For readers
Researchers
- The study isolates harness impact by keeping the model constant across three different tools.
- SGC rewards consistency rather than a lucky run, making it the most useful number for this context.
Developers
- CUGA offers features like automatic reflection and verification before execution for enterprise tasks.
- Hermes and Deep Agents are better suited for personal assistants and long research tasks respectively.
Business
- Enterprise agents require rule enforcement and secure deployment options like sovereign cloud support.
- Choosing the right harness is critical for ensuring agents can handle complex business system interactions.
Ask the article
Local keyword search of this article; no generated answers.Did this article give you the evidence you needed?
Your choice stays on this device. No public tally.Questions readers ask
Which model was used in the experiment?
The experiment used the GLM-5.3-flash open-weight model for all harness comparisons.
What benchmark was used for testing?
The researchers used AppWorld, a benchmark featuring ninety tasks across thirty scenarios involving multiple applications.
Did the study test policy enforcement?
No, the runs did not include policy enforcement, so the results do not measure that capability.
Connected stories
Reading receipt
- Changing the agent harness while keeping the model constant significantly alters task completion rates on AppWorld benchmarks.
- CUGA achieved higher scenario consistency than Hermes or Deep Agents in the controlled experiment involving ninety tasks.
- The study highlights that harness selection should align with specific workload requirements rather than seeking a universal best option.
Still open: How do these results translate to real-world enterprise environments with complex policy constraints?
Still open: Will other benchmarks adopt similar controlled harness comparison methodologies?