← ResearchAction state

A technical summary of Nenya

The July results in five pages: the problems long-running agents meet, and what changes when their commitments are kept in force.

Nenya at a Glance: A Technical Summary

Lindell S. Cumes

This is our July research in five pages. It starts from three problems every long-running agent meets: rules fall out of the context window, summaries turn tentative points into settled facts, and rules nobody wrote down get broken without anyone noticing.

In the paper’s thirty-day model, an agent running Nenya keeps 95% of its standing commitments. Unaided agents keep fewer than half.

What we found

  1. On a regulated task with four kinds of constraint, Nenya reached a fidelity of 1.000. The closest alternative scored 0.886, and an unaided agent 0.429.
  2. An unaided model breaks an unwritten rule 55 to 61% of the time, every time it meets it, and does not improve. Nenya breaks it at that rate the first time, then less than 1% of the time. Over eight encounters that is 0.61 expected violations against 4.64.
  3. More commitments barely move the result: fidelity is 0.995 with four in force at once and 0.986 with forty-eight. An unaided agent falls from 0.429 to 0.250.
Continue reading →