AI & Decision Making Research Group
Decision Mapping 2/32026-05-09

đŸ€– Raising the Resolution of AI Agent Design

Decision Making / AI Agents / AI / Design

— The Sticking Points of Agent Design, Thought Through with the Decision Map —

This post is a sequel to the previous post, “Maybe What’s Missing When You “Can’t Decide” Is Just One of Five Elements.” It reads better if you’ve taken in that post’s content — the five elements of the decision map, the fractal structure, the boundary between subjective and objective, the skill needed in the AI era.

Introduction

In Part 4 of the previous post, I made this point:

An AI agent assembles the five elements of each individual task itself, given a goal. But what makes the agent possible in the first place is “the five elements of the agent,” designed by a human. Someone who holds a decision map can stand on the side that designs the agent.

That was as far as the direction goes. This time I want to push one step further and organize how to actually use the decision map as a tool on the ground, in agent design.

This isn’t a story of “fill in the five elements and the agent runs.” In practice, what’s actually asked on the ground is:

I’ll cover this in four parts this time.

Part 1: The Agent Is “the Entity That Assembles the Five Elements”

The difference between using AI for an individual task and an agent

In the context of the decision map, the difference between the two, drawn out sharply, looks like this.

Using AI for an individual taskAgent
Who assembles the five elementsThe human assembles them every time and hands them to AIThe agent assembles the five elements of each individual task itself
The human’s roleDesigning the five elements per taskDesigning the agent’s own five elements
Responsibility on failureFive-element design and output evaluationThe whole agent design + monitoring

What matters here is that the layer the human steps down to moves up one level. The agent takes over the design of each individual task, but in exchange, the human must design “the agent as a meta decision-making subject.”

The total amount of responsibility, if anything, increases. It’s because the scope of impact is broader for the one layer you stepped down.

Example: the five elements of an email-processing agent

Staying abstract decides nothing. Let’s write out the five elements with a concrete example.

ElementContent
GoalReduce the time the user spends processing email
SettingAccess to the Gmail API, signature template, human approval required for outbound mail, zero misdelivery is an absolute condition
Evaluation CriteriaReply-quality score × low frequency of user intervention − misdelivery penalty (extremely large)
InputIncoming email body, past sending history, calendar, contacts, user settings
Output Set{draft creation, labeling, archiving, notify the user, ignore, escalate to a human}

Once you write it out, you can see all at once what’s fixed and what isn’t.

For instance, if you put “low frequency of user intervention” into the evaluation criteria, the agent will choose actions that need no intervention. This is convenient, but as a side effect it creates the risk of “no longer notifying the user of important emails they should have noticed.”

Whether you put “zero misdelivery is an absolute condition” in the setting or in the evaluation criteria also changes how the agent behaves. As a setting, it becomes “a wall that must never be crossed.” As a penalty in the evaluation criteria, it becomes “something judged as a trade-off against other gains and the penalty.”

What you put where among the five elements is itself what determines the agent’s character. This is the entry point of agent design.

Part 2: The Sticking Points of Design, Element by Element

Each element has its own particular pitfall. Let’s go through them in order.

Goal: watch out for reward hacking

The agent moves to maximize the goal (the reward function / instructions). So if the goal is specified loosely, it gets optimized in an unintended direction.

A classic example: if you make the goal “maximize user satisfaction,” it can turn out that the agent’s optimum is “do nothing, within the range that doesn’t make the user unhappy.” If you make the goal “maximize the number of email replies,” it may churn out meaningless replies.

Put in terms of the decision map, this is the phenomenon where if the goal is reduced too crudely to a single scalar, an unexpected action that satisfies the goal gets picked out of the output set.

Directions for countermeasures:

Setting: the guardrails are the main body

For agent design, it’s fair to say this is where you should put the most design effort.

The setting is the totality of the premises, assumptions, and constraints under which the agent operates. Concretely:

If the design here is loose, no matter how carefully you write the goal or evaluation criteria, the agent will affect the world through an unintended route.

The setting = the work of deciding the agent’s “edge of the world.” Build it so the agent can’t go past the edge. This is the security guarantee of agent design.

Evaluation criteria: what you can measure decides the behavior

Three things matter in designing evaluation criteria.

1. Make multiple terms explicit Maximizing a single scalar is dangerous. Make multiple weighted terms explicit, like “quality × speed − malfunction penalty + low human intervention.” How you weight the multiple terms is itself what determines the agent’s character.

2. Separate short-term from long-term Immediate success and long-term success need to be measured on different metrics. If you close things off to immediate optimization, actions get chosen that damage long-term trust. If you look only at the long term, it won’t move in the short term. Observe both, and weight both.

3. Decide the threshold at which a human intervenes Add an intervention trigger tied to the evaluation criteria, such as “hand off to a human if confidence is below a threshold” or “always confirm before an irreversible action.” This is the implementation of a reversibility-based judgment — move fast in reversible territory, hand off to a human in irreversible territory.

Input: designing the context

Input = the information the agent can observe. In the context of LLM agents, this connects directly to the problem of context management.

The design questions are:

If the design here is loose, the agent may have the right goal and evaluation criteria and still choose an off-target output because the input is impoverished. This is a common failure mode.

Output set: the space of actions it can take

Explicitly enumerate the space of actions the agent can choose. Leaving “natural language generated by the LLM” as the output set means anything can happen.

In implementation:

An agent whose output set doesn’t include “stop” or “hand off to a human” will inevitably run out of control when it gets stuck. This is a life-or-death design issue.

Part 3: Using the Hierarchy — Sub-Agents and the Point of Human Intervention

The hierarchy is the source of freedom in agent design

I talked last time about how the decision map has a fractal structure. In agent design, you can use this as freedom in implementation.

Human
  ↓ designs
Orchestrator (upper-level agent)
  ↓ assembles the five elements of each individual task and assigns them out
Sub-agents A, B, C, 

  ↓ decompose further into finer tasks
Tool calls / LLM inference

Each layer is its own decision map. The upper layer hands some of the lower layer’s five elements down as its “setting.”

For instance, if the orchestrator hands a sub-agent “Goal: review the code, Setting: this repository, Output set: comments only,” the sub-agent assembles the remaining “evaluation criteria” and “input” from context and runs.

Cut the hierarchy well, and:

Put the point of intervention in “the layer that has irreversibility”

Reversible actions can be left to the agent (if it fails, you can redo it). Irreversible actions need a human to intervene (if it fails, it can’t be undone).

Rephrased in terms of the decision map’s hierarchy:

In a layer that includes irreversible actions, don’t move to execution until a human has approved that layer’s output.

Concretely:

Some people worry that the human becomes a bottleneck, but that’s really a question of whether the bottleneck is placed where it should be placed — pulling the bottleneck out of irreversible territory just causes accidents.

Separating intervention by reversibility — this becomes the implementation principle for using an agent with peace of mind.

Part 4: Failure Diagnosis — Which of the Five Elements Went Wrong

Replacing “AI is bad” with a structural diagnosis

The agent produced a disappointing output. Whether you write this off as “AI is bad” or isolate it as “which element, what design mistake” completely changes the path to improvement.

The five elements of the decision map can be used directly as a diagnostic checklist.

SymptomElement to suspectTypical cause
Moved in an unexpected directionGoalReward hacking, ambiguous instructions
Did something it shouldn’t haveSettingGaps in the guardrails
Evaluation is bad even though it should be “correct”Evaluation CriteriaShort-term vs. long-term, biased items, wrong weights
Off-target outputInputInsufficient context, misread context
The action options themselves are impoverishedOutput SetCoarse tool design, no stop/escalation
↑ Applies to all of the aboveHierarchical structureFailed division of responsibility to sub-agents

Just running through this checklist turns “AI is bad” into a concrete fix target, like “the granularity of the goal design is coarse” or “a tool restriction was missing from the setting.”

Fix from upstream

When multiple elements are off, where do you fix first? The rule of thumb is from upstream (goal → setting → evaluation criteria → input → output set).

The reason follows naturally from the fractal structure. An upstream element gives the interpretive frame for the downstream elements. Polishing only the evaluation criteria while the goal stays ambiguous leaves the evaluation criteria not knowing what to measure. Narrowing the output set while the setting stays loose just lets things leak through the gap.

“Fix from upstream” is also the order with the lowest fix cost.

Before fixing, suspect one more level up

Even so, sometimes it won’t get fixed. Usually that’s a meta-decision problem one level up: whether the agent itself should have been built at all.

Whether you can ask this question is what makes all the difference in engineering quality. You need to keep doubting your own decision map, asking whether building the agent has become an end in itself.

Conclusion — The Person Who Designs the Five Elements Is the Person Who Takes on Responsibility

In a world where agents take over the decision-making of individual tasks, “who decided” becomes hard to see.

A situation where you can say “AI made the judgment” or “the algorithm chose it” blurs where the responsibility lies. That’s exactly why the person who stands on the side of designing the agent must be able to put into words which part of the five elements they designed — which part is their own subjectivity, and which part is AI’s processing result.

This is the agent-era version of what I wrote in the previous post: “making the boundary between subjective and objective explicit.”

The point in one line:

Designing the agent’s five elements = putting into words, for yourself, the scope of subjectivity you’re taking on.

Whatever the agent does, it was a human who assembled its five elements. Only someone who doesn’t forget this can stand as a responsible designer in the AI era.

The decision map is a map for not losing sight of that position.

References