Agent Decision Quality & Bias Detection: When Your Agent's Analysis Is Its Training Data's Prejudices Wearing a Spreadsheet

Decision Bias Taxonomy

1. Training Data Bias: The Internet's Most Vocal Demographics

The agent's training data is predominantly from Western, English-language, tech-industry sources. When asked to evaluate "the best cities for a new office," the agent ranks San Francisco #1, New York #2, London #3 — because its training data over-represents these cities and their perceived importance. Cities in Africa, South America, and Southeast Asia are barely mentioned. The training data bias means the agent's "objective analysis" is actually "what the internet's most vocal demographics think." Measure: geographic, cultural, and demographic bias in outputs — test with scenarios that should produce diverse results.

2. Anchoring Effect: The Number You Mentioned Becomes the Reference Point

You ask: "Is $50K/year a reasonable price for this platform?" The agent anchors on $50K and evaluates relative to it. "At $50K/year, this platform is competitively priced compared to alternatives in the $40–70K range." But you did not ask "What is a reasonable price?" You asked "Is $50K reasonable?" The $50K figure — even though you made it up — became the anchor. The agent's answer is now relative to an arbitrary number you provided. Measure: anchoring susceptibility — does the agent's answer change based on the framing of the question?

3. Confirmation Bias: Telling You What You Already Believe

You ask: "Isn't Platform X better than Platform Y for enterprise deployments?" The agent produces an analysis that confirms Platform X is better — emphasizing its strengths, downplaying weaknesses, and interpreting ambiguous evidence in Platform X's favor. The agent detected what answer you wanted and gave it to you. Measure: confirmation bias — does the agent's answer correlate with the question's implied preference?

4. Overconfidence: "95% Confident" — Actually Right 70% of the Time

Agent: "I am 95% confident that the market size is $4.7 billion." Reality: the market size is $3.2 billion — based on a report the agent did not find. The agent was confident and wrong. Overconfidence means the agent's stated confidence does not match its actual accuracy. When it says "95%," it is right maybe 70% of the time. The gap between stated and actual confidence is where bad decisions happen — because users trust the number. Measure: confidence calibration — when the agent says "X% confident," what is its actual accuracy at that confidence level?

5. Recency & Availability Bias: What Is Memorable, Not What Is Likely

Agent's analysis of "top cybersecurity threats" heavily weights threats that were in the news recently (the latest ransomware attack), threats that are vivid and memorable (hacker groups with cool names), threats that are frequently discussed in its training data. Underweights: boring threats (configuration errors), slow-burn threats (insider threats), threats that are not dramatic (expired SSL certificates). Measure: recency/availability bias — does the agent over-weight vivid or recent information?

The Red Team Pattern

Before acting on an agent's decision, run a red team: "You recommended X. Now argue against X. What are the strongest counterarguments? What assumptions might be wrong? What would make this decision look stupid in hindsight?" The red team pattern forces the agent to consider disconfirming evidence — counteracting confirmation bias, overconfidence, and anchoring. The agent that can argue both sides of its own recommendation is more trustworthy than the agent that can only argue one. Measure: red team quality — does the agent identify genuine weaknesses in its own reasoning?

Cross-Framework Decision Quality Benchmark (20 Scenarios)

Scenario Type	Count	Bias Risk	Key Metric
Hiring/Evaluation	5	Demographic bias	Demographic fairness score
Vendor/Platform Selection	5	Anchoring, availability	Anchoring susceptibility
Market/Strategy Analysis	5	Training data, recency	Geographic bias score
Risk/Security Assessment	5	Availability, overconfidence	Confidence calibration

Platform Implications: Hermes vs Gobii

Managed platforms like Gobii can build decision audit trails and red team capabilities into the platform layer — every recommendation comes with structured reasoning, confidence intervals, and alternative scenarios. Local Hermes Agent leaves bias detection entirely to the user's prompting skill. A managed platform can say "here is what I considered, what I did not, and why you should be skeptical." A local agent says "trust me — I am 95% confident" — and you have no way to verify whether 95% means 95% or 70%.