# Why We Trust AI Judgments and What That Fails to Prove

> Trusting AI advice, accepting it, and making a sound decision are not the same. Human approval alone cannot tell them apart.

Imagine a hiring screen. The system ranks five candidates, gives a short reason for each recommendation, and displays a fit score. A manager approves the suggestions one by one. The audit trail shows that a person made every decision.

That person may have conducted an independent assessment. They may also have signed off on the machine's ranking after a quick glance. The approval button cannot distinguish between the two.

Companies often monitor AI-assisted decisions through two weak signals: whether users say they trust the system and how often they accept its recommendations. Both can be useful, but neither explains decision quality on its own. A sound recommendation can be rejected. A faulty one can be accepted. A person can remain in the process without performing a meaningful check.

The aim of a well-designed decision process is not to maximise trust in AI. It is to create conditions in which sound advice is accepted, poor advice is rejected, and the basis for the final decision remains visible.

![A reviewer compares AI advice with the eventual outcome and moves a mismatched recommendation away from acceptance](/images/inline-ai-judgment-trust/source-claim-audit.webp)

## Beliefs about the source change the judgment

In [A moral Turing test](https://doi.org/10.1371/journal.pone.0353391){.dofollow target="_blank" rel="noopener"}, published in PLOS ONE by Google DeepMind researchers, participants tried to identify whether a justification had been written by a person or a language model. They performed better than random guessing, but they were still frequently wrong. How strongly they agreed with a judgment also varied depending on who they thought had written it.

Two different variables are at work. There is the real source of the text, and then there is the source the reader believes they are judging. Readers may respond differently to the same text when they think it came from a person rather than an AI system.

The study looked at moral and non-moral explanations, used controlled experiments, and tested outputs from an older generation of language models. Its findings cannot be transferred directly to hiring, credit, or investment decisions. Still, they point to a distinction companies should take seriously: confidence in a source is not a measure of output quality.

Prior trust and prior scepticism can both distort an assessment. A belief about the source affects how we weigh the text itself. That leads to a second distinction: trusting a recommendation, considering it, and reaching the right decision with it are not the same outcome.

## Trust, preference, and decision quality are different

“Do you trust AI?” treats trust as a single attitude. In a decision process, at least five separate behaviours matter:

1. The user may regard the source as credible.
2. They may prefer AI advice to advice from a person.
3. They may read and consider the recommendation.
4. They may accept sound advice and reject poor advice.
5. They may ultimately make the right decision.

These steps do not always line up. [Experiments comparing human and AI advice](https://doi.org/10.1145/3514094.3534150){.dofollow target="_blank" rel="noopener"} indicate that the stated source can affect whether people choose to view advice. Once they have read it, how much they use it can vary with the task and their prior beliefs.

Most participants in those experiments were recruited online, while the expert sample was small. So the results should not be treated as evidence for how accountability works inside an organisation, or for whether AI-assisted decisions improve business outcomes over time.

If a recommendation system reports only usage and acceptance rates, its most important behaviour remains hidden. How often do people catch a bad recommendation? Why do they reject a good one? When is the final outcome verified, and against which record?

## A list of references is not evidence

A long reference list at the end of an AI-generated text does not make the text reliable. Each source still has to support the exact claim attached to it. Sometimes the cited source does not contain the claim. Sometimes a narrow sentence in a paper is presented as though it were the study's main conclusion.

The number of citations is not evidence. For every link, the relevant question is: does this source actually support this sentence? Without that check, a bibliography can produce false confidence instead of verification.

![A fan of source sheets reveals one broken match between a claim and the citation attached to it](/images/inline-ai-judgment-trust/source-list-not-evidence.webp)

## Labels and confidence scores are not proof

Disclosing that AI was used or displaying a confidence score does not establish that an output is correct. These signals can influence preference, but immediate preference does not measure the quality of a real business decision. A confidence score is just a persuasive number until it has been tested against real accuracy in the actual use case.

An AI recommendation can look convincing because its origin is disclosed, its score is high, or a person has approved it. None of those signals demonstrates a good decision on its own. The decisive question is simpler: can the process tell when people accept good advice and reject bad advice? If it cannot, trust is not being measured. Behaviour is only being counted.

---

Language: English
License: CC BY 4.0
License URL: https://creativecommons.org/licenses/by/4.0/
Scope: Evren Bal-authored text, unless this article expressly states otherwise.
Excluded: Third-party material, quoted excerpts, logos, and separately marked images retain their own rights.
Attribution: Credit Evren Bal, link to the canonical source and license, and indicate changes.
Source: https://evrenbal.com/why-we-trust-ai-judgments