As AI systems take on greater responsibility for analysis, synthesis, and recommendation, the human role shifts decisively towards evaluation and judgement. The quality of human-AI collaboration depends less on the sophistication of the AI and more on the rigour with which humans assess what it produces. Kahneman's (2011) framework for understanding cognitive processing provides a useful foundation: AI outputs engage our fast, intuitive System 1 thinking, which tends to accept confident-sounding information at face value. Effective orchestration demands that we engage System 2, the slower, more deliberate analytical process, before acting on AI-generated recommendations.
This is not an abstract concern. Dell'Acqua et al. (2023) found that knowledge workers who used AI for tasks inside its capability boundary saw substantial productivity gains. Those who used AI for tasks outside that boundary, and lacked the critical thinking to recognise the difference, produced work that was measurably worse than what they would have achieved without AI. The researchers describe this as "falling off the jagged frontier," a phenomenon that occurs precisely when humans trust AI outputs without sufficient scrutiny.
The anatomy of AI error
Critical thinking in the context of AI collaboration requires understanding the characteristic failure modes of large language models. These differ fundamentally from the types of errors that human colleagues typically make, and so the evaluation strategies must differ accordingly.
Confabulation (sometimes called hallucination) occurs when AI systems generate plausible, internally consistent, but factually incorrect information. Unlike a human who guesses and signals uncertainty, AI systems present confabulated content with the same confidence as verified facts. Mollick (2024) notes that this creates a particular trap for users who evaluate AI outputs based on how confident they sound rather than whether they are correct.
Sycophancy is the tendency of AI systems to agree with the user's implied position, tell them what they appear to want to hear, and avoid challenging their framing. For an orchestrator seeking genuine analytical input, this behaviour pattern can reinforce existing biases rather than surfacing the alternative perspectives that robust decision-making requires.
Anchoring on prompt framing means that the way a question is posed significantly influences the answer the AI produces. Two different framings of the same question can produce contradictory recommendations, each presented with equal confidence. The critical thinker tests for this by deliberately reframing questions and comparing outputs.
There is also a verification asymmetry worth noting: AI systems can generate a 2,000-word analysis in seconds. Verifying whether that analysis is accurate, complete, and appropriately nuanced takes considerably longer. Organisations that measure AI productivity purely by output volume, without accounting for the verification burden, systematically underestimate the true cost of AI-augmented work and create incentives for insufficient review.
Calibrating trust dynamically
Effective critical thinking in human-AI collaboration is not about adopting a fixed posture of either trust or scepticism. It requires calibrated trust: a dynamic assessment that adjusts based on the specific task, the specific AI system, and the specific context in which the output will be used.
High trust is warranted when the task is well within the AI's documented capability, the stakes of an error are low, and the output is easily verifiable. Lower trust is warranted when the task involves current events or rapidly changing information, niche expertise, complex causal reasoning, or output that is difficult to verify independently.
Jarrahi (2018) observes that the most effective human-AI collaboration patterns emerge when humans understand their own cognitive biases as clearly as they understand the AI's technical limitations. The orchestrator who knows that they are susceptible to confirmation bias, for example, will be more vigilant when an AI output aligns conveniently with their existing view. This metacognitive awareness is a core component of the critical thinking capability that effective orchestration demands.
Cognitive biases in human-AI interaction
Several well-documented cognitive biases become particularly hazardous in human-AI collaboration contexts. The automation bias describes the tendency to over-rely on automated systems and to under-weight contradictory information that arrives through other channels. Raisch and Krakowski (2021) observe that this creates a subtle shift in decision-making: rather than forming independent judgements and then consulting the AI, humans increasingly start from the AI's position and adjust from there. The adjustment is typically insufficient.
The fluency effect describes our tendency to perceive information that is presented clearly and coherently as more accurate than information that is awkward or difficult to read. AI systems produce text of high superficial fluency. This creates a systematic risk: outputs that contain errors are evaluated more charitably than they deserve because the presentation quality implies quality of content.
The STOP protocol for AI output review
Building a critical evaluation practice requires a systematic framework. The STOP protocol provides a structured approach applicable to any AI output before it is used in a decision:
S: Source plausibility. Could the AI have reliable information on this topic? Is the output drawing on well-established knowledge, or does it require access to current, niche, or proprietary information that the AI is unlikely to have?
T: Tone consistency. Does the level of confidence in the output match the complexity and ambiguity of the question? Outputs that present uncertain topics with excessive certainty warrant additional scrutiny.
O: Omissions. What has the AI left out? AI outputs typically present what they include with fluency and coherence, making it easy to overlook what is absent. Asking "what is missing from this analysis?" is one of the most valuable questions an orchestrator can pose.
P: Precision of claims. Are specific claims, including numbers, dates, attributions, and causal relationships, verifiable? The more specific the claim, the more important verification becomes, as confabulation often manifests in precise but incorrect details.
From individual to collective critical thinking
While critical thinking is fundamentally an individual cognitive capability, its impact is magnified when it is embedded in team norms and organisational processes. Teams that establish shared expectations for AI output review, including explicit escalation criteria for high-stakes decisions, create a collective defence against the errors that individual reviewers might miss.
Edmondson's (2019) research on psychological safety is directly relevant here. Team members are more likely to challenge AI-generated recommendations when they feel safe to voice doubt without risking their standing. In teams where AI outputs are treated as authoritative, or where questioning AI is perceived as technophobia, the critical evaluation that effective orchestration demands is suppressed.
Reflection prompts for practitioners
Think about a recent decision you made using AI-generated analysis. What would have changed if you had applied the STOP protocol before acting on it?
When was the last time you challenged an AI recommendation that appeared plausible? What gave you the confidence to challenge it, and what was the outcome?
Consider your team's norms around AI outputs. Is there an implicit expectation that AI-generated content is reviewed critically before use, or is it typically accepted at face value?
References
Dell'Acqua, F. et al. (2023) 'Navigating the Jagged Technological Frontier', Harvard Business School Working Paper, No. 24-013.
Edmondson, A.C. (2019) The Fearless Organization. Hoboken, NJ: Wiley.
Jarrahi, M.H. (2018) 'Artificial Intelligence and the Future of Work', Business Horizons, 61(4), pp. 577-586.
Kahneman, D. (2011) Thinking, Fast and Slow. London: Penguin.
Mollick, E. (2024) Co-Intelligence: Living and Working with AI. New York: Portfolio/Penguin.
Raisch, S. and Krakowski, S. (2021) 'Artificial Intelligence and Management', Academy of Management Review, 46(1), pp. 192-210.