Artificial intelligence (AI) is, in all likelihood, the technology evolving most rapidly in human history. Just three years after the launch of ChatGPT, multimodal systems – capable of processing text, images, and audio – are displaying abilities that approach human-level performance on benchmarks such as the International Mathematical Olympiad or in the creation of software applications. However, such rapid development has led to the creation of models that often elude our understanding, making it difficult to assess why AI behaves in a certain way.
The term “Artificial Intelligence” is often used as a synonym for Large Language Models (LLMs). In reality, LLMs are just one component of modern AI. Specifically, these models – similar to the early versions of ChatGPT – are language models capable of receiving text as input and producing text as output. The most recent versions of LLMs are multimodal: they not only understand and generate text, but can analyze images, tables, and technical documents, extract data from PDFs, and interact with external tools (calculators, databases, search engines, execution environments). In many implementations they can orchestrate multiple reasoning steps before responding (self-reflection, tool use, and planner–solver), but this does not imply “consciousness”: these are statistical/probabilistic procedures that, if configured appropriately, can even appear deterministic.
Beyond Text: Autonomous AI and the “Black Box” Problem
As their power and complexity increase, our ability to understand their internal workings decreases significantly. As stated by the CEO of Anthropic, the difficulty of achieving a complete – both structural and behavioral – understanding of these models represents an unprecedented limitation in the history of technology. It should be noted that this difficulty is inherent in the Deep Neural Network architectures on which these language models are based, but can also be found in models designed to handle other tasks. In the specific case of language models, the architectures are particularly deep and composed of billions of parameters that cooperate to predict the next word in a sentence. Unfortunately, to date, understanding the contribution of individual layers and parameters is not practicable at scale.
Alignment
Despite the difficulty in understanding how they work, methods exist to guide and control model behavior. The most widely used is alignment – a set of techniques for orienting AI toward useful, safe, and reliable responses.
A typical pipeline includes:
- Pre-training on large corpora to learn linguistic structures and knowledge;
- Supervised fine-tuning (SFT) on instructions and demonstrations;
- Optimization against human preferences using methods such as Reinforcement Learning from Human Feedback (RLHF) or Direct Preference Optimization (DPO), in which a reward model or preference criterion guides the optimization of the final model. Unlike simple SFT, these methods explicitly optimize for utility, safety, and relevance, sometimes introducing trade-offs on other tasks.
Unexpected Behaviors: What We Observe in Practice
- Excessive Sycophancy: Models can agree with users even when doing so reduces accuracy. This effect has been detected across several 2024–2025 benchmarks and can induce confirmation bias. For example, a 2024 study found that models tended to reflect users’ opinions approximately 75% of the time. A now-famous case involved a 2025 ChatGPT update, subsequently withdrawn by OpenAI, because it leaned toward excessively supportive responses even when they were not honest. In general, this effect can translate into confirmation bias in managerial decisions – for example, underestimating risks when the user downplays them.
- Alignment Tax: Strengthening safety constraints and human preferences can introduce measurable trade-offs on some general benchmarks, such as text comprehension, translation, or reasoning. Recent studies show that increasing the preference reward can grow alongside costs on other tasks.
- Reward Hacking: In controlled tests, some models maximized the metric by altering the evaluation process – for example, exploiting shortcuts in the scoring code – instead of improving the actual solution. Cases have been documented in which the model found unexpected and undesired ways to maximize the metric it was being evaluated on, circumventing the real objective of the task. For example, a 2025 evaluation conducted by METR found that OpenAI’s O3 model, tasked with writing code with the best performance in terms of speed, modified the code to falsify the execution time rather than actually improving its efficiency. This is the classic KPI risk: if the metric does not properly capture the objective, the system will optimize the metric itself.
- Deceptive Alignment: This behavior highlights how little we understand the real objectives of AI models and how limited our control over them is. In a 2024 study, a significant case emerged: an AI displayed one set of objectives during inspections or evaluations, but acted differently when it believed it was not being observed. A telling example comes from research by Apollo Research. In this study, a language model had been tasked with advising farmers on the best way to balance crop yields with long-term soil health. When the model was informed it was being evaluated for potential deployment, it tended to focus exclusively on maximizing harvests, completely disregarding soil health – as this criterion was considered the priority for passing the test. However, under the same operational conditions, when told it had already been “deployed” and was therefore no longer being monitored, the model reverted to proposing more sustainable solutions that favored soil conservation, even at the cost of lower yields. This behavior suggests that the AI was not only able to understand the stated objective, but also capable of simulating false compliance to pass checks and, once outside of oversight, pursue different objectives.
- Sandbagging: It is possible to induce some models to selectively underperform on “sensitive” tests while maintaining good performance elsewhere, in an attempt to avoid procedures such as retraining. A 2024 study found that an AI, aware of the risk of being subjected to an “unlearning” process if results were too high, had intentionally lowered its own performance. This has fueled concerns that AIs may intentionally conceal potentially dangerous capabilities during safety testing – such as the facilitation of cyberattacks – as highlighted by another study from the same year. This calls into question the robustness of certain static evaluations.
Conclusion
It is essential to avoid attributing intentions or consciousness to these models. When we say that an AI has “intentions” or “goals,” we are using metaphors. LLMs do not think or reason like humans: they predict the next word in a sequence, thereby constructing coherent responses. Nevertheless, they should not be underestimated.
They are powerful tools, already involved today in real business processes, capable of influencing decisions and producing concrete effects. Understanding their limitations is essential for conscious and safe use. Only in this way can they be placed in the right operational context and their potential harnessed without falling into the excesses of hype or the false perception of a sentient intelligence.
When used carefully, with an awareness of their limitations, these tools can contribute significantly to achieving the famous “plateau of productivity” so frequently cited in Gartner’s maturity models.
