In 2016, Cathy O’Neil published the essay Weapons of Math Destruction, whose title might have seemed provocative: the author intended to denounce how the use of algorithms risks amplifying social inequalities, acting as a weapon that targets the most vulnerable groups. Ten years later, this title describes our society more deeply than it might appear. Indeed, an algorithm decides which content is recommended to platform users, evaluates credit scores and job applications, to such an extent that today we live, more or less consciously, in a world structured by statistical procedures.

The integration of AI models into decision-making processes represents a decisive shift in our relationship with the production of technology. It has always been natural for human beings to build a machine and understand how it works; contemporary systems, by contrast, often operate in a state of opacity, even from the developers’ point of view. Opacity refers to the lack of transparency or comprehensibility regarding how an algorithm processes data to arrive at a specific decision: this refers in particular to machine learning systems, which learn in a partially autonomous manner from the data on which they are trained. This semi-independent learning makes the machine’s decisions difficult to justify and explain, even for their own creators.

A system whose internal workings are obscure to the observer, who sees only the inputs and outputs without understanding the logic of transformation, is defined as a black box, comparing models to black boxes. In AI development, this phenomenon entails a lack of transparency that increases the risk of obtaining distorted results and hinders programmers in detecting errors and correcting outcomes. As mentioned above, the adoption of algorithms in decision-making processes turns the black box problem into a concrete social risk: Frank Pasquale, in The Black Box Society, highlights how those who use these algorithms possess “an unprecedented knowledge of the details of our daily lives, while we know little or nothing about how they use this knowledge to influence the important decisions that we—and they—make” (Frank Pasquale, The Black Box Society: The Secret Algorithms That Control Money and Information, 2015). The inscrutability of the models that govern creditworthiness or access to employment thus ends up placing these choices beyond public scrutiny: behind the curtain of technological complexity or industrial secrecy, power asymmetries take hold in which individuals are deprived of the possibility of understanding or challenging the principles that now determine their social and digital existence.

Explainable AI (XAI) is the field of research that seeks to provide tools for understanding the decisions made by algorithms, in pursuit of more reliable and transparent artificial intelligence. It uses a set of techniques to describe a given model, its expected effects and potential biases, in order to explain how the algorithm produced certain results and verify that the system works as hypothesized. XAI is based on two fundamental concepts: the interpretability of the model, that is, the possibility, in principle, of understanding the decision-making process of an AI system, and explainability, which indicates the possibility of communicating this process in ways that are accessible to the end user.

XAI Techniques

The attempt to shed light on the black box has taken shape around various methods. One of the most relevant is proxy modeling, or the surrogate model method. The underlying idea is intuitive: since the original model is too complex to be understood, a second model is built that is interpretable, such as a decision tree or a linear regression model. The second model is trained to mimic the behavior of the first—to act as a proxy, precisely—seeking to approximate the latter’s responses in a form that the human mind can decode. The risk of this technique is creating an explanation that is faithful to a model that works differently: the proxy model is a simplification and, by reducing complexity, may omit precisely those nuances or distortions that made the original version critical.

An alternative method, which uses symbolic techniques, is called design interpretability, and shifts the solution upstream: instead of explaining a complex system, it chooses to use only models that are “transparent by design.” This is a safe choice but often costly in terms of performance: in many domains, such as image recognition or natural language, simple models cannot achieve the necessary accuracy, forcing a compromise between the accuracy of the result and its comprehensibility.

In recent years, attention has instead shifted to post-hoc explanation techniques such as SHAP and LIME, which have now become the industry standard. LIME works on an individual decision: it perturbs the input data to analyze how the output changes and constructs an explanation that is valid only for that specific case. SHAP, based on game theory, assigns each part of the input a value indicating its contribution to the output, thus explaining what had the greatest influence on the AI model’s final decision.

Despite their mathematical elegance, these explanation techniques suffer from a structural limitation: they offer standardized explanations that do not take into account who the user is. These systems, in fact, merely show the phenomena that influenced the calculation without explaining in an understandable way why it should or should not be considered acceptable. In other words, the limitation of post-hoc techniques is that they treat human understanding as a process devoid of interaction. This gives rise to the need to move beyond the static nature of traditional models toward personalized and adaptive paradigms, in order to obtain an explanation that is not only correct with respect to the model, but also calibrated to the user’s profile, and that takes into account the distinction between a developer’s technical requirements and a citizen’s practical needs. Without this flexibility, XAI risks remaining an exercise in computational rhetoric that, although useful to those with technical skills, is incapable of truly bridging the gap between the system and the user.

Cognitive Mismatch and Interactive AI

The structural limitation of current explanation techniques emerges clearly in the review by Kim, Maathuis and Sent (Jenia Kim, Henry Maathuis, Danielle Sent, Human-centered evaluation of explainable AI applications: a systematic review, 2024). The authors raise a fundamental issue here: until now, XAI has been approached as a “software engineering” problem, overlooking the fact that explanation is, first and foremost, a communicative act between two agents with different cognitive structures. In their review, the researchers document what they define as a cognitive mismatch: the tendency to confuse the model’s technical transparency with its intelligibility to the user.

Specifically, Kim, Maathuis and Sent observe that exposing a non-expert user to data that require specific expertise to be understood (such as SHAP values or the results obtained through LIME perturbations) does not produce genuine understanding, but rather generates information overload: the excess of technical details prevents the user from applying their own mental frameworks instead of guiding them toward understanding. An overly complex explanation can, in fact, induce unjustified trust on the part of the user who, far from being critical and aware, is intimidated and delegates responsibility to the machine and to the explanation received, assuming that a truth they cannot challenge must lie behind such complexity.

To overcome this gap, a paradigm shift is necessary, moving the focus from the output to the process. The work by Raees et al. (Muhammad Raees, Inge Meijerink, Ioanna Lykourentzou, Vassilis-Javed Khan, Konstantinos Papangelis, From Explainable to Interactive AI: A Literature Review on Current Trends in Human-AI Interaction, 2024) moves in this direction, proposing a move beyond passive XAI in favor of Interactive AI. While the overview by Kim, Maathuis and Sent highlights the failure of explanation understood as a ready-made product that is the same for everyone, Raees et al. suggest that true transparency lies in establishing genuine user agency, in which interactivity is understood as a reshaping of the power relationship between the user and the algorithm. In an interactive AI system, the user becomes an active participant in a continuous feedback loop. This method makes it possible to address cognitive mismatch in real time: the user has the opportunity to question the model, ask for clarification on specific points, explore alternative scenarios and correct the model’s internal logic during the interaction.

To understand how Interactive AI can overcome the limitations of traditional explanation, let us consider the integration of an algorithmic system in an oncology department.

In the passive XAI scenario, the doctor uploads the diagnostic image scans relating to a patient, and the system returns the probability that the tumor being examined is malignant, accompanied by a saliency map highlighting which pixels in the image led the algorithm to that output. This is where cognitive mismatch occurs: the explanation is purely visual and statistical; it does not explain why those pixels are relevant from a clinical point of view. This opacity can lead to excessive trust or, conversely, to the doctor’s complete rejection of the system.

The paradigm proposed by Raees et al., on the other hand, understands explanation as an interactive process between the user and the algorithm, transforming AI from an oracle into an advisor. Returning to the previous example, the doctor should have an interface available to question the algorithm and verify in real time how its response changes according to variations in the patient’s clinical parameters.

According to Raees et al., this transition transforms explanation from a static report into a process of knowledge co-creation. Transparency, from being a simple technical requirement, becomes an emergent property of collaboration, so that explanation is an adaptive process calibrated to the skills, language and specific needs of the person being addressed. Restoring agency to the user therefore means recognizing that artificial intelligence must be capable not only of providing answers but also of negotiating their meaning with human beings.

Sparse Autoencoders and Neuro-symbolic AI

While traditional XAI seeks to explain a decision from the outside, new techniques aim to make the AI architecture itself closer to human concepts. One of the most recent and widely discussed innovations concerns the application of Sparse Autoencoders (SAEs) to large language models (Large Language Models).

Most current systems are based on artificial neural networks, computational models inspired by the structure of the human brain, in which information is processed through layers of interconnected nodes: neurons. The problem with artificial neurons is that they are often “polysemantic.” A single neuron does not contribute to the representation of a single concept, but may contribute to the representation of entirely different concepts, making it impossible to interpret its role in the computation precisely; this happens because models are built to be capable of processing enormous quantities of data while making economical use of available resources. SAEs solve this problem: they decompose the chaotic activations of the neural network into thousands of individual, interpretable concepts, making it possible to map the machine’s internal workings onto semantic concepts familiar to humans. This process is fundamental to interpretability because, from an interactive perspective, it means that the user can, theoretically, remove a concept from the equation to see how the model’s behavior changes, exercising the agency mentioned by Raees et al.

In continuity with the conceptual decomposition carried out by Sparse Autoencoders, Neuro-symbolic AI is an approach that makes it possible to connect these discoveries within a coherent structure of thought. While the previous techniques merely extract fragments of meaning from the chaos of data, the neuro-symbolic approach attempts to unite the two major historical branches of artificial intelligence: the statistical learning power of neural networks and the logical rigor of symbolic systems. In this hybrid model, the neural component handles perception and the recognition of complex patterns, while the symbolic component organizes this information according to logical rules.

The advantage of Neuro-symbolic AI for the end user lies in the nature of the explanation it produces. It is no longer a statistical approximation calculated after the fact, but the actual logical path of the reasoning carried out by the machine: in a decision-making context, the XAI system can expose the deductive steps followed, using symbols and categories familiar to humans. This convergence transforms AI from a black box into a transparent structure in which every conclusion can be examined and, above all, challenged through logic, finally offering a basis for shared rationality.

Conclusion

Ultimately, it is important that, in the future, Explainable AI includes among its objectives the resolution of the communication gap that still separates ordinary users and developers. As observed, technical transparency is not in itself a guarantee of usefulness; for AI to become a tool that is truly integrated into society, it is necessary to move from a logic of data disclosure to one of personalization of meaning. The cornerstone of this change lies in emerging tools, such as Sparse Autoencoders and neuro-symbolic AI, which transform abstract calculations into comprehensible concepts. Designing the XAI of the future therefore means imagining systems that do not merely passively deliver a verdict, but enable dynamic interaction, overcoming the rigidity of traditional techniques in favor of an interactive approach based on user agency.

Benedetta Bonomi

(translated by MIRAI)