The interpretability of an Artificial Intelligence (AI) system is the property that allows the system’s actions and algorithmic mechanisms to be studied, understood, and analyzed by humans. In practical terms, interpretability requires the model under consideration to be transparent. Indeed, in an interpretable system, users must be able to clearly examine how input data are transformed and mapped into the resulting outputs. Interpretable systems are therefore built with the aim of providing a clear view of the mechanisms governing the model, explaining how the system works and how it reaches a conclusion.

Although, ideally, the decisions made by a transparent model should be understandable to any user, interpreting the mechanisms that lead to the final decision often requires a certain level of technical expertise. Statistical or graphical models, such as regression models, decision trees, and analysis of variance (ANOVA), are generally considered interpretable. However, although these models are inherently transparent, their results may appear to non-specialist users as nothing more than meaningless numbers and graphs. By contrast, to an expert, such as a statistician or mathematician, these data have clear meanings and provide crucial information about how the system maps inputs to outputs.

For example, regression models are transparent because they allow the relative importance of each feature to be explicitly analyzed by examining the extent to which each variable affects the calculation. Nevertheless, rigorously assessing the impact of a variable on the final result requires specific statistical skills. Similarly, decision trees offer a high degree of transparency because they explicitly represent the logical path leading from input to output. However, correctly interpreting the decision-making contribution of each node requires specialized technical expertise. Finally, analysis of variance (ANOVA) also explicitly reveals certain significant relationships between data, but still requires a degree of technical knowledge to fully understand its results.

By contrast, so-called opaque systems, or black-box systems, are not interpretable insofar as they map input data to the outputs they produce through algorithmic mechanisms that may have no meaning comprehensible to users, even those with specialized technical knowledge. For example, sub-symbolic algorithms, such as deep neural networks (Deep Learning), process data by automatically transforming them through extremely complex nonlinear functions. Due to this high level of complexity and the enormous number of internal parameters, it is not necessarily possible for a human being to analyze and understand the underlying mathematics. This opacity makes such systems unsuitable for fields in which a meaningful and comprehensible interpretation of the mathematical steps is a fundamental requirement.

Nevertheless, a black-box system that is not directly interpretable can still be made explainable. The explainability of an AI system refers to its ability to provide humans with a clear and understandable reason or justification for a specific decision or action. For a system to be considered “explainable,” it must therefore produce an account that clarifies why a particular result was reached, ensuring that the decision is supported by rational arguments and, where necessary, does not conflict with ethical or legal standards. It is therefore clear that explainability is a fundamental concept for the adoption of AI in critical fields that have a significant impact on people’s lives, such as medicine, finance, and security. In these contexts, extremely robust guarantees regarding the technology’s overall reliability are essential.

To overcome the opacity of black-box systems and improve their explainability, it is possible to use Explainable Artificial Intelligence (XAI) techniques, which analyze their behavior and translate their decisions into representations that humans can understand. Examples of such techniques include Model Induction Methods and Model Agnostic Methods. These techniques make it possible to infer an interpretable model from a black-box model without having to know its internal structure. The approach involves querying the black box: the system submits inputs that have been slightly modified from the input used for a specific prediction and observes how the output changes. Systems such as LIME are based on similar ideas and provide extremely intuitive and understandable explanations of individual decisions.

In summary, interpretability addresses the question of “how” a model made a decision and depends on the model’s transparency. A system is interpretable when a user (typically an expert) can analyze it directly and understand the mathematical procedures through which input data are mapped to the resulting outputs. Understanding these mechanisms sheds light on how a system works, but not necessarily on why a particular decision is reached in the real-world context in which the system is used. Explainability, on the other hand, addresses precisely the question of “why” the model reached a particular decision.

Gaia Vilma Polce

(translated by MIRAI)